CI/CD Pipelines

CI merges and verifies every change automatically (build, unit tests, static checks). Continuous Delivery keeps every green build releasable with a one-click, usually approved, production release. Continuous Deployment removes the click: every green build goes to production automatically.

• CI is judged by feedback time and broken-main frequency.
• Delivery vs deployment is a business/risk decision, not a technical one; both need automated tests, feature flags and fast rollback.

Order stages from cheapest and fastest to most expensive so failures surface early: compile and unit tests, static analysis and dependency scan, package and image build, integration tests (Testcontainers), image scan, publish, deploy to staging, smoke/e2e, promote to production.

• Build the artifact once and promote the same immutable image through environments; never rebuild per environment.
• Run independent stages (lint, unit tests, SAST) in parallel.
• Keep the PR pipeline under about 10 minutes; push slow suites to post-merge or nightly.

GitHub Actions is a hosted, event-driven workflow engine: YAML workflows trigger on repo events, jobs run on ephemeral runners, and reusable steps come from a marketplace. Jenkins is a self-hosted controller with agents, configured as code through a Jenkinsfile (declarative or scripted Groovy) and extended by plugins.

• Actions: low ops burden, ephemeral clean runners, tight GitHub integration (OIDC, environments, checks). Cost grows with minutes unless you use self-hosted runners.
• Jenkins: maximum flexibility and on-prem control, but you own upgrades, plugin compatibility, agent scaling and controller security.
• Both support shared libraries/reusable workflows to avoid copy-pasted pipelines.

Use actions/setup-java with cache: maven, which keys on pom.xml hashes and restores ~/.m2/repository.

name: ci
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 21
          cache: maven
      - run: mvn -B -ntp verify

• -B batch mode and -ntp suppress download progress noise.
• Declare permissions at least privilege; the default token can be broader than needed.

Cache dependency stores (~/.m2/repository, ~/.gradle/caches) keyed by build file hashes, and for Gradle additionally use the build cache for task outputs. Dependency caching saves download time; build cache saves compile and test work.

- uses: gradle/actions/setup-gradle@v4
  with:
    cache-read-only: ${{ github.ref != 'refs/heads/main' }}
- run: ./gradlew build --build-cache

• Let only main write the cache; PRs read-only, which prevents cache poisoning from untrusted code.
• Key must include lockfile/pom.xml hash plus OS/JDK; use restore-keys for partial hits.
• Docker layer caching (cache-from/cache-to with buildx) is separate from dependency caching.

CI publishes immutable, versioned artifacts: libraries to Maven repositories (Nexus/Artifactory/GitHub Packages) and services as container images to a registry (ECR, GHCR, Harbor). Deployments reference the image by digest, which is content-addressed and immutable.

• Tag with the git SHA (and semver for releases); latest is mutable and unsuitable for deployments.
• workflow artifacts are short-lived build outputs between jobs, not a distribution mechanism.
• Apply retention/lifecycle policies on the registry and sign images (cosign).

docker buildx build -t ghcr.io/acme/orders:${GITHUB_SHA} --push .

GitHub Actions environments hold scoped secrets/variables and protection rules (required reviewers, wait timers, branch restrictions). Jenkins uses the input step and the Credentials plugin with folder-scoped credentials.

deploy-prod:
  needs: deploy-staging
  environment: production      # requires reviewer approval
  runs-on: ubuntu-latest
  permissions:
    id-token: write
    contents: read
  steps:
    - uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-prod
        aws-region: eu-west-1

• Prefer OIDC federation to cloud roles over long-lived access keys in secrets.
• Secrets are masked in logs but can be exfiltrated by a malicious step; restrict who can modify workflows.

Rolling replaces instances gradually; blue-green runs a full new environment and switches traffic atomically; canary sends a small percentage of traffic to the new version and increases it based on metrics.

• All require backward-compatible DB changes because old and new versions overlap.
• Canary needs good metrics and an automated analysis (Argo Rollouts, Flagger).

GitOps stores the desired cluster state in Git and an in-cluster controller (Argo CD, Flux) continuously reconciles the cluster to it. CI only builds the image and commits a new tag to the config repo; it holds no cluster credentials.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: orders, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/acme/deploy-config
    path: apps/orders/prod
    targetRevision: main
  destination: { server: https://kubernetes.default.svc, namespace: orders }
  syncPolicy:
    automated: { prune: true, selfHeal: true }

• Benefits: audit trail, rollback by git revert, drift correction, no CI-to-cluster credentials.
• Costs: image-tag update automation, secrets need sealed-secrets/External Secrets, two repos to manage.

Developers integrate into main at least daily through short-lived branches (hours to a day or two) instead of long-lived feature branches. Incomplete work is hidden behind feature flags so main is always releasable.

• Requires fast CI, strong tests and small PRs.
• Releases are cut from main (tag) or short release branches for hotfixes.
• Long-lived branches cause merge pain, integration surprises and large risky releases.

Flakiness comes from shared state, time dependence, async waits with sleep, order-dependent tests, port collisions, external services and resource-starved runners. A retry masks it but does not remove it.

• Quarantine: tag the test, run it in a non-blocking job, track an owner and deadline.
• Measure flake rate per test from CI history; fix top offenders.
• Use Testcontainers with random ports, Awaitility instead of Thread.sleep, fixed Clock injection.
• Failsafe/Surefire rerunFailingTestsCount can be a stopgap, with reporting of reruns.

Protecting everything that goes into and through the build: dependencies, base images, build tooling, the pipeline itself and the artifacts.

• Dependency scanning (OWASP Dependency-Check, Dependabot, Trivy), plus lockfiles/pinned versions.
• Generate an SBOM (CycloneDX/SPDX) per build.
• Sign images and verify at admission (cosign, Kyverno/Sigstore policy controller); produce provenance attestations (SLSA).
• Pin third-party actions to a commit SHA, not a movable tag.
• Least-privilege tokens, OIDC instead of static keys, ephemeral runners, protected branches.

Profile first, then attack the longest stage: parallelize, cache, and move slow suites out of the PR path.

• Cache dependencies and Docker layers; use Gradle/Maven incremental builds and -T 1C for Maven.
• Split tests into shards across matrix jobs; run only affected modules where the build tool supports it.
• Unit tests on PR; full integration/e2e after merge or in a merge queue.
• Use bigger runners if CPU-bound; a faster runner is often cheaper than developer waiting time.

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: mvn -B verify -Dsurefire.shard=${{ matrix.shard }}/4

(The shard property depends on your test splitting setup, for example a custom Surefire grouping or JUnit tags.)

Pipeline success only meant manifests were applied. Add post-deploy verification: readiness-based rollout status, smoke tests, and metric-based analysis, with automatic rollback on failure.

kubectl rollout status deploy/orders -n prod --timeout=180s \
  || { kubectl rollout undo deploy/orders -n prod; exit 1; }

• With GitOps, roll back by reverting the commit so Git stays truthful.
• Roll back application first; DB changes must have been backward compatible (expand/contract) so the old version still works.
• Feature-flagged changes are rolled back by flipping the flag, which is faster than a deploy.

Treat it as compromised: rotate first, clean up second.

• Revoke/rotate the credential immediately and check provider access logs for use.
• Delete logs/artifacts that contain it; removing from Git history (git filter-repo) does not un-leak it, because forks and caches may hold it.
• Find the cause (set -x, echoing env, debug flags) and add secret scanning (gitleaks, GitHub push protection) as a blocking check.
• Move to short-lived credentials (OIDC, Vault dynamic secrets) so leaks expire.

Use a declarative Jenkinsfile with Kubernetes or Docker agents, parallel test reporting and an input gate only on main.

pipeline {
  agent { docker { image 'maven:3.9-eclipse-temurin-21' } }
  options { timeout(time: 30, unit: 'MINUTES'); disableConcurrentBuilds() }
  stages {
    stage('Build & Test') {
      steps { sh 'mvn -B -ntp verify' }
      post { always { junit '**/target/surefire-reports/*.xml' } }
    }
    stage('Approve') {
      when { branch 'main' }
      steps { input message: 'Deploy to prod?', submitter: 'release-managers' }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps { withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
        sh 'kubectl set image deploy/orders app=registry/orders:$GIT_COMMIT'
      } }
    }
  }
}

• Place input outside an agent block in production so it does not hold an executor while waiting.

Use Argo Rollouts with stepped weights and a metric-based analysis; abort and revert when error rate or latency breaches thresholds.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: error-rate }]
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 100

The analysis template queries Prometheus: sum(rate(http_server_requests_seconds_count{status=~"5..",version="canary"}[5m])) / sum(rate(http_server_requests_seconds_count{version="canary"}[5m])) < 0.01.

• Compare against baseline, not absolute values, to avoid time-of-day noise.
• Payment-specific: ensure idempotency so retried requests across versions are safe.

Likely a stale or over-broad cache: restore-keys falling back to an old cache, caching target/ or build/, or a SNAPSHOT dependency resolved from the cached local repository instead of the remote.

• Re-run with the cache disabled to confirm; add -U (update snapshots) or avoid SNAPSHOTs.
• Cache only the dependency store, key by lockfile hash.
• Run a periodic clean build (nightly) without cache as a control.
• Use --no-build-cache or clean in the release workflow.

Each PR was tested against an old main, not against the other. Use a merge queue (GitHub merge queue, or Bors/Mergify) that tests each PR on top of the current queue head, and require "branch up to date" for protected branches.

on:
  pull_request:
  merge_group:    # required so checks run for queued merges

• Required status checks must run on merge_group or the queue stalls.
• Costs additional CI minutes but protects main.

Use expand/contract: old and new app versions run concurrently, so every migration must be compatible with both.

1. Expand: add nullable column/new table (deploy migration, old code ignores it).
2. Deploy code that writes both and reads new.
3. Backfill in batches.
4. Contract: remove the old column in a later release after nothing uses it.

-- release N
ALTER TABLE orders ADD COLUMN currency VARCHAR(3);
-- release N+2, after code stops reading the old field
ALTER TABLE orders DROP COLUMN legacy_currency;

• Run Flyway/Liquibase as a pre-deploy job rather than in every replica to avoid races (Flyway locks anyway).
• Beware long locks: CREATE INDEX CONCURRENTLY on PostgreSQL.

Interpolating untrusted context (github.event.pull_request.title, branch names) directly into run: lets an attacker inject shell commands. Pass it through an environment variable.

# vulnerable
- run: echo "PR is ${{ github.event.pull_request.title }}"

# fixed
- run: echo "PR is $TITLE"
  env:
    TITLE: ${{ github.event.pull_request.title }}

• With the fix, the value is data in a variable, not part of the script text.
• Avoid pull_request_target combined with checking out the PR head; it runs untrusted code with secrets.
• Pin actions by SHA and set permissions: contents: read by default.

selfHeal is working as designed: the live state differs from Git, so Argo CD reverts it. The fix belongs in Git.

• Commit the hotfix to the config repo (or revert), then let sync apply it.
• For fields legitimately changed at runtime (HPA-managed replicas, injected sidecars), use ignoreDifferences rather than disabling self-heal.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas]

• In an emergency, pause auto-sync for that app temporarily, and record it.

Use path filters or build-tool change detection so only affected modules build, and publish independent versions.

on:
  push:
    paths: ['services/orders/**', 'libs/common/**']

• A change in a shared library must trigger all dependents; compute the dependency graph (Gradle --build-cache with composite builds, Maven -pl :orders -am).
• Use mvn -pl <module> -am to build the module and what it depends on.
• Version independently with tags per component (orders-v1.4.0).
• Required checks with path filters can leave PRs pending; add an always-running aggregator job.

Use OIDC federation: the workflow requests a short-lived JWT from GitHub, and AWS exchanges it for temporary credentials via an IAM role with a trust policy scoped to the repo and environment.

{
  "Effect": "Allow",
  "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com" },
  "Action": "sts:AssumeRoleWithWebIdentity",
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
      "token.actions.githubusercontent.com:sub": "repo:acme/orders:environment:production"
    }
  }
}

• The sub condition is the critical control; a wildcard allows any repo/branch to assume the role.
• Requires permissions: id-token: write in the job.

Assess reachability and fix availability rather than reflexively blocking or ignoring. If a patched base image exists, rebuild; if not, document an expiry-dated exception.

trivy image --severity CRITICAL,HIGH --exit-code 1 --ignore-unfixed ghcr.io/acme/orders:$SHA

• --ignore-unfixed avoids blocking on vulnerabilities with no available patch.
• Keep a .trivyignore with justification and an expiry review date.
• Use minimal bases (distroless, Temurin JRE slim) to reduce attack surface; rebuild images regularly to pick up patches.
• Also scan continuously in the registry, since new CVEs appear after build.

Distinguish infrastructure from code: check exit code 137 (OOM kill), runner size and parallel job density, then bound JVM and test memory explicitly.

• Set MAVEN_OPTS=-Xmx2g and Surefire argLine limits, and size the runner/pod requests=limits accordingly.
• Reduce parallel forks (forkCount) when memory-bound.
• Proxy dependency downloads through an internal mirror (Nexus) to avoid network resets and rate limits; add retries to downloads.
• Use ephemeral clean runners to avoid disk fill from previous builds.