Mini-guide: Shipping a Spring Boot Microservice to Production

Production-ready means it can be deployed safely, observed, operated by someone else at 3am and fails gracefully. Walk these areas:

• Build and deploy: reproducible image, CI gates, rollback tested.
• Config and secrets externalized; no secrets in the image or Git.
• Health probes, graceful shutdown, resource requests/limits, at least 2 replicas with a PodDisruptionBudget.
• Resilience: timeouts, retries with backoff, circuit breakers, idempotency.
• Data: backward-compatible migrations, backups, connection pool sizing.
• Observability: metrics, structured logs, traces, SLO alerts with runbooks.
• Security: authn/authz, TLS, dependency/image scanning, least-privilege.
• Ownership: on-call, dashboards, capacity estimate, load test.

Build one image and inject environment-specific config at runtime (env vars, ConfigMaps, profile files), following twelve-factor. Secrets come from a secret manager, never from the image.

# Kubernetes
env:
  - name: SPRING_PROFILES_ACTIVE
    value: prod
  - name: SPRING_DATASOURCE_URL
    valueFrom: { configMapKeyRef: { name: orders-config, key: db-url } }
  - name: SPRING_DATASOURCE_PASSWORD
    valueFrom: { secretKeyRef: { name: orders-db, key: password } }

• Relaxed binding maps SPRING_DATASOURCE_URL to spring.datasource.url.
• Use @ConfigurationProperties with @Validated so bad config fails fast at startup.
• Spring Cloud Config/Vault are options; ConfigMap changes need a restart unless you wire refresh.

Readiness decides whether to route traffic; liveness decides whether to restart; startup protects slow starts from liveness. Spring Boot provides /actuator/health/liveness and /readiness when probes are enabled.

readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8081 }
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8081 }
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet: { path: /actuator/health/liveness, port: 8081 }
  periodSeconds: 5
  failureThreshold: 30

• Do not include external dependencies (DB) in liveness: a DB outage would restart every pod, making things worse. Dependencies belong in readiness, and even there cautiously.
• Too aggressive liveness causes restart loops under load or GC pauses.

On SIGTERM, Kubernetes removes the pod from endpoints concurrently with sending the signal, so some traffic still arrives. Delay shutdown briefly, then let Spring finish in-flight requests.

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

# pod spec
terminationGracePeriodSeconds: 60
containers:
  - name: app
    lifecycle:
      preStop:
        exec: { command: ["sleep", "10"] }

• Order: preStop sleep (endpoint propagation) then SIGTERM, graceful phase drains requests, then exit.
• terminationGracePeriodSeconds must exceed preStop + shutdown timeout.
• Stop consuming messages (Kafka listeners) cleanly and finish current batch.

Always set timeouts first; then retry only idempotent calls with backoff and jitter, add a circuit breaker and bulkhead, and provide a fallback or fail fast.

resilience4j:
  circuitbreaker:
    instances:
      payments:
        slidingWindowSize: 20
        failureRateThreshold: 50
        waitDurationInOpenState: 30s
  retry:
    instances:
      payments:
        maxAttempts: 3
        waitDuration: 200ms
        enableExponentialBackoff: true

@CircuitBreaker(name = "payments", fallbackMethod = "pending")
@Retry(name = "payments")
public PaymentResult charge(ChargeRequest r) { return client.charge(r); }

• HTTP client: connect timeout 1-2s, read timeout from the provider's p99 plus margin.
• Retries multiply load during outages (retry storms); cap attempts and budget them.
• Send an Idempotency-Key so retried charges cannot double-charge.

Use Flyway/Liquibase migrations that are backward compatible with the currently running version (expand/contract), applied before or at the start of the new rollout.

-- V12__add_customer_email_verified.sql  (expand)
ALTER TABLE customer ADD COLUMN email_verified BOOLEAN NOT NULL DEFAULT FALSE;

spring:
  flyway:
    locations: classpath:db/migration
    validate-on-migrate: true

• Run migrations as a pre-deploy Kubernetes Job or Helm hook to avoid replica startup contention and long probes failing.
• Large tables: avoid locking DDL (CREATE INDEX CONCURRENTLY, batch backfills, adding a NOT NULL column with default is cheap in PostgreSQL 11+ but not everywhere).
• Never edit an applied migration; Flyway checksum validation will fail.
• Drop/rename in a later release.

Use Little's Law: concurrent DB connections = throughput x time in DB = 500 x 0.04 = 20 across all replicas. Add headroom (about 1.5-2x) but stay below the database connection limit.

• With 5 replicas, 20 x 2 / 5 = about 8 connections per pod; set maximum-pool-size: 8-10.
• Total across pods must stay under DB max_connections minus admin/other clients (consider PgBouncer).

spring:
  datasource:
    hikari:
      maximum-pool-size: 10
      connection-timeout: 2000
      max-lifetime: 1500000   # below DB/proxy idle timeout

• Bigger pools do not help when the DB is the bottleneck; they increase contention.
• Tomcat threads default 200 per pod; if each holds a connection for the request, pool becomes the throttle, which is fine with a short connection-timeout.
• Verify with load test and hikaricp_connections_pending.

Validate JWTs issued by an identity provider with Spring Security's resource server support, enforce authorization on endpoints and methods, and use TLS everywhere.

@Bean
SecurityFilterChain api(HttpSecurity http) throws Exception {
  return http
    .csrf(c -> c.disable())   // stateless token API, no browser session cookies
    .sessionManagement(s -> s.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
    .authorizeHttpRequests(a -> a
      .requestMatchers("/actuator/health/**").permitAll()
      .requestMatchers(HttpMethod.GET, "/orders/**").hasAuthority("SCOPE_orders.read")
      .anyRequest().authenticated())
    .oauth2ResourceServer(o -> o.jwt(Customizer.withDefaults()))
    .build();
}

spring.security.oauth2.resourceserver.jwt.issuer-uri: https://idp.acme.io/realms/main

• Validate iss, aud, expiry; check object-level authorization (users access only their orders) in the service.
• Keep Actuator on a management port, secured or network-restricted.
• Scan dependencies and images; run as non-root with read-only root filesystem.

Kubernetes Secrets are base64, not encrypted by default, so source them from a secret manager and sync them in: External Secrets Operator, Secrets Store CSI driver or Vault agent. Enable etcd encryption at rest and restrict RBAC.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: orders-db }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets, kind: ClusterSecretStore }
  target: { name: orders-db }
  data:
    - secretKey: password
      remoteRef: { key: prod/orders/db, property: password }

• Use workload identity (IRSA) so pods have no static cloud keys.
• Rotation: dynamic DB credentials or a restart/reload procedure; test it.
• Never log config properties; mask in /actuator/env.

For a stateless service, a rolling update with surge capacity, readiness gating and a PodDisruptionBudget is the baseline; add canary for risky changes.

spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 2, maxUnavailable: 0 }
  minReadySeconds: 20
  progressDeadlineSeconds: 300

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 4
  selector: { matchLabels: { app: orders } }

• maxUnavailable: 0 keeps capacity; readiness must be accurate.
• progressDeadlineSeconds marks a stuck rollout failed; CD runs rollout status and undoes.
• Pair with feature flags for decoupling deploy from release.

A runbook gives a tired on-call engineer ordered, concrete steps: impact, quick checks, mitigations, escalation. Link it from the alert.

1. Impact: which endpoints and customers (dashboard link).
2. Recent changes: deploy, config, flag changes in the last hour (annotations); if yes, roll back first.
3. Dependencies: DB, payments, Kafka health panels; circuit breaker state.
4. Saturation: Hikari pending, CPU throttling, memory, pod restarts.
5. Commands:

kubectl -n prod rollout history deploy/orders
kubectl -n prod rollout undo deploy/orders
kubectl -n prod logs deploy/orders --since=15m | grep -i error

6. Mitigations: scale out, disable flag, shed load, fail over.
7. Escalation: owner team, dependency owners, incident commander criteria.

• Review and test runbooks in game days; stale runbooks are worse than none.

Measure, model, test, then add headroom.

1. Baseline: peak RPS, per-pod throughput at target latency and CPU (say 150 RPS/pod at 60% CPU).
2. Forecast: 5x peak of 400 RPS = 2000 RPS; pods = 2000 / 150 = 14, plus N+1/zone-loss headroom, so about 20.
3. Check downstream limits: DB connections and CPU, cache, third-party rate limits, Kafka partitions.
4. Load test in a production-like environment with realistic data (k6/Gatling), ramp to 1.5x forecast, soak for hours to find leaks.
5. Autoscale with HPA on CPU or custom metric (RPS/pod), with a floor set to expected launch minimum since scale-up lags.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 20
  maxReplicas: 40
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }

A post-mortem explains the system conditions that allowed the failure and produces owned, dated actions; it does not assign personal blame.

• Summary and impact: duration, error rate, customers affected, SLO budget consumed.
• Timeline with timestamps: detection, escalation, mitigation, resolution.
• Root cause and contributing factors: config had no validation, no canary, alert took 12 minutes.
• What went well and what was lucky.
• Detection and response analysis: time to detect, time to mitigate.
• Action items: type (prevent, detect, mitigate), owner, due date, tracked in the backlog, e.g. schema-validate config in CI, staged config rollout, tighten alert latency.

Measure first (--debug condition report, Actuator startup endpoint with BufferingApplicationStartup), then remove the dominant cost.

• Trim classpath scanning and unused auto-configuration; lazy initialization (spring.main.lazy-initialization=true) trades startup time for first-request latency and hides wiring errors.
• Avoid heavy work in @PostConstruct; warm caches asynchronously.
• Class Data Sharing / AppCDS (Boot 3.3+ supports CDS), or Spring AOT/GraalVM native for sub-second starts at build-time cost.
• Give CPU: JVM startup is CPU-bound; a 250m CPU limit makes startup very slow. Allow burst or set higher requests.
• Use a startup probe so liveness doesn't kill a slow-starting pod.

Slow dependencies are worse than down ones: threads and connections pile up waiting. Bound every call and isolate resources.

• Timeouts on all clients (connect and read); total request deadline smaller than the caller's timeout (timeout budget).
• Circuit breaker to stop calling a failing dependency; bulkhead to cap concurrent calls per dependency so one cannot eat all threads.
• Load shedding: reject early with 429/503 when saturated.
• Retries with backoff plus a retry budget; no retry at every layer (3x3x3 amplification).

resilience4j.bulkhead:
  instances:
    inventory: { maxConcurrentCalls: 20, maxWaitDuration: 0 }

• Test with fault injection (Toxiproxy, chaos tools) to prove behavior.

Choose a few user-centric SLIs, set targets from business need and measured baseline, and alert on burn rate.

• Availability SLI: non-5xx / total on user-facing endpoints. Target 99.9% (43 min budget per 30 days).
• Latency SLI: share of requests faster than 300ms. Target 95%.
• Freshness/correctness SLIs for async pipelines (consumer lag, age of oldest message).

1 - (
  sum(rate(http_server_requests_seconds_bucket{le="0.3",uri!~"/actuator.*"}[30m]))
  / sum(rate(http_server_requests_seconds_count{uri!~"/actuator.*"}[30m]))
)

• Exclude health checks and known-client-error 4xx from availability.
• Burn-rate alerts (fast and slow) plus a policy: when budget is exhausted, freeze features and fix reliability.

Consumers must be idempotent, handle poison messages, shut down cleanly and expose lag.

spring:
  kafka:
    consumer:
      enable-auto-commit: false
      auto-offset-reset: earliest
      properties:
        max.poll.interval.ms: 300000
    listener:
      ack-mode: record

• At-least-once delivery means duplicates: dedupe by event ID (unique key) or make updates idempotent.
• Poison messages: DefaultErrorHandler with backoff and a dead-letter topic (DeadLetterPublishingRecoverer).
• Alert on consumer-group lag and age, not just rate; scale consumers up to the partition count.
• Graceful shutdown: let the listener container stop and commit offsets; long processing beyond max.poll.interval.ms triggers rebalances.

Use a multi-stage build (or Spring Boot buildpacks / Jib), a slim JRE base, non-root user, and layered jars for cache efficiency.

FROM eclipse-temurin:21-jdk AS build
WORKDIR /src
COPY . .
RUN ./mvnw -B -DskipTests package && \
    java -Djarmode=tools -jar target/app.jar extract --layers --destination /layers

FROM eclipse-temurin:21-jre
RUN useradd -r -u 10001 app
WORKDIR /app
COPY --from=build /layers/dependencies/ ./
COPY --from=build /layers/application/ ./
USER 10001
ENTRYPOINT ["java", "-XX:MaxRAMPercentage=70", "-jar", "app.jar"]

• Pin base image digests; rebuild regularly for patches; scan with Trivy.
• Set readOnlyRootFilesystem, drop capabilities, runAsNonRoot in the pod security context.
• ./mvnw spring-boot:build-image produces a buildpack image without a Dockerfile.

Require clients to send an Idempotency-Key header; store the key with the request fingerprint and result under a unique constraint, and return the stored result on repeats.

@Transactional
public PaymentResult charge(String key, ChargeRequest req) {
  return repo.findByKey(key)
      .map(IdempotencyRecord::result)
      .orElseGet(() -> {
        PaymentResult r = provider.charge(req, key);   // pass key downstream too
        repo.save(new IdempotencyRecord(key, hash(req), r));
        return r;
      });
}

CREATE UNIQUE INDEX uq_idem_key ON idempotency_record(key);

• Concurrent duplicates: the unique constraint makes the second insert fail; handle by returning the first result or 409.
• Reject the same key with a different payload (fingerprint mismatch).
• Expire keys after a retention window (24-72h).

Requests drive scheduling and QoS; limits cap usage. Memory: request = limit, sized from observed working set plus headroom. CPU: set a request near typical usage; consider no CPU limit, or a generous one, since CFS throttling causes latency spikes.

resources:
  requests: { cpu: "500m", memory: "1Gi" }
  limits:   { memory: "1Gi" }       # no CPU limit; avoids throttling

• Observe with container_cpu_cfs_throttled_periods_total and working-set memory.
• JVM sees the CPU limit to size GC/compiler threads; fractional limits give 1 processor, which affects GC choice (Serial under 2 CPUs/1792MB).
• Spread replicas across nodes and zones with topologySpreadConstraints.

Ship code dark, enable gradually (internal, 1%, 10%, 100%) per cohort, watch SLO metrics, and keep a one-step off switch that needs no deploy.

@GetMapping("/quote")
Quote quote(@RequestParam String sku) {
  return flags.isEnabled("new-pricing", currentUser())
      ? newPricing.quote(sku)
      : legacyPricing.quote(sku);
}

• Use a flag service (Unleash, LaunchDarkly, OpenFeature) or a refreshable config; default to the safe path if the flag service is unreachable.
• Tag metrics with the flag variant to compare outcomes.
• Remove flags after rollout; stale flags are technical debt and risk.

Define RPO (acceptable data loss) and RTO (acceptable downtime) with the business, then pick mechanisms to meet them and rehearse restoring.

• Backups: automated daily snapshots plus WAL archiving for point-in-time recovery (PITR); RPO of minutes.
• HA: a synchronous or async replica in another AZ with automated failover (RDS Multi-AZ, Patroni); cross-region replica for regional DR.
• Restore drills: restore into a scratch instance regularly and verify with checksums/queries; an untested backup is not a backup.
• App behavior on failover: connection pool re-resolves DNS, max-lifetime short enough, retry transient errors, idempotent writes.
• Protect against logical errors (bad delete) with PITR, not just replication, which copies mistakes.

Treat the first weeks as hyper-care: higher observation, quick iteration on alerts, and learning from real traffic.

• Daily review of dashboards, error budgets, top slow endpoints, log error clusters, and pod restarts.
• Tune alerts: remove noisy ones, add missing ones found by incidents or near misses.
• Compare real traffic and resource usage with the capacity model; adjust requests, HPA, pool sizes.
• Confirm backups ran and a restore was tested; confirm certificates and secrets rotation dates are tracked.
• Hold a short retrospective: what surprised us, what needs automation.
• Define on-call rotation, escalation paths, and handover notes.