Microservices & Resilience

Draw boundaries around business capabilities (DDD bounded contexts) so a service owns its data and changes independently; avoid splitting by technical layer.

• Signals of a good boundary: high cohesion, low coupling, one team owns it, independent deploy and scaling, its own database.
• Smells: services that must deploy together, chatty synchronous call chains, shared DB tables, distributed transactions everywhere (distributed monolith).
• Start with a modular monolith and extract when team size or scaling needs justify it.

Instances register their address in a registry (or the platform tracks them); callers resolve a logical name to healthy instances.

• Client-side: client queries a registry (Eureka, Consul) and load balances itself (Spring Cloud LoadBalancer).
• Server-side: a router/LB looks up instances (Kubernetes Service + kube-proxy/DNS, AWS ALB).
• Registrations are kept alive with heartbeats/TTL or health checks; stale entries cause failed calls, so combine with retries and outlier ejection.
• On Kubernetes prefer built-in DNS service.namespace.svc over a separate registry.

A saga is a sequence of local transactions, each with a compensating action, replacing a distributed ACID transaction. Choreography uses events between services; orchestration uses a central coordinator.

Order -> Payment -> Inventory -> Shipping
fail at Inventory: refund Payment (compensate), cancel Order

• Sagas give ACD (no isolation): intermediate states are visible; use semantic locks, pending states, commutative updates.
• Compensations must be idempotent and retryable; steps use outbox for reliable messaging.

A circuit breaker stops calling a failing dependency to fail fast and give it time to recover. States: CLOSED (calls pass, failures counted), OPEN (calls rejected immediately), HALF-OPEN (a few trial calls decide to close or reopen).

CLOSED --failure rate > threshold--> OPEN --wait duration--> HALF_OPEN
  ^                                                              |
  +---------------- trial calls succeed -------------------------+ (else back to OPEN)

• Resilience4j settings: slidingWindowSize, failureRateThreshold, waitDurationInOpenState, permittedNumberOfCallsInHalfOpenState, slow-call thresholds.
• Combine with a fallback (cache, default, degraded feature) and timeouts.

Bulkheads isolate resources (thread pools, semaphores, connection pools, even instances) per dependency or tenant so one slow dependency cannot exhaust shared capacity.

• Thread-pool bulkhead: dedicated pool per downstream; semaphore bulkhead: cap concurrent calls (cheap, works with virtual threads/reactive).
• Separate connection pools and queues per critical path (checkout vs browse).
• Cell-based architecture isolates whole user cohorts.
• Size using Little's Law: concurrency = rate x latency.

Retry only idempotent, transient failures, with a bounded count, exponential backoff, and jitter, within an overall deadline; otherwise retries amplify outages.

• Backoff: delay = min(cap, base x 2^n) with full jitter (random 0..delay) to desynchronize clients.
• Retry on timeouts, 503, connection reset; not on 4xx or validation errors.
• Retry at one layer only; stacked retries multiply (3 layers x 3 retries = 27 calls).
• Use retry budgets (e.g. retries <= 10% of requests) and propagate deadlines.

Every remote call needs a timeout; the caller's timeout should exceed the sum of downstream latency plus retries, and ideally a deadline is propagated so downstream work stops when the caller has given up.

• Set connect and read timeouts separately; defaults are often infinite.
• Downstream timeouts must be shorter than upstream ones, or upstream times out while downstream keeps working.
• gRPC propagates deadlines natively; for HTTP pass X-Request-Deadline or use budget headers.
• Choose from observed p99 plus margin, not guesses.

A trace is the end-to-end path of a request through services; it consists of spans (timed operations) linked by parent IDs. A trace ID is generated at the edge and propagated via headers.

• W3C traceparent: 00-<trace-id>-<span-id>-<flags> is the standard; B3 is legacy Zipkin.
• Spring Boot 3: Micrometer Tracing with OpenTelemetry/Brave bridge; Boot auto-instruments RestTemplate/WebClient/messaging.
• Propagate through message headers (Kafka) and async executors (context-aware wrappers).
• Sampling (head-based percentage or tail-based on errors/latency) controls cost.
• Put the trace ID in logs (MDC) to join logs and traces.

They trade rollout speed, cost and rollback ease.

• Database changes need expand/contract migrations so old and new versions run together.
• Automate canary analysis (error rate, latency vs baseline) with auto-rollback.

Consensus lets a cluster agree on a value (a log) despite failures, tolerating f failures with 2f+1 nodes. Raft elects a leader per term; the leader replicates log entries and commits once a majority acknowledges.

• Election: a follower whose election timeout expires becomes candidate, increments term, requests votes; a majority wins; randomized timeouts avoid split votes.
• Safety: a candidate must have an up-to-date log; committed entries are never lost.
• Use odd cluster sizes (3 tolerates 1, 5 tolerates 2); latency is bound by the slowest node in the majority.
• Used in etcd, Consul, Kafka KRaft, CockroachDB; ZooKeeper uses ZAB.

A distributed lock gives mutual exclusion across processes, typically via a lease (TTL) in Redis, or a consensus store (ZooKeeper/etcd). Leases can expire while the holder is paused (GC, network), so use fencing tokens.

Client A gets lock token=33, pauses (GC)
Lease expires; Client B gets token=34, writes
Client A resumes, writes with token=33 -> storage rejects (33 < 34)

• Redis: SET lock:x <uuid> NX PX 30000; release with a Lua check-and-delete on the uuid.
• Redlock is contested for correctness-critical use; prefer etcd/ZooKeeper with fencing for strict safety.
• Often avoidable: use DB unique constraints, optimistic locking, or idempotent operations.

Logs (events), metrics (aggregated numbers over time), and traces (request paths); together they let you ask new questions of a running system.

• RED for services: Rate, Errors, Duration. USE for resources: Utilization, Saturation, Errors. Google's four golden signals: latency, traffic, errors, saturation.
• Metrics: Micrometer to Prometheus, with low-cardinality labels (never user IDs).
• Logs: structured JSON with correlation/trace IDs; centralize (ELK/Loki).
• Alert on symptoms (SLO burn) and use dashboards plus traces for diagnosis.
• Health: separate liveness (restart me) from readiness (do not send traffic).

Sync calls give simple request-response but couple availability and latency; async messaging decouples in time at the cost of eventual consistency and harder flow tracking.

• gRPC: HTTP/2, protobuf, streaming, strong contracts; REST: ubiquitous, cacheable.
• Rule of thumb: query synchronously, propagate state changes asynchronously.

Requests to the slow service hold threads (Tomcat 200 default) until all are blocked, so unrelated endpoints starve; the failure cascades.

• Add connect/read timeouts (e.g. 500 ms-2 s) on every client.
• Circuit breaker with a fallback so calls fail fast once the failure rate trips.
• Bulkhead: limit concurrent calls to that dependency (semaphore or separate pool) so remaining capacity serves other paths.
• Load shed with bounded queues; use async/virtual threads to cut thread cost but still cap concurrency.
• Test with fault injection (Toxiproxy, chaos experiments).

Use an orchestrated saga: state machine persisted in the orchestrator; each step is a command with a matching compensation.

• Payment fails: orchestrator issues ReleaseInventory, marks order CANCELLED.
• Order state PENDING until success so users see consistent intermediate state.
• Commands and replies use outbox plus idempotent handlers; compensations retry until success.
• Timeouts on each step trigger compensation; reservations carry TTL as a safety net.

Use a progressive canary with automated analysis: 1% then 5%, 25%, 50%, 100%, gating each step on SLI comparison against the baseline and rolling back automatically.

• Argo Rollouts/Flagger with Prometheus queries (error rate, p99 vs baseline).
• Segment canary by cohort (internal users first, then region) to limit blast radius.
• Pair with feature flags to disable the new behaviour without redeploying.
• Keep database changes backward compatible (expand, migrate, contract) so rollback is possible.
• Soak long enough to cover rare paths and cache-warm effects.

Each replica runs its own @Scheduled method. Ensure only one runs per tick using a lock or a leader.

• ShedLock with a DB or Redis lock row: @SchedulerLock(name = "dailyEmails", lockAtMostFor = "10m", lockAtLeastFor = "1m").
• Or run the job as a Kubernetes CronJob (single run), or use Quartz clustered mode, or leader election.
• Still make the work idempotent: locks with TTL can overlap after a pause.

Retries multiply: A's 3 attempts each cause B's 3 attempts, giving up to 9 calls on C per original request (27 with a third layer), a retry storm that prevents recovery.

• Retry only at the outermost safe layer or use retry budgets.
• Exponential backoff with jitter, and circuit breakers so calls stop while C is down.
• Propagate a deadline so B stops retrying once A has given up.
• Load shedding at C (reject early with 503) and rate limit retries per client.
• Return Retry-After/gRPC retry throttling to coordinate clients.

Use a lease-based election in a consensus-backed store: the Kubernetes Lease object (client-go style, Spring Cloud Kubernetes leader), etcd, or ZooKeeper (Curator LeaderLatch).

• Candidate acquires a lease with TTL and renews periodically; if renewal fails, it must stop work immediately.
• Fencing: pass a monotonic token (lease generation) to downstream writes so a stale leader is rejected.
• Renew interval < TTL/3 and account for clock skew and GC pauses.
• Keep processing idempotent because brief dual-leader windows can occur.

Orchestration: a central workflow engine (Temporal, Camunda, Step Functions) keeps the flow explicit, handles retries, timeouts and visibility.

• Choreography across 8 steps becomes an implicit, hard-to-trace graph of event reactions with cyclic dependency risk.
• A durable workflow engine persists state, offers timers, human tasks, compensation and replay.
• Keep the orchestrator free of business logic; services own rules.
• Choreography remains fine for 2-3 loosely related reactions (e.g. send welcome email on UserRegistered).

Add correlation and measurement: a trace ID everywhere, RED metrics per endpoint and dependency, and SLO-based alerts.

• Micrometer Tracing plus OpenTelemetry collector; inject trace ID into MDC and JSON logs.
• Metrics per downstream: request rate, error rate, latency histograms, circuit breaker state, pool saturation, queue depth, consumer lag.
• Exemplars link a slow histogram bucket to a trace.
• Dashboards by golden signals, runbooks per alert, deploy markers on graphs to correlate changes.
• Structured logs with request context, not stack-trace spam.

The lock lease expired during the pause; another client acquired it, and the paused client resumed and wrote, believing it still held the lock.

• Fix: fencing tokens (monotonic counter returned at lock time) validated by the storage layer (e.g. UPDATE ... WHERE version < :token).
• Alternatively skip the lock: optimistic concurrency (version column) or DB constraints give safety without time assumptions.
• Use a consensus-based lock service when correctness (not just efficiency) depends on it.
• Choose TTL > worst-case operation, plus a watchdog to extend while healthy.

Combine readiness gating, a preStop delay, and graceful draining.

lifecycle:
  preStop:
    exec: { command: ["sleep", "10"] }
terminationGracePeriodSeconds: 40
readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8080 }

• Spring Boot: server.shutdown=graceful, spring.lifecycle.timeout-per-shutdown-phase=30s; readiness flips to REFUSING_TRAFFIC on SIGTERM.
• The preStop sleep lets endpoints/LB deregister before the process stops accepting connections.
• Liveness separate and lenient; startup probe for slow starts.
• Consumers: stop polling and finish in-flight work before exit, commit offsets.

Use expand and contract so old and new app versions work with the schema at every step.

1. Expand: add new nullable column/table (backward compatible).
2. Deploy code that writes both old and new, reads old (or new with fallback).
3. Backfill existing rows in batches.
4. Switch reads to the new column; deploy.
5. Contract: stop writing old, drop it in a later release.

• Avoid long locks: CREATE INDEX CONCURRENTLY (Postgres), online DDL tools (gh-ost, pt-osc).
• Never rename or drop in the same release as code that depends on it.

Classify dependencies as critical or optional, then apply timeouts, breakers, bulkheads and fallbacks per class; fan out in parallel with a global deadline.

• Parallelize independent calls (CompletableFuture/structured concurrency) so latency = max, not sum.
• Optional (recommendations, reviews): short timeout, fallback to cache or empty section; page still renders.
• Critical (inventory, pricing): longer timeout, retry once with jitter, then fail clearly.
• Per-dependency circuit breaker and bulkhead; propagate overall deadline (e.g. 800 ms budget).
• Availability math: five 99.9% critical deps = 99.5%; making three optional raises it to ~99.8%.
• Observe per-dependency error and latency to see which one is spending the budget.