Data Stores & Messaging

Start from access patterns, consistency needs and scale; default to a relational database unless a specific requirement points elsewhere.

• A single Postgres node handles most workloads up to terabytes and thousands of QPS.
• NoSQL usually trades joins and multi-row transactions for horizontal scale and schema flexibility; queries must be designed up front.

Kafka is a partitioned, replicated, retained log for high-throughput streaming and replay; RabbitMQ is a smart broker for flexible routing and task queues; SQS is a fully managed queue with minimal ops.

• Choose Kafka for event streams, multiple independent consumers, replay, CDC; RabbitMQ/SQS for job queues and per-message retry/DLQ.

At-most-once may lose messages (no retry); at-least-once never loses but may duplicate (retry on failure); exactly-once means each message's effect is applied once, achieved via idempotence or transactions, not by the network.

• Practical recipe: at-least-once delivery plus idempotent consumers = effectively-once.
• Kafka: idempotent producer (enable.idempotence=true, default since 3.0) dedupes producer retries per partition; transactions plus read_committed give atomic consume-transform-produce within Kafka.
• Exactly-once across external systems (DB, email) is not possible by the broker alone; make those operations idempotent.

It solves the dual-write problem: updating the DB and publishing an event are two systems and cannot be atomic. Write the event into an outbox table in the same DB transaction as the business change; a separate relay publishes it.

Tx: INSERT order; INSERT outbox(event)  --commit-->  DB
Relay (poller or CDC/Debezium) -> Kafka -> consumers (idempotent)

• Relay options: polling SELECT ... FOR UPDATE SKIP LOCKED, or log-based CDC (Debezium), which avoids polling load.
• Delivery is at-least-once, so events carry a unique ID for consumer deduplication.
• Order within an aggregate is preserved by using the aggregate ID as the partition key.

Instead of storing current state, store an append-only sequence of domain events; state is derived by replaying them (with snapshots for speed).

• Benefits: full audit trail, temporal queries, rebuild new read models, debugging by replay.
• Costs: event schema evolution (upcasters/versioning), eventual consistency of projections, harder ad-hoc queries, learning curve, GDPR deletion (crypto-shredding).
• Rule: events are immutable facts; fix errors with compensating events.
• Fits finance, ledgers, workflows; overkill for plain CRUD.

CQRS separates the write model (commands, invariants) from one or more read models (queries optimized for views), usually updated asynchronously via events.

Command -> Write model (normalized/ES) -> events -> Projector -> Read store (denormalized/ES/Redis)
Query -----------------------------------------------------------^

• Justified when read and write shapes or scale differ greatly (dashboards, search, feed).
• Costs: eventual consistency (read-your-writes UX handling), duplicated models, projection lag and rebuilds.
• Can be used without event sourcing (a simple replica or materialized view).

Order is guaranteed only within a partition. The producer picks the partition by key hash, so all events for a key stay ordered. Within a consumer group, each partition is consumed by exactly one member.

• Max parallelism of a group = number of partitions; extra consumers sit idle.
• Rebalances reassign partitions on member join/leave; cooperative-sticky assignor reduces stop-the-world.
• Offsets committed per group/partition; commit after processing for at-least-once.
• Increasing partition count later changes key-to-partition mapping and breaks per-key ordering for new data.

An operation is idempotent if repeating it has the same effect as doing it once. Implement with an idempotency key stored atomically with the effect.

INSERT INTO processed_events(event_id) VALUES (:id)
ON CONFLICT DO NOTHING;   -- in the same transaction as the business update

• API: client sends Idempotency-Key; server stores key plus response for 24h and replays it on retry (Stripe style).
• Natural idempotency: PUT with full state, SET status = 'PAID', upserts by business key.
• Not idempotent: POST /charge, balance = balance + 10 without a dedup key.
• Same key with different payload should return an error (422/409).

A DLQ stores messages that failed after bounded retries so they stop blocking the queue and can be inspected or replayed.

• Distinguish transient errors (retry with exponential backoff and jitter) from poison messages (non-retryable: deserialization, validation; send to DLQ immediately).
• Kafka has no per-message retry: use retry topics (orders.retry.5s, .1m) then orders.dlq, or Spring Kafka @RetryableTopic.
• SQS: maxReceiveCount redrive to DLQ; RabbitMQ: dead-letter exchanges with TTL for delays.
• Alert on DLQ depth; keep headers with error cause and original topic/offset.

Backpressure is the mechanism by which a slow consumer signals producers to slow down, preventing unbounded buffering and memory exhaustion.

• Pull-based systems (Kafka, SQS) have natural backpressure: consumers fetch at their own pace; lag grows instead of the consumer being overwhelmed.
• Push-based: bounded queues, prefetch limits (RabbitMQ basic.qos), reactive streams request(n), TCP window, HTTP 429/503.
• Strategies when saturated: block, drop (newest/oldest), sample, or shed to a lower tier.
• Monitor consumer lag and queue depth; autoscale consumers on lag up to partition count.

Relational modeling normalizes data and lets the planner choose indexes per query; NoSQL (Dynamo/Cassandra) models tables around known queries, with denormalization and partition-key design.

• DynamoDB: partition key picks the node, sort key orders within it; query by key only; GSIs for alternate access (eventually consistent, extra write cost). Single-table design co-locates related items.
• Cassandra: one table per query, primary key = (partition key, clustering columns); no joins; avoid unbounded partitions (keep < ~100 MB).
• Relational: B-tree indexes, composite index column order (leftmost prefix), covering indexes.
• Hot partition risk in NoSQL when the key is low-cardinality or time-based.

A queue delivers each message to one consumer and removes it (work distribution); a stream is a retained log that many consumers read independently at their own offsets (event distribution and replay). Pub/sub fans a message to all subscribers; point-to-point to one.

• Queue example: image-resize jobs, email sending.
• Stream example: order events consumed by billing, analytics, search indexer.
• Kafka does both: consumers in one group = queue semantics; multiple groups = pub/sub.
• SNS + SQS fan-out gives managed pub/sub with per-consumer queues.

ACID transactions group operations atomically; isolation level determines which concurrency anomalies are allowed.

• *Postgres RR is snapshot isolation, which prevents phantoms but allows write skew; MySQL InnoDB RR uses gap locks.
• Defaults: Postgres read committed, MySQL repeatable read.
• Serializable in Postgres (SSI) may abort transactions; applications must retry.
• Prevent lost updates with SELECT ... FOR UPDATE, optimistic version column, or atomic UPDATE ... SET x = x + 1.

The service does a dual write; a crash or broker outage after commit drops the event. Adopt the transactional outbox.

@Transactional
public void place(Order o) {
    orders.save(o);
    outbox.save(new OutboxEvent(UUID.randomUUID(), "order", o.getId(),
        "OrderCreated", toJson(o)));   // same transaction
}

• A relay (Debezium CDC or scheduled poller) reads the table, publishes with key = order id, then marks/deletes rows.
• Consumers dedupe on event id.
• Expected result: no lost events; duplicates possible but harmless.

Offsets are committed after processing; if a rebalance or crash happens between side effect and commit, another consumer reprocesses the message (at-least-once). The side effect is not idempotent.

• Make the charge idempotent: pass the event/payment ID as the payment provider's idempotency key and record it in a unique-constrained table in the same DB transaction.
• Commit offsets only after success; handle ConsumerRebalanceListener to finish/abandon in-flight work.
• Tune max.poll.interval.ms / max.poll.records so slow processing does not trigger a rebalance.
• Do not rely on Kafka transactions for non-Kafka side effects.

Use the user (or aggregate) ID as the partition key so all of a user's events land in one partition and are handled sequentially by one consumer.

• Order across different users is not needed; this gives parallelism = #partitions.
• Hot users cause partition skew; use a composite key only if ordering is needed per smaller entity.
• Process within the partition sequentially (or per-key sequential workers), since parallel handling in one consumer breaks order.
• Include a per-aggregate sequence number so consumers can detect gaps or out-of-order arrivals.
• Do not increase partitions casually: key mapping changes.

Determine whether production rose or consumption slowed, then scale or fix the bottleneck.

• Check: producer rate, per-partition lag (skew implies hot key), consumer CPU, downstream latency (DB, HTTP), rebalance storms, GC pauses.
• Quick wins: batch DB writes, async I/O with bounded concurrency, increase max.poll.records, add consumers (up to partition count).
• If consumers = partitions already: repartition to a new topic with more partitions and migrate, or parallelize inside the consumer while committing offsets in order.
• Protect downstream with rate limits; temporarily shed low-priority topics.

Match the model to the workload.

• (a) Email jobs: SQS or RabbitMQ; per-message ack, retries, delays, DLQ, no replay needed.
• (b) Clickstream: Kafka (or Kinesis); very high throughput, retained log, replay, stream processing.
• (c) Order events, 5 consumers: Kafka (consumer groups per service, replay for new consumers), or SNS + SQS if managed, simple fan-out suffices.
• Prefer managed offerings (MSK, Confluent, SQS) unless the team can operate brokers.

Write services emit events (via outbox); a projector consumes them idempotently and maintains a denormalized Elasticsearch index optimized for queries.

Catalog DB --outbox--> Kafka --> Projector --> Elasticsearch  <-- Search API
Orders DB  --outbox--> Kafka ----^

• Projection is idempotent using the event version: ignore events with version <= stored version.
• Rebuild by replaying the topic into a new index and swapping an alias (blue/green index).
• Accept seconds of staleness; monitor projection lag; use a fallback to DB for details.

Require an Idempotency-Key header: store key, request hash and result atomically; a retry with the same key returns the stored response instead of charging again.

• Steps: insert key row (unique) in the same transaction as the charge record; concurrent duplicates block or get 409 "in progress".
• TTL keys (e.g. 24 hours); scope per client/account.
• Propagate the key to the payment provider as well.
• On a timeout the client does not know the outcome, so it must retry with the same key and not generate a new one.

For simple CRUD with heavy ad-hoc querying, short-lived data, strict immediate consistency needs, or teams without the tooling, event sourcing adds cost without benefit. Example: an admin settings service.

• Pain points: schema evolution of old events, replay time, projections lag, privacy deletion.
• Alternative: normal tables plus an audit/history table or CDC log to get auditability cheaply.
• Use it where history is the domain (ledger, order lifecycle, inventory movements).

Order only what needs ordering: partition by entity key so each entity is sequential while entities run in parallel.

• SQS FIFO: use MessageGroupId = entity; RabbitMQ: consistent-hash exchange across queues with one consumer per queue.
• If a handler can take arbitrary time, use per-key sequential executors inside the consumer.
• Alternatively make operations commutative or version-based (last-version-wins), so order stops mattering.
• Global total order caps throughput to one partition/leader.

Fix the access pattern first, then scale the same engine, then change engines.

1. EXPLAIN (ANALYZE, BUFFERS); add or fix composite/covering/partial indexes; avoid OFFSET pagination (use keyset).
2. Partition by time or tenant (declarative partitioning) so queries prune and old partitions drop cheaply.
3. Archive cold data; read replicas for reporting; cache hot reads.
4. Vertical scale, connection pooling (PgBouncer).
5. Shard (Citus/Vitess-style) or move specific workloads (search, time series, wide key-value) to purpose-built stores.

Accept events into a durable stream, expand them into per-channel queues, and have rate-limited workers per provider.

Event -> Kafka(notifications) -> Preference/Template service
      -> email queue | sms queue | push queue -> workers(rate limited) -> providers
                                                  \-> retry queue -> DLQ

• Idempotency key per (user, notification) prevents duplicates on retries.
• Worker concurrency and token buckets match provider limits; priority queues so OTPs bypass marketing.
• Per-provider circuit breaker with fallback provider; honour user preferences and quiet hours.
• Batching and campaigns: chunk 10M into pages and enqueue progressively to smooth load.

Dual-run: publish events alongside the existing API, migrate consumers one by one, then retire the sync call.

• Publish with the outbox so events are reliable; define a schema in Avro/Protobuf/JSON Schema in a registry with compatibility rules (BACKWARD for consumers upgrading first).
• Only additive changes: new optional fields, never rename or retype; use new event types for breaking changes.
• Shadow-compare results between old and new paths before cutover.
• Keep events as facts (past tense, OrderPlaced), not RPC in disguise.