Scalability, Caching & Load Balancing

Vertical scaling makes one machine bigger (simple, bounded, single point of failure); horizontal adds machines (near-unbounded, needs statelessness or partitioning, adds distributed-systems complexity).

• Vertical first when load fits one big node: no data partitioning, ACID intact, easiest ops.
• Horizontal for stateless tiers (trivial behind an LB) and when data or traffic exceeds a single node or you need fault tolerance.
• Stateful tiers scale horizontally via replication (reads) and sharding (writes/data).

Choose by request cost variance and statefulness.

L4 balances TCP/UDP connections on IP and port without parsing payload (fast, protocol-agnostic); L7 terminates HTTP and routes on path, headers, cookies (flexible, costs CPU).

• L7 enables: path routing, TLS termination, header rewriting, canary routing, WAF, gRPC-aware balancing, retries.
• L4 (AWS NLB, IPVS) suits extreme throughput and static IPs; L7 (ALB, Nginx, Envoy) suits HTTP microservices.
• Long-lived HTTP/2 or gRPC connections behind an L4 LB pin to one backend; use L7 or client-side balancing.

A CDN is a geographically distributed cache and proxy that serves content from edge locations near users, cutting latency and origin load.

• Good: static assets, versioned JS/CSS, images, video segments, cacheable API GETs.
• Control with Cache-Control (max-age, s-maxage, stale-while-revalidate), ETag, Vary.
• Invalidate by purge API or, better, by versioned URLs (app.3f9a.js) with long TTLs.
• Bad: personalized or authenticated responses (unless keyed correctly), rapidly changing data.
• Pull CDN fetches from origin on miss; push CDN has you upload content.

They differ in who loads/updates the cache and when the DB is written.

• For cache-aside updates, prefer delete-on-write over update-on-write to avoid races.

Stampede: many requests recompute one expired hot key. Penetration: requests for non-existent keys always hit the DB. Avalanche: many keys expire together or the cache dies.

• Stampede: per-key lock / single-flight, probabilistic early refresh, serve stale while revalidating.
• Penetration: cache negative results with short TTL, Bloom filter in front, input validation.
• Avalanche: add random jitter to TTLs, multi-tier caches, replicated cache, rate-limit and shed load to the DB.

Nodes and keys are hashed onto a ring; a key belongs to the first node clockwise. Adding or removing a node moves only ~1/N of the keys, whereas modulo hashing remaps almost everything.

        key K
          |
   N1 ----v---- N2          K -> N2 (next clockwise)
  /                \         add N4 between N1,N2:
 N4?               N3        only keys between N1 and N4 move

• Virtual nodes (100-200 per physical node) smooth load and allow weighted capacity.
• Used by Dynamo, Cassandra, memcached clients, CDNs, Envoy ring hash.
• Alternatives: rendezvous (HRW) hashing, jump consistent hash, fixed hash slots (Redis Cluster: 16384).

Each trades range-query support against distribution evenness and rebalancing cost.

• Choose a shard key with high cardinality, even access, and that matches the dominant query to avoid scatter-gather.
• Cross-shard joins/transactions are expensive; denormalize or use sagas.
• Resharding: many small logical shards (e.g. 1024) mapped to few nodes lets you move shards instead of re-splitting.

Single-leader is simplest with no write conflicts; multi-leader allows writes in several regions but needs conflict resolution; leaderless (Dynamo-style) uses quorums for availability.

• Leader-follower: reads scale on followers, failover requires promotion and risks losing unreplicated writes (async) or stalling (sync).
• Multi-leader: low local write latency, offline editing; conflicts via LWW, merge, CRDTs; hard to reason about invariants.
• Leaderless: any node accepts writes, read repair and anti-entropy, tunable R/W; no strict ordering.
• Sync vs async vs semi-sync trade durability against write latency.

Partitioning splits a dataset into disjoint pieces (can be within one DB); sharding is partitioning across machines; replication keeps copies of the same data on multiple nodes. They combine: each shard is replicated.

• Partitioning improves manageability and scan pruning (e.g. Postgres declarative partitions by date).
• Sharding adds write/storage capacity.
• Replication adds read capacity and durability/availability, not write capacity.

They bound request rates per client; they differ in burst handling and accuracy.

• Return 429 Too Many Requests with Retry-After.
• Distributed: keep counters in Redis using an atomic Lua script or INCR + EXPIRE, or approximate with local buckets that sync periodically.

It is the single entry point handling cross-cutting concerns: routing, authN/Z, TLS termination, rate limiting, request/response transformation, aggregation, caching, and observability.

• Patterns: Backend-for-Frontend (one gateway per client type), aggregation to cut chattiness.
• Risks: becomes a bottleneck/SPOF if not scaled and replicated; business logic creeping in creates a distributed monolith; extra hop latency.
• Examples: Kong, Spring Cloud Gateway, AWS API Gateway, Envoy/Istio ingress.

LRU is the default for temporal locality; LFU suits stable popularity; TTL bounds staleness; modern caches (Caffeine W-TinyLFU) combine frequency and recency for better hit ratios.

• Size from the working set: hit ratio vs memory curve; 80/20 rule suggests caching the hottest ~20%.
• Redis: maxmemory-policy allkeys-lru (cache) vs noeviction (store); volatile-* only evicts keys with TTL.
• Watch hit ratio, evictions, and p99 of misses; a 95% hit ratio still sends 5% to the DB, which may be the true limit.
• Local (L1) plus distributed (L2) caches cut network hops, at the cost of cross-node invalidation.

Sessions are stored in each instance's memory, so requests landing on a different node lack the session.

• Best: externalize the session (Redis via Spring Session) or use signed tokens (JWT) so any node can serve any request.
• Stopgap: sticky sessions (cookie affinity); loses sessions on node restart and unbalances load.
• Rolling deploys and autoscaling make local state unacceptable.

Use a central atomic counter store for accuracy, or local token buckets with periodic sync to cut latency; choose by tolerance for overshoot.

Client -> Gateway(local bucket) --sync--> Redis cluster (key=user:window)
              |  allowed/429

• Redis token bucket via a Lua script (atomic read-refill-decrement), key sharded by user id; ~1 round trip (0.5 ms) per request.
• Reduce load: lease tokens in chunks (e.g. 50 at a time) to the gateway; error bounded by the lease size.
• Fail open or closed on Redis failure: fail open for general APIs, closed for abuse-sensitive ones.
• Hot users: limit at the edge/CDN first.

Losing the node removed its share of keys, sending that traffic to the DB, which cannot absorb the full uncached load; rising latency causes retries and more load (cascade).

• Replicate cache nodes (Redis replicas/Sentinel/Cluster) and use consistent hashing so only 1/N keys move.
• Protect the DB: connection pool limits, circuit breaker, load shedding, request coalescing.
• Warm the cache before returning a node to rotation; avoid synchronized TTLs.
• Capacity-plan the DB to survive a cache outage at reduced traffic via degraded mode.

Sharding by tenant_id gives locality for tenant queries but a whale tenant overloads one shard; sharding by order_id hash spreads evenly but scatters tenant queries.

• Hybrid: shard by tenant_id with a directory that can pin large tenants to dedicated shards (or split by (tenant_id, hash(order_id) % k)).
• Keep small tenants packed on shared shards with a lookup table for easy moves.
• Use many logical shards so rebalancing means moving a logical shard, not rehashing.
• Global ID: Snowflake-style 64-bit (time + shard + sequence), sortable and coordination-free.

Store originals in object storage (S3), serve through a CDN with long TTLs on immutable, content-hashed URLs, and generate variants lazily.

Upload -> API -> S3 (original) -> event -> resizer workers -> S3 (variants)
Read:  Client -> CDN edge -> (miss) origin shield -> S3 / on-the-fly resizer

• Immutable URLs (/img/<hash>_w400.webp): Cache-Control: public, max-age=31536000, immutable; no purges needed.
• Origin shield collapses misses so S3 sees one request per object, not per edge.
• 50k RPS at 95% edge hit = 2.5k RPS to origin; cache variants to avoid repeated resizing.
• Presigned URLs for direct-to-S3 uploads keep bytes off the app servers.

Changing N from 4 to 5 remaps ~80% of keys (only keys where hash%4 == hash%5 stay), so most lookups miss and hit the DB at once.

• Fix with consistent hashing with virtual nodes (about 1/N of keys move) or fixed hash slots (Redis Cluster migrates slots incrementally).
• Throttle migration, pre-warm, and add the node during low traffic.
• Keep the DB protected with single-flight and rate limits.

Spread the hot key's load: replicate it across several keys or shards, and add a local in-process cache in front.

• Key replication: store profile:42#0..#9 and read a random suffix; writes update all copies (or tolerate TTL expiry).
• Short-TTL local cache (Caffeine, 1-5 s) on app servers absorbs most reads with no network.
• CDN caching for public data.
• Detect hot keys via sampling (Redis --hotkeys, client-side counters) and auto-promote.

Classify reads by staleness tolerance and route accordingly.

• Replica OK: listings, search, analytics, reports, product pages.
• Leader required: read-after-write on user's own data, balance checks, uniqueness checks, anything in a write transaction.
• Implement via separate read-only DataSource/routing (AbstractRoutingDataSource with @Transactional(readOnly = true) routing in Spring) plus lag-aware health checks.
• If lag keeps growing: reduce write bursts, tune replication parallelism, or cache more.

Use separate gateway configurations (BFF pattern) so each client class has its own auth, limits, payload shape and versioning.

• Mobile BFF: aggregate calls to cut round trips, trim payloads, device tokens, per-user limits.
• Partner API: API keys/OAuth2 client credentials, per-key quotas and plans, strict versioning, usage metering, IP allow-lists.
• Shared: TLS termination, WAF, request IDs for tracing, structured access logs, circuit breakers to backends.
• Scale gateway horizontally behind L4/L7 LB; keep it stateless, config as code.

Absorb with caches and queues, protect scarce resources, and shed excess load deliberately.

• Pre-scale and warm (autoscaling is too slow for 10-minute spikes); static content on CDN.
• Waiting room/virtual queue at the edge limits admission to what checkout can handle.
• Inventory: atomic decrement in Redis, async persist; reject early when sold out.
• Rate limit per user/IP, prioritize critical endpoints (checkout over browse), return cached/static fallbacks.
• Backpressure: bounded queues, timeouts, circuit breakers; load shed with 503 + Retry-After rather than collapsing.

Use logical shards and double-write/backfill with a verified cutover, never an in-place rehash.

1. Provision new shards; keep the routing table (directory) authoritative.
2. Backfill a logical shard's data via snapshot plus CDC (change stream) to catch up.
3. Verify with checksums/row counts; briefly pause writes or fence with a version on that logical shard.
4. Flip the routing entry to the new node; keep the old copy read-only for rollback.
5. Delete old data after a soak period.

• Move one logical shard at a time to limit blast radius.

Each places the balancing decision in a different spot with different freshness and operational cost.

• Global traffic: GeoDNS/anycast to the nearest region, then regional L7 LB.
• Pair with health checks and outlier ejection.