High-Level Design

Generate a unique short code per long URL, store the mapping, and serve redirects from a cache-fronted key-value store; the system is read-heavy and latency sensitive.

• Requirements: shorten, redirect (301/302), custom alias, expiry, click analytics; 99.99% availability on redirect.
• Estimates: 100M new URLs/month (40 writes/s), 100:1 read ratio (4K reads/s, peak 10x); 5 years = 6B rows x ~500 B = ~3 TB. 7 base62 chars = 62^7 = 3.5T codes.
• API: POST /v1/urls {longUrl, alias?, ttl?} -> {shortUrl}; GET /{code} -> 302 Location.
• Data model: urls(code PK, long_url, user_id, created_at, expires_at); key-value or sharded SQL by code.

Client -> CDN/LB -> Redirect Service -> Redis (code->url) -> DB (sharded by code)
                 \-> Create Service -> Key Gen Service (range allocator) -> DB
Redirect Service -> Kafka (click events) -> Analytics

• Key generation: a counter service hands each app node a range (e.g., 1M ids) from ZooKeeper/DB, encoded to base62; no collisions, no per-write coordination. Hashing (MD5 truncated) needs collision retries.
• 301 is cacheable by browsers (less load, lost analytics); 302 keeps every click visible.
• Trade-offs: sequential ids are guessable (permute bits); range allocation wastes ids on crash (fine).
• Failure modes: cache outage sends load to DB, so use replicas and request coalescing; allocator down means nodes use their remaining range; expired links are cleaned lazily plus a TTL sweep.

Store paste content in object storage (S3), metadata in a database, and generate IDs like the URL shortener; reads are served through a CDN.

• Requirements: create paste (text up to 10 MB), optional expiry and visibility, view by link, optional syntax highlighting.
• Estimates: 1M pastes/day (~12 writes/s), 5:1 reads; avg 10 KB gives ~10 GB/day = 3.6 TB/year.
• API: POST /v1/pastes {content, expiresIn, visibility}; GET /v1/pastes/{id}.
• Data model: pastes(id, owner, s3_key, created_at, expires_at, visibility); content blob in S3 keyed by id.

Client -> LB -> API -> Key Gen -> Metadata DB
                  \-> S3 (content) ; Client <- CDN <- S3 (reads)
Expiry Worker -> deletes expired metadata + objects

• Deep dive: write content to S3 first, then metadata; orphaned blobs are swept by a reconciler. Use S3 lifecycle rules for TTL.
• Dedup by content hash (SHA-256) saves storage but complicates deletion (refcount).
• Trade-offs: storing small pastes inline in DB speeds reads; big pastes in object storage keeps DB small.
• Failure modes: S3 write succeeded but DB failed (orphan, reconciled); abuse/spam needs rate limiting and malware scanning.

Run a shared limiter keyed by user/IP/API-key in Redis with atomic Lua scripts, evaluated at the gateway; choose token bucket or sliding-window counter based on burst tolerance.

• Requirements: per-key limits with multiple rules, low added latency (<5 ms), accurate enough, fail-open or fail-closed decided per route, return 429 with Retry-After.
• Estimates: 1M req/s peak across 50M active keys; each key ~100 B = ~5 GB in Redis, so shard by key.
• API: allow(key, rule) -> {allowed, remaining, resetAt} plus X-RateLimit-* headers.
• Data model: Redis hash per key {tokens, last_refill} with TTL; rules in a config store, cached locally.

Client -> API Gateway -> Limiter lib -> Redis Cluster (Lua: refill+decrement)
                              \-> local cache of rules (from Config Service)

• Atomicity: a Lua script reads, refills, decrements and writes in one step, avoiding race conditions between gateway nodes.
• Reduce Redis calls: each node pre-fetches a batch of tokens (local leases) at the cost of slight over/under-limit.
• Trade-offs: fixed window is cheap but allows 2x bursts at boundaries; sliding window log is exact but memory heavy; token bucket supports bursts.
• Failure modes: Redis down means fail-open for general APIs (availability) and fail-closed for expensive/abuse-prone ones; clock skew is avoided by using Redis TIME; hot keys are split across shards.

Use fan-out on write (push) for normal users, precomputing each follower's feed in a cache, and fan-out on read (pull) for celebrities; merge at read time.

• Requirements: post, follow, home timeline (reverse chronological or ranked), low-latency reads (<200 ms), eventual consistency acceptable.
• Estimates: 300M DAU, 5 reads/day per user-session x 10 = 1.5B timeline reads/day (17K/s avg, 50K peak); 100M posts/day (1.2K/s). Average 200 followers means 240K feed inserts/s.
• API: POST /v1/posts, GET /v1/feed?cursor=; cursor-based pagination.
• Data model: posts(post_id, user_id, text, media_url, ts), follows(follower, followee), feed cache feed:{userId} as a Redis sorted set of post ids (cap 800).

Post API -> Post DB -> Kafka(post-created) -> Fanout Workers -> Redis feed:{follower}
Feed API -> Redis feed (ids) -> Post Cache/DB (hydrate) ; + celebrity posts pulled & merged

• Hybrid: skip fan-out for accounts with >1M followers; at read time merge the cached feed with recent posts of followed celebrities.
• Ranking: candidate generation then ML scoring; keep the pipeline separate from storage.
• Trade-offs: push gives fast reads but heavy writes and wasted work for inactive users (fan-out only to active users); pull is cheap on write but slow on read.
• Failure modes: cache loss rebuilt from follows + posts; fan-out lag means a delayed post appears (acceptable); deleted posts filtered on hydration.

Keep persistent WebSocket connections to chat servers, route messages via a presence/session registry and a message queue, and persist messages in a wide-column store keyed by conversation.

• Requirements: 1:1 and group chat (up to 500), delivery and read receipts, online presence, offline delivery, history, media; message ordering per conversation.
• Estimates: 500M DAU x 40 messages = 20B msgs/day (~230K/s); 100 B/msg = 2 TB/day. 100M concurrent connections at ~100K per server = ~1000 servers.
• API: WebSocket frames send{convId, clientMsgId, body}, ack, typing; REST for history GET /conversations/{id}/messages?before=.
• Data model: Cassandra messages((conv_id), msg_id) with time-sortable ids (Snowflake); conversations, members.

Client <-WS-> Gateway(chat server) -> Session Registry (Redis: user->server)
Sender GW -> Message Service -> Kafka -> Delivery Worker -> Receiver GW -> Client
                          \-> Message Store (Cassandra) ; Offline -> Push (APNs/FCM)

• Delivery: sender sends with clientMsgId (idempotent); server persists, acks, then forwards; receiver acks; retries until acked.
• Ordering: sequence per conversation (one partition key = conv_id in Kafka).
• Groups: fan-out to member sessions; large groups use fan-out via queue.
• Trade-offs: store-and-forward vs transient; E2E encryption means servers cannot index content.
• Failure modes: gateway crash drops connections, clients reconnect and sync from last seen msg id; duplicates handled by clientMsgId; presence is heartbeat-based with TTL to avoid thundering updates.

An ingestion API validates requests, applies user preferences and rate limits, renders templates, and enqueues per-channel messages that workers send through providers with retry and dedup.

• Requirements: multi-channel, templating, user opt-out, scheduling, priority (OTP over marketing), at-least-once with dedup, delivery tracking.
• Estimates: 10M users, 5 notifications/day = 50M/day (~600/s avg, 10x peak at campaigns).
• API: POST /v1/notifications {userId, type, params, idempotencyKey}.
• Data model: preferences(user, channel, enabled), templates, notification_log(id, user, channel, status, ts), device tokens table.

Producers -> Notification API -> Kafka topics (otp / transactional / marketing)
Channel Workers (email|sms|push) -> Provider (SES/Twilio/FCM) -> status callbacks -> Log DB
Scheduler -> Kafka (delayed) ; DLQ for exhausted retries

• Separate queues per priority and channel so a marketing blast cannot delay OTPs.
• Retry with exponential backoff and jitter; fail over to a secondary provider; respect provider rate limits with a token bucket per provider.
• Dedup via idempotency key stored with a TTL.
• Trade-offs: at-least-once means rare duplicates; exactly-once is impractical with external providers.
• Failure modes: provider outage (circuit breaker + fallback), invalid device tokens (prune on feedback), poison messages go to DLQ.

Split files into content-addressed chunks, upload only changed chunks to object storage, keep file-to-chunk metadata in a database, and notify other devices to sync.

• Requirements: upload/download, multi-device sync, versioning, sharing, resumable transfers, large files (GBs), conflict handling.
• Estimates: 50M users x 10 GB = 500 PB raw; 10M DAU, 2 file changes/day = 230 changes/s. Chunk size 4 MB.
• API: POST /files/{id}/chunks:upload-url, PUT chunk to pre-signed URL, POST /files/{id}/commit {chunkHashes}; GET /changes?cursor=.
• Data model: files(file_id, owner, path, version), file_versions(file_id, version, chunk_list), chunks(hash, s3_key, refcount).

Client(chunker, hash) -> API -> Metadata DB (sharded by user)
Client -> pre-signed PUT -> Object Store ; Notification Svc (long poll/WS) -> other clients
Block dedup index: hash -> exists?

• Delta sync: client hashes chunks, asks which are missing, uploads only those; dedup is global by hash (cross-user dedup raises privacy side-channels).
• Conflicts: last-writer-wins with a "conflicted copy" file; vector of version numbers per file.
• Trade-offs: small chunks give better dedup but more metadata; large chunks fewer requests.
• Failure modes: partial upload resumes via chunk list; garbage collect chunks with refcount 0 after a grace period; metadata DB sharding must keep a user's data on one shard.

Drivers stream locations into a geospatial index; a matching service finds nearby drivers for a ride request and runs the trip through a state machine, with payments and ETA as separate services.

• Requirements: request ride, match nearest driver, live tracking, ETA and fare, surge pricing, trip history; match within a few seconds.
• Estimates: 1M drivers online sending location every 4 s = 250K writes/s; 20M rides/day (~230/s, peak 1K/s).
• API: POST /rides {pickup, dest}, PUT /drivers/location, POST /rides/{id}/accept, WebSocket for live updates.
• Data model: trips(id, rider, driver, state, fare, ...) in SQL; driver location in an in-memory geo index (H3/geohash grid in Redis or custom service), not a durable DB.

Driver App -> Location Svc -> Geo Index (sharded by cell, in-memory)
Rider App -> Ride Svc -> Matching Svc -> Geo Index -> offer to driver (push/WS)
Ride Svc -> Trip DB ; Pricing Svc ; Maps/ETA Svc ; Payment Svc ; Kafka events

• Geo index: partition the map into cells; query the rider's cell plus neighbours; rank by ETA (road distance) not straight line.
• Matching: offer to one driver at a time with a 10-15 s timeout, or batch matching in windows to optimise globally.
• Consistency: reserving a driver must be atomic (single owner per driver id, e.g., sharded by driver).
• Surge: compute demand/supply ratio per cell over a sliding window.
• Failure modes: location staleness (TTL on entries), matching service crash (trip state in DB, retry), double assignment prevented by the atomic reservation.

Accept payment requests with an idempotency key, orchestrate calls to a PSP through a state machine, and record every money movement in an append-only double-entry ledger with periodic reconciliation.

• Requirements: charge, refund, multiple payment methods, exactly-once effect, auditability, PCI compliance, high consistency.
• Estimates: 1M payments/day (~12/s, peak 100/s); low QPS, extreme correctness requirement.
• API: POST /v1/payments {orderId, amount, currency, method} with Idempotency-Key header; GET /payments/{id}; webhooks from PSP.
• Data model: payments(id, order, amount, state, idem_key UNIQUE), ledger_entries(id, txn_id, account, debit/credit, amount); amounts in minor units.

Checkout -> Payment API -> Payment DB (state) -> PSP Adapter -> PSP (Stripe/Adyen)
PSP webhook -> Webhook Handler (verify signature) -> update state -> Ledger -> Kafka events
Reconciliation Job: PSP settlement files vs ledger

• Idempotency: unique key stored with the request hash and response; replays return the stored result.
• State machine: CREATED -> PENDING -> SUCCEEDED/FAILED; timeouts are unknown outcomes, so query the PSP status before retrying and never blindly re-charge.
• Use the outbox pattern to publish events atomically with DB updates.
• Trade-offs: synchronous confirmation vs async webhook; never store raw card data (tokenize via PSP).
• Failure modes: PSP timeout (unknown state, poll and reconcile), duplicate webhooks (idempotent handler), ledger imbalance caught by reconciliation.

Serve top-K suggestions per prefix from a precomputed trie or prefix index in memory/cache, refreshed offline from aggregated query logs.

• Requirements: suggestions within 100 ms as the user types, top 5-10 by popularity, personalisation optional, freshness within hours.
• Estimates: 10B searches/day, each ~4 prefix requests with debouncing = 400K req/s peak, so aggressive caching and CDN.
• API: GET /v1/suggest?q=prefix&limit=5.
• Data model: trie nodes store top-K suggestions at each node (precomputed), or Redis prefix -> [top K].

Client(debounce) -> CDN/Edge cache -> Suggest Svc -> Trie/Redis (prefix -> top K)
Search logs -> Kafka -> Aggregator (Spark/Flink, counts per query) -> Trie Builder -> snapshot -> Suggest nodes

• Storing top-K at each node makes lookup O(prefix length); memory cost is controlled by limiting depth and pruning rare queries.
• Build offline daily plus a small real-time layer for trending terms; swap snapshots atomically.
• Trade-offs: freshness vs build cost; personalised results hurt cacheability.
• Failure modes: stale snapshot (serve the old one), hot prefixes (cache), offensive suggestions (blocklist filter).

Upload raw video to object storage, transcode into multiple resolutions and chunked adaptive streams (HLS/DASH) through a pipeline, and deliver via CDN with adaptive bitrate on the client.

• Requirements: upload, playback with minimal buffering, many devices, search, recommendations; global.
• Estimates: 500 hours uploaded per minute; a 1-hour video ~ 1 GB raw, ~5 GB across renditions; egress is dominated by CDN, petabytes/day.
• API: POST /videos (pre-signed upload), GET /videos/{id}/manifest.m3u8, segment fetches from CDN.
• Data model: videos(id, owner, title, status, renditions) in SQL/NoSQL; segments and manifests in object storage; view counts in a stream-aggregated store.

Uploader -> Object Store (raw) -> Kafka -> Transcoder Workers (DAG) -> Object Store (HLS segments)
Player -> CDN edge -> origin (object store) ; Metadata DB + Search Index

• Transcoding: split into GOP-aligned chunks, process in parallel, stitch the manifest; produce 240p-4K variants.
• Adaptive bitrate: the client picks the rendition per segment (2-6 s segments) from bandwidth estimation.
• CDN: push popular titles to edges (Netflix Open Connect), pull-through for the long tail.
• Trade-offs: more renditions cost storage and compute; generate rare ones lazily.
• Failure modes: transcode failures retry per chunk; CDN miss storms on viral videos mitigated by origin shielding and request collapsing.

A distributed frontier of URLs feeds fetcher workers that download, parse and extract links; dedup (URL seen set, content hash) and per-host politeness keep it correct and ethical.

• Requirements: crawl 1B pages/month, respect robots.txt, avoid duplicates and traps, prioritise important pages, refresh periodically.
• Estimates: 1B pages/month = ~400 pages/s; 100 KB/page = 100 TB/month; storage ~1.2 PB/year.
• API (internal): seed(urls), frontier.next(host), store(page).
• Data model: URL frontier (priority + per-host queues), seen_urls (Bloom filter + persistent store), page store in object storage/HDFS, link graph.

Seed -> URL Frontier (priority queues -> per-host queues) -> Fetchers (DNS cache, robots.txt)
Fetchers -> Parser -> Link Extractor -> URL Filter -> Dedup (Bloom/seen) -> Frontier
Parser -> Content Hash (simhash) -> Page Store -> Indexer

• Politeness: one queue per host with a crawl-delay; map hosts to workers (consistent hashing) so one worker owns a host.
• Dedup: Bloom filter for URLs (false positives skip a few new pages), simhash for near-duplicate content.
• Traps: infinite calendars/session ids; cap depth per host and normalise URLs.
• Trade-offs: BFS vs priority (PageRank, freshness); breadth vs recrawl rate.
• Failure modes: fetcher crash (lease expiry returns URL to frontier), slow hosts (timeouts, backoff), DNS bottleneck (local cache).

Partition keys across nodes with consistent hashing (virtual nodes), replicate for availability, evict with LRU/LFU, and let clients or a proxy route requests.

• Requirements: sub-ms get/put, horizontal scale, high availability, TTLs, eviction, optional persistence.
• Estimates: 1M ops/s, 1 TB data; with 64 GB per node, ~20 nodes (plus replicas).
• API: GET key, SET key value ttl, DEL key, INCR.
• Data model: in-memory hash table per node, LRU/LFU lists, slot or hash-ring mapping stored in a config service.

Client lib / Proxy -> Hash Ring (vnodes) -> Node A (primary) -> replica A'
                                         -> Node B ...      ; Config/Gossip for membership

• Routing: consistent hashing minimises key movement on node add/remove (about K/N keys); Redis Cluster uses 16384 hash slots instead.
• Replication: async primary-replica for speed (can lose last writes); failover through sentinel/gossip.
• Hot keys: replicate hot keys or add a local L1 cache; stampede: single-flight loading and TTL jitter.
• Patterns: cache-aside (default), write-through, write-behind; invalidation is the hard part.
• Failure modes: node loss (replica promoted, rewarm), network partition (split brain, so require quorum), thundering herd on mass expiry.

Use a Snowflake-style 64-bit ID composed of timestamp, machine id and sequence, giving roughly time-ordered, unique ids without central coordination.

• Requirements: globally unique, 64-bit, roughly sortable by time, 10K+ ids/s per node, no single point of failure.
• Layout: 1 sign bit + 41 bits ms timestamp (~69 years) + 10 bits worker id (1024 nodes) + 12 bits sequence (4096/ms/node).
• Estimates: 1024 nodes x 4M/s = ~4B ids/s theoretical capacity.
• API: nextId() -> long (library or service).

App/IdService node -> [timestamp | workerId | sequence]
Worker id assignment: ZooKeeper/etcd lease or DB at startup

• Same millisecond: increment sequence; on overflow wait for the next millisecond.
• Clock moving backwards: refuse/wait until the clock catches up, or use a logical clock; NTP slew helps.
• Alternatives: UUIDv4 (no ordering, 128-bit, poor index locality), UUIDv7 (time-ordered), DB ticket servers, range allocation, ULID.
• Failure modes: duplicate worker ids (lease them), clock drift, sequence reset.

Checkout is a saga across cart, inventory, payment and order services; reserve inventory with expiry, charge, confirm the order, and compensate on failure.

• Requirements: cart to order, price/tax/discount calculation, stock reservation (no oversell), payment, order confirmation emails; flash-sale resilience.
• Estimates: 10M orders/day (~120/s), flash-sale peak 50K checkout attempts/s on a few SKUs.
• API: POST /checkout {cartId, addressId, paymentMethod} (with idempotency key), GET /orders/{id}.
• Data model: orders(id, user, state, total), order_items, inventory(sku, available, reserved), reservations(id, sku, qty, expires_at).

Client -> API GW -> Checkout Orchestrator -> Pricing Svc
                         |-> Inventory Svc (reserve w/ TTL) -> Inventory DB
                         |-> Payment Svc -> PSP
                         '-> Order Svc -> Kafka -> Email/Shipping/Analytics

• Saga: reserve inventory -> create order PENDING -> charge -> mark CONFIRMED; on payment failure release the reservation (compensation).
• Oversell prevention: conditional update UPDATE inventory SET available = available - ? WHERE sku = ? AND available >= ?; for hot SKUs use Redis counters or queue-based serialisation.
• Orchestration (central, easier to reason about) vs choreography (events, looser coupling).
• Failure modes: reservation expiry job returns stock; duplicate checkout clicks (idempotency key); payment succeeded but order write failed (outbox + reconciliation).

Make seat or ticket reservation an atomic conditional update with a hold TTL, put a virtual waiting room in front to shed load, and confirm only after payment.

• Requirements: view availability, hold seats for ~10 minutes, pay, no double booking, handle 1M users hitting one event.
• Estimates: 1M concurrent users, 50K booking attempts/s on a hot event; inventory only 50K seats, so most requests must fail fast.
• API: GET /events/{id}/seats, POST /holds {eventId, seatIds}, POST /bookings {holdId, paymentToken}.
• Data model: seats(event_id, seat_id, status, hold_id, hold_expires_at), bookings, holds.

Users -> CDN -> Waiting Room (queue token) -> API -> Hold Svc
Hold Svc -> Seat DB (UPDATE ... WHERE status='AVAILABLE') / Redis (SET NX EX)
Payment -> Booking Svc -> Seat DB (HELD->BOOKED) ; Expiry Worker releases holds

• Atomic hold: UPDATE seats SET status='HELD', hold_id=? WHERE seat_id IN (...) AND status='AVAILABLE' and check affected rows equals requested count; else roll back.
• Alternatives: pessimistic SELECT ... FOR UPDATE (simple, contention), optimistic version column, Redis SET NX with TTL for hot events (database remains the source of truth).
• Waiting room admits N users per second so downstream stays within capacity; availability reads served from cache (stale OK).
• Failure modes: payment success after hold expiry (re-verify, refund if lost), Redis loss (rebuild from DB), bots (CAPTCHA, per-user limits).

Use a Redis sorted set keyed by score for O(log N) updates and rank queries; shard by leaderboard or score range when a single set outgrows one node.

• Requirements: update score, top-K, rank of a user, nearby ranks, daily/weekly/all-time boards, near real-time.
• Estimates: 50M players, 10K score updates/s; one sorted set of 50M members ~ 5 GB; top-100 reads at 20K/s (cacheable for a second).
• API: POST /scores {userId, delta}, GET /leaderboard/top?k=100, GET /leaderboard/rank/{userId}.
• Data model: Redis ZADD lb:2026-W40 score user; ZREVRANGE top-K, ZREVRANK rank; durable scores in SQL.

Game Svc -> Kafka(score events) -> Score Consumer -> Redis ZSET (per board)
API -> Redis ZSET / top-K cache ; Scheduled job snapshots + resets weekly board
Durable store (SQL) <- consumer (source of truth)

• Beyond one node: shard by score bucket and compute global rank as sum of higher buckets' sizes plus local rank; approximate rank for the tail (percentile).
• Time windows: separate sets per window with TTL; all-time is the largest.
• Ties: encode (score, timestamp) into a single score or secondary ordering.
• Failure modes: Redis loss (rebuild from the durable store/Kafka replay), hot top-K reads (cache), duplicates (idempotent events).

Agents on each host ship logs and metrics to Kafka; stream processors parse and aggregate; logs are indexed in a search store and metrics in a time-series database with tiered retention.

• Requirements: ingest from thousands of services, search logs within seconds, dashboards and alerts on metrics, retention 15 days hot / 1 year cold, no loss of critical logs.
• Estimates: 10K hosts x 1 MB/s logs = 10 GB/s ingest peak ~ 800 TB/day raw (sample/compress; 5x compression); metrics 10M series x 1 point/10s = 1M points/s.
• API: agents push over gRPC/HTTP; query APIs GET /logs?q=, PromQL-style metrics queries.
• Data model: logs as JSON with ts, service, level, trace_id; time-series metric{labels} -> (ts, value).

Host Agent (Fluent Bit / OTel) -> Kafka -> Stream Proc (parse, enrich) -> Elasticsearch/ClickHouse (hot)
                                        -> S3 (cold, Parquet) ; Metrics -> TSDB (Prometheus/M3) -> Alert Manager

• Kafka decouples bursts from storage; partition by service; backpressure through bounded agent buffers.
• Cost control: sampling, dropping debug logs, downsampling metrics (1m, 1h rollups), index only needed fields.
• Alerting evaluates rules on the TSDB, deduplicates and routes (PagerDuty).
• Trade-offs: index everything (expensive, fast search) vs index labels only (Loki style).
• Failure modes: pipeline lag (alerts on consumer lag), agent disk buffers for network loss, cardinality explosion in metrics labels.

A gateway is the single entry point that terminates TLS, authenticates, rate-limits, routes to services via a dynamic route table, and emits telemetry; it must be stateless and horizontally scalable.

• Requirements: routing, authN/Z (JWT/OAuth), rate limiting, request/response transformation, circuit breaking, canary routing, observability, <5 ms added latency.
• Estimates: 200K req/s peak; each node ~20K req/s, so ~15 nodes across AZs.
• API: admin API for routes, upstreams, plugins; data plane forwards by host/path.
• Data model: route config (path -> upstream, plugins, limits) in etcd/DB, pushed to nodes and cached; JWKS cache for token validation.

Client -> Global LB/WAF -> Gateway Cluster [TLS | AuthN | RateLimit | Route | LB | CircuitBreaker]
                                   -> Service A / B / C (service discovery)
Control Plane -> Config Store -> push to gateways ; Telemetry -> Metrics/Tracing

• Auth: validate JWT locally with cached keys (no remote call per request); introspect only opaque tokens, with a short cache.
• Resilience: timeouts, retries only for idempotent calls, circuit breakers, bulkheads per upstream.
• Trade-offs: central gateway risks becoming a bottleneck/SPOF and a team dependency; keep business logic out of it; BFF gateways per client type.
• Failure modes: bad config push (validate and roll out gradually), upstream slowness (timeouts, load shedding), key rotation (JWKS refresh).

Store job definitions and next-run times in a database, have schedulers claim due jobs with leases, push them to a queue, and workers execute with at-least-once semantics so jobs must be idempotent.

• Requirements: cron and one-off jobs, retries with backoff, priorities, millions of jobs, no missed or double runs at scale, visibility and cancellation.
• Estimates: 10M scheduled jobs, 5K executions/s peak at the top of the minute (cron spike); execution history ~1 KB/run.
• API: POST /jobs {cron|runAt, payload, retryPolicy}, DELETE /jobs/{id}, GET /jobs/{id}/runs.
• Data model: jobs(id, schedule, next_run_at, status, owner) indexed on next_run_at; job_runs(id, job_id, state, attempt, started_at, lease_until).

API -> Jobs DB (partitioned) <- Scheduler nodes (poll due jobs per partition, SELECT ... FOR UPDATE SKIP LOCKED)
Scheduler -> Queue (Kafka/SQS) -> Worker Fleet -> update run state ; failed -> retry/DLQ
Leader election / partition ownership via ZooKeeper/etcd

• Claiming: schedulers fetch WHERE next_run_at <= now() LIMIT n FOR UPDATE SKIP LOCKED, set a lease, enqueue, and advance next_run_at in the same transaction.
• Spike handling: add jitter to cron times, partition jobs by hash across scheduler instances.
• Execution guarantees: at-least-once plus idempotency keys (job id + scheduled time); heartbeats extend leases; expired leases are re-queued.
• Trade-offs: DB polling (simple, a few thousand/s per shard) vs hierarchical timing wheels in memory for sub-second precision.
• Failure modes: scheduler crash (lease expiry, another takes over), worker crash (visibility timeout), clock skew (use DB time), long-running jobs (heartbeats, timeouts).