A Kafka consumer processes payments and sometimes charges twice after a rebalance. Why and how to fix?
advancedSystem Design › Data Stores & Messaging
Offsets are committed after processing; if a rebalance or crash happens between side effect and commit, another consumer reprocesses the message (at-least-once). The side effect is not idempotent.
- Make the charge idempotent: pass the event/payment ID as the payment provider's idempotency key and record it in a unique-constrained table in the same DB transaction.
- Commit offsets only after success; handle
ConsumerRebalanceListenerto finish/abandon in-flight work. - Tune
max.poll.interval.ms/max.poll.recordsso slow processing does not trigger a rebalance. - Do not rely on Kafka transactions for non-Kafka side effects.
- Would auto-commit help? It can lose messages (commit before processing) or still duplicate.
- Does static membership help? It reduces rebalances for restarts but not for crashes beyond session timeout.