Cloud Computing Architecture Guide

Failure, Idempotency, Schema Evolution, and Replay

Goal

Design correct behavior when work is duplicated, delayed, reordered, incompatible, or only partly processed.

The Short Version

Distributed processing cannot make acknowledgement and every business effect one indivisible action. A consumer may commit a payment and fail before acknowledging its message. Retrying is necessary, so repetition must be safe.

Idempotency means repeating the same intended operation produces no additional business effect. It requires a stable business key, an atomic record of the decision, and a retained result. Transport deduplication alone is not enough.

A Recoverable Message Lifecycle

produce -> persist -> deliver -> deduplicate -> process -> acknowledge
                         |             |
                       retry       business result
                         |
                  quarantine -> diagnose -> replay -> reconcile

The transactional outbox records a state change and its message intent together, then publishes later. An inbox or processing ledger records a consumed message key with its business result. These patterns reduce gaps, but still require monitoring and repair.

Retries need timeouts, backoff, jitter, attempt limits, and classification. Permanent validation failures should not consume retry capacity. A dead-letter or quarantine path needs an owner, diagnostic context, retention, and a supported replay procedure.

Change Without Coordinated Deployment

Prefer additive contract evolution: add optional fields, keep existing meanings stable, and make consumers tolerant of fields they do not use. Breaking semantic changes need an explicit new version and migration period. Ordering guarantees usually apply only within a key or partition; design business invariants to that scope.

Replay is a privileged business operation. Before replaying, verify schema compatibility, idempotency retention, downstream side effects, ordering assumptions, current business state, and the intended time range. Reconciliation compares authoritative outcomes and repairs drift when replay alone is unsafe.

Example: Duplicate Payment Notification

A consumer records a payment and crashes before acknowledgement. Delivery repeats. The handler checks the provider transaction ID and order ID in the same transaction as its state change, returns the existing result, and does not charge or advance the order twice.

Later, a deployment rejects an older optional field and messages enter quarantine. The team fixes the tolerant parser, tests a sample against current consumers, replays a bounded range, and reconciles payment and order authorities. It does not press "replay all" and hope.

Common Mistakes

  • Treating exactly-once delivery as removal of application idempotency.

  • Retrying non-idempotent operations without a stable business key.

  • Letting dead-letter storage become an unowned data graveyard.

  • Making a field optional syntactically while changing its meaning.

  • Replaying old records without checking current consumers and side effects.

If Someone Asks You

How do you prevent duplicate messages from causing duplicate business actions?

"I give the intended action a stable idempotency key, atomically store the processing decision with the business effect, return the stored result on repetition, and retain that decision for at least the retry and replay window. I also reconcile authoritative outcomes."

What You Should Remember

Assume repetition and change; design business effects to remain correct.

Questions to Check Your Understanding

  1. Where must an idempotency decision be stored?

  2. What turns a dead-letter path into an operational recovery mechanism?

  3. What must be verified before historical replay?

Further Study