Durable Work, Retries, Dead Letters, and Idempotency

Goal

Event routing and handlers are enough for many reactions. They are not enough when the reaction represents business work that must be completed, tracked, retried, or investigated.

This is where durable processing enters the design.

The durable idea is:

Durable event-driven work needs retry, idempotency, dead-letter, and recovery design.

The Short Version

Event notification is good at saying:

Something happened.

Durable processing is good at protecting messages that mean:

This work must be processed reliably.

That distinction is the heart of the design.

Event notification
  -> Handler decides work is required
  -> Durable queue or broker
  -> Worker processes with retry
  -> Dead-letter or recovery path
  -> Idempotent side effect

An event can start the flow. A durable queue can protect the work.

When Notification Is Not Enough

Notification may be enough when a reaction is optional, lightweight, or easy to repeat. For example, a cache warmer may react to a product update, or an audit subscriber may record a fact that can be rebuilt from source data.

Notification becomes insufficient when the business says:

• This work must not disappear if the handler is offline.

• Processing can take longer than a quick handler should run.

• Traffic spikes need buffering.

• Failed items need investigation and replay.

• A receiver must process work at its own pace.

• Duplicate processing must be controlled.

At that point, the architecture needs a durable work boundary.

Retries and Idempotency

Retries are useful when failures are temporary. A database may throttle. A downstream API may time out. A worker may restart. Retrying can let the system recover without human intervention.

Retries are also dangerous when side effects are not safe to repeat. A duplicated fulfillment message should not ship two boxes. A repeated notification should not send five customer emails. A repeated payment operation should not capture money twice.

Idempotency protects against that risk. A handler can use business keys, event IDs, deduplication records, state transitions, or dependency-level safeguards to make repeated processing safe.

Dead-Letter Handling

A dead-letter queue or failure store is not a trash can. It is a signal that the system found work it could not safely process.

Messages may be dead-lettered because:

• The payload is malformed or missing required data.

• The handler repeatedly fails.

• A dependency rejects the request.

• The message is too old to process safely.

• The event contract changed and the consumer cannot understand it.

• The message represents a business case the handler does not support.

Dead-letter handling needs an owner. Someone must watch counts, inspect messages, decide whether to fix data, replay work, compensate manually, or update code. A dead-letter path without alerts and runbooks can hide failure for days.

Example: Fulfillment Work

An OrderPlaced event is useful for notification. Fulfillment is different. If the fulfillment worker is offline, the order should not disappear. If inventory is temporarily unavailable, the system needs controlled retry or a clear failed state. If the same message is processed twice, the team must avoid duplicate shipments.

A practical design is:

Orders service records order
  -> publishes OrderPlaced
  -> fulfillment handler creates durable work item
  -> fulfillment worker processes queue
  -> repeated failures move to dead-letter queue

The event announces the fact. Durable processing protects the fulfillment work.

Reliability Checklist

Before adding durable processing, answer:

Common Mistakes

• Using notification-only routing for work that must not be lost.

• Retrying non-idempotent handlers.

• Ignoring dead-letter queues.

• Treating backlog as invisible.

• Assuming ordering is global when the real need is usually per order, customer, or workflow.

• Sending large payloads when the message should carry an identifier and context.

If Someone Asks You

When does event-driven work need durable processing?

A good answer is:

I add durable processing when an event creates work that must complete, survive receiver outages, absorb traffic spikes, or be investigated and replayed after failure. Notification says something happened. Durable processing protects the work created by that fact. I also design idempotency, retry policy, dead-letter handling, backlog alerts, ownership, and safe replay.

What You Should Remember

If work must complete, design recovery as part of the flow.

Questions to Check Understanding

1. What problem do dead letters solve?

2. Why can retries create duplicate side effects?

3. How is notification different from durable work?