CH11-04: Reliable Processing with Service Bus and Dead-Letter Handling

Goal

The previous sections used Event Grid for routing and Functions or Container Apps for processing. That is enough for many reactions. It is not enough when the reaction represents business work that must be completed, tracked, retried, or investigated.

This is where durable messaging enters the design.

The durable idea is:

When work must not be lost, add durable processing instead of relying on notification alone.

The Short Version

Event Grid is good at saying:

Something happened.

Service Bus is good at protecting messages that mean:

This work must be processed reliably.

That distinction is the heart of the design.

Event Grid notification
    -> Handler decides work is required
    -> Service Bus queue
    -> Worker processes durable work
    -> Dead-letter path for repeated failure

An event can start the flow. A queue can protect the work.

When Notification Is Not Enough

Notification is often enough when a reaction is optional, lightweight, or easy to repeat. For example, an audit subscriber may record an upload event, or a cache warmer may react when a product changes.

Notification becomes insufficient when the business says:

This work must not disappear if the handler is offline.
Processing can take longer than a quick event handler should run.
Traffic spikes need buffering.
Failed items need investigation and replay.
A receiver must process work at its own pace.
Duplicate processing must be controlled.

At that point, the architecture needs a durable work boundary.

What Service Bus Adds

Azure Service Bus is a managed message broker for reliable work handoff. In event-driven architecture, it often appears after an event handler determines that the event creates business work.

Service Bus helps with:

Capability	Architecture Meaning
Queues	One durable work stream for one receiver or receiver group.
Topics and subscriptions	Multiple durable subscriber paths, each with its own processing responsibility.
Message locks	A worker can receive a message while the broker keeps control until completion.
Delivery count	Repeated processing attempts can be counted and limited.
Dead-letter queues	Messages that cannot be processed move to an investigation path.
Sessions	Related messages can be grouped when ordering matters within a business key.
Time-to-live	Work can expire when it is no longer relevant.

These features do not make the application correct by themselves. They give the team controlled places to handle delay, failure, retry, and investigation.

Dead-Letter Handling

A dead-letter queue is not a trash can. It is a signal that the system found work it could not safely process.

Messages may be dead-lettered because:

The payload is malformed or missing required data.
The handler repeatedly fails.
A dependency rejects the request.
The message is too old to process safely.
The event contract changed and the consumer cannot understand it.
The message represents a business case the handler does not support.

Dead-letter handling needs an owner. Someone must review counts, inspect messages, decide whether to fix data, replay work, compensate manually, or update code. A queue without dead-letter monitoring can hide failure for days.

Example: Order Fulfillment Work

Imagine an OrderCreated event. Several systems may want to know the order exists, so Event Grid can route the notification.

Fulfillment is different. If the fulfillment worker is offline, the order should not disappear. If inventory is temporarily unavailable, the system needs controlled retry or a clear failed state. If the same order message is processed twice, the team must avoid duplicate shipments.

A practical design is:

Order service records order
    -> publishes OrderCreated event
    -> Event Grid routes notification
    -> fulfillment handler creates Service Bus work item
    -> fulfillment worker processes queue
    -> repeated failures move to dead-letter queue

Event Grid announces the fact. Service Bus protects the fulfillment work.

Reliability Design Checklist

Before adding Service Bus, answer these questions:

Question	Good Design Signal
What work does the message represent?	The message has a clear business meaning.
Who owns the worker?	A team is accountable for processing and failures.
Can processing happen twice safely?	Idempotency or duplicate detection is designed.
What failures are temporary?	Retry policy distinguishes transient and permanent failures.
What goes to dead-letter?	Poison messages and unsupported cases have a visible path.
Who watches backlog?	Queue depth, message age, and dead-letter counts have alerts.
How is work replayed?	Replay steps are documented and safe.

Common Mistakes

Treating Event Grid retry behavior as a complete guarantee for business work.
Adding queues but never assigning ownership for backlog, message age, or dead-letter review.
Retrying non-idempotent work and creating duplicate business effects.
Sending large payloads through the broker when the message should carry an identifier and the data should live elsewhere.
Assuming ordering is global when the real requirement is usually narrower, such as ordering per order, customer, or document.

If Someone Asks You

When should Service Bus be added to an event-driven architecture?

A good answer is:

I add Service Bus when an event creates work that must be processed reliably. Event Grid can route the notification that something happened, but Service Bus gives the design durable queues or subscriptions, retries, message locks, delivery counts, and dead-letter paths. I also define idempotency, monitoring, ownership, and replay because durable messaging makes failure manageable, not automatic.

What You Should Remember

Use Service Bus when the design needs durable work handling, not just event notification.

Questions to Check Your Understanding

What problem does a dead-letter queue reveal?
Why is durable work different from event notification?
Who should own backlog and failure monitoring?