CH17-04: Durable Workflow, Compensation, and Exception Handling

Goal

This section explains how durable workflow state, retries, compensation, exception queues, and human review protect back-office automation when long-running work fails in pieces.

The previous section sketched an Azure target architecture for workflow, messaging, operational stores, and dashboards. Now we need the reliability model behind that architecture.

The Short Version

Long-running operations fail in pieces, so the architecture must know what has happened and what to do next.

Back-office workflows are rarely one quick transaction. They may wait for a warehouse scan, a carrier response, a supplier update, a fraud review, a refund approval, or a finance posting. Some steps are automated. Some depend on external systems. Some need a person. Some can be retried safely. Some cannot.

A reliable design records workflow state at meaningful checkpoints. It knows which commands have already been sent, which steps can be retried, which steps need compensation, which failures belong in an exception queue, and what evidence operators need to recover safely.

A Failure Path Mental Model

Every workflow step needs a planned outcome path.

Step succeeds
	-> record checkpoint
	-> continue

Step times out
	-> retry if safe
	-> pause if external state is uncertain

Step fails permanently
	-> compensate where possible
	-> send to exception queue

Sensitive or ambiguous step
	-> require review
	-> record decision and evidence

This model is simple, but it prevents a common production problem: the system knows that something failed, but nobody knows whether it is safe to run the step again.

Durable Workflow State

Durable workflow state records the progress of the business process, not just technical logs.

For an order fulfillment workflow, useful checkpoints might include:

Logs are still important, but logs alone are not workflow state. A workflow should answer, Where is this order now, what already happened, and what is the next safe action?

Retries And Idempotency

Retries are useful only when the command can be repeated safely.

An idempotent command can be processed more than once without creating duplicate business effects. For example, reserve inventory for order 123, line 2, reservation request abc is safer than reserve one unit because the receiver can detect that the same request has already been applied.

Design retryable steps with:

• Stable identifiers for order, line item, reservation, workflow, and command.

• Duplicate detection at the receiving service or integration boundary.

• Clear timeout behavior when the caller does not know whether the receiver completed the action.

• Retry policies that avoid overwhelming a struggling dependency.

• Dead-letter or exception paths when retry is no longer useful.

Without idempotency, retries can create duplicate reservations, duplicate shipments, duplicate refunds, duplicate purchase orders, or duplicate customer notifications.

Compensation

Compensation is the action that reduces harm when a completed step must be undone or balanced by another step.

It is not always a perfect rollback. Physical and financial work often cannot be erased. If inventory was reserved but the warehouse cannot fulfill, compensation may release the reservation, reallocate the order, notify the customer, or open a support case. If a refund was issued incorrectly, compensation may require finance review rather than automatic reversal.

Use compensation when:

• A later step fails after an earlier business effect has completed.

• The system needs to maintain a consistent operational story.

• A customer, warehouse, supplier, seller, or finance team must be informed of the changed outcome.

• Undoing the action automatically would be risky, impossible, or noncompliant.

The architecture should name compensation paths during design. Waiting until an incident to ask what should happen is how manual cleanup becomes the real system.

Exception Queues And Human Review

An exception queue is not a dumping ground for errors. It is an operational work surface.

Good exception handling includes:

Some exceptions can be resolved by automation. Others need warehouse operations, supply chain, customer support, finance, security, or platform teams. The architecture should make that routing explicit.

Example: Carrier Label Failure

Suppose inventory is reserved and a warehouse task is created, but the carrier label request fails.

A weak workflow might retry blindly. If the carrier actually created the label but the response timed out, retrying may create a second label. If the workflow gives up silently, the warehouse may have a task but no shipment path. If the order status simply says processing, customer support has little to explain.

A stronger workflow records the checkpoint before the carrier call, sends an idempotent label request with a stable request identifier, handles timeout separately from confirmed failure, checks for an existing label where possible, and moves the order to an exception queue if the carrier state remains uncertain. Operators can then see the order, carrier request, warehouse task, customer promise, and next recovery action.

The design avoids losing the order, double-reserving inventory, or leaving operators blind.

Replay And Reprocessing

Replay is powerful, but it should be controlled.

Back-office replay may mean rerunning a failed command, resuming a workflow from a checkpoint, rebuilding an operational dashboard, reprocessing a carrier event, or republishing a status notification. Each case has different risk.

Safe replay needs:

• Idempotent handlers.

• A clear scope, such as one order, one time window, or one event type.

• Authorization for sensitive replay actions.

• Audit records showing who replayed what and why.

• Protection against replaying obsolete events into current state without validation.

Replay should be a designed recovery tool, not an emergency script that only one person understands.

Common Mistakes

• Retrying non-idempotent actions without duplicate protection. This can create duplicate warehouse tasks, labels, refunds, or supplier orders.

• Treating manual review as outside the architecture. Human decisions need queues, context, permissions, and audit trails.

• Losing workflow context when one integration fails. Technical errors are not enough; operators need the business state.

• Designing compensation after production failure. Compensation paths should be discussed before the first incident.

• Letting dead-letter queues accumulate without ownership, severity, or recovery rules.

If Someone Asks You

How do you design reliable back-office workflows?

I design the failure path as deliberately as the happy path. Each workflow needs durable checkpoints, idempotent commands, retry and timeout policies, compensation rules, exception queues, human review paths, and audit evidence. The system should always be able to answer what happened, what is safe to do next, who owns the issue, and how customer impact will be handled.

What You Should Remember

Design the failure path before operations discovers it for you.

Questions to Check Your Understanding

1. What makes a workflow step safe to retry?

2. When should compensation be used?

3. Why are exception queues part of the user experience indirectly?