Durable Workflow and Compensation

Goal

The previous section separated communication meanings. Workflow deserves special attention because it is easy to hide a business process across API calls, messages, event handlers, and scheduled jobs until nobody can see where the process is stuck.

This section explains long-running business processes, visible state, retries, compensation, and human review.

The durable idea is:

Durable workflow makes long-running business state visible and recoverable.

The Short Version

A workflow coordinates work over time. It may call services, send commands, wait for events, pause for manual review, retry temporary failures, compensate completed work, and record progress.

Use workflow thinking when the architecture must answer:

Where is this process now?
What already happened?
What is the next safe action?
Who owns the exception?

Messages can move work. Events can announce facts. A workflow explains the process state across many steps.

Workflow State Model

A useful workflow model names the happy path and the failure path.

Start process
  -> Step succeeds
      -> record checkpoint
      -> continue
  -> Step times out
      -> retry if safe
      -> pause if outcome is uncertain
  -> Step fails permanently
      -> compensate where possible
      -> open exception
  -> Sensitive or ambiguous step
      -> human review
      -> resume, compensate, or reconcile

The checkpoint matters. Logs say what code did. Workflow state says what the business process believes has happened.

Compensation

Compensation is the action that repairs, balances, or reduces harm after an earlier step has completed and a later step fails.

It is not always a perfect rollback. In distributed systems, completed work may already be visible to another service, partner, customer, warehouse, finance process, or legal record. Compensation often applies business-specific rules.

Completed Step	Later Failure	Possible Compensation
Inventory reserved	Warehouse cannot fulfill.	Release reservation, reallocate, notify support, or ask customer to choose another option.
Payment authorized	Fraud review rejects order.	Void authorization or route to finance review.
Carrier label created	Warehouse task cancelled.	Cancel label if allowed, mark label unused, or reconcile with carrier billing.
Refund issued	Invoice already exported.	Create correcting finance entry or manual review task.

The important design question is not only how to undo work. It is how the system reaches a valid business state when a simple transaction is not available.

Retries Are Not Enough

Retries help with transient failures. They do not solve every workflow problem.

Retries are risky when:

The operation is not idempotent.
The previous attempt may have succeeded but the caller did not receive the response.
The downstream system is throttling or overloaded.
The failure is caused by invalid data.
A human decision is required.
Repeating the action could duplicate shipments, payments, refunds, notifications, or reservations.

A reliable workflow uses stable identifiers, idempotent commands, timeout rules, retry limits, dead-letter or exception paths, and correlation IDs. It also records enough state to resume or compensate safely.

Human Review Is Architecture

Human review is not an admission that automation failed. It is often the correct architecture for ambiguous, high-impact, regulated, or irreversible decisions.

Good review paths include:

Review Need	Design Support
Business context	Show order, customer, partner, amount, current state, and risk.
Evidence	Show inputs, outputs, timestamps, contract version, and correlation IDs.
Decision choices	Provide safe actions such as approve, reject, resume, compensate, or escalate.
Permissions	Limit who can take sensitive actions.
Audit history	Record who acted, when, why, and what changed.
Resume controls	Continue workflow from a known checkpoint instead of editing data manually.

If manual review happens in production but is not represented in the design, the real architecture is hiding in tickets, chat threads, and emergency scripts.

Example: Fulfillment Workflow

Consider an order fulfillment process:

Order accepted
  -> reserve inventory
  -> authorize payment
  -> allocate fulfillment node
  -> create warehouse task
  -> request carrier label
  -> notify customer

If carrier label creation times out, the workflow should not blindly call the carrier again forever. It should know whether the label request had a stable identifier, whether the carrier supports lookup by that identifier, whether duplicate labels are possible, and what support should see if the outcome remains uncertain.

A durable workflow records the checkpoint before the carrier call, retries safely where possible, pauses when the external state is ambiguous, opens an exception with evidence, and resumes or compensates after review.

Common Mistakes

Hiding long-running workflows across many services. The process exists, but nobody can see it.
Treating compensation as an afterthought. Recovery rules should be designed before production failure.
Retrying forever instead of surfacing exceptions. Endless retry can deepen incidents and hide customer impact.
Ignoring support dashboards and manual recovery. Operators need a safe work surface, not database access.
Assuming compensation returns the world to its original state. Often it only moves the business into a valid next state.

If Someone Asks You

When does a process need durable workflow?

A good answer is:

A process needs durable workflow when it spans multiple steps, systems, waits, decisions, or recovery paths and the business must know what happened, what is safe to retry, what must be compensated, and who owns exceptions. Workflow state is part of reliability because it makes progress, failure, and recovery visible.

What You Should Remember

Workflow state is part of reliability.

Questions to Check Understanding

What does compensation solve?
Why can invisible workflow state be dangerous?
When should human review be designed explicitly?