CH19-04: Idempotency, Consistency, Compensation, and Reconciliation

Goal

Explain the reliability patterns that protect financial workflows under retries, duplicates, partial failure, provider delays, chargebacks, and reconciliation differences.

The Short Version

In distributed finance workflows, the hardest question is often not "Did the code run?" It is "What happened, and is it safe to try again?"

Payment providers can time out. Queues can deliver messages more than once. A workflow can complete one step and fail before the next. A support agent can retry. A provider can later report a chargeback. A settlement file can disagree with the internal ledger. A strong design expects these realities.

The durable idea is:

Financial workflows should be designed so retrying is safe and differences can be found.

This does not mean every finance process becomes one giant database transaction. Across providers, queues, services, and finance systems, that is usually unrealistic. Instead, the architecture uses idempotency, correlation, state machines, compensation, and reconciliation.

Safe Retry Starts With Idempotency

Idempotency means a caller can repeat the same logical request and get the same logical outcome without creating duplicate side effects.

For a refund, the idempotency key might represent "refund order line X for return request Y." If the first attempt times out and the support agent tries again, the platform checks the key. If the refund already succeeded, it returns the known result. If it is pending, it returns or resumes the pending operation. If it failed in a known way, it shows the failure and next action. It should not create a second independent refund.

flowchart LR
	A["Financial command"] --> B{"Idempotency key exists?"}
	B -->|No| C["Create command record"]
	B -->|Yes| D["Return or resume known operation"]
	C --> E["Call provider or workflow step"]
	E --> F["Record provider correlation and internal evidence"]
	F --> G["Publish status event"]
	D --> G

Azure Service Bus duplicate detection can help discard duplicate messages in some messaging scenarios, but do not treat broker duplicate detection as the whole finance design. The business operation still needs an idempotency model, command status, provider correlation, and records that support investigation.

Consistency Is A Design Choice

A finance engine usually has several consistency zones:

The design should be honest about these boundaries. If a provider call succeeds but the internal ledger write fails, you need a recovery path. If the ledger write succeeds but event publication fails, you need a way to publish later or repair. This is where outbox-style thinking helps: record the state change and the message to be published as part of controlled work, then deliver the message reliably.

Compensation Instead Of Pretend Rollback

In a single database transaction, rollback can undo changes before commit. In a distributed finance workflow, many actions cannot simply be rolled back. If a payout was sent to a provider, the system might need an adjustment, reversal, hold, or compensating entry.

Compensation means doing a new controlled action that corrects or offsets a previous action. For example:

• If capture succeeds but invoice generation fails, continue invoice recovery rather than undoing payment immediately.

• If a seller payout is too high because a return arrived late, create an adjustment in the next payout cycle.

• If a refund succeeds at the provider but the internal workflow fails afterward, repair the internal records and reconcile instead of issuing another refund.

• If a chargeback arrives, create a dispute or adjustment workflow that updates customer, seller, ledger, and reporting views.

Compensation must be designed. It cannot be improvised only during incidents.

Reconciliation Is A Control Loop

Reconciliation compares records from different systems to find differences. In commerce finance, the comparison might include:

• Provider authorizations, captures, refunds, fees, disputes, and settlements.

• Internal payment intent, capture, refund, invoice, ledger, and payout records.

• Orders, returns, cancellations, and fulfillment status.

• Seller balances, commissions, taxes, adjustments, and payout batches.

• Finance reports and operational dashboards.

Provider settlement
	+ Internal ledger
	+ Invoices and refunds
	+ Seller payout records
	+ Orders and returns
	-> Matched, pending, exception, or adjustment required

Reconciliation should produce operational states, not just spreadsheets. A difference should have an owner, severity, investigation path, and resolution method.

Example: Retrying A Refund After Timeout

A support agent retries a refund after a timeout. The system should return the known refund status or continue the same operation, not create a duplicate refund.

The safe flow looks like this:

1. The refund request receives a stable idempotency key.

2. The refund workflow records the request and current status.

3. The provider call includes a provider correlation value when supported.

4. If the provider response times out, the internal status becomes pending or unknown, not failed by assumption.

5. A retry checks the idempotency key and resumes or returns the known operation.

6. Reconciliation later compares provider refund records with internal refund and ledger records.

This is how the design avoids both duplicate refunds and lost refunds.

Common Mistakes

• Assuming provider timeouts mean failure.

• Retrying without idempotency or correlation.

• Treating reconciliation as a finance report rather than a control loop.

• Letting dead-letter queues become storage for problems nobody owns.

• Designing compensation only after the first incident.

• Reporting success before the system has recorded enough evidence to investigate later.

If Someone Asks You

How do you protect payments from duplicates and partial failures?

Use idempotency keys for financial commands, record command status and provider correlation, make workflow steps durable, publish messages safely, design compensation for actions that cannot be rolled back, and reconcile provider records against internal ledger, invoice, order, refund, and payout records. The goal is safe retry and visible differences.

What You Should Remember

Safe retry and reconciliation are core finance architecture patterns.

Questions to Check Understanding

1. Why is provider correlation important?

2. When is compensation needed?

3. Why is reconciliation more than a finance report?