Reliability and Resilience Patterns

Goal

Show how to move from a failure concern to a pattern family without treating reliability as a single switch marked "high availability."

Earlier chapters introduced request paths, asynchronous flows, ownership boundaries, and failure handling. This section turns those ideas into a review lens. The durable idea is:

Reliability keeps the system useful; resilience limits damage and makes recovery possible when reliability controls are not enough.

The Short Version

Start with the failure, not the favorite pattern. Ask what can fail, how the failure spreads, what the user must still be able to do, and how the system returns to a trustworthy state.

Failure concern
  -> detect and bound the failure
  -> protect critical work
  -> continue, degrade, or stop safely
  -> recover and reconcile
  -> learn from operating evidence

Redundancy can provide another instance or location, but resilience is broader. A resilient design also knows when to retry, when to stop calling a broken dependency, how to isolate a blast radius, what to do with unfinished work, and how to verify recovery.

Map the Failure Before Choosing the Pattern

A useful review begins by classifying the failure shape.

The same pattern can help one failure while worsening another. Retries may hide a short fault, but uncontrolled retries can amplify an overloaded dependency. A queue can preserve work, but it does not guarantee that consumers process the work correctly. Failover can restore service, but it cannot repair corrupted state copied to every replica.

A Small Reliability Pattern Stack

Reliability controls work best as a sequence rather than as isolated decorations:

flowchart LR
    A["Request or work item"] --> B["Timeout<br/><small>Bound waiting</small>"]
    B --> C["Retry<br/><small>Only transient and safe work</small>"]
    C --> D["Circuit breaker<br/><small>Stop repeated harmful calls</small>"]
    D --> E["Bulkhead or queue<br/><small>Contain pressure and preserve work</small>"]
    E --> F["Fallback or recovery<br/><small>Degrade, replay, restore, reconcile</small>"]
    F --> G["Operating evidence<br/><small>Prove the journey recovered</small>"]

This is a reasoning sequence, not a rule that every request must pass through every control.

Retry only when the failure is temporary and the action is safe

A retry fits a transient failure: a short connection interruption, a temporary throttle, or a dependency that is briefly unavailable. The retry should be limited, delayed, and aware of the caller's time budget. Add randomness when many clients might retry together.

The operation also needs duplicate safety. An idempotent operation produces the intended result even if the same request is processed more than once. If the action cannot be made idempotent, use a request identifier, deduplication record, or explicit state check before repeating it.

Stop when persistence becomes pressure

A circuit breaker prevents a caller from repeatedly spending time and resources on a dependency that is unlikely to succeed. It gives the dependency room to recover and gives the caller a deliberate alternative: reject quickly, use cached data, accept work for later, or offer a smaller experience.

Isolate before the failure becomes everyone else's failure

Bulkheads, cells, separate worker pools, and bounded queues reduce the blast radius. Isolation can be organized by dependency, workload priority, tenant group, or failure domain. The cost is lower resource sharing and more partitions to observe. Use it where one noisy or broken path could otherwise consume resources needed by critical journeys.

Recover business state, not only infrastructure

Restarting a component is not the same as recovering the business. Durable work may need replay. A partially completed workflow may need compensation. Orders, inventory, or payments may need reconciliation against a trustworthy record. Recovery is complete only when the system's state and the user's outcome agree again.

Example: Checkout Under Dependency Failure

Imagine a checkout flow that performs four jobs:

1. validate the request and its idempotency key;

2. authorize payment and create a durable order outcome;

3. hand fulfillment and notification work to durable processing;

4. show the customer an answer that support can later explain.

Suppose the notification provider becomes unavailable during a traffic spike. A weak design waits for notification inside checkout and retries without limit. Customer requests become slower, connection pools fill, and an unimportant dependency threatens order acceptance.

A stronger design uses several patterns together:

• checkout places a strict timeout around the notification dependency;

• notification work is handed to a durable queue after the order outcome is recorded;

• retries are bounded and delayed;

• failed items move to visible exception handling instead of disappearing;

• checkout can confirm the order even while notification is degraded;

• operators monitor backlog age and replay work after recovery.

Now change the failure: payment authorization is unavailable. The system should not pretend that a payment succeeded. It may fail safely, offer another payment path, or accept a clearly pending order if the business supports that state. The pattern choice follows the business promise, not a generic desire to "stay up."

Review Questions That Reveal the Real Design

• Which customer or business journey is critical?

• What are its dependencies, deadlines, and durable state changes?

• Which failures are transient, persistent, overload-related, or destructive?

• Where can work be delayed, and where must the caller receive certainty now?

• What prevents one failure from consuming every shared resource?

• What state must be replayed, restored, compensated, or reconciled?

• Who declares recovery complete, and what evidence supports that decision?

These questions connect directly to the ownership and failure model and the durable work patterns introduced earlier in the guide.

Common Mistakes

• Calling a design resilient because it has duplicate instances, without testing failover or state recovery.

• Retrying every error, including validation failures and persistent dependency failures.

• Building a queue without an owner for backlog, poison messages, replay, and capacity.

• Measuring component uptime while ignoring whether the customer journey completed.

• Providing a fallback that returns stale or misleading business information.

• Writing a recovery plan that restores infrastructure but does not reconcile business state.

If Someone Asks You

Which patterns would you reach for when a system must survive failure?

A good answer is:

First identify the failure and the business outcome that must be protected. Use timeouts and bounded retries for safe transient faults, circuit breakers for persistent dependency failure, bulkheads or queues to contain pressure, graceful degradation for non-critical capabilities, and failover or restore patterns for larger outages. Then define how durable work and business state will be replayed or reconciled. The right combination limits the blast radius and has a tested recovery path.

What You Should Remember

Reliability keeps the customer moving; resilience keeps the system recoverable.

Questions to Check Your Understanding

1. Why can an uncontrolled retry reduce reliability rather than improve it?

2. What does a bulkhead protect that redundancy alone may not protect?

3. When is a system recovered from a business perspective?

Further Study

• Cloud Design Patterns - Review a catalog of failure, messaging, data, and workload-management patterns and their trade-offs.

• Bulkhead pattern - Explore failure isolation through pools, partitions, and cells.