Scalability and Performance Patterns

Goal

Show how load, throughput, latency, and bottleneck concerns lead to different pattern choices.

Reliability asked how a system behaves when something fails. Scalability and performance ask what happens when the system is healthy but the shape or amount of work changes. The durable idea is:

Performance is about the path of one unit of work; scalability is about how that path behaves as demand changes.

The Short Version

Do not answer overload with "add more compute" until you know what is saturated. A slow request can be waiting on a database, a remote dependency, a lock, a queue, storage, network transfer, or a constrained worker pool. Scaling a layer that is not the bottleneck mainly scales the bill.

Use this review path:

Demand shape
  -> critical journey and latency budget
  -> measured bottleneck
  -> absorb, reduce, partition, parallelize, or prioritize work
  -> verify performance and cost under realistic load

Separate the Concerns

The words are related, but they are not interchangeable.

Concern	Practical question	Useful evidence
Latency	How long does one request or task take, including waiting?	Percentile response time, dependency time, queue wait, and end-to-end trace
Throughput	How much useful work completes per unit of time?	Completed operations, messages, rows, bytes, or jobs per interval
Concurrency	How many requests or tasks are active at once?	Active connections, workers, threads, sessions, and in-flight operations
Scalability	How does the system behave as load grows or shrinks?	Load versus latency, throughput, error rate, saturation, and cost
Efficiency	How much resource and money produce a useful outcome?	Utilization, unit cost, cache effectiveness, and wasted work

Average latency can hide the experience of the slowest important requests. Total throughput can look healthy while one tenant or workflow starves. Review the critical journey, its percentiles, and its constrained resources together.

Concern-to-Pattern Map

Load concern	Pattern direction	Best fit	Trade-off to own
Sudden burst	Queue-based load leveling	Work can finish later than the request that created it.	Backlog age, delayed outcomes, and recovery capacity
Too many callers	Throttling, rate limiting, quotas, and admission control	The system must preserve critical capacity.	Some work is delayed or rejected and clients need clear behavior
Repeated expensive reads	Cache-aside, precomputed view, or read replica	Slightly stale data is acceptable or invalidation is manageable.	Staleness, eviction, warm-up, and cache failure
One data or work partition is hot	Partitioning, sharding, or cell-based placement	A stable key can distribute work.	Rebalancing, cross-partition operations, and skew
Independent work can run concurrently	Competing consumers or parallel processing	Ordering and shared-state constraints are limited.	Coordination, duplicate handling, and downstream pressure
Long work blocks the request path	Asynchronous handoff or workflow	The caller can receive an accepted or pending state.	More state, correlation, monitoring, and user-status design
Different work has different business value	Priority queues, reserved capacity, or workload isolation	Critical work must survive pressure from optional work.	Fairness and possible starvation
Payload or query does too much work	Pagination, batching, projection, and result shaping	Work can be reduced before infrastructure is expanded.	More client coordination and contract design

These patterns solve different problems. A cache reduces repeated read work; it does not increase write consistency. A queue absorbs arrival bursts; it does not make a slow consumer faster. Partitioning spreads work; it does not fix a single hot key. Throttling protects capacity; it does not create capacity.

Find the Control Point

flowchart LR
    U["Demand"] --> E["Edge<br/><small>admit and prioritize</small>"]
    E --> C["Cache<br/><small>avoid repeated work</small>"]
    C --> A["Application<br/><small>scale and isolate</small>"]
    A --> Q["Queue<br/><small>absorb and smooth</small>"]
    Q --> W["Workers<br/><small>parallelize safely</small>"]
    W --> D["Data or dependency<br/><small>partition and protect</small>"]
    D --> O["Outcome telemetry<br/><small>latency, throughput, errors, cost</small>"]

The diagram is a review aid, not a required architecture. Locate the pressure first, then choose the nearest effective control point.

Absorb work when arrival and processing rates differ

A queue is useful when work arrives in bursts but can complete over a longer window. Capacity planning must consider both the arrival rate and the rate at which consumers can drain the backlog. If demand remains above processing capacity, a queue only postpones failure.

Avoid work when repetition is the bottleneck

Caching can shorten a read path and protect a data store. The architecture decision includes freshness, invalidation, access isolation, cache warm-up, and what happens during a cache miss or cache outage. Do not cache a sensitive or user-specific result under a key that ignores its authorization context.

Split work when one resource cannot grow safely

Partitioning distributes data or processing by a key such as tenant, region, account, or entity range. A useful key spreads load and keeps common operations local. A poor key creates hot partitions or forces expensive cross-partition coordination.

Reject or defer work before saturation becomes collapse

Rate limits, quotas, bounded queues, and admission control make overload behavior explicit. They preserve capacity for critical work and give callers a signal to slow down or try later. This is often safer than allowing every request to enter the system and time out deep inside it.

Example: Catalog and Checkout Need Different Patterns

A commerce platform receives a surge after a product launch.

The catalog path is read-heavy. Many customers request the same product details, and slightly delayed price-display updates are acceptable if checkout revalidates authoritative price and availability. Useful pattern choices include caching popular product views, distributing read load, shaping large responses, and scaling stateless request handlers.

Checkout has a different concern. It changes business state and depends on payment, order, and inventory responsibilities. Caching does not solve order acceptance. A better design may reserve capacity for checkout, keep its synchronous path short, use idempotency for repeated submissions, and hand non-critical follow-up work to durable processing.

During the surge, the team might deliberately reduce recommendation refreshes, background indexing, or analytics exports so checkout keeps capacity. That is a prioritization decision, not merely a scaling configuration.

Verify the Result, Not the Mechanism

After selecting a pattern, test the claim it is meant to support.

For a cache, measure hit rate, miss latency, origin load, staleness, and failure behavior.
For a queue, measure arrival rate, completion rate, oldest-item age, and time to drain after a burst.
For partitioning, measure load distribution, hot keys, cross-partition work, and rebalancing behavior.
For autoscaling, measure scale-up delay, saturation before scale-out, scale-down safety, and unit cost.
For throttling, measure rejected work, retry behavior, fairness, and preserved success for critical journeys.

Run realistic load shapes: gradual growth, sharp bursts, hot tenants, slow dependencies, and partial failures. A pattern that works in a smooth benchmark may behave badly in the messier traffic shape the business actually produces.

Common Mistakes

Scaling the application tier when the data store or downstream dependency is saturated.
Using averages that hide slow percentiles and hot partitions.
Adding a queue without enough capacity to drain it.
Treating cache invalidation, authorization, and stale data as implementation details.
Allowing every workload to compete equally even when their business criticality differs.
Optimizing one component while making the end-to-end journey slower or more expensive.

If Someone Asks You

Which pattern would you use when the system is overloaded?

A good answer is:

First identify the bottleneck and the shape of demand. Use caching to avoid repeated reads, queues to absorb work that may complete later, partitioning to distribute independent work, throttling or admission control to protect capacity, and horizontal scaling where the constrained component can actually scale out. Then verify end-to-end latency, throughput, saturation, error rate, backlog, and cost under realistic load.

What You Should Remember

Performance is about the shape of work, not only the size of machines.

Questions to Check Your Understanding

Why does a queue not solve a permanent shortage of processing capacity?
What makes a useful partition key?
Which measurements would show whether a cache improved the customer journey?

Further Study

Cache-Aside pattern - Review cache loading, expiration, staleness, and consistency considerations.
Rate Limiting pattern - Explore how controlled consumption can reduce overload and throttling pressure.