Horizontal Scaling Patterns
LESSON
Horizontal Scaling Patterns
By the end of this lesson, you will be able to...
Explain why replicas add useful capacity only when they can share work correctly.
Trace how a bottleneck moves after an application tier is widened.
Choose between replication, caching, queued work, and partitioning from the constrained resource.
Idea in one sentence: Horizontal scaling widens one part of a work path; it succeeds only when the next shared constraint is visible and can handle the new pressure.
Core Insight
Suppose a learning platform launches a popular certification. Traffic rises from 200 to 1,000 requests per second. The team adds four API instances behind the load balancer. API CPU falls, and page latency improves for a few minutes. Then database connections reach their limit, cache misses create more database reads, and receipt jobs begin aging in the worker queue.
Adding API replicas worked. It just did not solve the whole request path.
Horizontal scaling means adding peers that can perform the same role. It is most useful when a layer has local, parallelizable work and its instances are interchangeable. It is much less useful when every new peer increases pressure on one shared database, lock, hot key, external provider, or worker queue.
The trade-off is that replicas improve availability and local serving capacity, but they add coordination, shared-state pressure, and a new need to observe the complete path. Lessons 009 and 010 established how replicas receive traffic and leave rotation. This lesson asks whether the replicas create throughput at the system boundary. The next lesson uses a deployment as a pressure test for that answer.
Interchangeable Does Not Mean “No State”
Plain meaning:
Any healthy API instance should be able to handle the next request without needing one particular machine to remember something essential.
In this scenario:
A learner may arrive at api-1 for one request and api-4 for the next. Both instances can read the session, course progress, and rate-limit state from shared services.
Technical name:
This is stateless request handling. The system still has state. The important distinction is that critical request state is not trapped in a single serving process.
interchangeable API tier:
client -> balancer -> any ready API -> shared session/data/object store
instance-bound design:
client -> balancer -> one required API -> local session/workflow state
Local memory is not forbidden. An instance may hold a cache, a connection pool, or temporary computation. The request becomes a scaling problem only when correctness requires that exact instance to survive or receive the next request. Then routing needs stickiness, replacement becomes risky, and a larger fleet behaves like several smaller constrained pools.
Externalizing important state has a cost: the shared store adds network latency, coordination, and its own failure mode. It buys the ability to replace, add, and balance serving instances. That is a design trade-off, not a magic property of containers.
Follow the Wider Path
Do not ask “Can we add instances?” Ask “What happens to every stage when we do?”
request -> load balancer -> API -> cache -> database -> receipt job -> worker -> email provider
Before the launch, two API instances are the visible narrow point. At 200 requests per second, each performs 100 requests per second and the database handles 80 reads per second because most course pages hit cache.
After adding four instances, the API tier can accept 1,000 requests per second. The new flow looks like this:
| Stage | Before expansion | After API expansion | What changed |
|---|---|---|---|
| API CPU | 85% | 35% | The original bottleneck is wider. |
| Cache hit rate | 95% | 82% | Cold local caches and more key churn produce extra misses. |
| Database reads | 80/s | 260/s | Misses and new traffic reach the shared database. |
| DB connection wait | low | 400 ms p95 | The database becomes the constrained stage. |
| Receipt queue age | 10 s | 7 min | More successful checkouts create more follow-up work. |
The path is the worked evidence:
input: four new API replicas accept the launch traffic
-> transition: requests are distributed across more ready instances
-> intermediate state: API work drops, but cache misses and DB connection waits rise
-> output: pages slow again and receipt jobs age
-> naive failure: declaring success from API CPU alone hides the moved bottleneck
This is bottleneck migration. It is not proof that scaling failed. It tells the team where the next constraint lives. The correct next change depends on why database load rose and why the queue aged; “add even more APIs” may make both worse.
Check: API CPU falls after a scale-out, while database connection wait and p99 page latency rise. Which layer should the team scale first?
Think first, then reveal.
Answer: Investigate the database path first. The evidence says the application tier is no longer the immediate limit. More APIs would add more competing database requests; it would not create database capacity.
Choose the Pattern From the Constraint
Horizontal scaling is a family of patterns, not a button.
| Constrained resource | First useful pattern | What it changes | What it does not solve |
|---|---|---|---|
| API CPU or connection handling | Replicate stateless API instances | More parallel serving capacity | A shared database, hot key, or slow dependency |
| Repeated read work | Cache or add read capacity | Fewer reads reach the authoritative store | Freshness and write coordination |
| Long request work | Accept work, then use a bounded worker queue | Removes slow work from request latency | Worker limits, queue age, or downstream capacity |
| One hot tenant/key/data owner | Partition work or data | Splits ownership across independent paths | Cross-partition coordination and skew inside a partition |
| External dependency saturation | Admission control, pacing, breaker, or fallback | Limits pressure on the dependency | The dependency's underlying outage |
Replication is appropriate when the work itself is parallel. Caching helps when it is safe to reuse a result. Queues help when a user need not wait synchronously. Partitioning helps when one shared owner is the limit. Each changes a different physical or logical constraint.
This also explains why a cache can make scale-out worse during a cold start. More application instances may miss their local caches together and stampede the same origin. A shared cache, request coalescing, warmup, or bounded rollout may be needed. The next lesson will show why deployments are a common moment for this pressure.
Scaling Signals Are a Comparison, Not a CPU Threshold
“Scale when CPU exceeds 70%” can be a useful local trigger. It is not a complete capacity model. CPU may be low while requests wait on database connections, locks, disk I/O, or a slow provider. CPU may be high during harmless warmup while latency remains inside its budget.
For this path, compare:
- request rate, p50 and p99 latency, and error rate at the API boundary;
- ready instance count, active requests, and per-instance skew;
- cache hit rate, miss rate, and origin/database reads;
- database connection wait, query latency, and saturation evidence;
- queue depth, oldest age, completion rate, and worker utilization;
- dependency timeouts and breaker state.
The word compare matters. A signal earns an explanation only when it changes alongside another stage. If p99 rises while API CPU falls and database wait rises, the causal hypothesis is much stronger than any single red graph. Validate it with traces or profiles before changing architecture.
Check: A service has six API replicas, but every request updates the same account balance row under one lock. What limit remains after adding a seventh replica?
Think first, then reveal.
Answer: The serialized balance update remains the limit. More replicas may create more contenders for the same lock. The team needs to examine the ownership and write design—perhaps batching, partitioning by account, or an asynchronous workflow—not merely add serving processes.
Autoscaling Is a Control Loop, Not an Architecture
Autoscaling can add or remove replicas from an already-interchangeable layer. It needs a target signal, warmup time, minimum and maximum limits, and a way to avoid reacting to a brief burst after the fact.
For example, scaling APIs from queue depth alone may be wrong if the queue grows because a downstream email provider is throttling. Scaling on API CPU alone may be wrong if the actual wait is in the database. A good trigger is tied to the layer it changes and is checked against the downstream evidence it might amplify.
Autoscaling also cannot decide where critical state belongs, make a primary database horizontally writable, or split a hot tenant. It automates a bounded replica decision. That is valuable, but it is not a substitute for the architecture that makes replicas useful.
Trade-offs and Limits
More replicas improve fault tolerance and local parallelism, but cost money, warmup time, network connections, cache memory, and operational complexity. They can make a shared dependency fail sooner by increasing its concurrency. Scaling the request tier can also increase asynchronous work and move the user-visible delay into a queue.
Caching reduces read pressure but adds freshness and invalidation policy. Queues smooth burst pressure but add waiting, retries, and a delivery budget. Partitioning reduces one shared limit but introduces ownership decisions and cross-partition cases. There is no free “scale out” control; each option replaces one bottleneck with a new policy boundary.
The boundary is visible when an added replica no longer improves the target outcome: p99, useful completion, queue age, or error rate. That is the moment to stop adding replicas and trace the next narrow stage.
Common Confusions
Confusion: Stateless means the system has no state
Why it is tempting:
The word sounds absolute.
Better model:
Critical state still exists. It is reachable by multiple serving instances so the request is not bound to one process.
Confusion: A larger API fleet means a scalable system
Why it is tempting:
The application tier is usually the easiest component to duplicate.
Better model:
The system scales only if the whole work path has capacity. A wider API tier often exposes the next shared limit.
Confusion: Autoscaling repairs any overload
Why it is tempting:
It reacts automatically, which feels comprehensive.
Better model:
It changes one replica count. It cannot repair a hot partition, an unsafe state boundary, or a saturated dependency and may amplify them.
Practice: Defend the Next Capacity Change
After adding API replicas, a service sees lower API CPU, unchanged cache hit rate, database connection wait at 600 ms p95, and a growing queue of report-generation jobs. The database handles both page reads and report queries. Propose one immediate intervention and one longer-term design question. Name the evidence that would tell you whether the immediate intervention helped.
Model answer: Immediately protect the database by moving or limiting report queries—perhaps queue them with a bounded worker concurrency or shed nonessential report work—rather than adding APIs. The longer-term question is whether report reads need an independent read model, cache, or partition. Watch database connection wait, page p99 latency, report queue age, and successful report completion; a lower API CPU would not prove success.
Resources
- [DOC] Kubernetes Horizontal Pod Autoscaling — Focus: See how a replica control loop uses observed metrics and where its scope ends.
- [BOOK] Designing Data-Intensive Applications — Focus: Connect replication, partitioning, and shared-state limits to real system design.
- [BOOK] The Site Reliability Engineering Workbook — Focus: Relate saturation and capacity evidence to operational decisions.
- [ARTICLE] The Tail at Scale — Focus: See why one slow component can dominate user-visible latency across a distributed path.
Key Takeaways
- Replicas add useful capacity only when serving instances are interchangeable and the next shared constraint can absorb the added pressure.
- Scale-out commonly moves the bottleneck from API CPU to cache, database, worker, or dependency paths; that movement is useful evidence.
- Replication, caching, queued work, partitioning, and pacing solve different constraints and create different policy costs.
- Judge a capacity change by its target outcome and the complete work path, not by one local CPU graph.
← Back to Caching, Workers, and Performance