Load Balancing Fundamentals

By the end of this lesson, you will be able to...

• Explain how a load balancer turns several instances into one service boundary.

• Choose a simple routing policy from request cost, active work, and instance capacity.

• Identify when local state, stale health evidence, or uneven requests defeats an apparently even distribution.

Idea in one sentence: A load balancer makes a fleet useful by sending each new request to capacity that can serve it safely, not by making every instance receive the same number of requests.

Core Insight

Suppose api.learn.example has three application instances behind one public address. At 09:00, most requests are quick course-page reads. At 09:05, a popular instructor opens a live dashboard that starts several long personalized report requests. One instance accepts two of those long requests. The other two instances remain mostly free.

If the balancer keeps using strict round robin, its next request may still go to the busy instance just because that instance is next in the rotation. From a request-count view, this is fair. From the learner's view, it creates an avoidable slow page.

The design promise of a load balancer is simple: clients see one stable service while the system chooses a suitable backend from a changing fleet. This separates clients from individual machines and gives the service a place to contain failures, drain instances, and use added capacity.

The trade-off is that the balancer must make decisions from incomplete and changing evidence. A simple policy is predictable but can ignore expensive requests. A more responsive policy can reduce hot spots, but it depends on accurate load and health signals. This lesson chooses that policy boundary; the next lesson examines what “healthy enough to receive traffic” should mean.

The Naive Model: Split Requests Evenly

The first model is:

one public address -> round robin -> equal request counts per instance

Round robin is a useful baseline. When instances have similar capacity and requests finish in similar time, it spreads arrival work cheaply and predictably.

But requests are not always equal. A cache hit may finish in 10 ms; a cache miss that performs an expensive database query may take 800 ms. A streaming request may hold a connection for minutes. One instance may be warming after deployment, or have fewer CPU resources than its peers.

Equal counts do not imply equal work. A balancer needs a model of what it is trying to protect: tail latency, capacity use, failure isolation, locality, or a combination of these.

Plain meaning:

The balancer is a traffic director. For every new request, it chooses a backend that should be able to handle it.

In this scenario:

The director should avoid giving a new course-page request to an instance already occupied by two long reports when another healthy instance is free.

Technical name:

This is load balancing: routing work across a backend set according to a policy and an eligibility view. The policy chooses among eligible backends; eligibility says which backends may receive new traffic at all.

Three Questions Before Choosing a Policy

Do not start with an algorithm name. Start with three questions.

1. Are requests similar in cost and duration?

If yes, round robin may be enough. If duration varies widely, a count-based rotation can create a hot instance even while every instance has received the same number of requests.

2. Are instances interchangeable?

If one instance has half the CPU of the others, or one is deliberately warming, a uniform share wastes information. A weighted policy can give a larger instance more traffic, while a drain state can give a warming instance none.

3. Can any healthy instance serve the request correctly?

This is the state question. If sessions, uploads, or workflow state live only in one process, routing becomes constrained. The balancer may need stickiness, which preserves an affinity but reduces its freedom to avoid a bad or overloaded instance.

These questions turn “load balancing” from a generic scalability noun into a design decision with explicit assumptions.

Policies Are Scheduling Rules

Each policy uses a different proxy for “best next backend.” None can see the future exactly.

Policies are usually applied only to new requests. A balancer cannot move an already-running database query from one application instance to another. That is why routing cannot undo a slow dependency or an overloaded handler. It can only avoid adding more pressure to the same place.

Check: Four identical API instances receive requests that usually complete in 20 ms. Is round robin automatically a poor choice?

Think first, then reveal.

Answer: No. With genuinely similar request cost and capacity, round robin is a good simple default. Replace it only when evidence shows that its assumptions no longer hold. A more elaborate policy is not automatically more correct.

A Worked Routing Trace

Return to the live dashboard. At one moment, the balancer sees three eligible instances:

The next three requests are: a course page, another report, and a profile update.

Trace A: strict round robin

Suppose api-1 is next in rotation.

input: course page
  -> transition: round robin selects api-1
  -> intermediate state: api-1 now runs 3 requests, including 2 long reports
  -> output: the cheap course page waits behind contention on the busy instance
  -> naive failure: request counts stay almost equal while latency becomes uneven

Round robin may next choose api-2, then api-3. The counts look balanced: two, one, and one new requests over a short interval. The work is not balanced because the initial work was uneven.

Trace B: least outstanding requests

input: course page
  -> transition: policy compares active work and selects api-2
  -> intermediate state: api-2 runs 1 short request; api-1 remains at 2 long requests
  -> output: the course page avoids the known busy instance
  -> decision: subsequent requests use the current counts again, not a fixed rotation

For the report request, the balancer may then choose api-3, and for the profile update it may choose whichever instance has completed work first. The policy does not promise equal latency. It uses the evidence available to avoid a visible hot spot.

So far: round robin spends no effort on load evidence. Least-outstanding-work spends more effort to react to uneven active work. The right choice depends on whether that extra signal is credible and worth operating.

State Changes the Routing Freedom

The easiest fleet to balance is one where any healthy instance can serve any request:

request -> load balancer -> any eligible app instance -> shared data/session store

When a browser session exists only in api-2 memory, the request path changes:

request with session S -> load balancer -> must return to api-2

This is sticky routing or session affinity. It can be reasonable for a legacy session design, a long-lived connection, or local cache locality. But it turns an unconstrained pool into smaller per-instance pools. If api-2 is slow, all users attached to it suffer even if api-1 is idle.

Externalizing session state does not make every system stateless in a magical sense. It makes the state reachable from more than one eligible instance, which gives the balancer more choices. The cost is a shared-state dependency, its latency, and its own failure boundary.

Check: A service uses a hash of customer_id so requests from the same customer reach the same instance. One large customer creates 40% of all traffic. Why might an otherwise healthy fleet still have high tail latency?

Think first, then reveal.

Answer: The hash preserves affinity, but it cannot split one hot customer across instances. The assigned instance becomes a hot shard. Inspect load by routing key and consider a design that partitions that customer's work or relaxes the affinity where correctness permits.

Health Is an Input, Not a Guarantee

Before applying any policy, a balancer filters the fleet to its eligible backends. It might exclude an instance that is down, draining, or not ready to accept new work. If that filter is wrong or delayed, even a sophisticated policy routes traffic badly.

For example, api-3 may respond to a shallow process check while its database connection pool is exhausted. A round-robin policy will still send it its turn. Least-connections may send it even more requests if its failed requests close quickly and make its active count look low.

This is why “balancing” and “health” are separate decisions:

health/readiness evidence -> eligible backend set -> routing policy -> selected backend

The next lesson will make those health and circuit states explicit. Here, retain the boundary: a policy is only as useful as the eligibility information it receives.

Trade-offs and Limits

Round robin improves predictability and has little coordination cost. It does not protect against unequal request cost. Least-connections or least-requests can improve tail behavior under uneven work, but they need fresh counters and can be fooled by idle keep-alive connections, delayed completion reports, or background work that continues after the HTTP response.

Weights improve use of known capacity, but require an owner who updates them as instance types and runtime limits change. Stickiness may preserve correctness or locality, but it trades routing freedom for affinity and can create hot keys. Shared state restores routing freedom, but introduces a new dependency that must be observed and scaled.

Watch for the boundary with per-instance request duration, active work, error rate, and routing-key skew. A fleet-wide average can look normal while one instance carries the long tail. A balancer distributes pressure; it does not remove a database bottleneck, repair an unhealthy dependency, or make a stateful design interchangeable by itself.

Common Confusions

Confusion: Equal request counts mean equal load

Why it is tempting:

Request count is easy to graph and round robin makes it look fair.

Better model:

Load depends on duration, CPU, memory, downstream waiting, and connection lifetime. Count is only one proxy for work.

Confusion: Least connections always finds the least busy server

Why it is tempting:

An active connection sounds like active work.

Better model:

Connections can be idle, multiplexed, long-lived, or uneven in cost. Choose the signal whose limitations you understand and validate it against latency and saturation evidence.

Confusion: Sticky sessions make horizontal scale impossible

Why it is tempting:

They restrict routing and can create hot instances.

Better model:

Stickiness is a constraint, not an automatic failure. Use it intentionally when affinity is required, observe its skew, and avoid using it to hide state that could safely be shared.

Practice: Review a Routing Decision

A service has four application instances. Three have 4 vCPUs; one has 8 vCPUs. Requests are mostly short, but 15% stream a report for several seconds. Sessions are stored in a shared database. The current design uses unweighted round robin. During report bursts, one small instance often has the worst p99 latency while another is mostly idle.

Propose a first routing change and name one signal you would use to decide whether it helped. Then name one thing that change cannot prove.

Model answer: Start with a policy that considers outstanding requests, optionally combined with weights so the 8-vCPU instance receives a larger baseline share. Compare p99 latency and active requests per instance before and after, grouped by request class. This may reduce hot spots from long reports, but it cannot prove that the report handler or database is healthy; traces and dependency latency still matter.

Resources

• [DOC] NGINX HTTP load balancing — Focus: Compare basic distribution methods and their configuration assumptions.
• [DOC] HAProxy load-balancing algorithms — Focus: See how routing policies use different workload signals.
• [DOC] Envoy load balancing — Focus: Relate locality, health, and policy selection in a modern proxy.
• [BOOK] The Site Reliability Engineering Workbook: Load Balancing — Focus: Connect balanced traffic to capacity, overload, and operational evidence.

Key Takeaways

• A load balancer presents a fleet as one service and chooses a backend for each new request from an eligible set.
• Round robin is a strong simple choice only when request cost and instance capacity are similar.
• Policies such as least outstanding work and weights use more evidence to react to skew, but their signals can be stale or misleading.
• Local state and sticky routing reduce the balancer's freedom; shared reachable state restores choices while adding another dependency boundary.