Cache Stampede Control and Request Coalescing

By the end of this lesson, you will be able to...

• Trace how one expired popular key can become many equivalent origin requests.

• Choose a safe coalescing key, leader policy, and follower outcome for a refill.

• Recognize when stale serving, jitter, or a distributed lease helps—and which freshness or failure boundary remains.

Idea in one sentence: A cache miss is not dangerous by itself; it becomes dangerous when equivalent misses all decide to refill independently.

Core Insight

For Atlas Shop, product:42:en expires just as a campaign sends 200 requests in one second. The cache says “miss” to every request. Each handler fetches the same product, renders the same public response, and writes it back. Origin CPU spikes even though the resulting cache value is identical.

The tempting model is that a miss means “this caller must fetch.” That is reasonable for a rare key or a request that cannot share a response. It fails when many callers have the same safe key and arrive before the first refill completes.

The stronger model is one refill decision per shareable key. One caller becomes the leader; compatible followers either wait for its result, receive a permitted stale value, or receive a bounded overload response. The mechanism reduces duplicate work. It does not decide freshness policy or make unrelated requests equivalent.

The Situation: One Expiry, Many Refill Decisions

Atlas has already made the public product key safe: it includes product id and language, but not analytics parameters. The origin render takes 120 ms. The following is illustrative traffic:

The unit of coordination is not “the cache” or “the route.” It is the exact response identity that may safely be shared. If the key omits language, authorization, or another representation-changing input, coalescing efficiently returns the wrong response.

The Initial Model: Raise TTL Until the Spike Disappears

Longer freshness reduces how often a key expires. It is a useful lever when the product can honestly serve an older value for that longer period. But it cannot stop a burst after a purge, an eviction, a deploy, or a genuine expiry. It also does not prevent duplicate in-flight fetches when a miss happens.

The parallel mistake is to lock every miss globally. That protects origin, but one slow key now delays unrelated keys. The missing boundary is a per-key refill state: calls for product:42:en may coordinate; calls for product:99:en should proceed independently.

HTTP caching makes the freshness distinction explicit: a response's reuse policy and its validators determine when it can be reused or checked again. Coalescing sits beside that policy; it does not turn stale data into fresh data. MDN HTTP caching.

The Mechanism Step by Step

In plain English, request coalescing records that a refill for one cache key is already in flight. The first eligible caller owns the work. Later callers with the same key do not start another equivalent refill.

state for product:42:en
  cache: expired value V7
  in_flight: none

request A: miss -> claim leader -> in_flight=A
request B: miss -> sees A -> follower
request C: miss -> sees A -> follower

The implementation can be a local in-process table, a shared coordination service, or a CDN feature. The mechanism matters more than the product name. Go's singleflight is one concrete local form: it suppresses duplicate calls for one key, and duplicate callers wait for the original and receive its result. Go singleflight.

Now trace Atlas's policy. Its numbers are illustrative.

The follower policy is a product contract. “Serve stale” fits only when V7 is safe within a named stale window. “Wait” protects freshness but can increase tail latency. “Fail fast” protects the origin but may lower availability. The right choice depends on the response's correctness promise, not on cache-hit aesthetics.

A follower must not inherit an unlimited leader deadline

It is easy to implement “wait for the promise” and accidentally turn coalescing into a queue with no escape. The leader has a refill budget; each follower has its own request deadline. Those are different clocks. A leader may still have 300 ms left to complete useful work while a browser request has only 40 ms before its page budget is exhausted. That browser must take its named alternative—permitted stale V7, a partial response, or a controlled error—rather than wait simply because the leader exists.

The reverse race matters too. Suppose A's 500 ms lease expires at 500 ms, B is allowed to become a replacement leader at 510 ms, and A finally gets V8 from origin at 530 ms. If A can blindly write after losing ownership, it can overwrite B's newer value or clear B's in-flight marker. A robust design checks an ownership token or generation when publishing and clearing state. The leader that still owns the current entry may publish; a late leader records its result as abandoned or discards it according to policy.

This is why “one request at a time” is not a complete specification. State who may take over, how many re-elections are allowed, what result followers see during the handoff, and whether origin work can be cancelled after all followers have left. For a public catalogue entry, one replacement attempt and a short stale window may be sensible. For a balance or permission check, a stale fallback may be forbidden, so a short bounded error is more honest. Coalescing coordinates work; the endpoint's correctness contract still decides the user-visible outcome.

Where It Breaks: Leaders Can Fail Too

What if A times out, crashes, or receives an origin error? Followers must not wait forever, and a lease must not let an old leader overwrite a newer result.

The in-flight entry therefore needs a deadline, an observable outcome, and careful cleanup:

leader succeeds -> publish V8 if still current -> clear entry
leader fails before deadline -> return named failure or allowed stale V7 -> clear entry
leader exceeds deadline -> followers may re-elect once policy permits

For cross-process coordination, a lease or lock needs an ownership token and a bounded validity window. A naïve key deletion can release a lock acquired by another client after expiration. Redis's lock guidance illustrates the reason for a unique token and conditional release; it also makes clear that safety guarantees depend on the chosen design and failure assumptions. Redis distributed locks.

This does not mean every cache refill needs a distributed lock. A local coalescer can be enough when duplicate refills across instances are acceptable. A shared lease may be justified when origin protection needs to span instances. State the scope honestly: local suppression reduces duplicate work per process; it does not guarantee one global refill.

Cost, Limits, and Signals

Coalescing improves origin protection and can make post-purge behavior predictable. It costs in-flight state, deadlines, follower policy, and observability. The trade-off is between some controlled follower waiting or stale delivery and uncontrolled duplicate work at origin.

Watch these signals by key class:

Jittered expiry can spread future refills across time, reducing synchronized expiry. It does not replace coalescing during a sudden burst. Stale-while-revalidate can keep readers fast, but it does not suit every response. A lease can coordinate instances, but it introduces clock, timeout, and recovery assumptions. None of these mechanisms fixes an origin that is slow for every unique key.

Check: Two requests differ only by utm_campaign and hit an expired public product response. A third request is for the same product in another language. Which requests may share one refill?

Think first, then reveal.

Answer: The first two may share only after verifying the campaign parameter does not affect the response and normalizing it out of the key. The other language must use a different key and refill decision because its representation differs.

Trace It Yourself

An account-summary key is allowed to serve at most 10 seconds stale. Fifty equivalent requests arrive after a purge. The origin normally answers in 80 ms but now takes 700 ms. Design the follower policy. State the coalescing key, leader deadline, stale rule, follower timeout, and the signal that would make you disable stale serving for this endpoint.

A good answer should mention:

• a key that includes account identity and any representation-changing authorization or locale dimension;
• exactly one leader attempt per key while its lease is valid, with a deadline below the caller's total timeout;
• stale data only when its measured age is within 10 seconds and the account contract permits it;
• a bounded wait or explicit overload response for followers rather than indefinite waiting; and
• a correctness signal such as an account-balance update, authorization change, or stale-age breach that requires fresh data or a different response.

Connections

The review lesson ended with an optimization loop: bound unnecessary work, then measure the new limit. Coalescing applies that loop to a cache miss before it multiplies origin work. The next lesson generalizes admission control: token buckets and shared counters decide which traffic may enter a constrained path at all.

Resources

• [DOCS] Go singleflight — Focus: Inspect duplicate-call suppression, keyed in-flight work, and follower result sharing.
• [DOCS] HTTP caching — Focus: Separate freshness, validation, and permitted stale reuse from refill coordination.
• [DOCS] Redis distributed locks — Focus: Study ownership tokens, conditional release, lease windows, and failure assumptions for shared coordination.

Key Takeaways

• Coordinate an exact safe response key, not every cache miss or every request on a route.
• One leader plus compatible followers can reduce an origin stampede, but follower behavior must be an explicit freshness and availability contract.
• Local coalescing and cross-instance leases have different scopes and failure assumptions.
• Deadlines, cleanup, stale-age limits, and per-key metrics make a refill mechanism operable.
• Coalescing reduces duplicate work; it does not make a non-shareable response safe or a unique origin request cheap.