Caching Across Storage Layers

By the end of this lesson, you will be able to...

• Trace a read through local, shared, edge, and authoritative storage layers.

• Predict the cost and pressure created by a cache miss or a clustered set of misses.

• Choose a freshness boundary and a refill rule for a small storage workload.

Idea in one sentence: A cache makes common reads cheaper by keeping a copy closer to the reader, but it also creates another copy whose freshness and refill must be controlled.

The page that is fast until it is not

At 09:00, a course platform publishes a popular lesson. Thousands of learners open the same page and request the same thumbnail:

/media/storage/lesson-03/thumbnail.png

The team knows the image lives in object storage. Their first model is simple: a request reads the image from object storage and returns it. That works for ten learners. At ten thousand, it means ten thousand network requests to the origin for identical bytes. The origin may be durable and correct, but it is now doing repeated work far from most readers.

So the team adds a CDN edge cache. Then the application keeps lesson metadata in a process-local cache and in a shared cache. The page becomes fast. It is tempting to call this a collection of independent performance tricks.

But consider the next event: the editor replaces the thumbnail after finding a typo. One learner still sees the old image from a browser cache. Another is served by an edge cache. A third reaches the origin because the edge entry has expired. The system has not merely added speed; it has created several temporary copies, each with its own age and refill behavior.

That is the useful cache model for storage work. A cache is not a sidecar to the real path. It creates a hit path and a miss path. It shifts work away from deeper storage when its copy is usable, and shifts that work back—sometimes all at once—when it is not.

Core Insight

The authority holds the storage system’s durable answer. A cache holds a reusable copy of an answer that may let a request avoid going to that authority.

Plain meaning:

Keep a useful answer near the next person likely to ask the same question.

In this scenario:

An edge cache keeps a copy of the thumbnail near learners; a process cache keeps lesson metadata near one application worker.

Technical name:

This arrangement is a cache hierarchy: several layers, each trading capacity and coordination for a shorter or cheaper read path.

The word hierarchy matters. The layers are not all interchangeable. They sit at different distances from the reader and share data with different groups.

thumbnail request
  browser cache
       ↓ miss
  CDN edge cache
       ↓ miss
  object-storage origin

lesson-title request
  application process cache
       ↓ miss
  shared cache
       ↓ miss
  metadata database (authority)

A browser cache saves a round trip for one learner. An edge cache saves geographic distance for many nearby learners. A process cache avoids even a network hop for one application instance. A shared cache avoids repeatedly asking the database across many instances. The correct layer depends on who is likely to reuse the answer and which expensive path it can remove.

Start with the expensive path

Caching is worthwhile only when a miss has enough cost and an answer has enough reuse. Cost can mean latency, network transfer, origin requests, database work, or pressure on a limited service. Reuse can be many readers requesting the same image, or one process repeatedly consulting the same course metadata.

The naive rule is: “If a request is slow, cache it.” That breaks in several ordinary cases:

• A value changes so often that most cached copies expire or require invalidation before they are reused.
• The underlying read is already cheap, so the cache adds more moving parts than saved work.
• The cache is too small for the working set and spends its time evicting values just before reuse.
• The data is user-specific or permission-sensitive, so a shared copy needs careful keying and access rules.

The better question is: which repeated, expensive path are we trying to avoid, for whom, and for how long may its answer remain valid? That question makes the cache a storage design choice instead of an automatic response to a latency graph.

A worked read trace

Follow one thumbnail request after a learner opens the lesson page. The input is a request for version v7 of the image. The version is part of the name on purpose: it lets the new asset be a new durable object rather than an ambiguous mutation of the old one.

GET /media/storage/lesson-03/thumbnail-v7.png

The output is an image rendered from a known version. The path is short on a hit and deep on a miss, but both paths are part of correct behavior.

Now contrast the naive failure. If the editor had overwritten a stable key such as thumbnail.png, layers could hold different copies under the same name. Whether a learner sees the new image would depend on each layer’s expiration and invalidation rules. Versioned content does not remove every freshness problem—someone still has to publish the pointer from “current thumbnail” to v7—but it narrows the problem. Immutable content can be cached aggressively; the small mutable pointer needs a clear freshness rule.

Check: The thumbnail origin is healthy, but the edge hit rate falls from 98% to 80% just after a deployment. What should you investigate before concluding that object storage is slow?

Think first, then reveal.

Answer: Inspect why the edge is missing: changed keys, a cache-policy change, evictions, expired entries, or a new request pattern. A lower hit rate can send far more work to the origin even when the origin itself has not changed. Compare miss rate, origin request rate, refill latency, and error rate together.

A shared cache changes the ownership question

The thumbnail is content-like: one key can serve many readers. Lesson metadata has a different shape. The application may ask for the lesson title, publishing state, and current thumbnail key on each page request. A process-local cache is very fast, but each process owns a separate copy. A shared cache lets many application instances reuse one answer, while the database remains the authority.

request reaches application A
  → local cache? no
  → shared cache? yes: {title, current_thumbnail: v7}
  → return answer and optionally warm A's local cache

This gives a useful division of responsibility:

• The database owns the durable metadata record.
• The shared cache reduces repeated database reads across the application fleet.
• Each local cache reduces repeated shared-cache reads for a single busy process.

The copy closest to the reader is often fastest, but it is also hardest to update everywhere. When the editor changes current_thumbnail from v6 to v7, the team must decide what the system promises. Is a thirty-second old title acceptable? Must a publishing dashboard show the new value immediately? Can readers tolerate a short stale window but not an old permission decision?

There is no universal TTL that answers those questions. A time-to-live is a budget for staleness, not a magic correctness number. It says that a cache may serve its copy for some period before asking again. Explicit invalidation can make a change visible sooner, but invalidation itself is another distributed action that may be delayed or missed. Versioned keys, bounded TTLs, revalidation, and event-driven invalidation are tools; the promised freshness boundary decides which combination is appropriate.

Where a fast path becomes a failure path

The most dramatic cache failure happens on a miss shared by many callers. Suppose current_thumbnail expires in the shared cache just before class begins. Five thousand requests arrive while the database lookup is slow.

09:00:00  shared entry expires
09:00:01  5,000 application requests miss
09:00:02  each request asks the database for the same row
09:00:03  database queue grows; responses slow down
09:00:04  callers retry or wait; cache still has no refill

The cache was meant to protect the database, yet its empty entry has directed a burst toward the database. This is a cache stampede (also called a thundering herd): many callers simultaneously take the expensive miss path for the same reusable value.

One remedy is request coalescing. The first miss owns the refill; the others wait briefly for its result rather than each making the same database request.

first miss:  acquire refill lock → read authority → store answer → release
later miss:  wait for refill, then read stored answer

Other designs use jittered expirations, background refresh, stale-while-revalidate, or a short-lived negative cache for absent values. Each changes a different failure mode. Serving a slightly old thumbnail while one worker refreshes it can protect the origin. It is a poor choice for a revoked access decision, where stale permission data may be a security bug. Caching reduces duplicate work; it does not decide which stale answers are safe.

Check: A team adds a five-minute cache entry for “may this user view this private course?” because the database check is expensive. What is the missing design question?

Think first, then reveal.

Answer: They must define the acceptable revocation delay. A permission that remains cached after access is removed may expose data. The cache key, invalidation path, TTL, and fallback behavior must reflect the security promise, not only the database cost.

Trade-offs and limits

Caches improve latency and origin load when their copies are reused. They cost memory or edge capacity, add observability work, and introduce a second state that can disagree temporarily with the authority.

Do not judge a cache by hit rate alone. A 95% hit rate might be excellent when the remaining 5% is cheap, or dangerous when every miss triggers an expensive metadata query. Pair it with the miss cost: latency, origin requests, queue depth, retries, and errors. Also monitor the age of served values when freshness matters. A cache with perfect hits can be confidently wrong.

This lesson also stops at a boundary. It explains how copies and refill paths shape storage reads; it does not make a cache authoritative or explain how replicas agree after failures. The next lesson adds that missing pressure: multiple durable copies may have seen different prefixes of a write history. A cache can hide a read from its authority; replication must decide which durable history the authority may promise.

Common confusions

Confusion: A cache is just faster storage

Why it is tempting:

The cache returns bytes or metadata, so it looks like a smaller, faster version of the origin.

Better model:

A cache is a copy with a policy. Its usefulness depends on hit reuse, its correctness depends on freshness, and its risk appears on misses and invalidation.

Confusion: A high hit rate proves the cache is healthy

Why it is tempting:

Hit rate is easy to graph and usually moves in the expected direction when performance improves.

Better model:

Read hit rate together with miss cost, entry age, refill latency, and downstream saturation. The value of a hit depends on the path it prevented.

Confusion: A TTL is an invalidation strategy

Why it is tempting:

Expiry eventually removes an old value, so it can feel like freshness is solved.

Better model:

A TTL only bounds how long a layer may reuse its copy before checking again. If a change needs faster visibility, define how updates invalidate, version, or revalidate the relevant copies.

Practice: design the two paths

The course platform adds a “current enrollment count” beside every lesson. The count changes after enrollments and cancellations. It is acceptable for ordinary course pages to show a count up to one minute old, but an instructor dashboard must show a newly completed enrollment within a few seconds. At 09:00, many learners open the same course page.

Design the smallest cache plan. Specify:

1. the authority and the key for each cacheable answer;
2. which readers may use a shared or edge cache;
3. the acceptable freshness rule for each reader; and
4. how you would prevent a clustered miss from overwhelming the authority.

Model answer: Keep the enrollment database as authority. Cache course_id → public_count in a shared cache with a bounded one-minute TTL; an edge cache may serve the public page if the page’s own policy permits the same age. Use a different instructor-dashboard path that reads a shorter-lived shared value, revalidates after a completed enrollment, or receives an explicit update; do not reuse a one-minute public value when the product promise is seconds. For the hot public key, coalesce refills or refresh in the background and watch cache misses, database query rate, refill latency, and the age of served counts. This plan accepts a visible trade-off: public readers get lower origin load in exchange for bounded staleness, while the instructor path pays more for fresher information.

Resources

• [DOC] Redis eviction — Focus: See how capacity pressure determines which reusable values remain cached.
• [DOC] HTTP caching — Focus: Compare freshness, validation, and stale-serving rules at the browser and edge boundary.
• [BOOK] Designing Data-Intensive Applications — Focus: Connect caching and derived data to invalidation, replication, and consistency trade-offs.

Key Takeaways

1. A cache hierarchy creates distinct hit and miss paths; each layer saves a different kind of distance or duplicate work.
2. The authority owns the durable answer, while cache policies decide how long a copy may stand in for it and how it is refilled.
3. Cache design is a correctness and stability problem as well as a latency problem: define freshness, prevent clustered misses, and observe miss cost.