Cache Fundamentals - CPU to CDN

By the end of this lesson, you will be able to...

• Describe a cache using locality, key, capacity, hit path, miss path, and authority.

• Transfer that vocabulary across CPU, page, application, and edge caches without equating their guarantees.

• Judge a cache with miss cost and correctness evidence, not hit rate alone.

Idea in one sentence: Caches at different layers reuse nearby copies, but each layer defines “nearby,” “valid,” and “expensive miss” differently.

Core Insight

A product page loads an image, a price, and a recommendation list.

One request may benefit from several caches:

• the CPU reuses recently accessed memory lines while running the code;
• the operating system may serve file-backed pages from memory instead of storage;
• the application may reuse a decoded object or a database result;
• an edge cache may return the image without contacting the origin service.

It is tempting to summarize all four as “a faster copy.” That is a useful start, but it is too weak to guide a design. The copies do not share one key space, one freshness rule, one eviction mechanism, or one authority.

The stronger model is a set of six questions:

What is close to the consumer?
What key identifies reusable work?
What does a hit avoid?
What does a miss cost?
What limits capacity?
Who decides whether the copy is valid?

These questions reveal the repeated cache pattern without pretending that CPU coherence and HTTP freshness are the same mechanism.

One Request, Four Meanings of “Near”

Follow the product image through a simplified request path.

browser request
  -> edge cache
      -> origin application
          -> application cache
              -> database or object store

The application code executing this path also uses CPU caches and the operating-system page cache. Those lower layers are not extra network hops in the diagram. They reduce the cost of computation and local data access inside each hop.

“Near” therefore changes with the consumer:

The common structure is locality: keep likely-to-be-reused work closer to where it is consumed. The unit of locality differs. A CPU cares about addresses and cache lines. An application chooses domain keys. An edge cache reasons about request and response metadata.

A Hit Is a Conditional Shortcut

A cache hit is not merely “the key exists.” It means the layer is permitted to use the copy for this access.

At the application layer, a value might exist but be too old for the product's freshness promise. At an edge, a response might match the URL but not the correct authorization or variation rules. In hardware, the coherence state determines whether a local cache-line copy is valid for the requested operation.

Plain meaning:

A hit is a shortcut whose reuse conditions currently hold.

In the product-page scenario:

The edge can return the image only when its key and validity policy match the request.

Technical name:

The hit path is the work performed when the cache may serve the copy. The miss path is the deeper work needed when the copy is absent or unusable.

This distinction matters because a nominal hit can still be wrong. Fast reuse is not useful when it violates freshness, isolation, or authorization.

The Miss Path Carries the Hidden Cost

Suppose the application cache reports a 95% hit rate.

That sounds strong. Now add illustrative costs:

• a hit takes 2 ms;
• a miss takes 400 ms because it queries a slow dependency;
• 50 requests for the same expired key can miss together.

For an isolated request, the approximate average is:

0.95 × 2 ms + 0.05 × 400 ms = 21.9 ms

The calculation is a teaching model. It ignores queueing and concurrent refill. That omission is exactly where production trouble can hide. If the 5% misses arrive together, they may overload the dependency, increase the 400 ms cost, and push tail latency much higher.

So hit rate is evidence about one branch of the path. It does not answer:

• how expensive misses are;
• whether misses cluster;
• whether refills duplicate work;
• whether hits return acceptable data;
• whether memory is spent on useful entries.

Check: Two caches both report a 90% hit rate. Cache A has a 10 ms miss; Cache B has a 2 s miss. Are they equally effective?

Think first, then reveal.

Answer: No. The same miss rate exposes very different latency and downstream pressure. Compare hit cost, miss cost, tail latency, refill behavior, and correctness boundaries.

Capacity Turns Reuse into Policy

A cache is useful because it is smaller, faster, or closer than the full source. That advantage creates a limit: it cannot keep every possible copy.

Once capacity fills, the cache must decide:

admit this new entry?
keep which existing entries?
evict which entry?
spend how much metadata making that choice?

The next lesson studies LRU, LFU, and ARC because those policies make different predictions about future reuse. At this point, the important concept is simpler: capacity is not a neutral number. It forces the cache to choose which work deserves fast reuse.

Different layers expose capacity differently. An application cache may have a configured byte budget. A CDN account may enforce product-specific policy. Hardware caches have fixed structures and replacement behavior selected by the processor design. Shared vocabulary helps us ask the question; it does not give us control at every layer.

Authority Is Not the Same at Every Layer

“The cache is not authoritative” is a useful application-level rule, but it needs careful transfer.

For a product price, the database or owning service may be the business authority. The cache holds a derived copy and needs a freshness policy.

For file data, the operating system coordinates cached pages with filesystem semantics. For a coherent CPU, hardware maintains rules about which cache-line copies are valid and who may write. These mechanisms preserve architectural behavior; application code does not manually invalidate an L1 entry after changing a product price.

The shared question is:

Which mechanism decides that this copy may be used now?

The answer changes by layer. This is the boundary that prevents a useful analogy from becoming a false equivalence.

A Cross-Layer Comparison

Return to the product page and classify each reuse decision.

The table is not a claim that all layers implement the same algorithm. It is a transfer tool: every row makes locality, identity, avoided work, and boundary explicit.

Check: An API caches personalized recommendations only by URL. Which of the six questions exposes the likely bug first?

Think first, then reveal.

Answer: “What key identifies reusable work?” The URL alone may merge users whose responses are not interchangeable. Authority and validity questions then reveal the privacy and correctness consequence.

Trade-offs and Limits

Caching buys lower latency, less repeated work, and protection for deeper systems. It costs memory, metadata, refill work, invalidation or validation logic, and new failure modes.

That balance also changes over time. A policy that is sensible while the authority has ample headroom may become unsafe when a miss storm can saturate it. Revisit the contract when either the workload or the dependency budget changes.

The common vocabulary is strongest at the level of pressure and policy. It becomes weak when used to claim identical guarantees. CPU coherence does not teach HTTP cache-control syntax. A CDN purge does not explain MESI state transitions. Redis eviction does not control a processor's replacement policy.

Use the analogy to ask better questions, then study the layer-specific mechanism before making a correctness claim.

Practice: Name Two Different Contracts

An API returns a course title and a course image. Title edits must appear within ten seconds. The image may remain unchanged for an hour. Both currently share one cached response with a one-hour expiry.

Propose two cache contracts and name evidence for each.

Model answer: Separate title data from image content or give them independently validatable keys. Use a short validation or explicit invalidation policy for the title and a longer policy for the image. Measure stale-title age and invalidation delay for the title; measure hit rate, miss latency, and origin load for the image. A high combined hit rate would hide whether the title promise is met.

Resources

• [DOC] Redis: Key Eviction — Focus: See how a production cache connects memory limits, approximate policies, and runtime evidence.
• [DOC] MDN: HTTP Caching — Focus: Compare HTTP validity and revalidation with application-level freshness assumptions.
• [BOOK] What Every Programmer Should Know About Memory — Focus: Build the lower-level locality intuition behind processor and memory caches.

Key Takeaways

• Caches repeat a locality pattern, but each layer defines identity and validity differently.
• A hit is a permitted shortcut; a miss exposes the deeper cost that hits hide.
• Hit rate is incomplete without miss cost, refill pressure, tail latency, and correctness evidence.
• Cross-layer analogies are useful for questions, not for importing one layer's guarantees into another.