Cache Fundamentals and Core Patterns

By the end of this lesson, you will be able to...

• Explain why a cache is a non-authoritative copy governed by a policy.

• Trace a cache hit, a cache miss, and a stale read through one request path.

• Choose a basic cache pattern from freshness, latency, and failure constraints.

Idea in one sentence: A cache reuses a previous answer, so every speed improvement comes with a decision about when that answer is still safe to trust.

Core Insight

A course page is suddenly popular. Ten thousand learners ask for the same title, description, instructor, and rating summary during one hour.

The backend can ask the database to rebuild the answer ten thousand times. The answer is correct, but most of that work is repeated. The course description may change once during the hour. The read rate and the change rate are very different.

A cache exploits that difference. It keeps a copy of a previous answer and lets later requests reuse it.

The tempting summary is “a cache makes reads faster.” That is true, but it hides the design problem. The cached value is not automatically current. It came from somewhere, it was stored at some time, and a rule decides whether the system may still use it.

The durable model is:

cache = non-authoritative copy + reuse policy

The copy reduces repeated work. The policy controls the risk introduced by reusing it.

A Small Situation

Consider one endpoint:

GET /courses/204

The response contains:

{
  "title": "Distributed Systems Foundations",
  "description": "Learn how partial failure changes system design.",
  "instructor": "Mira",
  "rating": 4.8
}

The course database is the source of truth. It owns the current course record. The API may create a cache entry with the key course:204:details, but that entry is only a reusable copy.

The components have different jobs:

This separation matters. A cache stores a value. It does not know whether an old course description is acceptable to the user. The application and product contract must decide that.

The Naive Model

The first design has no cache:

request -> API -> database -> API -> response

This design is easy to reason about. Every read asks the authoritative source. If the database has the latest committed value, the response sees that value.

It becomes wasteful when many requests ask the same question while the answer changes slowly. Database connections, query execution, serialization, and network hops are paid for again and again.

So we add a map in front of the database:

request -> API -> cache
                    | hit  -> response
                    | miss -> database -> fill cache -> response

That version looks simple until an editor changes the course description.

The database now contains version 42. The cache still contains version 41. Both values exist at the same time. The system needs a rule for which copy may be served and for how long.

This is where “put it in memory” stops being a complete design.

A Cache Is a Copy With a Policy

Plain meaning:

A cache lets the system answer an old question again without doing all the original work.

In this scenario:

The API reuses a recent course-details response instead of querying and serializing the course record for every request.

Technical name:

The stored response is a non-authoritative copy. The rules for creating, trusting, and replacing it form the cache policy.

A useful policy answers five questions:

1. Authority: Which component owns the current value?
2. Identity: Which key names the reusable answer?
3. Fill: What event puts a value into the cache?
4. Trust: How does the read path decide that the value is fresh enough?
5. Recovery: What happens when the cache or source cannot answer?

For the course page, one policy might say:

authority: course database
key: course:204:details
fill: first read after a miss
trust: up to 60 seconds after fill
recovery: bypass cache and read the database

The 60-second rule is a time to live, usually shortened to TTL. A TTL does not make the cached value correct. It limits how long this particular policy is willing to trust the copy without asking again.

A Worked Request Trace

Let us make the hidden states visible.

Starting state:

database: version 41
cache: empty
TTL: 60 seconds

Now follow six events:

The cache improved the first part of the path. Learner B avoided a database query and received a faster answer.

The same policy gave Learner C a stale answer. The cache behaved exactly as designed: version 41 was still inside its 60-second trust window. The design question is whether that behavior matches the user promise.

The naive failure contrast is useful:

naive claim: the cache contains the course details
better claim: the cache contains a course-details copy that this policy trusts until 60s

So far, we have seen a complete path: input request, cache lookup, miss transition, source read, cached intermediate state, later hit, source update, stale hit, expiry, and refresh. This matters because cache bugs often come from reasoning only about the hit and forgetting the other states.

Core Cache Patterns Are Maintenance Rules

Named cache patterns answer who maintains the copy and when.

Cache-aside

The application manages the cache on the read path.

def get_course(cache, database, course_id):
    key = f"course:{course_id}:details"

    cached = cache.get(key)
    if cached is not None:
        return cached

    current = database.fetch_course(course_id)
    cache.set(key, current, ttl_seconds=60)
    return current

Cache-aside is a common starting point. Only requested values enter the cache. A miss is slower because it pays for both cache lookup and source read. Concurrent misses can also repeat the same fill work.

Write-through

The write path updates the authoritative source and the cache as part of one logical operation.

write -> source update -> cache update -> acknowledge

Reads can observe fresh cache entries sooner. Writes become more expensive and gain another failure boundary. If the database update succeeds but the cache update fails, the design still needs a repair rule.

Write-behind

The write path updates a fast layer first and sends the change to durable storage later.

write -> fast layer -> acknowledge
                      -> durable source later

This can reduce write latency, but it changes durability and authority assumptions. A failure between acknowledgement and the later write can lose data. Write-behind is not simply a faster write-through. It is a different reliability contract.

The pattern name is less important than the maintenance rule. Ask who changes the copy, when the source changes, and what a partial failure leaves behind.

Freshness Is Part of the User Promise

Not every field can tolerate the same staleness.

A course description that is 40 seconds old may be acceptable. A cached enrollment decision based on the last available seat may not be acceptable at all.

This gives us a practical sequence:

user promise -> acceptable staleness -> maintenance policy -> cache technology

Starting from the technology reverses the reasoning. “Redis supports a five-minute TTL” does not tell us whether five minutes is safe.

A TTL also gives only a rough upper bound for time-based freshness. With a 60-second TTL, a value may be almost current or almost 60 seconds old, depending on when the source changed. Delayed invalidation or clock and propagation effects can extend the real boundary.

The central trade-off is explicit:

more reuse -> lower latency and source load
more reuse -> more opportunity to serve an older answer

The correct point depends on the request path. There is no universally safe hit rate or TTL.

Failure Boundaries and Signals

A cache adds a component and therefore adds failure choices.

The cache is unavailable

The API may bypass it and read the database. That preserves availability, but a fleet-wide cache outage can suddenly send all traffic to a source sized for cache-protected load.

Useful signals include cache error rate, source query rate, database connection pressure, and request tail latency.

The cache is healthy but stale

The request is fast and wrong relative to the latest source value. Latency dashboards alone will not reveal this. Version mismatches, freshness age, invalidation delay, and user-visible inconsistency are more relevant.

Many requests miss together

When one popular key expires, many requests may query the source before the first fill completes. This is a cache stampede. Later lessons will develop the controls; the important point here is that a miss is a workload transition, not just an empty lookup.

The cache has a high hit rate

A high hit rate can still hide a bad policy. The system may be serving data that should not have been cached or trusting it for too long.

This helps when repeated reads dominate cost. It costs memory, an extra dependency, invalidation logic, and operational attention. It does not protect the source from every burst, and it does not make stale data correct. The boundary appears in miss rate, source load, age of served values, error rate, and tail latency.

Common Confusions

Confusion: A cache is the same as memory

Why it is tempting:

Many caches use memory because memory is fast.

Better model:

Caching describes reuse under a policy. The storage may be process memory, a shared service, local disk, a browser, or an edge server.

Confusion: A short TTL guarantees consistency

Why it is tempting:

Shorter trust windows usually reduce how long stale values survive.

Better model:

A TTL bounds one refresh mechanism. It does not coordinate concurrent writes, guarantee invalidation delivery, or prove that every read sees the newest value.

Confusion: More cache hits are always better

Why it is tempting:

Hits are usually faster and cheaper than misses.

Better model:

A hit is useful only when the reused answer is acceptable. A fast stale authorization or inventory answer can be worse than a slower authoritative read.

Check Your Understanding

Check: The course description changes at 35 seconds, and a 60-second cache entry was filled at 0 seconds. What can a read at 40 seconds return?

Think first, then reveal.

Answer: It can return the old description. The entry is still inside its trust window. The cache hit is expected behavior, not proof that the value matches the latest database version.

Check: A cache outage causes database traffic to increase tenfold. Was the cache “only an optimization”?

Think first, then reveal.

Answer: No. The system's safe operating load depended on the cache. The cache may not own authoritative data, but its availability is part of the production capacity model.

Practice

A course service exposes two values:

• course_description, read 20,000 times per hour and changed a few times per week
• seat_available, read 2,000 times per hour and changed whenever a learner enrolls

Design the smallest cache policy for each value. State:

1. the source of truth
2. whether you would cache it
3. the cache key
4. the acceptable staleness
5. the fill or update rule
6. the behavior when the cache is unavailable

A good answer should mention:

• The description is a strong cache candidate because reads greatly outnumber changes and modest staleness may be acceptable.
• Seat availability needs a much stricter promise. A cached value may be useful for display, but the enrollment write must verify availability against authoritative state.
• The two values should not inherit the same TTL merely because they appear on the same page.
• Bypassing the cache protects availability only if the database can absorb the resulting load.
• The policy should say what the user may observe, not only where bytes are stored.

Connections

The next lesson moves the copy outside one API process. A shared Redis cache lets many instances reuse the same entry, but it also adds a network hop and a fleet-wide dependency. The “copy + policy” model remains the same.

Later lessons will revisit invalidation, eviction, stampede control, and edge caches. Each mechanism answers one of the five policy questions introduced here.

Resources

• [DOC] Redis Caching — Focus: See cache-aside and expiration in one concrete shared-cache system.
• [DOC] HTTP Caching — Focus: Compare application caches with freshness and validation rules carried in HTTP metadata.
• [ARTICLE] AWS Caching Best Practices — Focus: Review hit rate, TTL, failure, and operational trade-offs in production cache design.
• [BOOK] Designing Data-Intensive Applications — Focus: Connect cached copies to storage, consistency, and system-design trade-offs.

Key Takeaways

• A cache is a non-authoritative copy plus a policy for filling, trusting, refreshing, and recovering it.
• Cache hits reduce repeated work, but a hit says nothing by itself about whether the value is fresh enough.
• Cache-aside, write-through, and write-behind differ mainly in who maintains the copy and what partial failures can leave behind.
• Freshness starts from the user promise; TTL and technology come after that decision.
• Cache health requires more than hit rate: watch source load, value age, errors, misses, and tail latency.