Caching and Worker Performance Capstone

By the end of this lesson, you will be able to...

• Defend an end-to-end request and worker design from explicit freshness, burst, retry, and tail-latency constraints.

• Trace where authority, waiting, admission, cache reuse, and asynchronous side effects occur in one workload.

• Propose evidence that can distinguish a helpful optimization from a displaced or hidden bottleneck.

Idea in one sentence: A responsive backend is not a pile of caches and workers; it is a set of bounded promises about which work may happen now, later, once, or from a safely reused result.

Core Insight

For Atlas, the moment its product launch begins brings 2,000 requests per second for the same product page, one partner retrying inventory updates too quickly, and merchants continuing to submit image work. The system cannot make all three pressures disappear. It must decide which public copy may be reused, which write may enter, which job may wait, and which result must be checked before it is repeated.

The lessons in this track replace one vague response—“make the backend faster”—with a chain of smaller decisions. A copy can be fast without being authoritative. A request can be rejected without failing the system. A job can be accepted without being complete. A retry can preserve useful work without proving that an effect did not already happen. The capstone tests whether those boundaries still hold when all four pressures arrive together.

The Scenario

Atlas Shop is about to run a product launch. A campaign can send 2,000 product-page requests per second to one popular item. Merchants also upload new product images, and a partner integration occasionally retries inventory updates far faster than normal. The storefront must stay useful while these pressures overlap.

The first instinct is to add more cache time, more workers, and more API instances. That can improve an average graph while making the important promises less clear. A long cache lifetime can show the wrong price. More workers can create more retries against a failing image service. More API instances can multiply a local rate limit. A fast queue drain can conceal duplicate side effects.

This capstone asks for one defensible workload design. You will not choose a universal architecture. You will state what Atlas promises, draw the synchronous and asynchronous paths, predict failures, and name the evidence that would force a revision.

Constraints

Treat these as the product contract for the exercise. The quantities are illustrative constraints, not observed production measurements.

The key separation is deliberate. A product page can reuse a public description. Checkout cannot treat that copy as authority for stock. An image job can be accepted before it completes. A retry can preserve an image intent, but it cannot be assumed safe merely because a queue delivered it again.

Design Goal

Design a path that preserves four promises at once:

1. Public product rendering can reuse a boundedly stale public representation.
2. Checkout consults authoritative price and availability before accepting an order.
3. One noisy partner or one failed job cannot consume unbounded shared capacity.
4. An operator can locate the pressure with evidence before changing a timeout or adding replicas.

The goal is not “zero latency” or “exactly once everywhere.” Those phrases hide the trade-offs. The goal is controlled degradation: a public page may be slightly old under a named rule; a partner may receive a clear rejection; an image may remain pending; an uncertain external effect may be reconciled instead of repeated.

Proposed Model

Atlas uses separate contracts for the public read, authoritative order, partner write, and background render paths:

public product request
  -> edge / application cache key: product + language + public variant
  -> fresh value, permitted stale value, or one coalesced refresh
  -> catalog origin only for the elected refresh

checkout request
  -> checkout service
  -> authoritative price + inventory reservation
  -> accepted order or clear rejection

partner inventory update
  -> shared tenant-and-route token bucket
  -> authoritative inventory write
  -> invalidation or versioned refresh signal for product copies

merchant image upload
  -> durable image intent + job enqueue
  -> worker receives with visibility lease
  -> idempotent render / publish
  -> status update, acknowledgement, or bounded retry -> DLQ

The components are intentionally not one giant “performance layer.” Each owns a smaller decision:

This is the central capstone move: authority, admission, reuse, and completion are different facts. Naming them prevents one fast component from accidentally making a correctness promise owned by another.

Walkthrough

First, trace a normal product page. The cache holds public representation P41:en:v7, generated 20 seconds ago. It includes the description, public image URL, and a display hint that is not checkout authority. The request key includes product id, language, and the public variant; it excludes analytics parameters only after Atlas has verified they do not change the response.

At t=0, the page is fresh. The cache serves v7. At t=65s, the 60-second freshness period has ended, but the item is within its 30-second stale window. One request becomes refresh leader for P41:en; compatible followers receive the named stale v7 or wait briefly according to the page policy. They do not each render from origin. When origin returns v8, the leader stores it and clears the in-flight marker.

Now add the campaign burst and the partner error:

The trace shows a useful asymmetry. A product page can degrade by serving a boundedly stale copy because the contract permits it. Checkout should not convert that availability choice into a stock promise. An image upload can degrade to processing because the user knows completion is deferred. A partner update can degrade to a clear admission rejection because allowing unbounded work would damage everyone else.

So far, the design has not proved that latency will be good. It has made the work visible enough to ask the right question: which boundary is now consuming the budget?

Failure Review

1. The cache keeps pages fast but hides an unsafe field

Suppose a developer adds live availability to the cached public payload without changing the freshness contract. The cache hit rate rises and product-page p95 improves, but customers can see a quantity that checkout will not honor. This is not a cache-performance success. It is an authority error.

Correction: keep checkout's availability decision on the authoritative path, or explicitly make the displayed availability a non-binding, boundedly stale hint. Track cache version, stale age, and the difference between display and checkout rejection separately. A higher hit rate cannot settle that correctness question.

2. The rate limit works on one server but not during the launch

If each API process holds its own partner counter, four instances turn a two-per-second tenant policy into roughly eight per second. The design looks fine in a one-instance test. It breaks when the load balancer distributes the retry burst.

Correction: use a shared, atomic token decision for the tenant-and-route key when the quota must span instances. Redis documents shared per-tenant limits and atomic read-decide-update operations as the relevant distributed boundary. Redis rate limiter. If the shared counter times out, name the policy: Atlas may fail closed for inventory writes to protect the database, while a public read might use a tightly bounded local emergency allowance. Neither choice preserves every benefit.

3. More image workers make the renderer incident worse

When the renderer returns temporary errors, doubling worker concurrency can double the retry load. The queue depth may decrease briefly because workers take messages, while the number of in-flight attempts and renderer failures rises. This is a displaced bottleneck, not recovery.

Correction: bound worker concurrency by the renderer's tolerance, classify temporary versus permanent versus unknown failures, and use delayed retries with jitter. A message that exceeds its attempt budget enters the DLQ with its image intent and provider evidence. A DLQ separates work that needs diagnosis from new useful work; it does not make redrive safe. Amazon SQS dead-letter queues.

4. A worker visibility timeout creates overlapping publication

If rendering typically takes 90 seconds but visibility expires after 60, another worker may receive the image job while the first still runs. Extending the lease can reduce premature redelivery, but a crash or duplicate delivery remains possible. The image publication step must use image:I-88 as its stable intent and make “publish version” idempotent or reconcilable.

Correction: treat the queue receipt as a delivery lease, not as an identity. The source queue's visibility setting manages timing; application state and the downstream API determine whether repeating the effect is safe. SQS visibility timeout.

Trade-offs

The design intentionally spends complexity in several places. That is not a flaw if each cost buys a named protection.

The overall trade-off is lower tail latency and bounded overload versus freshness limits, delayed completion, more state, and operational discipline. There is no free path around the constraints. Tail latency can dominate an interactive service even when averages look acceptable, which is why Atlas watches p95/p99 and queue delay alongside mean latency. The Tail at Scale.

Evidence and Readiness

Before the launch, Atlas should run a controlled load test and a failure drill. The test need not reproduce the whole internet. It must change one pressure at a time and make the decision visible.

The proposed dashboard is deliberately cross-layer:

request: p50 / p95 / p99, cache hit and stale age, coalesced follower wait
admission: allowed / rejected by tenant and route, remaining tokens, limiter latency
origin: render count per key, database latency, pool or queue wait
workers: ready / delayed / in-flight counts, oldest job age, delivery count
effects: idempotency outcomes, publish versions per intent, DLQ arrivals and age

Do not infer success from a single green queue graph. A shrinking queue with growing DLQ arrivals can mean the system is giving up on work. A higher cache hit rate with wrong checkout expectations can mean the system is faster at serving the wrong copy. Evidence must connect the intervention to both the protected resource and the user promise.

Final Challenge

Write a one-page design review for Atlas's launch path. Include a small diagram or table for the synchronous product/checkout path and the asynchronous image path. State the cache key and freshness contract, the authoritative checkout read, the partner rate-limit key and failure policy, the worker's idempotency identity and retry budget, and five signals you would alert on or inspect during the launch. Then choose one deliberate degradation policy and defend why it is acceptable.

A strong answer earns each of these points:

• Authority and freshness: separates a reusable public product representation from authoritative checkout price and stock; names stale limits rather than saying “cache it.”
• Burst control: uses a safe coalescing key for correlated public misses and a shared tenant-and-route admission key for partner updates; does not confuse either with successful completion.
• Async safety: gives image work a durable intent, bounded concurrency and retries, and a DLQ path; treats uncertain external results as reconciliation work.
• Failure reasoning: predicts at least three specific failures, including one that could improve an average metric while harming correctness or tail latency.
• Evidence: pairs each important mechanism with a signal that could falsify its expected benefit, such as stale age plus cache version, 429s plus downstream queue depth, or retry count plus renderer errors.
• Trade-off defense: names what is intentionally allowed to degrade—such as a 30-second-old public description or an image processing state—and what must not degrade, such as checkout authority or duplicate publication.

Resources

• [PAPER] The Tail at Scale — Focus: Relate tail latency to system size, utilization, and an evidence-driven responsiveness goal.
• [DOCS] Redis rate limiter — Focus: Review why shared tenant limits need key scope and atomic state updates across instances.
• [DOCS] Amazon SQS visibility timeout — Focus: Separate delivery leases from idempotent business effects and retry safety.
• [DOCS] Amazon SQS dead-letter queues — Focus: Plan quarantine, evidence, retention, and measured redrive for jobs that cannot make progress.

Key Takeaways

• A full request path needs separate contracts for reuse, authority, admission, deferred work, and completion.
• A cache can make a public page fast without becoming checkout authority; a queue can defer work without proving its effect happened once.
• Coalescing, shared rate limits, worker concurrency, retries, and DLQs each bound a different way that repeated work can multiply.
• Good performance evidence follows pressure across layers: request tails, stale age, shared admission, origin work, queue age, and effect outcomes.
• A defensible design states what may degrade and what may not, then tests that boundary before trusting a better average latency number.