Zero-Downtime Deployments

By the end of this lesson, you will be able to...

• Trace how ready capacity, cold caches, open connections, and queued work change during a rolling deployment.

• Distinguish a process that is alive from an instance that is safe to receive production traffic.

• Diagnose rollout-time saturation and choose a bounded mitigation.

Idea in one sentence: A deployment avoids downtime only when usable capacity stays ahead of demand while old and new instances overlap, warm up, and drain.

Core Insight

A checkout service normally runs eight instances. At 10:00, a deployment starts replacing them with version B.

The first new process starts in four seconds. Its liveness check passes. The deployment controller sees a running container and continues.

Then p99 latency jumps.

The new process is alive, but it is not yet carrying useful capacity. Its database pool is still opening connections. Its local cache is empty. The first requests trigger expensive reads that the old instances had already cached. Meanwhile, old instances are draining long requests and briefly remain part of the system.

The tempting model is:

one old instance stops
-> one new instance starts
-> capacity stays constant

That model works only when a started instance is immediately ready and every request is interchangeable. Real deployments add intermediate states:

starting -> initializing -> ready but cold -> warm -> draining -> stopped

Zero downtime is therefore not a property of the deployment command. It is a property of the capacity timeline observed by users.

The Production Symptom

Use this synthetic rollout as a worked example. The numbers are illustrative, not measurements from a real system.

Before the deployment:

• eight warm instances serve 800 requests per second;
• each warm instance can safely serve about 125 requests per second under this workload;
• database connections and local cache entries are already established;
• p99 latency is 180 ms.

The rollout allows two old instances to stop while two new instances start.

At 10:01, the dashboard shows eight running processes. That looks safe. But the useful capacity is different:

Nominal capacity is still above 800 requests per second, but the margin has fallen from 200 to about 20. A small traffic burst or a slower dependency now creates a queue. Cold misses also add database work, so the deployment reduces service capacity while increasing downstream pressure.

This is the first correction:

running replicas != ready replicas != warm capacity

What the System Knows

Different signals answer different questions.

Liveness asks whether the process should be restarted. A liveness success does not prove the instance can serve the real workload.

Readiness asks whether the instance should receive regular traffic. A useful readiness check waits for critical initialization: configuration loaded, required connections available, and the request path able to complete.

Warmth asks whether the instance can serve its expected share without creating a correlated miss storm. Many platforms do not have a built-in boolean called warm. The team must infer it from evidence such as cache-fill progress, representative request latency, connection-pool state, or a bounded warmup procedure.

Drain state asks whether an old instance has stopped receiving new work and finished or safely transferred in-flight work. Removing it from discovery is not the same as proving every active connection has ended.

Queued-work compatibility asks whether jobs created by one version can be consumed by another. A request deployment may finish while old background jobs remain in a queue.

Each signal sees only one part of the transition. A robust rollout combines them instead of treating one green health check as proof of full capacity.

A Deployment Capacity Timeline

Now trace one batch from start to finish.

10:00:00  eight old instances are warm
10:00:04  two new processes are alive
10:00:12  connection pools are established
10:00:20  readiness passes; bounded traffic begins
10:00:45  cache-miss rate and p99 settle near the old baseline
10:00:50  two old instances stop receiving new requests
10:01:20  their in-flight requests finish; old instances stop

The important intermediate state occurs between 10:00:20 and 10:00:45. Both versions receive traffic, but the new instances are not yet equivalent to warm ones.

The controller must preserve enough surplus capacity for that period. It can do this by adding temporary replicas, replacing fewer instances at once, or shifting traffic more slowly. The exact control depends on the platform; the invariant is stable:

Do not remove known-good capacity faster than replacement capacity proves itself useful.

Check: Why is 8/8 processes running weak evidence during this rollout?

Think first, then reveal.

Answer: It collapses several states into one count. Two processes may be alive but cold, while old processes may be draining. The useful question is how much ready, warm capacity can serve the current workload without overloading dependencies.

Where the Rollout Creates Pressure

Three mechanisms commonly turn a healthy service into a slow one during change.

Correlated cold misses

Several new instances start with empty local caches at the same time. Requests that were cheap on old instances now reach Redis, the database, or another origin together.

The cache is not the only component under pressure. Connection pools grow, authentication keys load, code paths compile or initialize, and remote clients establish new connections. A deployment synchronizes work that was previously spread over time.

Premature capacity removal

If the rollout stops old instances before new ones prove readiness, queueing begins. Average latency may still look acceptable while p99 grows because only requests arriving during short capacity gaps wait.

Incomplete draining

An old instance may leave normal load balancing while still owning long-lived connections or in-flight work. Stopping it too soon turns a clean replacement into resets, duplicate retries, or lost progress. Draining buys completion time, but it temporarily keeps resources and mixed versions alive.

These mechanisms explain why deployment-time symptoms can look like ordinary overload. The distinguishing evidence is timing: the capacity loss, cold-miss rise, connection growth, or reset rate begins with a rollout step.

Mitigation and Prevention

Start with the smallest control that protects useful capacity.

1. Separate startup, readiness, and liveness. Give initialization time to finish before regular traffic arrives. Do not make liveness depend on a temporarily slow downstream service unless restarting is truly the right response.
2. Replace in bounded batches. Keep enough old capacity until new capacity is ready. Temporary surge capacity is useful when the normal fleet has little headroom.
3. Warm deliberately. Use representative, bounded work. An uncontrolled warmup that scans every key can create the incident it was meant to prevent.
4. Drain before stopping. Remove old instances from new traffic, then wait for in-flight requests or connections according to an explicit timeout and recovery policy.
5. Watch the whole path. Compare ready capacity, p99 latency, error rate, cache misses, database load, connection pressure, and request resets with rollout events.

Check: Two new instances pass readiness, but database CPU rises sharply as soon as they receive traffic. What should the team do first?

Think first, then reveal.

Answer: Pause further replacement and inspect whether cold cache misses or connection initialization are amplifying database work. Keep old capacity, bound traffic to the new instances, and continue only when downstream pressure and latency settle.

Trade-offs and Limits

Slower batches and surge capacity reduce rollout-time saturation, but they consume more time and temporary resources. Warmup reduces cold-path cost, but it can distort traffic or overload the origin if it is too broad. Long drain windows protect in-flight work, but they extend mixed-version operation.

This lesson does not design a complete progressive-delivery program. Canary governance, feature flags, schema migration policy, and release approvals belong in the dedicated release-safety track. Here the narrower question is operational: does usable capacity remain sufficient while instances change state?

The boundary becomes visible when ready replica count stays healthy but latency, miss rate, remote dependency work, or resets worsen during rollout. That disagreement is a reason to inspect the capacity model, not to trust the green count more loudly.

Practice: Repair the Timeline

A service has six warm instances and no spare capacity. A rollout stops two old instances at a time. New instances pass liveness in five seconds, readiness in twenty seconds, and need another forty seconds before their cache-miss rate settles. During those forty seconds, each can serve only half a warm instance's load.

Propose the smallest safer rollout and name one abort signal.

Model answer: Replace one instance at a time or add temporary surge capacity. Keep the old instance until its replacement passes readiness and the cold-path signals settle; only then drain it. Pause the rollout if p99 latency, origin load, cache-miss rate, or request resets cross the agreed boundary. A process count alone is not an adequate abort signal.

Resources

• [DOC] Kubernetes: Liveness, Readiness, and Startup Probes — Focus: Distinguish restart health from traffic readiness and follow the termination/drain references.
• [DOC] Kubernetes: Deployments — Focus: Inspect rolling-update controls such as surge and unavailable capacity.
• [BOOK] Site Reliability Engineering: Reliable Product Launches — Focus: Connect launch evidence and rollback preparation to user-facing reliability.

Key Takeaways

• Zero downtime depends on useful capacity, not on the count of running processes.
• Startup, readiness, warmth, and draining are different states with different evidence.
• Rolling replacement can synchronize cache misses, connection work, and downstream pressure.
• Preserve known-good capacity until replacement capacity proves that it can carry the workload.