Health Checks and Circuit Breakers

By the end of this lesson, you will be able to...

• Distinguish liveness, readiness, and dependency health for a serving instance.

• Trace how a circuit breaker limits repeated failing calls without claiming that the dependency is repaired.

• Choose a safe traffic or failure response from a partial-outage symptom.

Idea in one sentence: Health checks decide whether an instance should receive new work; circuit breakers decide whether a caller should keep sending work to a failing dependency.

Core Insight

Suppose a learner clicks Pay for a course. The request reaches api-2, one of four instances behind the load balancer. api-2 can still answer a tiny /live endpoint, but it has lost its Redis connection and cannot read the checkout session. At the same time, the payment provider begins timing out for every API instance.

These are two failures at different boundaries. Sending more traffic to api-2 turns its local problem into more broken checkouts. Continuing to call the timing-out payment provider turns one remote failure into exhausted request threads, retry queues, and long user waits across the fleet.

The controls should be different:

instance fitness -> readiness evidence -> load balancer includes or removes api-2
dependency call  -> breaker state     -> caller allows, rejects, or carefully probes payment

The trade-off is deliberate. Removing an instance reduces available serving capacity. Opening a breaker can reject a request that might have succeeded. Both are preferable when their alternative is a larger, slower, less understandable failure. The previous lesson showed that a routing policy can only choose from eligible instances. This lesson defines eligibility and protects the call path after routing.

Three Meanings of “Healthy”

“The service is healthy” is usually too vague to be useful. Separate the questions.

Plain meaning:

Liveness asks whether the kitchen still has a cook. Readiness asks whether that cook can take another order. Dependency health asks whether the supplier is delivering ingredients on time.

In this scenario:

api-2 is live because its process runs. It is not ready for checkout traffic because its required Redis session state is unavailable. The payment provider is a separate dependency problem shared by all otherwise-ready instances.

Technical names:

These are liveness probes, readiness probes, and dependency health signals. They may use some of the same measurements, but they should lead to different control actions.

Readiness Is a Traffic Contract

A load balancer needs an eligible set before it chooses round robin, weighted routing, or any other policy:

all instances:       api-1  api-2  api-3  api-4
readiness result:    ready  no      ready  ready
eligible set:        api-1          api-3  api-4

For this checkout path, the application cannot serve a correct checkout without the session data in Redis. It is reasonable for readiness to become false when the necessary Redis capability is unavailable. The balancer then stops sending new checkout traffic to api-2. Existing requests need their own timeout and error handling; removing an instance from rotation does not erase them.

Do not put every dependency into readiness automatically. If the recommendation service is optional, removing every API instance because recommendations are slow could create a total outage where a degraded course page would have been acceptable. Readiness should represent the dependencies that truly decide whether this instance may accept the relevant class of traffic.

This may require separate routing or endpoints for different paths. An instance might be ready for browsing but not for checkout, depending on how the service boundary is designed. The important part is to state the contract rather than returning a reassuring 200 from a probe that does not match real serving ability.

Check: A course-page instance can render the catalog without the recommendation service. Recommendations are missing, but the catalog and checkout remain correct. Should the entire instance become unready?

Think first, then reveal.

Answer: Usually no. If recommendations are an optional feature, readiness should not evict otherwise useful catalog capacity. Record the dependency failure, degrade the feature explicitly, and reserve readiness failure for conditions that make the instance unsafe for the traffic it is meant to receive.

A Circuit Breaker Changes Caller Behavior

Now follow the payment call. A timeout is the first boundary: it stops one request from waiting forever. A circuit breaker uses the recent outcomes of many calls to decide whether another attempt is likely to waste more resources.

closed:    calls flow normally; outcomes are counted
open:      calls fail fast locally for a cool-down period
half-open: allow a small number of probe calls
closed:    probes succeed enough to resume normal calls
open:      probes fail; continue protecting the caller

The breaker is owned by the caller. It does not repair the payment provider, and it does not tell the load balancer whether an API process should be removed. It protects caller resources such as connection pools, request threads, queue slots, and latency budgets.

A breaker needs a failure policy. For a payment authorization, the safe response may be a clear “payments are temporarily unavailable; try again later.” For an optional exchange-rate widget, the safe response may be a cached value with a visible age. Do not invent a successful payment result from a failed remote call. Degrade only where the product contract permits it.

A Worked Failure Trace

At 10:00, the payment provider begins taking longer than the API's 800 ms call budget. The service has a 10-second rolling window and opens the breaker after 12 failures or 60% failed calls. It allows two half-open probes after a 20-second cool-down.

The path is visible:

input: checkout requests arrive while payment calls time out
  -> transition: timeout records failure; window crosses breaker threshold
  -> intermediate state: breaker opens and blocks ordinary remote calls
  -> output: checkout fails quickly with a clear retry-later response
  -> naive failure: without the breaker, each request waits, retries, and consumes more fleet capacity

The threshold numbers are not universal. They depend on traffic volume, error cost, timeout budget, and acceptable false openings. At low traffic, 12 failures may be too many; at high traffic, it may happen in milliseconds. The operational goal is not to find a magic number. It is to bound failure amplification and make the choice observable.

Signals and a Small Runbook

For readiness, watch the number of eligible instances, probe failures by reason, request error rate, and per-instance latency. A sudden drop from four ready instances to one changes the capacity and blast-radius calculation even if the remaining instance is technically healthy.

For a breaker, watch state transitions, failed or timed-out dependency calls, fast-fail count, half-open probe outcome, and fallback usage. An open breaker with zero user-visible errors can be good evidence if a safe optional fallback is working. An open breaker for mandatory checkout is a product outage even though the API itself may have low CPU and clean process liveness.

A compact runbook asks:

1. Is the failure local to one instance or shared across the fleet?
2. Which traffic contract is broken: liveness, readiness, or one dependency call?
3. What are callers doing now: waiting, retrying, fast-failing, or using a permitted fallback?
4. Which signal would show safe recovery before traffic or calls are restored?

This avoids a common harmful response: restarting every API instance during a shared payment outage. Restarting may remove useful capacity while the breaker is already doing the safer job of reducing remote pressure.

Check: All four API instances are ready. Payment breaker state is open, and checkout attempts fail quickly. Should the load balancer remove all four instances from rotation?

Think first, then reveal.

Answer: No, not solely because the payment breaker is open. The instances may still serve browsing, course progress, and other safe paths. The breaker protects the payment call. Remove instances only when their readiness contract for the routed traffic is false.

Trade-offs and Limits

Deep readiness checks improve routing decisions, but they can flap when a transient dependency blip removes too much capacity. Keep their scope tied to the actual traffic contract, add sensible thresholds, and observe the ready-instance count. A liveness check that depends on every remote service can cause restart storms instead of recovery.

Circuit breakers reduce futile calls, but they introduce thresholds, windows, state storage, and recovery behavior to operate. They can open falsely during a brief spike, and a shared breaker state can create its own coordination question. A breaker also cannot distinguish every cause of failure; pair it with timeouts, bounded retries, traces, and a clear owner for the dependency.

Neither pattern makes a dependency reliable. They limit how far a failure spreads. The boundary is visible when the service has no safe fallback for a mandatory path: it should fail fast and honestly rather than claim that protection equals success.

Common Confusions

Confusion: A liveness endpoint is enough for traffic routing

Why it is tempting:

A process returning 200 is easy to test and looks like a complete health check.

Better model:

Liveness decides whether a process should continue to exist. Readiness decides whether it should receive new requests. A live but unready instance must not be treated as normal serving capacity.

Confusion: An open circuit means the API instances are unhealthy

Why it is tempting:

Users see errors on one path, and the breaker state is visible from the API.

Better model:

The breaker describes a caller-to-dependency relationship. The instance may still serve other paths correctly. Use readiness only when the instance cannot safely serve the traffic assigned to it.

Confusion: A circuit breaker replaces timeouts and retries

Why it is tempting:

The breaker has a memorable state machine and sounds like a complete resilience feature.

Better model:

Timeouts bound one attempt; retry policy decides whether and when to try again; a breaker changes behavior after a pattern of failures. They work together and each needs an explicit limit.

Practice: Choose the Boundary

For each condition, write the first control that should react and one signal to watch:

• api-3 has deadlocked and stops answering its local event loop.
• Redis is unavailable only to api-2, and checkout needs Redis session state.
• The payment provider returns timeouts for every API instance, while browsing remains healthy.

Model answer: The deadlocked instance needs liveness detection and replacement; watch restart reason and ready-instance count. api-2 needs readiness false so new checkout traffic avoids it; watch probe failures and checkout errors by instance. The shared payment failure needs timeout and breaker protection at the caller; watch breaker state, fast-fail count, and provider latency. None of these signals alone proves recovery: restore normal behavior only after the relevant probes or controlled calls succeed.

Resources

• [DOC] Kubernetes liveness, readiness, and startup probes — Focus: Map probe types to lifecycle and traffic decisions.
• [ARTICLE] Circuit Breaker — Focus: See the caller-side state machine and why it limits failure amplification.
• [DOC] Resilience4j CircuitBreaker — Focus: Inspect failure windows, open state, and controlled recovery probes.
• [BOOK] Release It! — Focus: Connect breakers, timeouts, and bulkheads to production failure containment.

Key Takeaways

• Liveness, readiness, and dependency health are different observations because they lead to different actions.
• Readiness controls whether a load balancer should send new traffic to an instance; it should reflect the actual serving contract, not every optional dependency.
• A circuit breaker protects a caller from repeated costly dependency attempts by failing fast and testing recovery carefully.
• Protection is not success: pair these controls with explicit fallback or fail-fast behavior, timeouts, bounded retries, and recovery evidence.