Health Checks, Load Balancing, and Traffic Steering

By the end of this lesson, you will be able to...

• Explain the difference between liveness, readiness, and request-specific health.

• Map health signals to routing actions such as serve, drain, shed, or fail closed.

• Review a load-balancing policy for overconfidence, overreaction, and partition-sensitive behavior.

Idea in one sentence: Health is a traffic promise, not a heartbeat; routing is the act of choosing where a request is safe to go with incomplete evidence.

Core Insight

The previous lesson showed that a replica can be alive but not authoritative during a partition. This lesson moves from failure theory to traffic control.

Imagine the learning platform has three progress-service replicas behind a load balancer:

P1: process is running, but its database connection pool is exhausted
P2: process is healthy, but it is draining during a deploy
P3: can serve cached progress reads, but cannot reach authoritative storage for writes

From far away, all three may look "up." They answer a simple ping. They have open ports. They may even return 200 OK from a shallow endpoint.

From the user's point of view, they are not equal.

The naive idea is:

If a replica is up, send it traffic.

That is too coarse. A replica can be alive and still be unsafe for some traffic. It may be safe for reads but not writes. It may be safe for existing connections but not new work. It may be safe from one zone but unreachable from another.

Health checks and load balancing are not just traffic distribution features. They are how a system turns partial evidence into routing decisions while failure is still ambiguous.

The Small Incident

During a deploy, learners start reporting that the lesson page loads, but completing a lesson sometimes hangs.

The router sees this endpoint pool:

progress-service endpoints
  P1 zone A version v2
  P2 zone B version v2 draining=true
  P3 zone C version v1 read_only=true

The user request is:

POST /complete-lesson
learner_id=7
lesson_id=3016
idempotency_key=req-3016-a

If the router sends this write to P1, it may queue behind an exhausted database pool. If it sends the write to P2, it may interrupt deploy draining. If it sends the write to P3, the replica may accept traffic but fail later because it cannot reach authoritative storage.

Plain meaning:

Readiness means "this target is a responsible destination for this traffic."

In this scenario:

P3 might be ready for GET /progress-summary but not ready for POST /complete-lesson.

Technical name:

This is request-class-aware readiness. The health signal depends on what kind of request the router is about to send.

Liveness, Readiness, And The Traffic Promise

A service can be running and still be unsafe for user traffic.

Separate these ideas:

liveness: should the platform restart this process?
readiness: should the router send traffic here?
startup: has the process finished becoming ready for checks?
request-specific readiness: is this replica safe for this class of request?

For the progress service:

A shallow health endpoint might say:

GET /health -> 200 OK

A more useful traffic signal says:

process_alive=true
draining=false
db_pool_available=false
authoritative_storage_reachable=false
cached_reads_allowed=true

The router cannot invent meaning from a shallow ping. It can only act on signals the service exposes.

Check: P3 can serve cached reads but cannot reach authoritative storage. Should the load balancer send POST /complete-lesson to P3?

Think first, then reveal.

Answer: No. P3 may be ready for a read class, but not for an authoritative write class. One generic "up" signal is too broad.

A Worked Routing Trace

Follow one request through traffic steering.

Input:
  POST /complete-lesson
  requires authoritative progress write
  user deadline remaining = 500 ms

Transition:
  router reads endpoint health and metadata
  P1: alive, db_pool_available=false
  P2: alive, draining=true
  P3: alive, read_only=true

Intermediate state:
  no endpoint is currently a safe write target

Output or decision:
  reject quickly, shed write traffic, or return retryable unavailable
  do not route blindly to an "up" replica

Naive failure contrast:
  round-robin over alive endpoints sends writes to unsafe targets
  request-aware routing fails closed for writes and may keep reads degraded

Here is the decision as a table:

The best action may be disappointing: do not accept the write right now. But a fast, honest unavailable response is often better than sending the write to a target that will hang, duplicate, or create ambiguous state.

So far:

• liveness protects the process lifecycle
• readiness protects traffic routing
• request-class readiness protects semantics
• routing should use health as evidence, not as magic truth

Load Balancing Is A Policy Choice

Load balancing sounds like spreading requests evenly. In a real incident, it is more like choosing a destination under uncertainty.

Different policies optimize for different pressures:

round-robin: simple distribution
least-connections: avoid already-busy endpoints
weighted routing: shift traffic during rollout
locality-aware routing: prefer nearby healthy capacity
request-class routing: separate reads, writes, and expensive operations
outlier detection: reduce traffic to endpoints with bad behavior

The learning platform should not necessarily route every request the same way.

This connects directly to the partition lesson. A replica can be healthy from one path and unsafe from another. Health checks should be read as observations from particular paths, not as universal truth.

The trade-off is simplicity versus precision. One generic policy is easy to operate, but it treats unlike requests as if they had the same risk. More precise routing can improve behavior under failure, but it adds configuration, tests, and debugging work.

Check: A router retries a timed-out completion write on a different replica because that replica has fewer connections. What extra fact must the router or application know?

Think first, then reveal.

Answer: It must know whether the write is safe to repeat, usually through idempotency or deduplication, and whether the new replica is write-ready for the authoritative path.

Draining Protects Change

Healthy replicas still need to leave rotation during deploys, scale-downs, and maintenance.

Connection draining is controlled disappearance:

mark draining
  -> stop new requests
  -> let in-flight requests finish until deadline
  -> close or migrate long-lived connections
  -> terminate safely

Without draining, a rolling deploy can create avoidable failures. A replica disappears while users are mid-request. A retry lands on a new version with slightly different behavior. A long request loses its connection.

Draining does not mean infinite patience. Streaming responses, long-running jobs, and slow writes may outlive the deploy budget. The system needs deadlines and a policy for what happens when work cannot finish in time.

A good draining policy says:

new requests: no
in-flight idempotent reads: finish if within deadline
in-flight writes: finish or return clear uncertain/retryable status
long-lived streams: close with reconnect signal before shutdown deadline

The trade-off is deploy speed versus user disruption. Fast removal is simple but abrupt. Careful draining is gentler but requires more state and more patience.

Health Checks Can Also Cause Trouble

Health checks are not free. They create traffic, make decisions from imperfect signals, and can synchronize bad behavior.

Common failure patterns:

• every replica probes the same dependency at the same time
• one flaky dependency causes all replicas to mark themselves unready
• strict readiness removes too much capacity and makes overload worse
• shallow readiness leaves broken replicas in rotation
• health from one network path is treated as global truth

Use thresholds, jitter, and separate signals.

bad:
  one failed DB probe -> unready immediately

better:
  several failed probes over time -> degrade write readiness
  cached reads may continue
  overload signal can shed low-priority traffic

The goal is not to make health checks optimistic. The goal is to make them honest enough for routing and stable enough that they do not become a new incident source.

Common Confusions

Confusion: "Liveness means ready"

Why it is tempting:

Both signals often look like health checks, and both may return an HTTP status.

Better model:

Liveness asks whether the platform should restart the process. Readiness asks whether the router should send traffic. A process can be alive and not ready.

Confusion: "The load balancer understands the product"

Why it is tempting:

Routers make important decisions, so it feels like they know whether a request is safe.

Better model:

Routers know endpoints, metadata, health, latency, and policy. Applications must expose operation meaning through methods, route classes, idempotency keys, or explicit readiness signals.

Confusion: "Removing unhealthy replicas always helps"

Why it is tempting:

Taking bad targets out of rotation sounds obviously good.

Better model:

Removing replicas can protect users, but removing too many at once can reduce capacity and deepen overload. Health policy must balance speed of reaction with capacity preservation.

Practice

Review this endpoint pool:

P1: alive=true, db_pool_available=false, draining=false, read_cache_fresh=true
P2: alive=true, db_pool_available=true, draining=true, read_cache_fresh=true
P3: alive=true, db_pool_available=false, draining=false, read_cache_fresh=false

Requests arriving now:

A: GET /progress-summary
B: POST /complete-lesson
C: GET /lesson-metadata

Write a routing review:

1. Which endpoints, if any, can receive A?
2. Which endpoints, if any, can receive B?
3. What should happen to P2?
4. What signal is missing if the router must choose for C?

Model answer:

A can go to a replica with fresh enough read data, so P1 may be acceptable if the product allows bounded staleness; P2 should not receive new requests while draining unless the policy explicitly allows a graceful read path. B should not go to P1 or P3 because neither has database capacity for authoritative writes, and P2 is draining, so the safer response may be fast unavailable or shed/retry later. P2 should stop new work and finish in-flight requests within a deadline. For C, the router needs metadata-specific readiness or a clear statement that lesson metadata can be served from a different service or cache.

Trade-offs and Limits

Health checks improve routing only when they match the promise being routed. A shallow ping is useful for process survival. It is not enough for write safety, partition behavior, draining, or overloaded dependencies.

Precise health improves correctness but adds operational surface area. Teams must design probes, thresholds, jitter, route classes, dashboards, and incident language. Too little signal makes routing overconfident. Too much fragile signal can make routing overreact.

Load balancing can reduce user-visible failure, but it cannot remove application semantics. It does not know whether a duplicate write is safe unless the application exposes that fact. It does not know whether stale data is acceptable unless the product and service contract say so.

You can see the boundary when a replica says "I am up" and the routing question is "up for what?"

Resources

• [TUTORIAL] Kubernetes Liveness, Readiness, and Startup Probes - Compare process survival, traffic readiness, and startup grace as separate signals.
• [ARTICLE] Envoy Load Balancing Overview - Study load-balancing policy as a configurable decision under imperfect endpoint knowledge.
• [BOOK] Site Reliability Engineering - Revisit overload, cascading failure, and service interaction as operational design problems.

Key Takeaways

• Liveness asks whether a process should be restarted; readiness asks whether it should receive traffic.
• A replica can be ready for reads, unready for writes, and draining for new work at the same time.
• Load balancing is a routing policy built from incomplete health, latency, locality, request-class, and rollout signals.
• Connection draining makes deploys and scale-downs less abrupt, but it still needs deadlines and fallback behavior.
• Traffic steering improves resilience only when services expose health signals that match real request semantics.