Service Discovery, Naming, and Routing Control

By the end of this lesson, you will be able to...

• Explain why a service name is not the same thing as a network endpoint.

• Trace how discovery, routing, health, and identity combine before a request is sent.

• Identify stale discovery as a normal failure mode, not a rare control-plane bug.

Idea in one sentence: Discovery gives a client candidate destinations, routing chooses one for this request, and identity proves the chosen endpoint is allowed to answer for the stable service name.

Core Insight

The previous lesson looked at health checks and traffic steering. A router cannot choose a healthy target until it has a list of possible targets. That list usually comes from service discovery.

Imagine the learning platform has split into services:

frontend
  -> progress-service
  -> catalog-service
  -> certificate-service
  -> recommendation-service

The frontend and gateway should not need to know that progress-service currently runs on 10.0.1.7:8080, 10.0.2.4:8080, and 10.0.3.9:8080. Those addresses are temporary. Replicas restart, deployments roll, zones partition, nodes drain, certificates rotate, and endpoint lists change.

The naive model says:

service discovery = lookup table
progress-service -> 10.0.1.7:8080

That model is useful for the first minute of learning, then it starts lying.

In a live distributed system, service discovery is not merely a phone book. It is part of the control plane that connects stable intent to temporary destinations. A client asks for progress-service because it wants "the service allowed to handle progress reads and writes." The discovery system returns candidates that might be able to serve that intent right now. Routing policy then chooses a destination for this particular request. Identity checks make sure the endpoint is actually allowed to answer as that service.

The important distinction is:

service name: stable intent
endpoint: temporary destination
discovery: candidate list
routing: request-specific choice
identity: proof that the endpoint is allowed to answer
health: evidence about whether the endpoint should receive traffic

Once you separate those meanings, several confusing network failures become easier to explain. A request can fail because the name resolved to stale endpoints. It can fail because routing chose a candidate that was healthy for reads but unsafe for writes. It can fail because an endpoint answered from the wrong rollout group. It can fail because the endpoint existed, but could not prove the right workload identity.

The useful design shift is to treat naming as a control surface, not a string replacement trick. Names, endpoint freshness, routing policy, health state, and identity all shape where traffic goes.

Stable Names, Temporary Endpoints

A service name should express what the caller means, not where the process happens to run today.

For the learning platform:

progress-service = the API that owns lesson completion state
catalog-service = the API that exposes lesson and track metadata
certificate-service = the API that issues durable completion evidence

Those names should survive normal operational change. The progress service may run three replicas this morning, six replicas during peak traffic, and two replicas during a quiet deploy. The name stays stable while the destination set changes.

An endpoint is different:

progress-service endpoints
  P1 10.0.1.7:8080  zone=A  version=v2  write_ready=true
  P2 10.0.2.4:8080  zone=B  version=v2  write_ready=false read_ready=true
  P3 10.0.3.9:8080  zone=C  version=v1  draining=true

Each endpoint is a current option, not a promise forever. P3 may be removed because it is draining. P2 may stay in the pool for reads but not writes. P1 may be the only write-ready target for a short period. A minute later the table may look different.

That difference matters because application code often wants a stable name, while infrastructure has to deal with unstable destinations. If application code hard-codes endpoints, every deployment becomes a coordination problem. If infrastructure hides too much, callers may assume that a successful lookup means "safe to use for anything," which is also false.

Plain meaning:

The name says what you want. The endpoint says where you might send this request. They are related, but they are not the same fact.

Technical name:

This is indirection. A stable service identity is mapped to a changing set of concrete network destinations.

Discovery Is Not Routing

Discovery answers:

What endpoints could currently receive traffic for this service name?

Routing answers:

Which endpoint should receive this request now?

Those are different decisions.

Suppose a gateway receives:

POST /complete-lesson
target service: progress-service
operation: authoritative write
deadline remaining: 700 ms
idempotency key: req-3017-a

Discovery might return three candidates:

P1: zone A, version v2, write_ready=true, identity=progress-service
P2: zone B, version v2, read_only=true, identity=progress-service
P3: zone C, version v1, draining=true, identity=progress-service

Routing should not randomly pick one of the three just because discovery returned them. The request is a write, so P2 is not safe even if it can serve reads. P3 is draining, so it should not receive new work. P1 is the only candidate that matches the request class.

For a different request, the right decision may change:

GET /progress-summary

That read might be acceptable on P2 if the product allows bounded staleness. The same endpoint can be a good destination for one request class and a bad destination for another.

This is the bridge to the previous lesson. Health and load balancing act on the candidates that discovery provides. If discovery gives stale or incomplete candidates, routing starts from bad evidence. If routing ignores request meaning, a good candidate list still produces bad traffic decisions.

The trade-off is where to place the intelligence. A smart gateway or service mesh can centralize routing policy. A simple client can be easier to reason about. But no design escapes the semantic question: "safe for which request?"

Check: Discovery returns endpoints P1, P2, and P3 for progress-service. P2 is read-only. Should the gateway send POST /complete-lesson to P2 because it appeared in discovery?

Think first, then reveal.

Answer: No. Discovery only says P2 is a candidate for the service name. Routing still has to check whether P2 is safe for this request class.

A Worked Discovery And Routing Trace

Follow one request through the mechanism.

Input:
  POST /complete-lesson
  service name = progress-service
  request class = authoritative write
  idempotency key = req-3017-b

Transition:
  gateway resolves progress-service
  discovery returns endpoint metadata
  gateway evaluates health, version, zone, request class, and identity

Intermediate state:
  P1 = write-ready, version v2, identity valid
  P2 = read-only, version v2, identity valid
  P3 = draining, version v1, identity valid but stale in one client cache

Output or decision:
  route the write to P1
  do not route the write to P2
  do not start new work on P3

Naive failure contrast:
  cached address sends the write to P3 because P3 was valid earlier
  round-robin sends the write to P2 because P2 is alive
  lookup-table thinking treats all discovered endpoints as equivalent

Here is the same mechanism as a table:

The output is not "we found an IP." The output is a controlled decision:

send POST /complete-lesson to P1
because P1 is write-ready, current, and identity-valid

If no endpoint satisfies those conditions, the right answer may be a fast 503 Service Unavailable, a retry after backoff, or a degraded response depending on the operation. For a progress write, failing closed is usually safer than sending the request to a destination that cannot responsibly commit it.

This is why service discovery belongs in failure-model thinking. Discovery data is evidence. Routing policy interprets that evidence. Identity constrains trust. None of them is perfect truth.

Stale Discovery Is A Normal Failure Mode

Discovery systems are distributed systems too. They have caches, watches, registries, DNS TTLs, control-plane delays, resolver behavior, dropped updates, and clients that continue using old data.

Stale discovery is therefore not strange. It is one of the everyday ways a networked system fails.

Imagine this deploy:

t0: P3 is serving progress-service traffic
t1: P3 starts draining for a deploy
t2: orchestrator marks P3 not ready
t3: registry removes P3 from the ready endpoint list
t4: one gateway has a cached endpoint list that still includes P3
t5: gateway sends POST /complete-lesson to P3

The discovery system may be behaving reasonably and still produce this situation. The cache existed to reduce control-plane load and survive brief registry problems. The stale entry is the cost of that resilience.

Now P3 needs defensive behavior. It cannot assume "if traffic arrived, I must be a good target." A draining endpoint can reject new writes, finish in-flight work, or redirect only if that is safe. The gateway can refresh discovery after a failed attempt. The application can require idempotency so a retry does not create duplicate completion state.

The key lesson is that discovery should reduce bad routing, not become the only safety barrier.

There is a real trade-off:

No setting eliminates failure. The goal is to know what kind of failure each setting makes more likely.

Check: A team lowers DNS TTL to one second and claims stale discovery is now solved. What is missing from that claim?

Think first, then reveal.

Answer: TTL helps, but it does not remove client caches, resolver behavior, delayed health updates, watch lag, partitions, or endpoints that must still defend themselves when stale traffic arrives.

Identity Prevents Blind Trust

Finding an endpoint is not the same as trusting it.

Suppose discovery returns:

progress-service -> 10.0.2.4:8080

The client still needs to know whether the workload at that address is allowed to answer as progress-service. In modern service fleets, that proof may come from mTLS certificates, SPIFFE identities, service accounts, signed tokens, or platform workload identity.

Without identity, discovery can accidentally become trust:

I found an address, therefore I trust whatever answers there.

That is dangerous. A stale registry entry, misconfigured DNS record, reused IP, compromised side path, or accidental port exposure can send traffic to the wrong workload. Strong identity gives the client or proxy a way to reject the connection even when the address was reachable.

The better model is:

name requested:
  progress-service

endpoint reached:
  10.0.2.4:8080

identity verified:
  workload is allowed to represent progress-service

route allowed:
  only if health and request policy also match

Identity does not replace discovery. It also does not replace health checks. It answers a different question: "is this endpoint who it needs to be?"

The trade-off is operational complexity. Certificates expire. Identities need policy. Rotation can break traffic. Debugging a failed handshake is harder than debugging a plain TCP connection. But without identity, a discovery mistake can turn into an authorization mistake.

Design Pressure Points

A good service discovery design makes several choices explicit.

First, decide what a service name means. If progress-service means "any process with a progress HTTP port," the name is too loose. If it means "the workload identity and API surface allowed to handle progress state," the name carries useful intent.

Second, decide how clients learn endpoint changes. DNS, a registry API, Kubernetes Services, sidecar proxies, and service meshes all draw the boundary differently. The right answer depends on operational maturity, traffic volume, latency needs, and how much policy you want outside application code.

Third, decide what metadata routing needs. A bare IP list may be enough for simple read traffic. A platform that routes writes, canaries, regional traffic, and degraded reads probably needs metadata such as version, zone, readiness class, weight, and identity.

Fourth, decide what happens when discovery is unavailable. Some clients can continue with cached endpoints for a short time. Some operations should fail closed. A certificate issuance path should not blindly use old discovery data if it cannot verify authority.

Fifth, decide what the endpoint does when it receives traffic it should not have received. The defensive answer should live in the service too, not only in the router.

Observability Needs The Decision Trail

The next lesson focuses on observability because discovery and routing bugs are painful when the decision trail disappears.

For this lesson's worked trace, useful telemetry would include:

requested service name = progress-service
discovery source = gateway cache
endpoint candidates = P1, P2, P3
selected endpoint = P1
route reason = write_ready=true and identity_valid=true
rejected candidates = P2 read_only, P3 draining
endpoint version = v2
identity verification = success
cache age = 4 seconds

Without those facts, engineers may only see "one progress request was slow" or "P3 received traffic during deploy." They will not know whether the problem was stale discovery, bad routing policy, an identity mismatch, a health signal delay, or a client ignoring the gateway.

This connects the track arc:

• lesson 4 explained that partitions can make authority ambiguous
• lesson 5 explained that health and load balancing are traffic decisions under uncertainty
• this lesson explains how names become candidate endpoints and trusted routes
• lesson 7 will explain how to record the path so incidents can be reconstructed

Practice

Take a service in a system you know and write five lines:

stable service name:
request classes:
endpoint metadata needed:
identity proof:
what stale discovery would break:

For example:

stable service name: progress-service
request classes: progress reads, completion writes, admin repair jobs
endpoint metadata needed: zone, version, read_ready, write_ready, draining
identity proof: workload certificate for progress-service
what stale discovery would break: writes may go to a draining or old-version replica

Then ask one review question:

If discovery returns this endpoint, what still has to be true before routing a write there?

If the answer is only "it was in discovery," the design is under-specified.

Resources

• [TUTORIAL] Kubernetes Services - Study how a stable Service fronts a changing set of Pods and endpoints.
• [ARTICLE] Consul Service Discovery - Compare registry-based discovery, health state, and service lookup.
• [ARTICLE] SPIFFE Overview - Use workload identity as the trust layer that keeps naming separate from blind address trust.

Key Takeaways

• A service name expresses stable intent; an endpoint is a temporary destination.
• Discovery creates a candidate list, while routing chooses a destination for the current request.
• Health, request class, rollout metadata, and identity all affect whether a discovered endpoint is safe.
• Stale discovery is a normal failure mode caused by caches, delayed updates, watches, TTLs, and partitions.
• Strong workload identity prevents a successful lookup from becoming blind trust in the wrong endpoint.