Linearizable Reads, Leader Leases, and Fencing

The core idea: Linearizable reads make authority decisions reflect the current committed order, with a trade-off between low-latency reads and the evidence needed to prove that a leader, lease holder, or controller is still current.

Core Insight

Imagine a controller reads "I own the scheduler lease" from a consensus store and starts placing workloads. Then it pauses during a long garbage collection or loses contact with the cluster during a network partition. Another controller obtains a newer lease and starts placing workloads too. If the old controller wakes up and can still act on stale authority, the consensus store may have chosen correctly while the external system still suffers split-brain behavior.

Linearizable reads are about making authority visible at the moment a decision is made. Leader leases are a performance tool for serving some reads without a quorum round trip each time. Fencing is the defensive layer that makes stale authority harmless when an old actor wakes up and tries to act anyway.

The common misconception is that a leader can safely serve every read from local memory because it was leader recently. In reality, leadership can become stale. A read that grants permission to mutate the world needs evidence that it reflects the current committed order, not merely a plausible local cache.

Linearizable Reads Need Fresh Authority

A linearizable read behaves as if it occurred at one instant between request and response. For a coordination service, this matters when the read answers questions like:

• who owns this lease?
• what is the current configuration?
• has this lock been released?
• what revision should my controller act on?

Serving those reads from a stale leader can break correctness. A former leader may still have a plausible local log, but after a partition it might not know that a newer term exists or that a newer owner has been elected.

Consensus systems use different techniques to make authority reads safe:

• route the read through the normal log path,
• confirm current leadership with a quorum before serving,
• use a lease only under explicit timing assumptions,
• attach revisions so clients can reason about freshness.

The fastest method is not always the right method. A dashboard read can tolerate staleness more often than a lease read that grants permission to mutate the world.

Read Paths and Their Cost

There are several common read paths, and each one buys latency with a different kind of evidence.

The most conservative path is to put the read through the replicated log, sometimes as a no-op or read barrier. Once that entry is committed, the leader knows all prior writes that could affect the read are ordered before it. This is simple to reason about, but it adds log traffic and commit latency.

A cheaper path is to confirm leadership with a quorum before serving the read. Raft-style read index mechanisms use this idea: the leader establishes that it is still the current leader and that its commit index is safe enough for the read. This avoids appending every read to the log, but it still needs communication with a majority when the leader does not already have fresh evidence.

The fastest path is a lease read. The leader serves locally while it believes a lease is valid. That can be correct only if the system's timing assumptions are strong enough to prevent another valid leader from existing during the same interval.

Some systems also expose explicitly stale or serializable reads. Those are useful for metrics, browsing, or low-risk caches, but they should not be used to grant locks, elect owners, approve writes, or make irreversible decisions.

Leader Leases Are Timing Contracts

A leader lease says, roughly, "for this bounded interval, no other leader should be valid." That can make reads cheaper because the leader may serve them without a full round trip to a quorum each time.

But a lease is a timing contract, not a proof of permanent leadership. Its safety relies on assumptions:

• clocks do not drift beyond the tolerated bound,
• pauses and scheduling delays stay within the model,
• lease expiry is interpreted conservatively,
• election and renewal rules prevent overlapping authority,
• clients and downstream systems respect the lease token.

If those assumptions are wrong, the system may create overlapping authority.

old leader pauses
new leader obtains lease
old leader resumes and acts

This is why lease design must include conservative expiry, monotonic revisions, and operational limits around clock behavior and pauses. A lease-based read can be a good optimization, but it should be treated as a claim with preconditions, not as a shortcut around consensus safety.

Fencing Makes Stale Actors Lose

Fencing is the defensive move that protects the external world from stale lease holders. Each grant carries a monotonically increasing token, epoch, term, or revision. Downstream systems reject actions with older tokens.

lease token 41 -> old controller
lease token 42 -> new controller

storage accepts writes with token 42
storage rejects token 41

Without fencing, a consensus store can correctly elect the new owner while an old owner still corrupts an external resource. With fencing, the stale owner may wake up, but its authority is no longer accepted.

This pattern is essential for locks, schedulers, primary ownership, storage writers, and any system where a coordinator's decision authorizes side effects outside the consensus store.

Worked Example: A Stale Scheduler Lease

Consider a scheduler controller that writes assignments to a separate workload database.

1. Controller A reads the coordination store and receives lease token 41.
2. A pauses before writing the assignment.
3. The lease expires, and controller B receives token 42.
4. B writes new assignments with token 42.
5. A resumes and tries to write with token 41.

If the workload database does not check tokens, both controllers can write. The consensus store has not failed; the authority did not reach the resource being protected.

With fencing, the workload database records the highest token it has accepted and rejects lower tokens. A can still send a delayed request, but the protected resource refuses to honor stale authority. The correctness boundary moves from "the old actor never wakes up" to "the old actor cannot successfully act."

Operational Failure Modes

The main failures are predictable:

• stale leaders serving authority reads,
• leases built on clock assumptions nobody measured,
• external systems accepting old tokens,
• clients treating a cached read as a current grant,
• APIs that expose a lock but no revision, token, or epoch,
• dashboards and controllers sharing the same stale-read path.

The fix is to tie reads, grants, and actions to evidence: quorum confirmation, committed revisions, lease tokens, and downstream fencing checks. A design review should ask where authority is created, how freshness is proven, and which downstream system refuses stale requests.

Connections

The previous lesson explained how quorum intersection preserves commit evidence across leadership changes. This lesson asks how a read proves it is observing that current evidence before granting authority.

The next lesson turns these ideas into coordination APIs: locks, leases, watches, and compare-and-swap. Those APIs are safe only when they expose revisions, tokens, and recovery behavior clearly enough that clients cannot mistake stale observations for current authority.

Resources

• [PAPER] The Chubby Lock Service for Loosely-Coupled Distributed Systems
  • Focus: Study locks, leases, sessions, and client-visible coordination semantics.
• [DOC] etcd API Guarantees
  • Focus: Use the docs to connect revisions, watches, leases, and consistency guarantees.
• [PAPER] In Search of an Understandable Consensus Algorithm
  • Focus: Read the sections on leader completeness, terms, and linearizable reads.
• [PAPER] Spanner: Google's Globally-Distributed Database
  • Focus: Read for timing assumptions, external consistency, and the cost of stronger time guarantees.

Key Takeaways

• Linearizable reads need evidence that the answer reflects current committed authority, especially when the read grants permission to act.
• Leader leases reduce read latency only under explicit timing assumptions about clocks, pauses, renewals, and elections.
• Fencing tokens protect external systems by making stale actors lose even when they resume and send delayed requests.