Clocks, Leases, and Safe Reads

By the end of this lesson, you will be able to...

• explain why a newly elected leader is not automatically ready for a lease-based fast read;

• trace the conservative interval checks that avoid overlapping read authority;

• choose a safe fallback when time uncertainty or quorum communication erodes the lease margin.

Idea in one sentence: A lease read is fast only because a system has a time-bounded proof that no other replica may still serve the same read as current.

Core Insight

At 09:17, Harbor Point moves shard 184 from New York to Madrid. The consensus group has chosen md-db-2 as the new leader. A trader then asks for the remaining capacity of an issuer.

The tempting answer is immediate: “Madrid is leader now, so read from Madrid's memory.” That works when no previous leader can still act. But the old leader, ny-db-3, may be alive, isolated, and looking at a clock that differs from Madrid's. If its old read lease appears valid to it, two replicas may both answer a question that promised one current value.

The important distinction is small:

elected leader
    -> may be allowed to order writes for a new term

safe lease reader
    -> also has a current proof that earlier read authority cannot overlap

Consensus establishes the ordered authority history. A lease is an additional fast-path argument, not a substitute for consensus. When the argument is unavailable, the correct response is usually a slower linearizable read, not a hopeful local read.

The Small Situation: What Does “Now” Mean?

Humans say “the lease expires at 09:17:01.500” as if every machine sees the same instant. A distributed program cannot assume that. It receives time from a local clock, and the clock may be early or late relative to real time.

For this lesson, use a simplified bounded-time model. A replica reads an interval:

now = [earliest, latest]

The interval means that real time is somewhere between those two values. It is a teaching model for systems that can bound their clock error; specific databases may use different clock, lease, or read protocols.

Suppose ny-db-3 holds a read lease until T = 09:17:01.500. At one moment it sees:

ny-db-3 now = [09:17:01.480, 09:17:01.492]

The old leader may serve via this lease only if its pessimistic upper bound is still before T:

latest < T

Why use latest? Real time might already be 09:17:01.492. If even that value is before the expiration, the old leader has not stepped beyond the interval it was granted. The midpoint is not enough evidence; it hides the possibility that the local clock is behind.

The successor must be conservative in the opposite direction. It may enable a replacement lease only after:

earliest > T

If Madrid's interval is [09:17:01.494, 09:17:01.506], earliest has not passed the old expiry. Madrid waits or uses a quorum-confirmed read. When its interval becomes [09:17:01.512, 09:17:01.524], the time condition is satisfied.

This creates a deliberate gap. Near T, neither replica gets the local lease fast path:

old holder: latest < T         safe to use old lease
              [uncertain gap] use another safe read path
new holder: earliest > T       eligible for new lease

The gap is not a bug. It is the cost of replacing “their clocks probably agree” with a proof that does not allow two lease holders to overlap.

The Initial Model and Where It Breaks

An initial model says: “A leader may serve a local read until its wall clock passes the expiry, and a new leader may serve as soon as it is elected.” It is attractive because it is fast and uses familiar timestamps.

It breaks when a partition separates the former leader from the quorum. The former leader may not hear about the new term. If it compares only its own raw wall clock, a slow clock can make an expired lease look valid. Meanwhile, a fast clock at the new leader can make the old lease look expired. The two local decisions overlap in real time.

The stronger model has two parts:

1. The lease-grant protocol must ensure that a current quorum will not grant conflicting authority in the same interval. That protocol is product-specific and normally ties leases to terms, membership, and quorum acknowledgements.
2. A holder and successor must consume the time bound conservatively. The holder stops before the possible end; the successor starts after the possible end.

The interval formulas alone do not create safety. They only preserve safety after the protocol has supplied a valid lease boundary. A database with no bounded-clock assumption should use a different proof, such as confirming leadership with a quorum for the read.

The Mechanism Step by Step

Use this illustrative sequence. The times and margins are examples, not observed measurements.

Lease for term 41 held by ny-db-3: expires at T = 01.500

01.480  ny-db-3 sees [01.480, 01.492]
        latest < T, so its existing lease fast path is still valid.

01.490  network partition isolates ny-db-3 from the quorum.

01.496  quorum elects md-db-2 for term 42 and records its authority.
        Election alone is not permission to reuse the old time window.

01.498  md-db-2 sees [01.494, 01.506]
        earliest <= T, so it does not use a replacement lease yet.
        It can wait or use a quorum-confirmed read.

01.518  md-db-2 sees [01.512, 01.524]
        earliest > T. With the protocol's term and quorum proof in place,
        it may enable its own lease fast path.

Notice what each replica can decide. ny-db-3 cannot learn from the partition that it was replaced. It can only obey the lease it already has and stop conservatively. md-db-2 can learn a new term from the quorum, but it still waits out the possible remainder of the prior lease. The overlap disappears because the old node's safe interval ends before the new node's begins.

What if Madrid needs a result at 01.498? It does not need to invent a weaker answer. A common alternative is a quorum-confirmed linearizable read: the leader verifies that it is still current with the quorum, then serves state at a safe log position. The details vary by system—some call this a read-index path—but its purpose is stable: trade an extra coordination round for a proof that is not relying on the lease fast path.

So far, the lease does not make a read correct by magic. It lets a system reuse a recent proof for a limited interval. When time or communication makes that reuse unclear, the system obtains fresh evidence.

The scope is narrow. A valid leader lease can justify a read from the leader's committed local state; it does not make an arbitrary follower current. A follower still needs its own freshness evidence, such as a required log position, before it can answer a session-sensitive request.

Margin Is an Operational Signal

Lease safety is binary at the boundary, but its performance value fades before the boundary. Harbor Point can track the available margin:

lease expiry                        01.800
latest safe local-read instant      01.784
remaining local-read margin           16 ms

Those numbers are illustrative. The exact margin comes from a product's lease duration, clock bound, renewal timing, and safety reserve. Its use is practical: a shrinking margin predicts that the fast path will close more often.

Two causes can shrink it:

The trade-off is explicit. Longer leases can reduce coordination on ordinary reads, but they make handoff waits and the consequences of a poor clock bound more visible. Shorter leases reduce the duration of any old authority but require more frequent renewal. Neither setting repairs an unbounded clock or a broken lease-grant protocol.

The boundary also explains an important operational choice: a rise in read_fallback_total after a time-quality or heartbeat alert can be protective behavior. It means the service chose a more expensive proof. Treat it as a reason to investigate clocks, pauses, network delay, or follower overload—not as permission to turn off safety checks to recover latency.

Where Lease Reads Do Not Apply

Do not infer that every “read from leader” implementation is a lease read, or that every lease is a leader read lease. A system may offer linearizable reads by quorum confirmation, route reads to a leaseholder, use timestamp ordering, or expose an explicitly stale replica read. The word lease names a particular time-bounded authority argument; it does not name a universal API guarantee.

Likewise, a clock bound is an assumption to operate, not a number to declare once. If a host pause, time-service failure, virtualization problem, or monitoring signal makes the bound untrustworthy, the fast path must close until the system can re-establish its assumptions. The safe fallback preserves semantics but may not preserve the latency objective.

Check Your Understanding

Check: The former leader sees [01.489, 01.503] and its old lease expires at 01.500. May it serve a lease read?

Think first, then reveal.

Answer: No. latest is 01.503, already beyond the expiry. The former leader must stop using that lease even though earliest is still before the boundary.

Check: The new leader has been elected, but sees [01.497, 01.509] while the old expiry is 01.500. What is safe?

Think first, then reveal.

Answer: It may not start a replacement lease because earliest has not passed 01.500. It can wait or use a quorum-confirmed read path if the system provides one.

Practice: Review a Read Fast Path

Your service proposes a 400 ms leader-read lease. Write the smallest safety review for it.

1. Name the term, membership, and quorum event that grants the lease.
2. State the clock-bound assumption and the conservative stop/start checks.
3. Describe what the old leader does after a partition.
4. Name the fallback that preserves the read guarantee during the guard interval.
5. Choose one signal for time quality and one for renewal health.

A good answer should mention: a proof beyond “this node is leader,” a guard interval with no overlapping lease authority, a defined fallback, and a response when the clock or communication assumption is no longer trustworthy.

Connections

• Secondary Indexes Across Shards separates an optimized read path from the authoritative base row; leases similarly make a fast read path conditional on proof.
• Membership Changes and Replica Set Evolution shows why a lease also depends on the current term and committed replica configuration.
• Replication Flow Control and Backpressure explains why delayed follower communication can turn a read optimization into fallback traffic.

Resources

• [DOC] Spanner: TrueTime and external consistency — Focus: Study an explicit bounded-time model and how timestamp ordering supports strong reads across regions.
• [DOC] etcd API guarantees — Focus: Compare serializable and linearizable reads, and notice that an API can expose different proofs for different read modes.
• [PAPER] In Search of an Understandable Consensus Algorithm — Focus: Review terms, quorum agreement, and leader authority as prerequisites that a lease optimization cannot replace.

Key Takeaways

1. Election and lease-read authority are related but different proofs; a new term does not erase the possible remainder of an old lease.
2. With a bounded-time model, an old holder stops using its lease on the conservative upper bound and a successor starts only after the conservative lower bound passes expiry.
3. Lease reads buy lower normal-path latency, but their honest degraded mode is a slower confirmed read when time or quorum evidence weakens.