Read Repair and Anti-Entropy Basics

Learning / Distributed Systems / Consistency and Replication / 038

LESSON

Consistency and Replication

038 30 min advanced

Day 437: Read Repair and Anti-Entropy Basics

The core idea: Once ordinary replication has already allowed replicas to diverge, systems need explicit repair paths: read repair fixes stale replicas encountered on the foreground read path, while anti-entropy finds and heals divergence that no client is actively touching.

Today's "Aha!" Moment

In 037.md, Harbor Point used flow control to keep lag from turning one recovering replica into a shard-wide failure. That is the prevention story. This lesson starts after prevention was not enough. During the opening auction, shard 184 kept serving writes from Madrid while ny-db-3 spent twelve minutes disconnected. When the WAN link returns, New York is not merely "behind." Some keys are stale, some repair cleanly from the current leader, and some no longer exist in the retained log window that ordinary replication depends on.

That is where teams often mix up two very different mechanisms. Read repair is opportunistic. A client performs a read, the coordinator notices replicas disagree, returns the value selected by the consistency rule, and pushes the stale replica toward that same value. Anti-entropy is deliberate background maintenance. It walks data that no client may read for hours, compares replicas, and schedules repair work so cold divergence does not sit in the system forever.

The important correction is that neither mechanism "creates consistency" on its own. They only work because the system already has version metadata and a conflict rule that tells it which replica state should win or whether an application-level merge is required. Without that metadata, repair degenerates into guesswork. With it, repair becomes a controlled reconciliation path instead of a blind overwrite.

Why This Matters

Harbor Point stores reservation state for exchange orders. At 09:41, reservation R-184-7731 was updated from pending to confirmed on md-db-2 and replicated to local quorum follower md-db-4. Because ny-db-3 was disconnected, it still holds pending. A trader in New York refreshes the order page after the link comes back and sees a stale answer if the system reads only from the local replica. Worse, a back-office audit job that scans old reservations overnight might never touch this key again, so a purely read-driven repair strategy would leave whole cold ranges inconsistent for days.

Production systems care because divergence is not just a cosmetic stale-read issue. A stale replica can feed bad decisions to nearby services, inflate conflict counts during failover, and make later reconciliation far more expensive because now the cluster must distinguish "temporarily stale" from "genuinely conflicting" state. Read repair and anti-entropy are the mechanisms that keep a bounded outage from becoming long-lived data drift.

They also force an operational trade-off. Every repair creates extra reads, writes, CPU work, and network traffic. If you repair too aggressively on the foreground path, tail latency rises for the exact users already hitting a busy shard. If you delay anti-entropy too much, cold keys remain wrong and recovery confidence drops. The engineering problem is not "should we repair?" but "which inconsistencies deserve immediate synchronous attention, and which are cheaper to heal in the background?"

Learning Objectives

By the end of this session, you will be able to:

  1. Explain the difference between read repair and anti-entropy - Identify which mechanism runs on the client read path and which one repairs divergence in the background.
  2. Trace how a system decides what value to repair toward - Use version metadata, quorum evidence, and conflict rules to reason about repair outcomes.
  3. Evaluate the operational trade-offs of repair strategies - Decide when foreground repair is worth the latency cost and when background anti-entropy should absorb the work.

Core Concepts Explained

Concept 1: Read repair piggybacks on a real read, so it is best at healing hot keys

Harbor Point exposes a QUORUM read mode for reservation lookups during market hours. When the New York trader reads R-184-7731, the coordinator asks three replicas for the row version:

md-db-2  -> version=57 status=confirmed
md-db-4  -> version=57 status=confirmed
ny-db-3  -> version=52 status=pending

The coordinator can answer the client because the latest durable value is clear: two replicas agree on version 57, and the stale replica is strictly older. But a production coordinator usually does more than return confirmed. It also records that ny-db-3 is stale and schedules a repair write carrying the winning version and its metadata. In leaderless systems, that repair might be sent directly to the stale replica. In leader-based systems, the coordinator may route the repair back through the normal write authority so the same ordering rules still apply.

The mechanism depends on version evidence, not intuition. "Newest timestamp wins" is one possible rule, but it is not the only one. Some systems use vector clocks or dotted version vectors to tell stale copies from concurrent ones. Others use a leader term plus log index, or an application merge policy for objects that can be combined safely. Read repair works only if the system can answer a precise question: is this replica behind, or are these replicas showing concurrent writes that need a different conflict path?

This makes read repair effective for hot data. Keys that users actually read tend to converge quickly because each read is also an inspection point. The trade-off is that foreground repair is not free. A QUORUM read now performs comparison work, may wait for a second replica, and may enqueue follow-up writes. Tail latency rises, especially if the read path insists on confirming that the repair landed before returning. Harbor Point therefore treats read repair as a selective tool for correctness-sensitive reads, not as a blanket fix for every stale replica.

Concept 2: Anti-entropy repairs cold divergence, but it must compare replicas efficiently enough to run all the time

Now consider the reservations no one is reading. Harbor Point's overnight compliance scan checks a tiny fraction of the shard, so relying on read repair alone would leave many stale rows untouched. The platform runs an anti-entropy job that periodically compares replica state for key ranges, identifies mismatches, and schedules targeted repair streams.

At the conceptual level, anti-entropy has three stages:

choose a range
   -> compare compact summaries or row versions across replicas
   -> stream missing or stale records toward the chosen source of truth

The reason this is a separate mechanism from ordinary replication is that the live log is no longer enough. ny-db-3 may have missed writes that already aged out of retained WAL, or it may have applied entries out of order before a crash and now hold internally inconsistent state for a subset of keys. Anti-entropy treats the replicas themselves as the evidence source. It asks, "What data do you each have now?" rather than, "What entries are still left in the log?"

The central design constraint is efficiency. A naive anti-entropy process that full-scans every row on every replica can heal divergence, but it will do so by consuming the same disk and network budget that foreground traffic needs. Real systems therefore compare compact summaries first and fetch detailed rows only for mismatching ranges. The next lesson, 039.md, focuses on Merkle trees because they are a common answer to that scaling problem. For this lesson, the important baseline is simpler: anti-entropy is background comparison plus targeted repair, not a magical consistency sweep that comes for free.

Concept 3: Repair safety comes from choosing the right authority and bounding the blast radius

A dangerous implementation mistake is to treat repair as "copy data from the first replica that answered." Harbor Point cannot do that safely. Suppose ny-db-3 contains version 58 for R-184-7731, but that version was written by a failed leader term that never reached quorum. If anti-entropy or read repair copies it blindly into Madrid, the repair path has just resurrected an uncommitted write.

Safe repair therefore begins by choosing an authority model. In a leader-based database, the current leader or a quorum-backed version usually defines the repair target. In a leaderless database, the repair logic may need to compare vector clocks and send sibling values to an application resolver rather than flatten them into one winner. Either way, repair is part of the consistency model, not an afterthought bolted onto storage cleanup.

Harbor Point also constrains how much repair work can happen at once. Background anti-entropy is rate-limited by shard and by replica pair so that recovery traffic does not starve normal writes. Foreground read repair is often asynchronous for non-critical reads, because the user needs the correct answer more urgently than the system needs immediate convergence on every replica. When the cluster is already stressed, operators may even prefer to defer anti-entropy and keep the repair backlog visible rather than let repair traffic create a second incident.

The trade-off is subtle but important. Aggressive repair shortens inconsistency windows, reduces failover surprise, and increases confidence that a rejoining replica is trustworthy. It also burns I/O budget and can amplify hotspots. Conservative repair protects latency and throughput but accepts longer periods where some replicas are known to be wrong. Good production tuning makes that choice explicit and observable instead of pretending both goals can be maximized simultaneously.

Troubleshooting

Advanced Connections

Connection 1: 037.md limits divergence; this lesson explains how to clean up after divergence exists

Flow control and backpressure try to keep a replica close enough to the live stream that normal replication is still the cheapest recovery path. Read repair and anti-entropy start once that assumption is already broken for at least some keys or ranges. Together they form a layered policy: first prevent drift from exploding, then repair the drift that still happened.

Connection 2: 039.md makes anti-entropy scalable by avoiding full row-by-row comparison

This lesson defines anti-entropy at the behavioral level: compare replicas, find mismatches, and stream repairs. The next lesson adds the data structure that lets large systems do that efficiently. Merkle trees matter because anti-entropy becomes operationally viable only when replica comparison can skip the vast majority of matching data.

Resources

Optional Deepening Resources

Key Insights

  1. Read repair is opportunistic convergence - It piggybacks on a real read, so it repairs the keys users touch most often and leaves untouched data to other mechanisms.
  2. Anti-entropy is background reconciliation - It exists because cold ranges can stay divergent long after foreground traffic stops noticing them.
  3. Repair is only as safe as the version rule behind it - Repair must follow quorum, leader, or conflict-resolution authority instead of copying whichever replica answered first.
PREVIOUS Replication Flow Control and Backpressure NEXT Merkle Trees for Replica Divergence Detection

← Back to Consistency and Replication

← Back to Learning Hub