Replication, Logs, and Storage Consistency
LESSON
Replication, Logs, and Storage Consistency
By the end of this lesson, you will be able to...
Trace one write through a leader and replicas using log positions.
Distinguish locally stored, replicated, and committed progress in a durable history.
Predict what a read or a leader failure may expose under a stated commit rule.
Idea in one sentence: Replication is not “several copies exist”; it is a rule for advancing several durable copies through one ordered history and deciding when that history may be promised to a user.
A completion that appears, then disappears
A learner finishes the final lesson of a course. The platform accepts the completion, shows a green checkmark, and begins creating a certificate. A few seconds later, the primary storage node fails. The replacement node starts serving requests. The learner refreshes the page and the completion is gone.
The immediate reaction is often: “But we had replicas.” That sentence hides the important question. Which replica had received the completion? Had it stored the update durably? Had the system decided that the update was committed before it acknowledged the learner? Was the new read served from a replica that had caught up?
The naive picture of replication is a photocopier: write bytes on one machine, copy them to others, and assume the copies are equivalent. It works as a picture only when nothing is changing. During a write, replicas can be at different points. One may have a record in memory. Another may have flushed it. A third may not have received it. The application needs a shared way to describe that uneven progress.
An ordered log provides that language. It records changes as positions in a history. A commit rule says which positions are safe enough to acknowledge or expose. Together, those ideas turn “replicated” from a vague compliment into a testable storage promise.
Core Insight
Plain meaning:
When several durable copies must tell the same story, give each change a place in one story and record how far each copy has read it.
In this scenario:
complete(course-42, learner-ada)is not merely a new value in one database row. It is an entry at a numbered position that each storage node may receive, make durable, and apply.
Technical name:
The numbered ordered history is a replicated log. A commit rule determines when a log position is part of the history the system may safely promise.
This is a deliberately small model. Real products use different replication protocols, failure detectors, and consistency guarantees. The portable storage question is still the same: after an acknowledgement, which durable copies are known to contain which ordered prefix of history?
log position: 41 42
entry: publish title record completion
The log is not the same as the final materialized state. A storage node may append an entry before it updates a page, index, or derived certificate view. That distinction is useful: a durable history can be replayed after a crash, while the convenient current-state structures can be rebuilt or caught up. The next lesson examines that local recovery boundary in more detail.
Three words that should not be synonyms
Teams often use saved, replicated, and committed as if they all mean “nothing can go wrong now.” They describe different steps.
| State of position 42 | What it means | What it does not prove |
|---|---|---|
| Received | A node has been sent the entry. | The entry survived a crash. |
| Locally durable | One node has flushed the entry according to its local storage contract. | Another node has it or a failover can preserve it. |
| Replicated | One or more other nodes have received, and perhaps durably stored, the entry. | The system’s configured commit condition has been met. |
| Committed | The configured rule has declared the position part of the durable history it will honor. | Every replica is already caught up or every read will see it. |
| Applied | A node’s queryable state reflects the entry. | Every other node has applied it too. |
The exact definitions vary by system. Some use acknowledgements after local durability; others wait for another durable copy or for a quorum. Some distinguish an entry being committed from it being applied to a queryable state. Do not borrow a word’s strongest meaning from another product. Read the promise that accompanies the acknowledgement.
A worked write trace
Use a small primary-replica group: leader L, replica A, and replica B. The commit rule for this example is: a position is committed after the leader and at least one replica have durably stored it. This is an example rule, not a recommendation for every system.
The input is the completion write at position 42.
write: complete(course-42, learner-ada)
| Time | L | A | B | Decision visible to the client |
|---|---|---|---|---|
| T0 | committed through 41 | applied through 41 | applied through 41 | No new write yet. |
| T1 | appends 42, then flushes it | through 41 | through 41 | Locally durable at L; do not acknowledge under this rule. |
| T2 | sends 42 | receives 42 and flushes it | network delay; still 41 | L and A durably have 42. |
| T3 | marks 42 committed | learns commit and may apply 42 | still 41 | Acknowledge success to the learner. |
| T4 | serves current reads | applies 42 | catches up later | Reads from B may still be stale unless the read rule prevents that. |
The transition at T3 is the heart of the model. Before it, the completion exists on the leader and perhaps on A, but the system has not yet made its configured promise. After it, the leader can answer: “this position belongs to the committed history.” B is not broken just because it is behind. Its position describes its lag.
committed log: [1] ... [41] [42]
L applied: [1] ... [41] [42]
A applied: [1] ... [41] [42]
B applied: [1] ... [41]
Now introduce the naive failure contrast. Suppose the leader acknowledged at T1, immediately after only its own local flush, then failed before A made 42 durable. A replacement chosen from the surviving replicas might know only through 41. The user saw success, but the new authority cannot prove that 42 belongs to a surviving history. Earlier acknowledgement lowers latency; it changes what failures can take away.
Check: At T4, is replica B necessarily unusable?
Think first, then reveal.
Answer: No. B may safely serve reads whose required freshness permits position 41, such as an older analytics view. It is unsuitable for a read that promises the learner will immediately see the acknowledged completion. The key is the read promise, not the label “replica.”
Commit rules turn latency into a product choice
The table above shows one point in a design space. A system can acknowledge after several different conditions:
| Acknowledge after… | What improves | Trade-off and boundary |
|---|---|---|
| leader receives the write | Low apparent latency | A crash can lose a write that was never durable. |
| leader persists locally | Survives some local process failures | Leader loss can still remove the only durable copy. |
| leader plus one durable replica | Better protection from one-node loss | Network delay or a slow replica enters the write path. |
| a larger configured set of durable replicas | Stronger evidence that a committed prefix survives | More failure or latency conditions can delay acknowledgement. |
The correct choice depends on the product’s loss tolerance, latency budget, availability requirements, and failure model. A course thumbnail can often be republished. A completion that unlocks a certificate may require a stronger acknowledgement rule. The lesson is not “always wait for more replicas.” It is “state the promise before choosing the wait.”
This also separates replication from caching. A cache can serve a temporary copy that may be old. A replica participates in a durable history and can become an authority after failure only under rules that preserve the committed prefix. A cache hit is a performance event; a committed log position is a correctness event.
Reads have a position too
After the learner receives success at T3, what should a refresh return? There are several valid policies, but each needs to say which replica may serve it.
- A leader read can observe the committed position if the leader is healthy and has applied it.
- A read from
Acan observe it afterAapplies the committed entry. - A read from
Bmay return the old state until it catches up. - A session can carry a minimum required position—“do not answer below 42”—and wait, route, or retry until a suitable replica can serve it.
That last idea is a useful bridge from plain language to precision. “Read your writes” means that after a client is told a write succeeded, later reads for that client should not be served from an older position than the acknowledged one. It does not require every reader everywhere to see 42 at the same instant. It requires an explicit connection between the acknowledgement and the permitted read path.
Check: The platform reports replica_lag = 0 seconds for B, but a learner just acknowledged at position 42 still receives the old completion from B. Which extra fact is missing?
Think first, then reveal.
Answer: Time-based lag alone does not prove the required position has been received and applied. Check B’s durable and applied log positions relative to the client’s required position, plus the read-routing rule. A small time value can hide an important recent write.
Trade-offs and limits
Replication improves availability and fault tolerance by giving the system more than one durable copy of a history. It costs network work, coordination, storage space, and time on the write path. It does not eliminate all failure: a bug can be replicated, a bad commit rule can make a weak promise, and a lagging replica can still serve an old answer if routing allows it.
Useful operational signals follow the mechanism:
- Received, durable, committed, and applied positions show different stages. A single “replica healthy” light cannot replace them.
- Replica lag by position and by time reveals whether replicas are falling behind and whether a given read is eligible.
- Commit latency and acknowledgement errors show the cost of the configured durability rule.
- Catch-up rate and backlog reveal whether a recovering replica is closing or widening the gap.
- Read-routing and stale-read counts show whether product promises are being met in practice.
These signals do not implement a consensus protocol for you. They reveal whether the storage promise you chose is being kept. Consensus mechanics, election rules, and full consistency models belong in deeper tracks; here, log position and commit rule are enough to audit the meaning of an acknowledgement.
Common confusions
Confusion: More replicas automatically mean stronger consistency
Why it is tempting:
Three copies sound safer than one copy.
Better model:
The number of copies matters only together with ordering, durable acknowledgement, commit rules, and read routing. Three asynchronous copies can still leave a recent acknowledged write absent from the replica chosen after failure.
Confusion: A replica with the bytes may serve the write
Why it is tempting:
If the entry arrived, it feels as though the replica knows the new state.
Better model:
Received, durable, committed, and applied are separate transitions. A query should be served only when the replica’s position and the read policy make its answer acceptable.
Confusion: A log is only a debugging record
Why it is tempting:
Logs are often first encountered as text examined after an incident.
Better model:
In storage, an ordered durable log can be the history used to recover state, bring replicas forward, and define the committed prefix. It is a data structure with a promise, not merely a diagnostic file.
Practice: review the acknowledgement promise
The platform plans two writes:
- A learner saves a draft note. Losing the final few seconds after a regional failure is acceptable if the UI says the note is “syncing.”
- A learner completes a paid certification exam. Once the UI says “completed,” support staff must be able to rely on that outcome after one storage node fails.
For each write, state a plausible acknowledgement rule, a read rule immediately after acknowledgement, one trade-off, and one operational signal that could show the rule is under pressure.
Model answer: For the draft note, the system might acknowledge a locally durable leader append while displaying a syncing state until a replica catches up; a later read can prefer the leader or carry the session’s recent position. This reduces latency but accepts that leader loss can discard a recently acknowledged draft, so monitor replica backlog and time-to-replication. For the paid exam, wait until the configured failure-surviving set has durably recorded the position before showing final completion; route immediate reads to a replica that has applied at least that committed position. This costs more write latency and may reject or delay writes during replica trouble, so monitor commit latency, acknowledgement failures, and durable/applied position gaps.
Resources
- [BOOK] Designing Data-Intensive Applications — Focus: Relate replication, log positions, read guarantees, and acknowledgement trade-offs.
- [DOC] PostgreSQL write-ahead logging — Focus: See why durable ordered intent supports recovery before applying it to replicated systems.
- [PAPER] In Search of an Understandable Consensus Algorithm — Focus: Extend the log-position and committed-prefix model into a full replicated-agreement protocol.
Key Takeaways
- Replicas are not interchangeable boxes: each has a received, durable, committed, and applied position in an ordered history.
- A commit rule gives an acknowledgement its meaning by stating what durable evidence must exist before the system makes a promise.
- Replication trades latency and coordination for failure tolerance; read routing and position-aware signals determine whether the promised result is actually served.