Consensus, Quorums, and Coordination

By the end of this lesson, you will be able to...

• explain why some decisions need one official answer across several machines.

• trace how quorum overlap prevents two groups from committing separate histories.

• decide whether a workflow needs consensus by naming what would fork without coordination.

Idea in one sentence: Consensus is how independent machines make one official decision when two official answers would break the system.

Core Insight

Imagine an invoicing service.

Every invoice needs a unique official number:

INV-1042
INV-1043
INV-1044

The promise is strict. Two customers must not receive different invoices with the same official number. The company also needs one auditable history of issued invoice numbers.

Now run the invoice-number service on five machines:

A B C D E

The naive idea is:

Any healthy machine can issue the next invoice number.

That works while every machine can talk to every other machine. Then the network splits:

side 1: A B C
side 2: D E

If both sides keep issuing official invoice numbers, the system can fork:

A B C issue: INV-1045 to customer Ana
D E issue:   INV-1045 to customer Bao

Both sides may be alive. Both sides may have logs. Both sides may believe they are helping customers.

The problem is not that a machine crashed. The problem is conflicting authority.

Plain meaning:

Some decisions are dangerous if two groups make different official answers.

In this scenario:

The next invoice number must come from one shared history, not from two isolated histories.

Technical name:

Consensus is the mechanism that lets several machines agree on one official decision despite failures and delayed messages.

The Decision That Must Not Fork

Not every decision needs consensus.

Refreshing a cache twice is usually fine. Sending two internal metrics may be annoying but repairable. Recomputing a preview image can waste work without breaking the product promise.

Invoice numbers are different. The decision is:

the next official invoice number is INV-1045
and it belongs to this invoice record

If two groups make different official decisions for the same number, repair becomes painful. You may need to void invoices, explain audit gaps, contact customers, or patch accounting records.

That is the signal that coordination is needed:

disagreement would cost more than waiting

Consensus is not for making everything faster. It is for protecting decisions that must not fork.

Check: Does every write in the invoicing system need consensus?

Think first, then reveal.

Answer: No. The protected decision is the official ordered invoice-number history. Other work, such as rendering a PDF preview or refreshing a cache, may not need consensus if disagreement is cheap or repairable.

The Moving Parts

A typical consensus group has:

participants:
  A B C D E

leader:
  the node currently allowed to propose ordered decisions

log:
  the ordered list of official decisions

term or epoch:
  a numbered leadership period

quorum:
  enough participants to make a decision count

For five nodes, a common quorum is three.

nodes: A B C D E
quorum size: 3

The important property is overlap:

any group of 3
shares at least 1 node
with any other group of 3

Example:

quorum 1: A B C
quorum 2: C D E
overlap: C

That overlap is how evidence of one official history can be carried into future official decisions. A later quorum cannot be made entirely of machines that had no possible contact with the earlier quorum.

The Naive Threshold

Try the dangerous version first.

Suppose five machines allow any two machines to issue the next official number:

quorum size: 2

During a partition:

side 1: A B C
side 2: D E

Both sides can form a group of two:

A B agree:
  INV-1045 = invoice for Ana

D E agree:
  INV-1045 = invoice for Bao

There is no overlap between A B and D E. No participant is forced to carry evidence from one decision into the other. The system now has two official-looking histories.

This is split brain: more than one group acts with official authority.

The tempting benefit is availability. Smaller groups can keep going during more failures.

The cost is safety. The system can create two incompatible answers.

The Mechanism: Quorum Overlap

Now require three of five machines.

quorum size: 3

During the same partition:

side 1: A B C
side 2: D E

Only side 1 can form a quorum.

A B C can commit:
  INV-1045 = invoice for Ana

D E cannot commit:
  not enough participants

The minority side is not dead. It may still serve old reads if the product allows stale reads. It may still accept local work into a queue marked "not official yet." But it must not publish a new official invoice number.

Consensus chooses refusal over fork.

That refusal is the point. The system is saying:

I would rather stop issuing new official numbers
than issue two official histories.

A Worked Trace

Here is a simplified consensus trace.

Starting state:

nodes: A B C D E
leader: A
term: 7
last committed invoice number: INV-1044
next log index: 19
quorum size: 3

The business service asks for the next invoice number.

The input is a request for one official decision. The transition is leader proposal plus follower acceptance. The intermediate state is important: A may have proposed the entry before it is committed. The output becomes official only when a quorum accepts it under the protocol rules.

Naive failure contrast:

if A alone returned INV-1045:
  A could crash before anyone else saw the decision
  another leader might later reuse the number

if A waits for quorum:
  the decision has enough shared evidence
  a later official decision must overlap with that evidence

So far, consensus has turned "one machine says so" into "enough overlapping machines accepted the same ordered entry."

Check: In the trace, why is A alone not enough to make INV-1045 official?

Think first, then reveal.

Answer: A single node can crash, be isolated, or be stale. A quorum creates shared evidence that future quorums cannot completely avoid.

Leaders, Terms, And Logs

Real consensus protocols need more than the word "quorum."

A leader gives the system one place to send proposed changes for a while. Without a leader or some equivalent ordering rule, different nodes can propose different next entries at the same time.

A term or epoch numbers the leadership period:

term 7: A is leader
term 8: C is leader

Terms help nodes reject old authority. If A was leader in term 7 but later sees term 8, it should stop acting as leader.

A log gives the system an ordered history:

index 17, term 6: INV-1043 -> Mei
index 18, term 7: INV-1044 -> Omar
index 19, term 7: INV-1045 -> Ana

The core promise is not merely "choose a value." It is:

once a log entry is committed,
future official histories must preserve it

This is why consensus appears in systems that need leaders, locks, configuration versions, shard ownership, cluster membership, and ordered logs. The exact decision changes. The pressure is the same: one official history must survive failure.

During A Partition

Return to the partition:

side 1: A B C
side 2: D E
quorum size: 3

Side 1 can continue committing invoice numbers if it has a valid leader.

Side 2 cannot commit new official invoice numbers. It may know the last committed number. It may have customers waiting. It may be tempted to "just keep going locally." But doing that would create a second official history.

Now change the partition:

side 1: A B
side 2: C D
isolated: E

No side has three nodes. The protected path stops. That may feel painful, but it is safer than issuing duplicate official numbers.

This is the central trade-off:

consensus improves safety for decisions that must not fork
but costs latency and availability when quorum cannot be reached

What This Changes

Before consensus, you might ask:

Which machine is alive enough to answer?

After consensus, you ask:

Which group has enough overlapping evidence to make this answer official?

That changes the design. A node can be healthy and still lack authority. A minority group can have CPU, disk, memory, and logs, but still be forbidden from accepting protected writes.

That also changes user experience. If the invoice-number group cannot reach quorum, the product should not pretend the invoice was officially issued. It can show "invoice pending," queue the request, or route to a healthy quorum. It should not invent a local invoice number and hope to merge later.

Consensus is useful exactly where "hope to merge later" is not a real plan.

Trade-offs and Limits

The trade-off is that consensus buys one official history by spending latency and sometimes refusing availability.

It helps when disagreement is more expensive than waiting.

It costs extra messages, leader management, disk writes, quorum checks, and operational complexity.

It can still fail when the implementation has bugs, disks lose acknowledged writes, operators misconfigure membership, or clients read from stale followers without a read rule.

It does not make every replica instantly up to date. A follower can lag behind the committed log. The system may still be safe, but reads from that follower may be stale unless the read path accounts for lag.

It also does not replace idempotency. Consensus can decide the official next invoice number. Idempotency can still be needed so a client retry does not submit the same invoice request twice.

You can see the boundary when leader elections spike, commit latency rises, quorum is unavailable, followers lag, or rejected writes increase because the system is protecting safety.

Signals To Watch

A consensus system should make its coordination state visible.

Useful signals include:

• current leader and current term
• proposal latency and commit latency
• number of leader elections
• quorum availability
• follower replication lag
• committed log index versus applied log index
• rejected writes because no quorum was reachable

These signals help separate two cases that feel similar to users.

In the first case, the system is healthy but refusing progress to protect safety. For example, only two of five invoice nodes are reachable, so the service rejects new official invoice numbers. That is painful, but correct.

In the second case, the coordination path itself is unstable. Leaders keep changing, followers fall behind, or commit latency rises even though enough nodes should be reachable. That is not the safety trade-off working as designed; that is the mechanism struggling.

The operational question is:

Are we waiting because quorum is truly unavailable,
or because the consensus group is unhealthy?

The answer changes the response. A real partition may require routing users to the side with quorum or showing a pending state. An unstable leader may require investigating networking, disk latency, overloaded nodes, or configuration mistakes.

Common Confusions

Confusion: Consensus means every node agrees instantly

Why it is tempting:

The word "consensus" sounds like everyone has the same value right now.

Better model:

Consensus protects the official committed history. Some followers may learn or apply that history later.

Confusion: A majority is good because it is faster

Why it is tempting:

Three responses can be faster than waiting for all five.

Better model:

The key property is overlap. Quorums prevent disjoint groups from making separate official decisions.

Confusion: The minority side is down

Why it is tempting:

It cannot accept protected writes.

Better model:

The minority side may be alive but lacks authority. It can do only work that is safe without a new official decision.

Confusion: Consensus is needed for every distributed action

Why it is tempting:

Consensus sounds like the strongest tool.

Better model:

Use consensus where two official answers would be expensive or unsafe. Avoid it where disagreement is cheap, temporary, or mergeable.

Practice

Pick one decision:

issue invoice number
choose shard leader
publish configuration version
grant write lock
refresh cache
increment analytics counter
assign support ticket

Fill in:

decision:
what would fork if two answers existed:
participants:
quorum or owner rule:
what becomes official only after quorum:
what the minority side may still do:
what the minority side must not do:
why waiting is or is not worth it:
signal that the coordination path is unhealthy:

Then change one parameter. In a five-node group, compare quorum size 2 with quorum size 3.

Predict:

what becomes more available:
what becomes less safe:
what overlap is lost or preserved:

A good answer should name the protected decision and the damage caused by two official histories. If you cannot name what would fork, consensus may be unnecessary for that decision.

Resources

• [PAPER] In Search of an Understandable Consensus Algorithm
  • Focus: Raft's leader, term, log, and commitment model as a readable consensus example.
• [PAPER] Paxos Made Simple
  • Focus: The core safety problem behind choosing one value despite failures.
• [PAPER] The Chubby Lock Service for Loosely-Coupled Distributed Systems
  • Focus: How a real lock and coordination service uses consensus-backed state.

Key Takeaways

• Consensus protects decisions that must have one official history.
• Quorum overlap prevents disjoint groups from committing separate official answers.
• Leaders, terms, and logs help turn proposals into ordered committed decisions.
• Consensus trades latency and some availability for safety.
• Use consensus where disagreement is costlier than waiting; avoid it where work is cheap to merge or repair.