Sharding and Authority Boundaries

The core idea: Sharding scales a data system by splitting authority across partitions, so every request must know which owner can decide the answer and every rebalance must move that ownership safely.

Core Insight

Harbor Point's reservation service has already learned how to replicate one authority domain. It can keep copies, route fresh reads, and even use an ordered chain for a scarce allocation bucket. That helps with resilience and some read load, but it does not remove the deeper limit: one authority still decides every write for the whole dataset.

Now the platform is growing. New desks, issuers, and reservation buckets are arriving faster than one primary path can absorb. Adding more replicas only makes more copies of the same decision point. Sharding changes the model: instead of one owner for all reservations, the system creates many owners, each responsible for a subset of keys.

The non-obvious insight is that sharding is not "put rows on more machines." It is a change in who is allowed to decide. A write to bucket B-173 must reach the shard that owns B-173; a transaction that spans B-173 and B-912 crosses an authority boundary; a rebalance is not just copying bytes, but changing which group is authoritative for some keys.

That makes sharding both powerful and expensive. It can turn one overloaded write path into many parallel local decisions, but it also pushes routing, transaction boundaries, fan-out, and ownership migration into the application and control plane.

Replication Copies Authority; Sharding Splits It

Before sharding, Harbor Point has one logical authority domain:

one reservation authority

          writes
            |
            v
       [primary]
       /   |   \
 [replica][replica][replica]

The replicas may serve reads, provide failover capacity, or participate in a stronger replication protocol. But for an authoritative write, the system still has to use the same ownership boundary. If that boundary is saturated, more replicas do not make the write decision parallel.

After sharding, ownership is divided:

reservation authority split by keyspace

bucket 000-255  -> shard A replica group
bucket 256-511  -> shard B replica group
bucket 512-767  -> shard C replica group
bucket 768-999  -> shard D replica group

Each shard is now an authority domain for the keys it owns. Shard A can accept writes for bucket 173 while shard D accepts writes for bucket 881. The system has not eliminated consistency questions. It has made many of them local.

This is the core payoff. Local decisions stay cheaper than global decisions. A reservation update that touches one bucket can be handled by the one shard that owns that bucket. A workflow that needs two buckets now needs coordination across two authorities, or it needs to be redesigned so the invariant does not cross that boundary.

Routing Is Part of the Contract

Once data is sharded, a request cannot simply "go to the database." It has to find the owner of the key before the database can make a correct decision.

POST /reservations/B-173/release
          |
          v
extract shard key: B-173
          |
          v
route through partition map
          |
          v
send to shard A, the owner of B-173

The partition map is now part of the correctness surface. If a router sends a write to an old owner during a rebalance, two groups may both believe they can decide the same key. If a read fans out to every shard because the API lacks the partition key, the system pays a cost that grows with shard count. If the application hides routing behind a helper library, that library becomes part of the data contract.

Healthy sharded systems make ownership visible in API and schema shape. The key needed to route a strong operation should be available at the start of the request. For Harbor Point, a release or confirm command should carry the reservation bucket or inventory identifier that selects the owner. Search, reporting, and dashboards may use derived views, but the decisive write path should not begin with "ask every shard."

Good strong-path shape:
  confirm(bucket_id=B-173, reservation_id=R-88421)
  -> one owner can decide

Risky strong-path shape:
  confirm(customer_id=C-7, issuer=CA-MUNI, maybe_bucket=?)
  -> router may need lookup, fan-out, or cross-shard coordination

The trade-off is design discipline. Sharding gives capacity only when the common strong operations can be routed to one owner. Otherwise, the system has many shards but still behaves like one distributed transaction engine on the hot path.

Authority Boundaries Shape Transactions

A shard boundary is also a transaction boundary unless the system pays for coordination above it.

Inside one shard, Harbor Point can often use the local database or local replicated log to enforce invariants. For example, moving one reservation bucket from held to released can be a single-shard transaction if all rows needed for that invariant live on the same owner.

single-shard invariant

shard A owns B-173:
  reservation row
  hold row
  bucket state

release B-173 -> local transaction

Across shards, the same style of operation becomes a different problem:

cross-shard invariant

shard A owns B-173
shard D owns B-881

swap allocations B-173 <-> B-881
  -> needs distributed transaction, saga, escrow, or redesign

This is why "we can shard later" is often too casual. If the domain model assumes many records can be updated together cheaply, sharding turns that assumption into coordination cost. The system either pays with distributed transactions, accepts weaker workflow semantics, or changes the data model so the strongest invariants fit inside one authority boundary.

The goal of this lesson is not yet to pick the perfect shard key. The next lesson goes deeper there. The goal here is to see what the key is really doing: it decides where authority begins and ends.

Rebalancing Moves Ownership

Shards do not stay balanced forever. One issuer may become hot, one range may grow faster, or a new region may need more capacity. Rebalancing changes which shard owns some part of the keyspace.

before:
  buckets 000-255 -> shard A

after split:
  buckets 000-127 -> shard A
  buckets 128-255 -> shard E

That looks like a storage migration, but the correctness issue is ownership. During the move, writes may still arrive. Routers may have cached old maps. Replicas may be behind. The system must avoid a period where both A and E accept authoritative writes for bucket 173, or where neither accepts them while clients believe the key is available.

A safe rebalance usually needs a protocol with phases:

1. copy data to the future owner
2. keep the future owner caught up
3. block or drain writes for the moving range briefly
4. switch the partition map to the new owner and epoch
5. reject stale requests that still target the old owner
6. clean up the old copy after the new ownership is proven

The exact implementation varies, but the principle is stable: rebalancing is a live authority migration. If the team treats it as a background file copy, it will eventually create stale reads, duplicate writes, or confusing partial outages.

Failure Modes

Adding replicas to a write-authority bottleneck. Replicas help availability and read scale, but they do not split the write decision if one primary or leader still owns the whole keyspace. The fix is to split authority, then replicate within each shard.

Hiding the shard key from strong APIs. If the router cannot identify the owner from the request, it may need lookups, fan-out, or cross-shard coordination. Shape decisive commands around the key that selects the authority boundary.

Letting business workflows casually cross shards. A workflow that touches many owners can still be correct, but it is no longer a local transaction. It needs a deliberate distributed transaction, saga, escrow model, or domain redesign.

Treating rebalance as bulk copy. Moving data without moving ownership safely can leave old and new owners accepting conflicting writes. Use epochs, routing-map updates, write drains, and stale-owner rejection to make the handoff explicit.

Resources

• [BOOK] Designing Data-Intensive Applications
  • Focus: Read the partitioning chapters for the difference between replication, partitioning, routing, skew, and secondary indexes.
• [DOC] Vitess Sharding Documentation
  • Focus: Study how a production sharding layer exposes keyspaces, routing, resharding, and control-plane responsibilities.
• [DOC] MongoDB Sharding
  • Focus: Use the overview to connect shard keys, routers, chunks, balancing, and distributed query behavior.
• [BOOK] Database Internals
  • Focus: Review partitioning and distributed storage internals as the physical side of authority boundaries.

Key Takeaways

1. Replication copies an authority domain; sharding creates multiple authority domains for different keys.
2. A sharded request must route to the owner that can decide the answer, so routing metadata becomes part of correctness.
3. Local shard transactions are cheaper than cross-shard workflows, so authority boundaries shape application semantics.
4. Rebalancing is a live ownership migration, not merely a storage copy.