Storage Foundations Capstone: Choose the Right Storage Contract

By the end of this lesson, you will be able to...

• Produce a storage plan that names contracts, authorities, paths, acknowledgement promises, and signals.

• Review a plan for cache, recovery, replication, and hotspot failure boundaries.

• Select the downstream implementation track that matches the remaining unknowns.

Idea in one sentence: A storage design is credible only when every workload has a fitting contract and every important promise can be traced to metadata, data, cache, durability, and operational evidence.

The launch review

The course platform is expanding from a small product to a regional launch. The team wants one answer to a deceptively simple question: “Where should all this data live?”

The tempting answer is one product for every byte. It sounds tidy. It fails as soon as the team lists the work: learner progress changes in small coordinated updates; videos are large immutable versions; report tools use paths and renames; backups need retention and restore evidence; course pages need fast but bounded-stale metadata; a popular launch creates hot names and refill bursts.

Your task is to review the storage plan before launch. Do not choose a vendor. Choose the promises each workload needs, then make the paths and evidence visible.

Core Insight

The track’s model is:

workload semantics
  → storage contract
  → metadata authority and data path
  → cache and freshness rule
  → durability and recovery promise
  → operational signals

If one arrow is absent, the plan has a hidden assumption. A contract without a recovery promise can lose acknowledged work. A cache without a freshness rule can be fast and wrong. A capacity chart without request-shape signals can miss a hotspot. A replica count without a commit rule says little about what success means.

The platform to design

The platform has six workloads:

1. Learner progress and certificates. Completion updates are small, ordered, and must remain credible after one storage-node failure once the UI says “completed.”
2. Video packages and thumbnails. Uploads create versioned media, read by many learners. A newly published version must not replace a playable old version with a missing object.
3. Shared report exports. Analysts and batch jobs use a shared path tree, create temporary files, rename completed reports, and browse directories.
4. Backup and restore artifacts. Nightly backups must be retained, tested by restore drills, and protected from accidental deletion.
5. Course catalogue metadata. Course title, publishing state, and current media version are read often. Public pages tolerate short staleness; editor actions need faster visibility.
6. Analytics event archive. Events are appended in batches, retained for later processing, and need economical large-volume storage rather than tiny interactive updates.

The simple model breaks because no single abstraction makes all six natural. The better model is a set of explicit boundaries.

Worked design review

This table is a design hypothesis, not a universal architecture. A database may use files and block devices below its own contract. Object storage may serve backups and media through the same substrate while their retention and publication rules differ. What matters is that the learner can state who owns each answer and what operation must be correct.

Check: Why is “put progress and videos into object storage because it is durable” an incomplete design?

Think first, then reveal.

Answer: Durability alone does not supply fine-grained mutable progress updates, transaction boundaries, read-after-write rules, or a certificate acknowledgement promise. Videos fit whole-object versioning well; progress needs a contract that owns coordinated mutation and recovery semantics.

Trace the two critical promises

First trace certificate completion:

completion request
→ progress authority appends ordered intent
→ configured durable commit condition
→ acknowledge “completed”
→ replicas apply and reads route by required position

The user promise is not “some node received bytes.” It is that a completed certificate can be recovered and served after the failure the team claims to tolerate. The plan must state whether local durability is enough or whether another durable replica is required before acknowledgement.

Then trace media publication:

upload package under media/course-42/v9
→ verify object availability
→ durably update current-version manifest
→ cache or revalidate manifest
→ readers fetch immutable v9

The key trade-off differs. Certificate writes may wait for stronger durable evidence. Media can use immutable objects and cached versioned keys, but the small manifest needs a clear freshness and recovery story. Never publish a pointer to an object whose availability has not been verified.

Review the failure boundaries

Use this short rubric to review every workload.

The rubric catches common weak plans. “We replicate it” is weak without a commit rule. “We cache it” is weak without a staleness budget. “We back it up” is weak without restore verification. “We have free capacity” is weak without request and metadata pressure.

Check: The report team wants to replace the shared export tree with object keys because the object store has lower cost. What constraint should decide the review?

Think first, then reveal.

Answer: Verify whether the workflow truly requires shared paths, directory browsing, temporary files, rename publication, and file-style coordination. If it does, keys alone do not provide equivalent semantics. A completed immutable export may be published as an object, but the workflow needs an explicit replacement for the file contract.

Operational launch plan

Before launch, record a small dashboard and a small drill for each promise.

• Progress: commit latency, replica position lag, read-routing failures, journal/checkpoint pressure, and a node-loss recovery exercise.
• Media: upload completion errors, manifest/object verification failures, cache hit and origin load, p99 start time, and a version rollback drill.
• Reports: metadata operation latency, hot-directory rate, rename errors, and small-file growth.
• Backups: last successful backup, restore-test age, restore duration, retention errors, and storage cost.
• Catalogue: cache value age, miss/refill duration, metadata queue age, p99 listing latency, and hot-prefix request rate.
• Archive: ingest backlog, lifecycle/retention completion, retrieval error rate, object-size distribution, and query cost.

These signals are not a vendor runbook. They are evidence that a promise is under pressure. The trade-off is instrumentation cost and attention: recording every object key as a metric can overwhelm telemetry, while aggregates can hide the hot prefix. Choose dimensions that can distinguish a plausible failure without exposing unbounded cardinality.

Practice: write and defend the plan

Create a one-page storage decision record for the platform. For every workload, include contract, authority, read/write path, acknowledgement or publication rule, cache/freshness policy, recovery/restore boundary, one operational signal, and one trade-off. Then answer these review questions:

1. Which promise would fail first if the primary progress node disappeared after an acknowledgement?
2. Which content can safely use long-lived versioned cache keys, and which small pointer needs a shorter freshness rule?
3. Which operation could be slow with free capacity, and which signal would prove it is metadata rather than byte throughput?
4. Which artifact is not trustworthy until a restore drill succeeds?

Model answer: Progress needs database-managed mutable storage, a durable commit condition appropriate for node loss, and position-aware reads; otherwise the acknowledged certificate is at risk. Versioned media objects can be cached aggressively, while the current-version manifest needs bounded freshness and a verified update. Course listings can be slow with free bytes when a namespace or metadata range is hot; listing p99, metadata queue age, and cache misses distinguish that from bulk throughput. A backup is only an untested copy until a restore drill proves the required data and recovery time are achievable.

What this capstone does not decide

The plan deliberately stops before implementation algorithms. If the unknown is page layout, buffer pools, WAL/checkpoint algorithms, and index maintenance, continue to database-engine-internals-and-implementation. If it is allocation, filesystem journaling modes, extents, snapshots, checksums, or repair, continue to filesystem-and-storage-implementation. If it is drivers, DMA, interrupts, hardware queues, and device scheduling, continue to io-stacks-device-drivers-and-storage-paths. If it is quorum safety, failover, and distributed guarantees, continue to consistency-and-replication.

Final design-review rubric

Score each workload from 0 to 2 on each row. A zero means the plan uses a label without a testable promise; a one names the promise but leaves a boundary unclear; a two gives an explicit rule and evidence.

A plan that scores highly is not guaranteed to be correct. It is inspectable: a reviewer can challenge one promise at a time. That is the practical purpose of the track. If a team cannot answer where a reader gets a value, what makes a write durable, or how it will observe a hotspot, it has not finished designing the storage behavior.

Consider a failure drill. A popular course is published during a replica catch-up and a catalogue cache expiry. Editors report success, learners see a mixture of old and new catalogue entries, and report exports are delayed. A weak response says “storage is degraded.” A stronger review separates the paths: verify the progress commit position for editor acknowledgements; check the manifest publication rule for media; measure catalogue cache misses and metadata queue age; inspect replica backlog; and check report-directory metadata operations. The signals may share a root cause, but they should not be assumed to be the same incident before the path is traced.

The final trade-off is scope. A foundation design should be detailed enough to make promises testable, but it should not pretend to contain every filesystem allocator, database recovery algorithm, device queue, or consensus rule. Naming the downstream question precisely is a success: it tells the next engineer what must be learned or verified before implementation.

Before approving the launch, run one tabletop exercise: remove a node after a progress acknowledgement, delay a media-manifest update, expire a catalogue entry under load, and start a restore from the latest backup. For each event, ask which user sees the effect, which authority decides the answer, which queue or cache changes the timing, and which metric confirms recovery. If the team can answer those questions without changing its vocabulary, the design has become an operational model rather than a diagram.

Record the answers with the launch decision. They become the first incident checklist, the first capacity review, and the evidence required when a proposed optimization changes a durability or freshness promise.

Resources

• [BOOK] Designing Data-Intensive Applications — Focus: Review storage, replication, logs, and data-system trade-offs.
• [BOOK] Operating Systems: Three Easy Pieces — Focus: Revisit filesystem, caching, and crash-consistency foundations.
• [REFERENCE] Google SRE Workbook — Focus: Turn storage promises into service-level indicators, error budgets, and drills.

Key Takeaways

1. Choose storage contracts from workload semantics, then state the authority, paths, promises, and evidence.
2. Metadata, caching, replication, recovery, and operational pressure are one design, not separate afterthoughts.
3. A good foundation plan makes its own implementation boundary and next learning track explicit.