Capacity, Hotspots, and Operational Storage Signals
LESSON
Capacity, Hotspots, and Operational Storage Signals
By the end of this lesson, you will be able to...
Separate stored capacity from request, metadata, queue, and maintenance pressure.
Diagnose a storage symptom by selecting the next signal that distinguishes competing causes.
Propose a mitigation with a stated trade-off and a way to verify its effect.
Idea in one sentence: Storage health is the shape of work moving through names, caches, queues, replicas, and maintenance—not just the percentage of bytes used.
The storage system with plenty of free space
At 09:00, the course platform publishes a new catalogue. It has 70% free storage capacity, low average device utilization, and no host is out of memory. Yet learners wait several seconds to browse /courses/new, while video playback and direct reads of known objects remain fast.
The first dashboard says “capacity healthy.” That answer is true and useless. It describes free bytes. It does not describe the work created by thousands of directory listings, tiny metadata reads, cache misses, or a maintenance job competing for the same path.
Storage incidents often start with this mismatch: an aggregate metric looks calm while one request shape is overloaded. The operational job is to turn the visible symptom into a smaller question. Is the wait in namespace metadata? Is one key range hot? Did a cache expiry expose an expensive origin path? Is writeback, replication catch-up, backup, or compaction consuming the headroom that users need?
Core Insight
Plain meaning:
Capacity tells us how much can fit. Pressure tells us where work is arriving faster than a layer can finish it.
In this scenario:
The catalogue occupies little additional byte capacity, but its many small course entries create a concentrated burst of path lookup and metadata work.
Technical name:
A hotspot is a concentrated demand on one key range, namespace, partition, replica, queue, or service boundary. Operational signals are observations that let us distinguish that pressure from other explanations.
The I/O-path model from the previous lesson gives the map. This lesson uses signals to locate the busy boundary.
symptom: slow catalogue listing
→ is the path lookup slow?
→ is metadata queued?
→ did cache misses rise?
→ is a lower service or maintenance job saturated?
Signals answer different questions
No single storage metric means “healthy.” Each signal describes a different resource or stage.
| Signal | Question it helps answer | What it cannot prove alone |
|---|---|---|
| Bytes used and free headroom | Can more stored data fit safely? | Whether requests are concentrated or slow. |
| Object/file count and small-object rate | Is metadata work likely to dominate? | Which metadata service is actually queued. |
| Request rate by key, prefix, directory, or shard | Is demand concentrated? | Whether the target can serve it. |
| Cache hit rate and miss cost | Did requests fall through to a deeper path? | Why the deep path is slow. |
| Queue depth and request age | Is work waiting at a boundary? | Whether queueing began upstream or downstream. |
| Replica lag and backlog | Is a durable copy falling behind? | Whether a particular read is allowed to use it. |
| Write amplification or writeback rate | Is internal work larger than user writes? | Which implementation detail creates it. |
| Error, retry, and throttle rate | Is the layer rejecting or delaying work? | Whether retries are making pressure worse. |
| Cost for requests, transfer, and storage | Is the workload economically sustainable? | Whether user latency is acceptable. |
The useful unit is a comparison: this prefix versus other prefixes, p99 versus median, before publication versus after, queue age versus completion rate. Averages flatten the shape that caused the incident.
An incident trace: free space, hot metadata
Return to the slow catalogue. The on-call engineer records the following:
| Observation | Value | First interpretation |
|---|---|---|
| Storage capacity | 70% free | No immediate byte-capacity emergency. |
| Direct object read | 40 ms p95 | Bulk data path is not visibly slow. |
/courses/new listing |
3.8 s p95 | The problem is tied to a namespace operation. |
| Metadata requests | 30× normal | Catalogue publication changed request shape. |
| Metadata queue age | rising from 5 ms to 2.5 s | Requests wait before metadata work completes. |
| Cache hit rate | fell from 97% to 61% | More requests reach the metadata authority. |
| Disk throughput | moderate | Low bulk throughput does not contradict metadata saturation. |
The input was a catalogue publication. The transition was a cache-miss and listing burst directed at one metadata path. The intermediate state is the growing metadata queue. The output is slow directory browsing while direct known-object reads stay normal.
The naive failure contrast is adding more data capacity. Extra free bytes do not shorten a metadata queue. A more relevant mitigation might cache completed catalogue pages briefly, spread publication reads over time, change pagination, or move a hot namespace range. Each has a trade-off: a cached catalogue may be stale, pacing delays visibility, pagination changes user behavior, and repartitioning costs operational work. The correct choice depends on the product promise.
Check: Why is “disk utilization is only 25%” weak evidence against a storage incident here?
Think first, then reveal.
Answer: The slow operation is a metadata-heavy listing, not a bulk byte transfer. The busy resource can be a namespace service, cache refill path, lock, or queue while disks have ample capacity and low throughput. Measure the path that the affected operation actually uses.
Maintenance work uses the same headroom
User requests are not the only work in a storage system. Writeback moves dirty pages. Replicas catch up. Backups read and copy data. Scrubs verify it. Compaction or cleanup rewrites it. Cache refill revisits an origin. These jobs are often correct and necessary, but they share queues, bandwidth, CPU, and capacity headroom with foreground work.
Imagine the catalogue incident happens during a replica rebuild. The maintenance backlog rises, tail latency rises, and the team sees retries from the metadata client. A simple signal sequence is:
maintenance backlog grows
→ queue age grows
→ p99 operation latency grows
→ callers time out and retry
→ arrival rate grows further
The response should preserve the durable and recovery promises already chosen. Pausing, rate-limiting, or scheduling background work may protect foreground traffic. It may also extend a replica’s exposure window or delay a backup. There is no free “maintenance off” switch; state the risk that moves elsewhere.
Check: A replica backlog grows during a busy write period, but free capacity remains high. Which next signal best distinguishes “not enough bytes” from “catch-up cannot keep up”?
Think first, then reveal.
Answer: Compare replication ingest and apply rate with the write arrival rate, plus replica queue age and lag by position. High free bytes do not explain whether the replica is receiving, persisting, or applying history quickly enough.
A small diagnostic loop
Use this loop instead of jumping from symptom to product setting.
- Name the affected operation and user promise: listing, read-after-write, upload completion, backup, or recovery.
- Trace the path it uses: cache, metadata, queue, device, network, replica, or remote service.
- Choose two competing causes and one signal that separates them.
- Apply the smallest reversible mitigation that protects the promise.
- Verify with the original symptom and the boundary signal, including tail latency and errors.
For the catalogue, “metadata hotspot” and “remote object latency” are competing explanations. Listing latency, metadata queue age, cache miss rate, and direct-object read latency separate them. A dashboard with only free bytes cannot.
Trade-offs and limits
Observability improves diagnosis, but it costs collection, cardinality control, storage, and attention. Per-key metrics can reveal a hot prefix, yet unbounded labels can overload the monitoring system. Caching a hot listing reduces authority pressure, yet introduces a freshness trade-off. Throttling maintenance protects users, yet delays redundancy, cleanup, or recovery progress.
Signals also do not automatically give causes. A high queue can be caused by a slow downstream service, an upstream burst, or a lock. A retry spike can be a symptom or an amplifier. Use a signal to decide the next measurement, not to announce certainty.
Watch these operational boundaries deliberately:
- p50, p95, and p99 latency by operation, not only an average;
- request and error rates partitioned by meaningful path or workload;
- cache misses and refill duration alongside origin load;
- queue age, completion rate, and retry rate together;
- metadata rate and small-object count alongside byte capacity;
- replication, writeback, checkpoint, backup, and compaction backlog;
- capacity headroom and cost before a maintenance emergency.
Common confusions
Confusion: Free bytes mean storage is healthy
Why it is tempting:
Space charts are easy to read and a full volume is a visible failure.
Better model:
Bytes are one capacity dimension. Metadata rate, hot keys, queues, replicas, and maintenance may be exhausted while byte capacity is comfortable.
Confusion: The average latency is the user experience
Why it is tempting:
An average is compact and usually stable.
Better model:
Queues and hotspots create tails. Inspect percentiles and the affected operation or partition; a good median can coexist with a painful p99.
Confusion: Maintenance is separate from production traffic
Why it is tempting:
Backup, compaction, and replication have their own job names.
Better model:
They consume the same paths and headroom. Treat their backlog and rate as first-class inputs to foreground latency and durability risk.
Practice: choose the next discriminating signal
At 18:00, the platform’s video uploads begin timing out. Storage bytes are 55% used. Network RTT is stable. Client retries doubled, upload-worker queues are growing, and the object service reports a rising throttle rate. Meanwhile, a lifecycle migration is copying old media to a new class.
State the likely pressure path, two safe actions, the trade-off of each, and the evidence you would use to decide whether the actions worked.
Model answer: The likely path is upload-worker queue → object-service request/throttle boundary, with lifecycle migration competing for request capacity. First, rate-limit or pause the migration; this protects new uploads but delays lifecycle completion and may keep old-class cost longer. Second, cap client retries and use backoff; this reduces self-amplified traffic but may make completion slower for individual uploads. Verify upload p95/p99, worker queue age, throttle and retry rate, object-service completion rate, and migration backlog. Free byte capacity and stable RTT are useful context but do not challenge the observed request-rate pressure.
Resources
- [BOOK] Designing Data-Intensive Applications — Focus: Relate workload shape, amplification, replication, and operational trade-offs.
- [DOC] Amazon S3 performance guidelines — Focus: Use a concrete object-service example to distinguish request pressure from stored capacity.
- [REFERENCE] Google SRE Workbook — Focus: Practice turning a user symptom into service-level indicators, saturation evidence, and a safe response.
Key Takeaways
- Capacity, metadata rate, request concentration, queues, maintenance, and cost describe different storage pressures.
- A hotspot can make one operation slow while free capacity and average throughput look normal.
- Operational diagnosis connects a user symptom to a path, chooses discriminating signals, and verifies a mitigation’s trade-off.