File Systems and Distributed Metadata

By the end of this lesson, you will be able to...

• Separate a storage namespace from the metadata and data paths behind it.

• Trace how a file name becomes a concrete read from one or more storage locations.

• Recognize a metadata hotspot even when disks have free capacity and data nodes are idle.

Idea in one sentence: A path is a name, not the bytes; metadata turns that name into a current plan for finding, validating, and protecting the bytes.

Core Insight

A learner presses play on this video:

/courses/storage/lesson-01.mp4

The path looks like a direct address. It is tempting to imagine that the storage system follows the text and arrives at the video, as if the path were a street address painted on one disk.

Now one storage node is taken out of service. The video chunks are copied to new nodes. The path shown to the learner does not change.

How can the same path still work when the bytes moved?

There must be a layer between the name and the bytes. That layer knows what the name currently refers to, which chunks form the file, where healthy copies live, and which version the client may read. This information is metadata.

The first useful storage model is therefore not:

path -> bytes

It is:

name -> metadata -> data locations -> bytes

This separation makes storage flexible. Bytes can move, replicas can be replaced, and permissions can change without changing every user-facing name. It also creates a critical control path. A system may have fast disks and abundant space yet still feel slow because name resolution or metadata lookup is overloaded.

The Confusion This Concept Solves

Suppose the learning platform has these symptoms:

• storage nodes use only 40% of their capacity;
• bulk reads from a known chunk id are fast;
• opening a file by path is slow;
• listing a directory is even slower;
• the slowdown becomes severe when a course publishes thousands of tiny subtitle files.

Looking only at stored bytes suggests that the storage system is healthy. The disks are not full. The data nodes are not saturated.

The missing distinction is between two kinds of work:

• metadata work decides what data the request means and where to look;
• data work moves the requested bytes.

The user experiences both as “storage latency,” but they may use different services, caches, locks, queues, and machines. We cannot diagnose the request until we separate them.

Three Pieces of the Model

1. The namespace gives names a structure

A namespace is the set of names the storage system accepts and the rules for organizing them.

In a file system, the namespace often looks hierarchical:

/
└── courses
    └── storage
        └── lesson-01.mp4

The path is meaningful to people and applications. It supports operations such as lookup, create, list, rename, and delete. But it still does not contain the video.

Plain meaning:

The namespace tells us how to ask for something.

In this scenario:

/courses/storage/lesson-01.mp4 is the stable name used by the player.

Technical name:

The organized set of accepted names is the namespace.

2. Metadata connects the name to storage state

The metadata record may contain information such as:

file id:       F-204
type:          regular file
size:          734 MiB
version:       17
chunks:        C1, C2, C3
chunk C1:      nodes A and D
chunk C2:      nodes B and E
chunk C3:      nodes C and F
permissions:   course-readers

Real systems organize this information in different ways. A local file system may use directory entries, inodes, and extents. A distributed file system may use a metadata service plus placement records. The exact structures differ, but their job is similar: translate a logical name or identity into a valid access plan.

Plain meaning:

Metadata is data about how other data is named, located, interpreted, and controlled.

In this scenario:

The metadata record says that the path identifies file F-204, which version is current, and where its chunks can be read.

Technical name:

The lookup and update work around these records forms the metadata path.

3. The data path moves the bytes

After the client has a usable location plan, it can fetch chunks from storage nodes. This is the data path.

client -> node A: read C1
client -> node B: read C2
client -> node C: read C3
client: validate and assemble the video stream

The metadata service does not need to carry all 734 MiB through itself. It can answer the smaller control question —“where and under what rules?”— while data nodes handle the larger transfer.

This split is common because metadata operations and data transfers have different shapes. Metadata records are comparatively small, but they may require frequent coordination. Data transfers are large, but they can often be spread across many nodes.

A Worked Read Trace

Let us follow one request after chunk C2 has moved from node B to node E.

Starting state

namespace:
  /courses/storage/lesson-01.mp4 -> F-204

current metadata for F-204:
  version 17
  C1 -> A or D
  C2 -> E or H
  C3 -> C or F

stale client cache:
  C2 -> B or E

The stable path has not changed. The current placement has.

Step-by-step trace

The result is a successful read even though the bytes moved before the request.

Now compare the naive path. Suppose the client skips fresh metadata resolution and uses its stale cached location for C2:

client -> B: read C2
B -> client: chunk not found here

That failure does not prove that the video is lost. It proves that one location hint is stale. A reasonable client can invalidate the hint, resolve the metadata again, and retry against E or H.

This distinction matters. “The name exists,” “this location is current,” and “the bytes are readable” are related claims, but they are not the same claim.

So far, we have seen a full path: a name becomes a file identity, the identity becomes a current chunk map, and the chunk map becomes byte reads. The intermediate metadata state is what lets the stable name survive changing physical placement.

Why Small Files Create Large Metadata Pressure

Compare two workloads with roughly the same data volume:

Workload A: 1 large video
Workload B: 1,000,000 tiny subtitle fragments

Workload A may need one namespace entry and a manageable chunk map. Workload B needs one million identities, names, ownership records, versions, and placement decisions. It may also create far more lookup, list, create, and delete operations.

The raw bytes do not explain this difference. Object count and namespace-operation rate do.

This is why a system can have plenty of storage capacity and still suffer metadata pressure. The data plane may be waiting for work while the metadata path queues millions of small decisions.

Check: A dashboard shows low disk throughput, low data-node CPU, and high latency for list and open operations. Which path should you investigate first?

Think first, then reveal.

Answer: Investigate the metadata path first. list and open must resolve names and metadata before significant byte transfer begins. Low data-node activity is evidence against the data path being the first bottleneck.

Trade-offs and Limits

Separating metadata from data has clear benefits, but the split is not free.

A concentrated metadata authority

A concentrated authority can make namespace rules and updates easier to reason about. There is a clear place to decide whether a rename succeeded or which placement is current.

The trade-off is concentrated pressure and failure risk. If every client must ask one metadata service, that service may become a latency bottleneck or availability dependency.

Signals include metadata CPU, lookup latency, namespace-operation queues, lock contention, and error rates. Disk throughput alone will not reveal the boundary.

Partitioned metadata

Partitioning metadata spreads load across several owners. Different directories, file ids, or key ranges can be resolved in parallel.

The trade-off is more coordination for operations that cross partitions. A rename between two metadata owners may be harder than a lookup contained within one owner. Hot ranges can still appear if many requests target the same course or prefix.

Cached metadata

Caching location information reduces repeated lookup cost.

The trade-off is freshness. A cached location can outlive a rebalance or node failure. Clients need version checks, leases, invalidation, retry, or another rule for deciding when a cached answer is still safe.

This model also has limits. It does not tell us which storage contract an application should choose, how metadata is implemented on disk, or which consistency protocol coordinates several metadata replicas. It gives us the map needed to ask those later questions precisely.

Common Confusions

Confusion: The path contains the data location

Why it is tempting:

The same path works repeatedly, so it feels like a physical address.

Better model:

The path is a logical name. Metadata may map it to different physical locations over time while preserving the same identity.

Confusion: Metadata is optional descriptive text

Why it is tempting:

Photo tags and document labels are commonly called metadata, so the word can sound decorative.

Better model:

Storage metadata can be on the critical path. Identity, size, type, permissions, version, chunk layout, and placement may all be required before the bytes can be accessed safely.

Confusion: More replicas remove metadata pressure

Why it is tempting:

More byte copies improve read placement and fault tolerance, so they seem to improve every part of storage.

Better model:

Replicas create more possible data locations, but some authority still has to track which copies exist and which are healthy. Replication can increase metadata work even while it improves the data path.

Check: A client receives chunk not found from one node after a rebalance. Does that prove the file was deleted?

Think first, then reveal.

Answer: No. The client may have a stale placement hint. It should distinguish a failed location from a missing file, refresh metadata, and try a current replica before concluding that the file is gone.

Practice: Diagnose a Thumbnail Store

A media service stores eight million thumbnails, each about 30 KiB. Aggregate capacity is 70% free. During a new-course launch:

• requests for already-resolved object ids remain fast;
• requests by course path become slow;
• directory or prefix listings time out;
• storage-node throughput stays moderate;
• one course prefix receives most of the traffic.

Answer these questions:

1. Which work belongs to the namespace, metadata path, and data path?
2. Where is the most likely first bottleneck?
3. Which three signals would you inspect next?
4. Name one mitigation and its trade-off.

Model answer

The course path and prefix structure belong to the namespace. Resolving each path to an object identity, version, and placement belongs to the metadata path. Returning the 30 KiB thumbnail bytes belongs to the data path.

The first bottleneck is probably the metadata owner or cache for the hot course prefix. Fast reads by resolved id and moderate storage-node throughput make the data path a weaker first suspect.

Useful signals include path-lookup latency, metadata request rate or queue depth for that prefix, and metadata-cache hit rate. Lock contention, throttling, and per-partition CPU would provide additional evidence.

One mitigation is to partition or replicate the hot metadata range, but that adds routing and freshness complexity. Another is to cache path resolutions, but stale placement or version information then needs an explicit refresh rule. A good answer names both the benefit and the new coordination cost.

Connection to the Next Lesson

This lesson separated names, metadata, and bytes. The next lesson asks a different question: what operations should the application be allowed to perform on durable data?

Block, file, and object storage expose different answers. Their contracts determine who owns naming, how mutation works, and how much coordination the storage layer provides.

Resources

• [BOOK] Operating Systems: Three Easy Pieces — Focus: Use the file-system chapters to connect directories, inodes, allocation, and the path from a name to local blocks.
• [PAPER] The Google File System — Focus: Follow how a stable file name is resolved into chunk handles and replica locations while clients read data directly from chunkservers.
• [DOC] HDFS Architecture Guide — Focus: Compare the NameNode metadata role with the DataNode byte-transfer role.

Key Takeaways

• A namespace tells clients how to name data; it does not contain the bytes.
• Metadata turns a logical name into identity, version, policy, and a current location plan.
• The metadata path decides where and under what rules; the data path transfers the bytes.
• Object count and namespace-operation rate can overload metadata even when raw capacity and data-node throughput look healthy.
• Stable names survive changing physical placement only because metadata keeps the mapping current.