Memory Allocators in Production - jemalloc, Arenas & Fragmentation

By the end of this lesson, you will be able to...

• Trace how an allocation request moves through size classes, local caches or arenas, and process pages.

• Explain why RSS can remain above live application data without proving a memory leak.

• Design a small allocator investigation that distinguishes fragmentation, retained free memory, contention, and a genuine growing-live-data problem.

Idea in one sentence: An allocator makes small allocations fast by grouping and retaining memory for reuse, so a service's resident memory and latency reflect allocation policy as well as the bytes its application currently needs.

Core Insight

A catalog-cache service has 18 GiB of live cached objects after a quiet morning. At noon, a bulk update briefly creates many new values while old ones are replaced. The update completes, and the service again reports about 18 GiB of live objects. Its RSS, however, fell only from 34 GiB to 29 GiB.

The first explanation is understandable:

The service freed the objects, so RSS should have returned to 18 GiB. If it did not, the service must be leaking.

That conclusion is too quick. RSS is a measure of memory resident in the process, not a direct count of the application's current useful objects. Between application objects and resident pages sits an allocator. It rounds requests into size classes, keeps some freed objects or pages ready for reuse, coordinates concurrent threads, and decides when returning memory to the operating system is worth the cost.

The stronger model is not “high RSS is harmless.” It is:

Live data, allocator-owned reusable memory, fragmentation, and resident pages are different states. Measure which state grew before choosing a remedy.

That distinction matters directly to cache capacity. If a process reaches its memory limit because pages cannot be reused for the current allocation shape, it can evict data or fail allocations even though the logical cache dataset looks stable.

The Small Situation

Use a simplified cache worker with three actors:

cache worker -> allocator -> operating-system pages

cache worker: creates and frees encoded cache values during an update burst
allocator: turns byte requests into reusable blocks and obtains larger regions from the OS
OS: accounts resident pages in the process RSS

The worker does not normally ask the operating system for one page per value. That would make tiny allocations expensive. Instead, a general-purpose allocator keeps a hierarchy of reusable blocks.

This lesson uses a deliberately simplified teaching model:

application request
    -> size class
    -> thread/CPU-local cache or arena
    -> shared allocator structures
    -> pages obtained from the OS

Real allocators have more paths and version-specific details. The model is useful because it exposes the decisions that affect production behavior: rounding, reuse, concurrency, and release.

The Initial Model: free() Returns a Page to the OS

The initial model fits a large allocation that the allocator can release directly: an object is no longer needed, free() runs, and the process may give pages back to the operating system.

It becomes insufficient for a service that repeatedly allocates small and medium objects. Suppose the worker asks for 24, 41, and 70 bytes. An allocator commonly serves each request from a size class that is at least that large. The exact class boundaries below are illustrative, not a promise about any allocator:

The difference between requested and allocated bytes is one source of internal fragmentation. It is not an error; it is a cost paid for a fast, reusable common path.

When the worker frees the 41-byte object, the allocator can put its block in a local cache or an arena's free list. The next compatible allocation may reuse it without a system call or a heavily contended global lock. Returning it immediately to the OS would make the next burst slower and might cause more page faults or allocator work.

So free() usually means “this block is available to the allocator,” not “this exact amount has left process RSS.”

The Better Model: Reuse, Contention, and Return Policy

In plain English, an allocator is a traffic controller for memory. It gives a thread a suitable block now, preserves enough structure to reuse it later, and asks the operating system for more pages only when its existing supply cannot satisfy the request.

In the cache-worker scenario, many threads may create values at once. A single global free list would be easy to picture but can become a lock-contended meeting point. Allocators reduce that pressure by partitioning work.

Technical names:

• A size class groups similarly sized requests so a freed block can be reused quickly.
• An arena is an allocator-internal partition that can reduce contention by giving concurrent threads more than one allocation path. It is not the same thing as an application-level arena allocator.
• A thread-local or CPU-local cache keeps a small supply of reusable blocks near the allocating execution context.
• Purging or decay is the policy that eventually releases unused pages or makes them reclaimable instead of retaining them indefinitely.

The exact implementation differs. jemalloc presents itself as a general-purpose allocator focused on fragmentation avoidance and scalable concurrency. glibc exposes tunables for allocator arenas and per-thread caches. TCMalloc documents per-thread or per-CPU front-end caches that avoid locks for most small allocation and deallocation work.

These are facts about the implementations, not a recommendation to change one today. The durable mechanism is the same: less shared coordination improves a busy common path, but memory kept close to many threads or CPUs may be less immediately reusable elsewhere.

A Worked Trace: Why RSS Stays High After the Burst

The figures in this trace are synthetic. They show how to reason, not a measurement from a real service.

At 11:55, the cache worker is steady:

live cache objects:            18 GiB
allocator reusable/free space:  3 GiB
other resident process memory:  1 GiB
RSS:                           22 GiB

At noon, a bulk update overlaps old and new encoded values. Sixteen workers allocate and free objects of mixed sizes.

This trace exposes two different kinds of waste:

• Internal fragmentation: a live object occupies a larger block than its requested payload.
• External fragmentation or stranded reuse: enough free memory may exist in total, but not in a contiguous region, compatible size class, or allocator partition that can satisfy the next request efficiently.

The allocator can also retain free blocks deliberately. That is not necessarily fragmentation and it is not necessarily a leak. It is a reuse policy. The relevant question is whether the retained memory is helping the next workload enough to justify the capacity and memory-limit cost.

Check: The live cache dataset drops back to 18 GiB, but RSS remains at 29 GiB. Which conclusion is justified immediately?

Think first, then reveal.

Answer: RSS is above live application data, but the observation alone does not identify a leak. It is consistent with retained reusable memory, size-class or arena fragmentation, other mapped process memory, or a true leak. Compare allocator and application-level evidence before changing allocators or restarting the service.

What Evidence Separates the Explanations?

Start with a question that the available measurements can answer.

On Linux, /proc/<pid>/status reports VmRSS as resident set size, and /proc/<pid>/smaps_rollup provides more detailed accounting when the inexpensive summaries are not enough. These are process-level observations. They do not say which cache key, allocator arena, or application object owns the memory.

Pair them with allocator-specific telemetry where available. jemalloc's statistics and controls, glibc's supported tunables, and TCMalloc's statistics describe different surfaces; use the documentation for the library actually linked into the process. Do not infer a jemalloc diagnosis from a glibc process, or the reverse.

Cost, Limits, and Signals

Local caches and multiple arenas can improve throughput and tail latency by avoiding shared allocator work. Size classes make reuse cheap. Delayed return can prevent a bursty service from repeatedly faulting pages in and out.

Those improvements cost memory slack and diagnostic complexity. A process with many execution contexts can retain more cached memory. Aggressive release may reduce RSS but hurt the next burst. A lower arena limit can reduce the amount of separately retained memory but may increase contention. These are situated choices, not universal tuning rules.

The boundary is visible when the allocator's policy stops matching the workload:

• RSS or cgroup memory approaches the process limit while live data is stable.
• p99 rises with allocation rate or allocator CPU time.
• Mixed-size churn leaves free memory that cannot satisfy common new requests efficiently.
• A memory reduction experiment improves RSS but causes more page faults or a worse next-burst latency.

An allocator cannot fix an unbounded cache, a value representation that creates too much churn, or a genuine object-retention bug. It can only make a particular allocation pattern cheaper, more observable, or less wasteful.

Common Confusions

Confusion: “RSS and live application data are the same metric.”

Why it is tempting: Both are expressed in bytes and both rise under a cache workload.

Better model: Live data is one contributor to resident process pages. Allocator metadata, reusable blocks, fragmentation, code, mappings, and other process memory also matter.

Confusion: “If memory is freed, immediate RSS reduction is always better.”

Why it is tempting: Lower RSS appears to mean lower cost and more headroom.

Better model: Immediate release can make a soon-following burst pay for allocation and page population again. The right release policy depends on reuse timing and the memory limit.

Confusion: “Switching to jemalloc, glibc, or TCMalloc fixes memory.”

Why it is tempting: Each allocator has useful design strengths and tuning surfaces.

Better model: An allocator choice is a controlled comparison under the real allocation sizes, thread count, traffic pattern, memory limit, and latency objective. It does not cure a leak or an unbounded cache model by itself.

Practice: Plan One Safe Experiment

A cache service has 24 GiB of live entries and 38 GiB RSS after a daily refresh. A restart returns RSS to 26 GiB, but p99 is acceptable before the restart. The service has a 40 GiB cgroup memory limit and the next refresh runs six hours later.

Propose one experiment before changing the allocator. State what you would measure, what outcome would support retained reusable memory rather than a leak, and one risk of an overly aggressive release policy.

A good answer should mention:

• application-level live or in-use bytes alongside RSS and allocator allocated/resident evidence;
• the same refresh workload, observed through the six-hour interval or a representative replay;
• evidence that live bytes remain stable while retained memory is reused or eventually decays; and
• the risk that reducing retained pages lowers RSS but increases page faults or p99 during the next refresh.

Connections

The previous lesson made cache value shape visible. That is upstream of allocator behavior: many tiny values, mixed encodings, or bursty replacement patterns create the allocation-size and lifetime distribution the allocator must handle.

The next lesson moves from memory inside one node to placement across a cache fleet. A well-balanced routing scheme cannot rescue a node whose usable memory is consumed by the wrong value shapes or allocation behavior; conversely, allocator tuning does not decide which cache node should own a key.

Resources

• [DOCS] jemalloc — Focus: Start with its stated goals of fragmentation avoidance and scalable concurrency, then consult the linked manual for the version in use.
• [DOCS] glibc memory-allocation tunables — Focus: Check the exact arena and per-thread-cache controls available in your distribution before changing process settings.
• [DOCS] TCMalloc design — Focus: Trace its front-end caches, size classes, and the memory footprint trade-offs of per-thread and per-CPU modes.
• [DOCS] /proc/<pid>/status — Focus: Distinguish process-level VmRSS from an allocator or application-level live-byte measurement.

Key Takeaways

• free() makes a block reusable by the allocator; it does not promise an immediate reduction in resident process memory.
• Size classes, arenas, and local caches trade some memory slack for fast allocation and less shared contention.
• High RSS is evidence to investigate, not proof of a memory leak; compare live data, allocator state, and process-level memory.
• The right allocator or release policy depends on reuse timing, allocation sizes, concurrency, memory limits, and tail-latency goals.
• Change allocator settings only after a controlled before-and-after experiment defines the problem and the cost of the fix.