Flame Graphs - Visualizing Performance

By the end of this lesson, you will be able to...

• Read a standard flame graph as an aggregate stack view rather than a request timeline.

• Trace a wide stack family to the decision that creates its work instead of choosing a colorful or tall frame.

• Compare two like-for-like graphs as evidence for, or against, a performance hypothesis.

Idea in one sentence: A flame graph makes repeated stack cost visible, but its width is a clue to investigate—not an automatic order to optimize the frame on top.

Core Insight

For Atlas Shop, the previous lesson produced a CPU profile for slow GET /home cache misses. The profile table showed response construction and JSON encoding consuming many samples. The on-call engineer opens the flame graph and sees a wide orange plateau labelled encode_json.

The tempting reaction is immediate: replace the JSON encoder. It would be sensible if the encoder's own implementation were the avoidable source of cost. But a flame graph has not yet said that. It has shown that many sampled stacks reach the encoder. Those stacks may be wide because an upstream handler is creating too many rows or too large a response.

The correction is to read the picture as aggregated call paths. Find the wide population, follow its ancestry, and ask what upstream work makes that population repeat. Then use the same workload and profile type to test one bounded change.

The Production Symptom: A Useful Picture With a Dangerous First Impression

Atlas keeps the same cohort used in the CPU profile:

route: /home
cache_state: miss
language: en
campaign: active
response: 200

The p99 is still 600 ms before the change. A CPU flame graph is appropriate because the preceding profile already found active CPU cost in this cohort. A graph made from a different resource, such as blocked time, would answer a different question even if it looked similar.

In a standard flame graph, each rectangle is a stack frame. The vertical axis is stack depth: callers are below their callees. The horizontal width represents how often a frame occurred in the profiled stack population, so wide frames represent more aggregated cost for that profile. The left-to-right placement is a sorted layout, not elapsed time; original colors are used to distinguish adjacent frames, not to grade severity. These interpretation rules come from the original flame-graph documentation. Flame Graphs.

That gives the graph a precise job:

The Initial Model: Read It Like a Timeline or a Bar Chart

Two visual habits create most bad readings.

First: “the frame on the right happened last.” This is tempting because traces and flame charts commonly put time on the x-axis. A standard flame graph deliberately does not. It reorders stacks to merge equal frame sequences and make frequent paths visible. A flame chart is a different visualization that does retain time order.

Second: “the tallest or hottest-colored block is the worst bottleneck.” Height is call-stack depth. Color is normally decorative. A very deep, narrow stack can be cheap; a short, wide block can dominate the profile.

These shortcuts sometimes appear to work when a graph has one obvious wide frame. They fail as soon as several paths share a leaf. The missing model is inclusive ancestry: a frame's width means that it appears in many stacks, but the reason it appears may be in a caller below it or in work it invokes above it.

The Better Model: Reconstruct the Merged Stack Population

In plain English, a flame graph turns many captured stacks into one hierarchy. Equal prefixes are merged, and a frame is drawn as wide as the stack population that includes it.

In Atlas's simplified CPU capture, the following counts are illustrative samples, not timings:

2,400  home_handler -> build_catalog_payload -> materialize_variant_rows
3,800  home_handler -> build_catalog_payload -> encode_json
1,400  home_handler -> render_navigation -> encode_json
  600  home_handler -> authenticate -> verify_token

After merging, the same information has this shape:

home_handler (8,200 samples)
├─ build_catalog_payload (6,200)
│  ├─ encode_json (3,800)
│  └─ materialize_variant_rows (2,400)
├─ render_navigation (1,400)
│  └─ encode_json (1,400)
└─ authenticate (600)
   └─ verify_token (600)

The actual graph draws those counts as widths. encode_json is wide across two branches: 5,200 samples in total. Its width is real CPU evidence. It does not prove that changing the encoder is the first useful intervention. The wider build_catalog_payload branch shows a more specific question: why does this homepage create so much catalog work before it reaches the encoder?

The FlameGraph tooling takes stack traces and produces an interactive SVG; the display is a way to inspect the stack data, not a replacement for knowing which workload and profile produced it. FlameGraph repository.

A Worked Reading: Follow the Width to Its Cause

Read the Atlas graph in a fixed order.

The team checks the page contract and finds that the homepage renders one offer per product but the handler materializes every variant row. That observation earns the hypothesis: reduce the catalog representation before encoding. It does not earn the stronger claim that JSON encoding is irrelevant; it may still be the next target after the upstream payload is bounded.

The intervention is the same narrow change selected in the previous lesson: create only the fields and variant rows needed by the homepage. The team preserves the language, price, and availability contract before release.

Afterwards, it captures the same CPU profile over the same route, cache state, and campaign conditions:

So far, the graph has made the shared encoder leaf understandable in context. It has not declared every wide frame a design flaw, and it has not established time order within a request.

Compare Graphs Without Comparing Accidents

A before-and-after flame graph is useful only when its inputs are comparable. Keep the profile type, route or job cohort, traffic shape, duration, build version, and relevant cache state visible. Otherwise a smaller plateau might simply mean fewer requests or a warmer cache.

This comparison is especially valuable in continuous profiling systems, where a team can correlate profiles with metrics, logs, and traces and select an incident interval rather than relying on a one-off screenshot. Grafana Pyroscope documentation.

When reading a comparison, ask three questions:

1. Did the predicted wide stack family shrink?
2. Did a nearby path widen, suggesting the cost moved rather than disappeared?
3. Did the latency, error, and correctness signals for the same cohort support the expected effect?

If the first answer is yes but p99 is unchanged, the CPU work was not the whole boundary. Return to the resource question: queueing, a lock, I/O, or a dependency may now dominate. The next lesson investigates those waiting paths.

Trade-offs, Limits, and Signals to Watch

Flame graphs improve a team's ability to see recurring stack cost and discuss it precisely. They cost profile collection, interpretation time, and discipline about workload labels. The trade-off is good when a stack table is too dense to reason about but the team still has a concrete performance question.

They can still mislead:

• A CPU flame graph does not show where wall-clock waiting accumulates; use a profile that measures the suspected resource.
• A single wide leaf can be a symptom of upstream volume rather than a local implementation bug.
• Frame names can be missing, merged poorly, or obscured by runtime frames, so symbol quality affects confidence.
• A graph with different traffic, duration, or cache state is not a valid before-and-after comparison.
• Reducing a wide path can break a response contract if its output was required.

The boundary signal is a mismatch between the graph and the user outcome. If the expected branch shrinks while p99 does not, do not keep polishing the graph. Capture evidence for the remaining time. If a CPU graph is narrow but latency is high, investigate waiting rather than declaring the service fast.

Check: A standard flame graph shows a thin but very tall authenticate stack at the far right, and a short build_catalog_payload plateau that occupies half the width. Which path should you investigate first for CPU cost, and why?

Think first, then reveal.

Answer: Start with build_catalog_payload. In a standard flame graph, width represents the larger aggregate stack population; height is depth and right-side position is not late time. The authentication stack may still matter for a different question, but the graph does not make it the leading CPU hypothesis.

Practice: Review a Graph Claim Before It Becomes a Fix

A worker CPU flame graph for image jobs has a wide compress_image leaf. It appears under two callers: create_thumbnail for 70% of its samples and rebuild_preview for 30%. A product change has made thumbnails four times larger, while preview output is unchanged.

Write a small next investigation. State the hypothesis, the first bounded change you would test, the graph comparison you need, and one signal that would make you choose a different resource or path.

A good answer should mention:

• that the width makes image compression a real CPU path, but the caller split points first to enlarged thumbnail input rather than an automatic codec rewrite;
• a bounded thumbnail-size or preprocessing change that preserves the required image quality;
• a like-for-like CPU graph comparison for the same image sizes and worker cohort, plus job latency and output-quality checks; and
• a disconfirming signal such as an unchanged thumbnail branch, a newly wide preview branch, or high wall time with little CPU, which would shift the investigation toward waiting or I/O.

Connections

The previous lesson chose a profile type from a resource hypothesis. This lesson makes the selected profile legible as stack families without confusing it with a trace. The next lesson asks what to do when latency is mostly waiting: locks, queues, and I/O can dominate while CPU graphs remain quiet.

Resources

• [ARTICLE] Flame Graphs — Focus: Learn the meaning of width, stack depth, colors, and the non-temporal x-axis in standard flame graphs.
• [DOCS] FlameGraph — Focus: Inspect how folded stacks become interactive SVGs and why the input profile matters.
• [DOCS] Grafana Pyroscope — Focus: Relate continuous profiles to metrics, logs, traces, comparisons, and flame-graph views.

Key Takeaways

• A standard flame graph aggregates stack populations: width is the main cost signal, height is stack depth, and x-position is not time.
• Trace a wide frame through its callers and children before assuming the visible leaf is the first fix.
• A flame graph is meaningful only for the resource and workload that produced it; CPU and waiting graphs answer different questions.
• Compare like-for-like graphs with latency and correctness evidence so a narrower path is not mistaken for a successful optimization.
• When a CPU path shrinks but latency remains high, move to the next resource hypothesis rather than continuing to tune the same leaf.