How Profilers Work, and How to Read a Flame Graph

Every profiler is built on one of two mechanisms, and they are not variations of the same idea.

Instrumentation records every event. The profiler inserts hooks at function entry and exit, so it knows exactly how many times each function was called and how long each call took. The data is exact and complete.

Sampling interrupts the program at intervals, typically tens to hundreds of times a second, and records the current stack. It does not know how many times anything was called. It knows what the program was doing at a set of moments, and infers proportions statistically.

Key idea: sampling is the default for production performance work, and the reason is not accuracy but honesty. Its overhead is small enough that the program it measures still behaves like the program you deployed, which is the property that matters most.

Instrumentation changes the thing it measures, and the distortion is systematically biased rather than uniform noise.

Gotcha: this is why an instrumented profile and a sampled profile of the same program can disagree about which function is the problem, and why the sampled one is usually right about proportions. If the two disagree, suspect the instrumentation before suspecting the sampler.

The single most common profiling mistake is using a CPU profiler on a problem that is not about the CPU.

CPU time counts only moments when your code was executing on a processor. A thread blocked on a network response, a disk read, a lock or a sleep is not consuming CPU and is invisible to a CPU profiler.

Wall-clock time counts elapsed time regardless of what the thread was doing, so waiting shows up as exactly what it is.

The consequence is stark for typical application code. A request that spends 700 ms waiting on a database and 50 ms computing will look, in a CPU profile, like a 50 ms request whose hottest function is something trivial. Nothing in that profile mentions the database, because from the CPU's perspective nothing was happening.

In practice: decide which one you need from the symptom. If the machine is at high CPU utilisation, profile CPU time, because something is genuinely computing. If the machine is idle and requests are still slow, profile wall-clock time or use distributed tracing, because the answer is waiting and a CPU profiler cannot see it.

Most web application slowness is waiting, which is why so many first profiling attempts produce a confusing report that seems to show nothing wrong.

The mechanism is simple enough to hold in your head, and holding it is what makes the output interpretable.

A timer fires at a fixed frequency. On each tick, the profiler interrupts the program and walks the call stack, recording the full chain from entry point to the function currently executing. That stack is one sample. The profiler aggregates thousands of them.

The inference is statistical: if a function appears in 30 percent of samples, it was on the stack roughly 30 percent of the time. With enough samples that estimate is reliable for anything substantial, and unreliable for anything rare, which is the trade sampling makes deliberately.

Two consequences follow directly. A function that never appears might genuinely be cheap, or might just be unlucky and fast, so absence is weak evidence. And a very short-lived program produces too few samples to say anything, which is why profiling a five-millisecond function requires running it in a loop rather than once.

Definition: the sampling rate sets the resolution. At 100 hertz you get one sample per 10 milliseconds, so a 2-millisecond function appears in samples only sometimes, and its measured share converges to the truth only over many runs.

Thousands of stack samples are unreadable as text. The flame graph, created by Brendan Gregg in 2011 while he was debugging a MySQL CPU problem, turns them into a picture you can read in seconds.

The construction: each sampled stack is drawn as a column of boxes, one per frame, with the entry point at the bottom and the currently-executing function at the top. Identical stacks merge, so a box's width is the number of samples containing that frame.

The reading rules, which are worth committing to memory:

• Width is everything. A wide box was on the stack in many samples. Look for the widest boxes; those are where time is.
• The y-axis is stack depth, not time. A tall tower is a deep call chain, not a slow one.
• The top edge is what was actually running. The leaf frame is the function that was on-CPU at that instant, which is where CPU time is genuinely spent.
• Left-to-right position means nothing. Boxes are typically ordered alphabetically to make merging work. A box to the left did not run first.

Gotcha: that last rule is the one people break constantly, reading a flame graph left to right as a timeline and constructing a story about execution order that the data does not contain. If you need order over time, you want a trace, not a flame graph.

Flame graphs have a small vocabulary of shapes, and two of them answer most questions immediately.

A wide plateau near the top. One function, or a small group, occupying a large horizontal share with little above it. This is a genuine hot spot: the program spends its time executing that code. It is the best possible finding, because the target is unambiguous.

A wide base narrowing into many thin towers. Time is spread across many different call paths with no single dominant one. This usually means the cost is structural rather than local, too many operations rather than one slow operation, and the fix is architectural: fewer calls, batching, caching, or a different algorithm. Optimising any individual thin tower is Amdahl's law in miniature and will achieve nothing.

Two more patterns worth naming. A wide box for a framework or runtime function often means you are calling it far too often rather than that the framework is slow. And a recursive function appears as a repeating stack of identical labels, which is normal and only interesting when the tower is unexpectedly deep.

In practice: the differential flame graph is the underused variant. It renders two profiles' difference, colouring what grew and what shrank, and it turns "the release got slower" from an investigation into a glance.

A profile is only as good as what was running while it was collected, and three mistakes routinely produce confident wrong conclusions.

• Profiling on the wrong data. A development database with a thousand rows exercises different code paths, different query plans and different cache behaviour than production with ten million. The profile is real and describes a system nobody uses.
• Profiling without warmup. In runtimes with just-in-time compilation, the first executions are interpreted and unrepresentative. Class loading, connection establishment and cold caches all inflate early samples.
• Profiling the whole process when you care about one path. Background jobs, health checks and metrics exporters all appear in the profile and dilute the thing you were investigating.

In practice: profile in production if you can. Sampling profilers are cheap enough that continuous production profiling is now normal, and it eliminates the entire class of "the benchmark was not representative" errors at a stroke. Where that is not possible, the next best thing is a load test built from recorded production traffic against a production-shaped dataset.

With the tooling understood, the remaining question is what the shapes actually mean, which is a question about the small number of things that make software slow.