Where Time Actually Goes: The Six Usual Suspects

Before diagnosing anything, calibrate on the costs, because the ratios are what make some designs obviously wrong.

The spans between rows are the point. A network round trip is roughly a thousand times a memory access; a cross-region call is a million times an arithmetic operation.

Key idea: these ratios mean an optimisation's value depends almost entirely on which row it removes. Eliminating one database query is worth more than making a million arithmetic operations twice as fast. That is why the six causes below are ordered as they are: the ones that touch the slow rows dominate almost everything else.

Each cause has a recognisable signature, and learning the signatures means you can form a hypothesis before opening the profile.

For typical application code, the largest cost is not computation. It is the program sitting idle while something else works.

The waits, roughly by frequency: database queries, calls to other services, cache lookups that miss and fall through to a slower store, file and object storage reads, and message queue operations.

The signature. Wall-clock time is high and CPU utilisation is low. The machine looks bored while requests take seconds. A CPU profile shows nothing interesting, which is itself the diagnosis: if the CPU profile is boring and the endpoint is slow, the answer is waiting.

The fixes are structural rather than clever:

• Remove the call. Cache the result, precompute it, or discover it was not needed.
• Overlap the waits. Three independent 100 ms calls made sequentially cost 300 ms; issued concurrently they cost about 100 ms. This is often the single largest win available in request-handling code.
• Move the work off the request path. If the user does not need the result to continue, do it asynchronously.

In practice: the sequential-await pattern is endemic in modern codebases, because writing await a(); await b(); await c(); reads naturally and quietly serialises three independent operations. Scanning request handlers for consecutive independent awaits is one of the highest-yield reviews available.

The most common database performance bug is not a slow query. It is a fast query executed far too many times.

Gotcha: it scales invisibly. With 5 orders in development it is 6 queries and nobody notices. With 500 in production it is 501, and the endpoint that was fine last quarter now times out, with no code change to blame.

The general pattern generalises past databases: the same multiplication happens with per-item API calls, per-item cache lookups and per-item file reads.

In managed-memory languages, creating objects is cheap individually and expensive in aggregate, and the cost appears somewhere other than where it was caused.

The signature. CPU time attributed to garbage collection, periodic latency spikes that do not correlate with any particular request, and memory usage that climbs and drops in a sawtooth. Crucially, the pause hits whichever request happens to be running, so the slow request is rarely the one that allocated.

Where excess allocation typically comes from:

• String building in loops, where each concatenation creates a new string, turning a linear task quadratic in memory traffic.
• Defensive copying, where a collection is copied on every access to avoid mutation.
• Boxing, where primitives become heap objects to satisfy a generic container.
• Intermediate collections, where a chain of transformations materialises a full list at every step.

In practice: the fix is almost always to allocate less rather than to tune the collector. Reuse buffers, build strings with a builder, stream rather than materialise, and avoid copying what nobody mutates. Collector tuning is a real discipline and it is the second thing to try, because a collector cannot be configured out of collecting garbage your code insists on producing.

Concurrency is supposed to make things faster, and contention is the mode where it stops doing that and starts doing the opposite.

When many threads need the same lock, only one proceeds while the rest wait. Adding threads adds waiters, not throughput. Past a point, more concurrency reduces total throughput, because the coordination overhead grows while the protected section does not.

The signature is distinctive and worth memorising: throughput rises with concurrency, plateaus, then declines, while CPU utilisation is moderate and threads are blocked rather than running. The system is busy waiting for itself.

Contention is not only locks. Connection pool exhaustion, where every thread waits for a free database connection, is the same shape. So is a queue with one consumer, a rate limiter, or any shared resource with a fixed capacity.

Key idea: the fixes reduce the amount of shared state rather than making the lock faster. Shorten the critical section so the lock is held briefly. Shard the data so different threads touch different locks. Use immutable data or per-thread copies so nothing needs protecting. Replace coarse locks with fine-grained ones. Only after that does a faster lock implementation matter at all.

Two further causes appear once the obvious ones are gone, and both are about data movement rather than logic.

Memory access patterns. Modern processors are far faster than memory, so performance depends on cache hits, and cache hits depend on locality. Walking an array sequentially is dramatically faster than chasing pointers through a linked structure scattered across the heap, even though both are the same number of operations. The catalogue's CPU course covers the hardware mechanics; the practical consequence is that data layout is a performance decision, and contiguous structures beat pointer-heavy ones for anything traversed in bulk.

Serialisation. Converting objects to JSON, protocol buffers or database rows and back is pure overhead that produces no business value, and it is routinely 20 to 40 percent of a service's CPU time in profiles. It hides because it is a library call nobody wrote.

In practice: the serialisation fix is usually to do less of it rather than to choose a faster library. Return fewer fields, avoid deserialising a payload you then discard most of, cache the serialised form when it is stable, and stop re-serialising the same object at every layer boundary. Swapping libraries is a constant factor; sending less data is a structural change.

In practice you meet a symptom before you meet a cause, so the useful skill is running the mapping backwards.

Key idea: the top two rows resolve most investigations, and they are distinguished by one number anyone can check in a minute: CPU utilisation while the system is slow. Idle means waiting, busy means computing, and those lead to entirely different halves of this list.

With the cause identified and a fix applied, one question remains, and it is the one that decides whether the work holds: how do you prove the change actually helped?