Distributed Tracing and the Art of Throwing Data Away

A trace is built from spans, and a span is a small, rigorously structured record:

• Identity: a trace id shared by every span in the request, a span id of its own, and its parent's span id. These three fields are the entire tree structure.
• Timing: start time and duration, which is all a flame graph needs.
• Attributes: key-value context following semantic conventions, the HTTP route, the database system, the status code, so tools can interpret spans from any service uniformly.
• Events and status: timestamped annotations, an exception with its stack trace, and whether the operation succeeded.

Instrumentation libraries create most spans automatically at the boundaries that matter: incoming requests, outgoing calls, database queries. Hand-written spans are added where the automatic ones are blind, around a business operation or an expensive computation.

Key idea: a span is cheap to create and expensive to keep. Everything difficult about tracing is downstream of that asymmetry: the instrumentation is a solved problem; deciding what to retain is not.

The spans of one request are created on different machines by different processes that never talk to each other about it. What stitches them into one tree is context propagation: every outbound call carries the trace id and the calling span's id, most commonly in the W3C traceparent header, and the receiving service adopts them.

When propagation works, it is invisible. When it breaks, traces silently shatter into orphaned fragments, and the breakages cluster in known places:

• Queues and async work. The HTTP header convention does not ride a message bus by itself; the context must be written into message metadata and restored by the consumer, which instrumentation only sometimes does automatically.
• Thread pools and background jobs. Work handed to another thread loses the context unless the runtime's instrumentation carries it across.
• Proxies and legacy services. One hop that strips unknown headers cuts the tree at that point.
• Manually-built HTTP clients. A hand-rolled request that skips the instrumented client library propagates nothing.

The diagnostic signature is characteristic: services report spans, but traces terminate at the same boundary every time. The fix is rarely deep, an instrumentation gap at one hop, but finding which hop is why teams audit propagation end-to-end before they need it.

Now the uncomfortable arithmetic. Suppose a modest platform: 2,000 requests per second, each crossing 8 services, each service contributing an average of 5 spans. That is 80,000 spans per second, roughly 7 billion spans per day, each carrying attributes, events and timing.

Almost all of them describe the same thing. Success, 60 ms, nothing unusual. The information density of trace data is spectacularly low: the millionth healthy checkout trace teaches you nothing the first thousand did not.

So the industry samples, and the honest framing is that sampling is not a compromise, it is the design. The goal was never to keep every trace; it is to keep every interesting trace, where interesting means: errors, outliers in latency, rare routes, and a small steady background of normal traffic to serve as the baseline you compare the anomalies against.

The entire engineering question collapses into one decision: at what point in a trace's life do you decide its fate? Before it happens, or after you have seen how it turned out? Those are head and tail sampling, and they produce very different debugging experiences.

The two sampling strategies differ in one variable: what you know when you decide.

Head sampling at 1 percent keeps 1 percent of everything, including 1 percent of the errors: when an incident produces a hundred failing requests, you will hold traces for one of them, and the on-call engineer discovers the gap at the worst moment.

Tail sampling inverts this: keep 100 percent of traces containing errors, 100 percent above a latency threshold, and 0.5 percent of boring successes as the baseline. Its price is that every span must exist and be routed to a component that can see whole traces before the verdict, which is real infrastructure with real memory.

The OpenTelemetry Collector is where sampling policy, and most other telemetry policy, is actually enforced. Services export everything they produce to a local or gateway collector; the collector's pipeline decides what continues.

For tail sampling, the pipeline routes spans so that all spans of one trace arrive at the same collector instance, buffers them for a decision window, applies the policy rules, and exports the survivors to the backend.

The same pipeline is where teams enforce the previous lesson's rules with a different hat on: dropping high-cardinality attributes, redacting secrets that leaked into span attributes, routing security-relevant telemetry to a second destination, and converting between protocols for legacy backends.

The architectural point to retain: policy belongs in the pipeline, not in application code. A sampling rate hard-coded into forty services is forty deployments to change during an incident; the same rate in the collector tier is one configuration change. During a fire, being able to say "keep everything for the checkout service for the next hour" is the difference between debugging and guessing.

Sampling changes what your data can honestly claim, and the traps are quiet.

Two further honesty rules. First, sampled counts need scaling: if a kept trace represents 100 unsampled ones, tooling must multiply accordingly when counting, and mixed policies make that scaling per-trace, which is why span metrics are best computed before sampling, in the collector. Second, publish the sampling policy where engineers see it. A developer who searches traces for a customer's failed request and finds nothing must be able to distinguish "it did not error" from "it was sampled away", and only the policy tells them which.

Synthesising the trade-offs, the configuration that serves most platforms well:

1. Errors: keep everything. Failed traces are the incident record; their volume is low by definition, unless everything is on fire, in which case you want them most.
2. Latency outliers: keep everything above a threshold set from your SLO, since these are the traces that explain the p99.
3. Baseline: a small percentage of healthy traffic, enough that any route has recent examples, because debugging needs a healthy specimen to compare the sick one against.
4. Rare routes: boost or keep fully. A percentage of almost nothing is nothing; low-traffic endpoints deserve floor rules.
5. Compute span metrics pre-sampling in the collector, so rates, error counts and duration histograms describe the true population while the trace store stays curated.
6. An incident override, one configuration flip that suspends sampling for a named service, with a timer so it cannot be forgotten on.

Small systems should skip all of it: below tens of requests per second, keep everything and spend the saved complexity elsewhere. Sampling is a scale tax, and paying it before you owe it is its own failure mode. The signal that you owe it is the trace bill or the collector's memory, not fashion.