Metrics, Logs, Traces: Three Signals, Three Cost Models

Monitoring and observability get used interchangeably, and the distinction is worth keeping sharp because it decides what you build.

Monitoring answers questions you wrote down in advance. Is the service up? Is the queue deeper than 10,000? Is the error rate above 1 percent? You knew the failure mode, so you pre-installed a question for it.

Observability is the property of being able to answer questions you did not write down in advance. Why are checkout requests from one country slow since Tuesday? Why does memory climb only on the pods running the new sidecar? Nobody pre-installed those questions. You answer them by interrogating telemetry that was collected without knowing what it would be for.

Key idea: monitoring is for the failures you predicted; observability is for the ones you did not. Production systems fail in unpredicted ways as a matter of routine, which is why teams that only monitor spend their incidents adding print statements and redeploying.

The raw material of observability is three signal types, and the rest of this lesson is what each one is actually for.

Every observability stack, whatever the vendor, is built from the same three signal types the OpenTelemetry project standardises.

The last column is the one people learn too late. Metrics are nearly free per event, because a counter increment does not grow storage, but every new label combination creates a new series to store forever. Logs cost per event, so they scale with traffic and verbosity. Traces cost per request across every service the request touches, which is why almost nobody keeps all of them.

Three signals, three different bills, and the next two lessons dig into the two expensive ones.

A metric throws information away on purpose. Ten thousand requests become one number per interval: a count, a sum, a histogram bucket. That compression is what makes metrics cheap to store for years, fast to query, and ideal for alerting: you can evaluate "error rate over 1 percent for 5 minutes" continuously against a tiny amount of data.

The same compression is the blindfold. Once aggregated, the individual request is gone. A metric can tell you that 2 percent of checkouts failed; it cannot show you a single failed checkout, because no single checkout exists in the data.

The practical grammar of metrics, standardised across Prometheus-style systems:

• Counters only go up: requests served, errors thrown. Rates are computed from them at query time.
• Gauges go both ways: memory in use, queue depth, connections open.
• Histograms record distributions cheaply by counting observations into buckets, which is what makes percentile estimates possible at all.

The skill of metric design is choosing what to count and, far more consequentially, what labels to attach, because labels are where the costs and the analytical power both live. That trade is the whole next lesson.

Logs are the oldest signal and the only one that preserves the individual event. When the question is about one specific request, one user, one payment, the answer lives in logs or nowhere.

Two practices separate useful logs from expensive noise:

• Structure. A log line that is a JSON object with fields, request id, user id, duration, outcome, can be filtered, grouped and joined. A prose sentence with values interpolated into it can only be regex-mined. Structured logging is the single highest-leverage logging decision a team makes.
• Wide events over chatty lines. Ten log lines scattered through one request handler are strictly worse than one line at the end carrying everything learned during the request. One event per request per service, as wide as needed, is the pattern the observability literature converges on: it is cheaper, and every field lands in the same record, so correlations are queryable.

Gotcha: log levels are not a cost strategy. Teams ship at info level, drown, and flip to warn, at which point the logs remember nothing useful. The durable levers are structure, one wide event per request, and retention tiers, not the severity dial.

In a system of one service, logs answer everything. In a system of forty services, the question "why was this request slow" has forty possible answers, and traces exist to pick one.

A trace is a tree. The root span is the request as the edge saw it; each child span is a timed operation performed on its behalf: a handler, a database query, a call to another service, which starts its own spans. Every span carries the same trace id, and the parent-child links are carried between services by context propagation, standardised as a header that rides along with every outbound call.

The result reads like a flame graph across machines: you see that a 900 ms checkout spent 40 ms in the gateway, 60 ms in the cart service, and 780 ms waiting on a single inventory query three hops deep. Without the trace, that conclusion takes an afternoon of correlating timestamps across four log systems. With it, the conclusion is one query.

The catch is cost and it is real: every request through every service emits spans. The industry's answer, sampling, is covered in lesson three, because doing it wrong quietly deletes the requests you most need to see.

For years each vendor shipped its own agents, formats and SDKs, and switching vendors meant re-instrumenting every service. OpenTelemetry, a CNCF project formed from the merger of OpenTracing and OpenCensus, ended that by standardising the boring parts:

• An API and SDK per language for producing metrics, logs and traces in one consistent way.
• A wire protocol, OTLP, that any backend can ingest.
• The Collector, a pipeline process that receives telemetry, transforms and filters it, and exports it to any number of backends.
• Semantic conventions: agreed attribute names, so an HTTP route or a database system is called the same thing in every service and every tool.

The strategic consequence is bigger than convenience: instrumentation became an asset you own rather than a vendor's hook into your codebase. You instrument once against the open standard, and the choice of backend, self-hosted or commercial, becomes a Collector configuration change instead of a migration.

That is also why every major vendor now accepts OTLP natively: once instrumentation is portable, the competition moves to storage, query and analytics, which is where it belongs.

The three signals overlap enough that teams misuse them constantly, usually by making one signal do another's job. A quick calibration:

The ordering is the point: aggregate to scope, trace to localise, log to explain. Teams without traces jump from metrics straight into log search across forty services, which is exactly the afternoon of grief the middle signal exists to delete.

Faced with three signals and finite time, where does a team start? The observability literature and the SRE tradition agree on a compact core.

For every service, emit the request-level basics, often summarised as rate, errors, duration: how many requests, how many failed, how long they took, as histogram metrics labelled by route and outcome. This alone powers alerting and the first minute of every incident.

For every request, one wide structured log event carrying the ids, the outcome, and the business context a future investigator will wish they had.

Across services, trace context propagation from the very first day the system has two services, because retrofitting propagation through a mesh of existing calls is the single most painful observability migration there is.

And for the system as a whole, a handful of resource and saturation gauges: queues, pools, memory, disk, the things that fill up before they fall over.

That set is smaller than most teams' actual telemetry, and more useful. The expensive failure mode is not missing data; it is unbounded data nobody designed, which is precisely the subject of the next lesson.