SLOs and Error Budgets: Turning Reliability Into a Number

The vocabulary comes from Google's Site Reliability Engineering practice, and the three acronyms are constantly conflated, so pin them first.

Definition: an SLI, service level indicator, is a measurement: the fraction of good events over total events. An SLO, service level objective, is your internal target for that measurement: what fraction is good enough. An SLA, service level agreement, is a contract with a customer, with money attached when it is missed.

The order matters. The SLI is engineering: what can we measure that tracks user happiness? The SLO is judgement: how good must it be? The SLA is business: lawyers set it below the SLO so that the contract breaches after the internal alarm, not before.

A well-formed example: SLI, the proportion of checkout requests answering successfully in under 500 ms, measured at the load balancer. SLO, 99.9 percent of them over a rolling 30 days. SLA, 99.5 percent monthly, with service credits.

Everything else in this lesson is what happens once you take the SLO seriously, because its most useful property is not the target itself but the remainder: the fraction you are allowed to fail.

An SLO's complement is a budget. At 99.9 percent, the allowed failure is 0.1 percent of events, or, read as time, 0.1 percent of the window.

The chart is worth staring at because the scale is logarithmic in disguise: each added nine divides the budget by ten. Two nines permits a long evening of downtime a month. Three nines permits three quarters of an hour. Four nines permits four minutes, which is less time than most teams need to notice an incident, let alone fix it: four nines is not a monitoring target, it is an architecture requirement, automated failover or it does not happen.

And each nine multiplies cost. The step from 99.9 to 99.99 typically means redundancy across zones, automated remediation, deployment machinery that can roll back in seconds, and on-call discipline to match. The SRE tradition's uncomfortable question is not "can we afford another nine" but "would our users even notice it", because past a point the user's own connectivity fails more often than your service does.

Here is the conceptual move that made SLOs famous: treat the allowed failure not as a shameful margin but as a budget to spend on purpose.

A 99.9 percent SLO gives the team 43 minutes of badness per month. Releases spend it: every deploy carries risk, and a bad one burns budget. Experiments spend it. Load tests against production spend it. Infrastructure migrations spend it.

The budget converts a permanent political fight into an arithmetic problem:

• Budget remaining? Ship. Velocity is free when reliability is ahead of target. The budget is there to be spent; chronically underspending it means you are over-investing in caution.
• Budget exhausted? The feature train stops. Engineering effort shifts to reliability until the rolling window recovers. Crucially, this rule was agreed by everyone in advance, so enforcing it is not a negotiation with a product manager mid-quarter; it is policy executing.

Key idea: the error budget is a peace treaty between velocity and reliability. Product wants to ship; operations wants stability; the budget replaces their argument with a number both signed up to. That, not the arithmetic, is the invention.

The claim sounds like engineering defeatism until the mechanism is laid out, and then it is just economics.

First, the marginal user cannot tell. Somewhere past three or four nines, your unreliability disappears beneath the noise floor of the user's own phone network, laptop wifi and browser crashes. Nines added below the noise floor cost real money and buy nothing observable.

Second, the cost curve is convex: each nine costs roughly an order of magnitude more engineering than the last, in redundancy, in automation, in the on-call machinery to hit four-minute response times.

Third, and least appreciated: a perfect-reliability target forbids all change, because every deploy risks the streak. The zero-risk posture is itself a product decision, and almost always the wrong one, since users want the features too.

The budget machinery is only as honest as the indicator underneath it, and indicator design has real craft in it.

• Measure as close to the user as possible. A server that returns errors the load balancer never sees, or a load balancer that fails while servers stay healthy, produce opposite lies. Edge measurement catches both; client-side measurement catches the most but adds noise you do not control.
• Define good events, not good minutes. "The service was up" is a weak SLI because up is not the user's experience. Request-based SLIs, good requests over total requests, weight badness by how many users actually hit it, and work naturally when the service is degraded rather than down.
• Latency belongs inside the SLI. A request answered correctly in 30 seconds is a failure to the user. Fold the threshold in: good means correct and under 500 ms. One indicator, honestly capturing both axes.
• Cover the journeys, not the endpoints. Users do not care about your 200 rate on one route; they care that search, add-to-cart and checkout each work. A handful of journey-level SLOs beats fifty per-endpoint ones nobody owns.

The test of a good SLI: when it degrades, support tickets should be arriving; when support tickets arrive, it should have degraded. Divergence in either direction means the indicator is measuring the system's opinion of itself, not the user's experience of it.

SLOs also repair alerting, which in most organisations oscillates between silent and unbearable.

The repair is the burn rate: how fast the error budget is being consumed, expressed as a multiple. Burn rate 1 means failing at exactly the pace that exhausts the budget precisely at the window's end, which is, by definition, acceptable. Burn rate 14 means the month's budget will be gone in roughly two days.

Alert on burn, not on raw error percentages:

The two-window design, popularised by Google's SRE workbook, kills both classic alerting failures at once. The brief 30-second error spike that used to page at 3am no longer does: it burns negligible budget. The slow 0.2 percent error rate that old threshold alerts never caught, because it never looked dramatic, now files its ticket, because over a day it burns budget faster than sustainable.

The alert has become the question you actually care about: is this incident material to the promise we made? Blips answer no. Slow leaks answer yes. Thresholds on instantaneous rates cannot tell them apart; burn rates exist because of it.

The machinery only compounds if it runs as a loop rather than a launch.

A workable operating rhythm, distilled from the SRE literature and practice:

1. Publish each SLO with its owner and its dashboard: current attainment, budget remaining, burn trend. If finding this takes more than one click, the SLO is decoration.
2. Review monthly, briefly. Consistently missing the target? Either invest in reliability or lower the objective honestly; a target nobody defends corrodes every other number. Consistently exceeding it with budget to spare? Spend braver: ship faster, run the risky migration.
3. Pre-agree the exhaustion policy, in writing, with product leadership: what stops, who decides, what unblocks. The budget's authority comes entirely from this agreement having been made before it was needed.
4. Let incident review consult the budget. Post-incident, the interesting number is not minutes down but budget consumed: an incident that spent 60 percent of the month's budget justifies engineering follow-up that a 2 percent incident does not.

Teams that skip the loop end up with SLO theatre: dashboards nobody reads, targets nobody enforces, and alerting quietly reverted to thresholds. The mechanism is sound, but it is a practice, not an artefact, and the practice is the part that was ever hard.