Why Deploys Break Things, and the Strategies That Answer It

A deploy replaces a running system with a different one while people are using it. Nothing else in normal operations changes so much at once, on purpose, with users attached.

The failure modes are not exotic, which is what makes them worth enumerating:

• The new version is simply wrong. Tests passed, production disagreed, usually about data or scale that no test environment had.
• The environment differs. Configuration, secrets, network policy, a dependency version, a resource limit that only exists in production.
• The transition itself breaks. Old and new run simultaneously for a period and disagree about a data format, a cache key, or an API contract.
• Load arrives before readiness. Traffic reaches instances that are up but not yet warm, connected, or migrated.

Key idea: deployment strategies are not about deploying faster. They exist to control one variable, how many users encounter a bad version before somebody notices, and to make undoing it cheap. Everything in this course follows from taking that variable seriously.

The instinct that follows from all this is to deploy less often. The evidence says the opposite.

The intuitive model is a dial: ship faster and break more, ship slower and break less. The DevOps Research and Assessment programme, whose findings were popularised in Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, measured this across tens of thousands of professionals and found the dial does not exist.

Their four metrics split cleanly into two pairs:

The finding that matters is that these do not trade against each other. The highest-performing teams score well on all four simultaneously, deploying on demand with short lead times and low change failure rates, while low performers are slow and unreliable at once.

The mechanism is not mysterious once stated. Deploying rarely means each deploy carries months of accumulated change, so it is large, poorly understood, and hard to attribute when it breaks. Deploying often means each deploy is small, its blast radius is one change, and the fix is obvious because only one thing moved.

Key idea: batch size is the hidden variable. Large infrequent deploys are risky because they are large, not because they are deploys, and the strategies in this lesson are what make small frequent deploys safe enough to be routine.

The named strategies differ in how the old version is replaced, and each is really an answer to how much exposure a bad version gets.

Recreate. Stop the old, start the new. Simple, and it means downtime plus every user meeting the new version simultaneously.

Rolling. Replace instances in batches. No downtime, no extra infrastructure, and both versions serve traffic during the roll. Rollback means rolling back through the batches, which takes as long as the deploy did.

Blue-green. Run two complete environments. Deploy to the idle one, test it, then switch all traffic. Rollback is switching back, which is close to instant, and the cost is running double the infrastructure during the deploy.

Canary. Route a small share of traffic to the new version, watch, then widen. Smallest exposure and the only strategy that finds problems using real traffic before most users see them, at the cost of the most machinery.

Blue-green sits at 100 percent for a reason worth noting: the switch is all-or-nothing, so its advantage is not smaller exposure but a much faster exit.

The choice is usually settled by three properties of the system rather than by preference.

The first is whether you can afford to run two full environments at once, which is what blue-green requires and what makes it expensive for large stateful systems and trivial for small stateless ones.

The second is whether traffic can be split by proportion, which canary needs. That is straightforward behind a load balancer or service mesh and awkward for a mobile application, a desktop binary, or anything where the client decides which version it runs.

The third is how quickly a fault would show up in metrics. A crash appears in seconds. A subtle data corruption or a slow memory leak may take hours, and no canary window is long enough to catch what only manifests at month end.

In practice: most teams end up with rolling as the default for routine changes because it needs no extra infrastructure, blue-green where instant rollback justifies double capacity, and canary reserved for changes to the highest-risk paths. Mixing by risk is normal; picking one strategy for everything is usually paying for machinery you do not need or accepting exposure you did not have to.

Every zero-downtime strategy has the same consequence, and it is the one that causes the subtlest bugs: for a window, two versions of your code are running at once against the same data and the same dependencies.

That window forces a compatibility requirement most teams discover the hard way.

Key idea: zero-downtime deployment is not a property of your deployment tool. It is a property of your changes, and the requirement it imposes is that any two adjacent versions must be able to run simultaneously. Every change that violates that must be split into steps that do not.

One practice removes an entire class of deployment failure, and it is the one that answers the "the environment differs" failure mode from the start of this lesson.

Build the deployable artifact exactly once, then promote that identical artifact through every environment. The build that ran in continuous integration is the container image or package that reaches staging, and the one that reaches production is the same bytes again.

What this rules out is the failure where a rebuild produces something subtly different from what was tested: a dependency that resolved to a newer patch version, a build tool that changed, a base image updated between builds, a compilation flag that differs by environment. Those differences are invisible in a diff and fully capable of breaking production alone.

The corollary is that anything varying by environment must be configuration injected at runtime, not baked in at build time. Database endpoints, credentials, feature defaults and log levels are supplied to the artifact; they are not compiled into three different artifacts.

In practice: the test is whether you can say precisely which build is in production and reproduce it exactly. If production runs a build nobody can identify, then every deploy is partly a new experiment, and the strategies in this lesson are controlling exposure to something you cannot fully name.

Every strategy depends on one mechanism to know a new instance is ready, and misconfiguring it is the most common way a technically correct deployment still causes an outage.

The distinction that matters is between two questions a system can ask an instance:

• Is it alive? A process that answers is running. Failing this should cause a restart.
• Is it ready to serve? Dependencies connected, caches warm, migrations applied, thread pools initialised. Failing this should remove it from the load balancer without restarting it.

Conflating them produces two opposite failures. A readiness check that merely confirms the process responds sends traffic to an instance that is up but cannot serve, so the deploy looks successful while errors climb. A liveness check that depends on a downstream service turns that service's brief outage into a restart loop across your entire fleet, converting a partial failure into a total one.

Gotcha: the readiness check that returns 200 unconditionally is worse than no check at all, because it actively asserts a readiness the platform then trusts. If a deploy "succeeds" and errors spike immediately afterward, suspect the readiness check before suspecting the code.

With the strategies mapped, the next lesson takes the most careful of them and shows what it takes to make the promotion decision automatic rather than a person watching a dashboard.