Making Rollback Possible: The Changes That Cannot Be Undone

Blue-green promises an instant switch back. Canary promises automatic rollback. Feature flags promise a kill switch. All three rest on the same premise: the previous version can be made to serve traffic again, and doing so restores correct behaviour.

That premise is about your changes, not your tooling, and it fails quietly.

A rollback is only a rollback if two things hold. The old code must still be able to run, meaning it does not depend on something the new version removed. And the old code must still be correct against the current state of the world, meaning the data it reads is in a shape it understands.

Key idea: deployment platforms give you the ability to run the old artifact again. Nothing about that guarantees the system returns to a working state, because the artifact is only half the system. The data is the other half, and it does not roll back with the code.

Most rollback failures are data failures, which is why this lesson is mostly about schemas.

Code is replaceable. Data is accumulated, shared between versions, and changed by every request that arrives during the deploy.

The general rule that falls out: a schema change is safe if the other version of the code, old or new, continues to work against it. Any change that fails that test is not one deploy, it is several.

The standard pattern for making a breaking schema change safely is called expand and contract, or parallel change. It converts one unsafe change into a sequence of individually safe ones, each of which can be rolled back on its own.

Taking the rename as the example, the phases are:

Expand. Add the new column. Do not remove the old one. This is additive, so old code is entirely unaffected and this deploy is trivially reversible.

Migrate. Deploy code that writes to both columns and reads from the old one. Backfill existing rows. Both versions of the code still work, because the old column remains authoritative.

Switch. Deploy code that reads from the new column while still writing both. Now the new column is authoritative, and rolling back to the previous step is safe because both are still populated.

Contract. Once you are confident, stop writing the old column, then drop it in a later deploy.

Key idea: the cost is four deploys instead of one, spread over days. What you buy is that at no point does a rollback break anything, and every intermediate state is a state where both code versions work. For a change to a table that matters, that trade is not close.

Expand and contract generalises into a small set of rules covering most changes that touch a contract, whether that contract is a database schema, an API or a message format.

The unifying principle is that additive changes are safe and subtractive ones are not, so every subtraction has to be preceded by a deploy that stops depending on the thing being removed.

In practice: the same logic applies to APIs and events, where the other version is not your previous deploy but somebody else's client. Removing a field from a response is exactly as breaking as dropping a column, and the mobile client running last month's build is the old version you cannot redeploy at all.

Some changes cannot be undone by any mechanism, and recognising them before shipping is a different skill from managing the reversible ones.

The recurring categories:

• Destroyed data. A dropped column, a deleted row, a truncated field. Once the bytes are gone, rollback restores the code that expected them and nothing more.
• Externally visible side effects. Emails sent, payments captured, webhooks delivered, messages posted, orders dispatched. The world outside your system has already reacted.
• Irreversible third-party state. A record created in a partner system, a DNS change that has propagated into caches you do not control, a published package version.
• Format changes applied in place. A migration that rewrote every row into a new encoding, where the original encoding is no longer recoverable from what remains.

The engineering responses are different in kind from rollback:

• Make them reversible where you can. Soft-delete instead of delete, so restoration is an update. Keep the original alongside the transformed value.
• Delay the irreversible part. Batch outbound side effects behind a short delay or a flag, so a bad deploy is caught before the emails go out.
• Guard them explicitly. A one-way door deserves a stricter release process, a smaller canary, a human confirmation, and rehearsal against a copy of production.

Gotcha: the most expensive incidents in this category are the ones where the deploy was fine and the migration was wrong. Rollback works perfectly, the code is correct, and the data it reads has been irreversibly mangled by a script that ran once.

Rollback is not always the right response, and treating it as the only one leads to bad decisions under pressure.

Rolling forward means shipping a fix rather than reverting. It is correct when the previous version cannot run against the current data, when the bug is small and understood, or when reverting would undo other changes that are fine and needed.

The honest comparison:

In practice: the useful default is to roll back first and diagnose afterwards, precisely because certainty is worth more than elegance while users are affected. Rolling forward is right when rollback is genuinely unavailable, and the failure mode to guard against is a team that chooses it because reverting feels like an admission, then ships a hurried fix that makes the incident longer.

The decision is much easier when it has been made in advance, which is what a documented rollback plan per risky change actually is.

Everything in this course reduces to one habit, applied before the change ships rather than during the incident.

If this is wrong, how do we get back, and how long does it take?

The answer sorts changes into three kinds, and each deserves different treatment:

• Trivially reversible. Stateless code changes behind existing contracts. Deploy them routinely, roll back automatically, spend no ceremony on them.
• Reversible with care. Schema and contract changes. Use expand and contract, and accept several deploys instead of one.
• Not reversible. Data destruction and external side effects. These need rehearsal, a backup verified by restoring it, a narrower rollout, and someone who has agreed to be responsible.

The course's through-line: deployment strategies control exposure, feature flags control release, and compatibility discipline controls reversibility. The first two are tooling you can buy or configure. The third is a property of your changes that no platform can provide, and it is what decides whether the first two actually work when you need them.

Key idea: the goal was never zero failed deploys, which is unattainable and would only be achieved by shipping nothing. It is that a failed deploy is a five-minute inconvenience rather than an incident, and every practice here exists to move failures from the second category to the first.