Feature Flags: Separating Deploy From Release

A feature flag is a conditional that decides at runtime whether some behaviour is active.

if flags.enabled("new-checkout", user=user):
    return new_checkout(cart)
return legacy_checkout(cart)

That is the whole mechanism. Its significance is not the branch but what the branch decouples: the code ships in one act, and the behaviour is exposed in another, at a time somebody else can choose.

What that separation unlocks, in rough order of value:

• Deploy incomplete work safely. Half-built features can live in the main branch behind an off flag, which removes the long-lived branch and the merge that comes with it.
• Release to a subset. Internal staff first, then beta users, then a percentage, then everyone.
• Turn something off without deploying. The fastest possible remediation, faster than any rollback, because nothing needs to be rebuilt or restarted.
• Time a launch. Marketing announces on Tuesday; the code has been in production for three weeks.

Key idea: the previous lesson ended by separating deployment from release. The flag is the mechanism that makes that separation real, and it converts "can we ship this?" from a deployment question into a product one.

Treating all flags as one thing is the root of most flag problems, because they differ in how long they should exist and who owns them.

The distinction that matters most is between the first row and the last. A release flag is temporary scaffolding, and its presence after the rollout is finished is debt. A permission flag is not a flag in the same sense at all: it is an entitlement, a permanent product concept that happens to be implemented with the same machinery.

Gotcha: the failure that produces unmanageable flag systems is a release flag that quietly becomes permanent because nobody deleted it. Six months later nobody remembers whether it is safe to remove, both branches are load-bearing for someone, and the conditional is now a permanent fork in the codebase that no test plan covers.

Deciding the kind at creation time, and recording it, is what makes cleanup a routine task instead of an archaeology project.

Flags multiply the number of states your software can be in, and the arithmetic is unforgiving.

The mitigations are about limiting the space rather than testing it:

• Keep the number of concurrently live release flags small. The count, not the total ever created, is the thing to watch.
• Prefer independence. Flags whose behaviour depends on other flags are the ones that produce genuinely surprising states.
• Test the combinations that exist, not all of them: the current production configuration and the one you are rolling toward.
• Delete aggressively, which is the only mitigation that actually shrinks the space.

The operational flag deserves separate treatment, because it is the one that saves incidents rather than enabling launches.

When something is going wrong, the available responses have very different speeds. Deploying a fix requires writing it, reviewing it, building it and rolling it out, which is minutes at best and often much longer. Rolling back requires a deployment cycle in the other direction. Flipping a flag propagates in seconds, and requires no build, no restart and no deployment pipeline that might itself be broken.

That makes a small set of deliberate kill switches one of the highest-value pieces of operational engineering available:

• Around any expensive or fragile dependency, so a failing third party can be bypassed rather than allowed to exhaust your threads.
• Around non-essential work, recommendations, enrichment, analytics enrichment, so load can be shed while the core path keeps serving.
• Around anything new, for the first weeks of its life.

In practice: the flag that saves an incident is the one that existed before the incident. Adding a kill switch during an outage is a deploy, which is exactly the slow thing you needed to avoid. Deciding what the switches are is a design activity, and it belongs in the same conversation as the timeouts and retries around the same dependency.

Release flags need a lifecycle with an ending, and making that ending explicit is what prevents accumulation.

A flag is created with a kind, an owner and an expected removal date. It is deployed off, so the code is in production and dormant. It is enabled progressively: internal users, then a small percentage, then wider, with the same evidence-based promotion the previous lesson described. Once it is fully on and has been stable, the flag is removed, meaning the conditional and the losing branch are deleted from the codebase.

That last step is the one that gets skipped, so it helps to treat it as part of the work rather than as follow-up. The cleanup ticket is created when the flag is created, not when the rollout finishes, because at that point nobody is thinking about the flag any more.

Key idea: a flag's removal is part of shipping the feature, not a separate tidying task. A feature is not done when it is fully enabled; it is done when the alternative path no longer exists, because until then the codebase still contains and must still support the thing you replaced.

Three implementation properties separate a flag system that helps from one that causes incidents.

Consistency per user. If a flag's value is computed independently on each request, a user can see the new checkout, refresh, and see the old one, which is worse than either version. Bucketing must be deterministic, typically a hash of the user identifier and the flag key, so the same user always gets the same answer.

A safe default when the flag service is unreachable. A flag system is a dependency in the request path, and if the code cannot decide, it must still do something. Cache the last known values locally and fall back to a defined default rather than blocking or erroring.

Low evaluation cost. Flags are checked on hot paths, often several times per request. Evaluation should be a local lookup against cached configuration, not a network call.

Gotcha: the worst possible failure mode is a flag system whose outage takes down every service that consults it. That is an entire product failing because the thing that decides which code path to run became unavailable, which is a strictly self-inflicted dependency. Local caching with a documented default turns that from an outage into a period where flag changes do not take effect, which is a very different severity.

The techniques are not the point; the workflow they enable is.

With flags in place, long-lived feature branches stop being necessary. Work merges to the main branch continuously, hidden behind an off flag, which eliminates the multi-week divergence and the painful merge that ends it. That practice, trunk-based development, is difficult without flags and largely straightforward with them.

It also changes who can decide things. Enabling a feature stops being an engineering deployment and becomes a product action, which means launch timing, staged regional rollouts and per-customer enablement no longer require the pipeline at all.

The honest costs, stated plainly: every flag is a branch in the code and an untested combination in the state space; flags in the request path add a dependency and a small latency; and a flag system without lifecycle discipline becomes a source of permanent complexity that outlives the features that motivated it.

Key idea: flags trade a permanent, manageable complexity for the removal of a temporary, dangerous one. That trade is usually excellent, and it stops being excellent the moment the permanent part is left unmanaged. Which leaves one question the whole course has been circling: what actually has to be true for the rollback to work when you need it?