The Failure Modes the Design Creates

Applying a change returns immediately and successfully. All that happened is a document was written to etcd.

Everything after that is controllers converging, and convergence has no completion event. The system moves toward the declared state and reports where it currently is, and if it cannot get there it keeps trying rather than failing.

So a successful command carries almost no information. It means the desired state was accepted and validated, and nothing at all about whether it can be achieved.

That is a genuine change from an imperative deploy script, which fails when a step fails, and it means a deployment pipeline must wait and check rather than treat the apply as the deploy.

kubectl apply -f deploy.yaml         # always succeeds
kubectl rollout status deploy/api    # this is the actual wait

The first line is the request. The second polls status until the new replicas are available or a timeout expires, and it is the one that can fail.

Pipelines missing that second line report green while the rollout is stuck, which is the most common way a broken deploy is announced as successful.

Continuous reconciliation means the system is always comparing reality to the declaration and correcting differences. That includes differences you introduced.

Editing a running deployment during an incident, scaling something by hand, patching a config file inside a pod: all of these are drift, and a controller will remove them. If a deployment pipeline reapplies from source, it happens on the next run; if a continuous-delivery agent is watching, it happens within a minute.

This is the system working correctly and it is genuinely surprising the first time, because the fix works, the incident resolves, and then it recurs later with no apparent cause.

The consequence is that the declaration is the system, and the cluster is a projection of it. Changing the projection is not changing the system.

The discipline that follows is the one usually labelled GitOps: the declared state lives in version control, changes are made there, and an agent reconciles the cluster to it. The benefit is not process ceremony. It is that the cluster and the repository stop being able to disagree, which is the same argument the lesson Types as Propositions makes about specification and code being one artefact rather than two.

The emergency exception is to suspend the reconciler explicitly rather than to fight it.

Four states cover nearly every failure, and each points at a different layer.

Pending means no node was chosen. The cause is in the pod's events, and it is almost always insufficient unrequested capacity, an unsatisfiable constraint, or a volume that cannot be attached.

ImagePullBackOff is outside the cluster: a wrong tag, a private registry without credentials, or a rate limit.

CrashLoopBackOff is the one people misread. It does not mean the container cannot start; it means it starts and then exits, repeatedly, with the runtime backing off between attempts. The current logs are usually empty because the container has just started, and the useful output is from the previous attempt, retrieved with --previous.

Running but not Ready is the quiet one. The pod looks fine in a list, and it receives no traffic because its readiness probe is failing, which is why a service can have healthy-looking pods and no capacity.

The general rule for all four: describe before logs. The events attached to an object explain scheduling and lifecycle decisions, and logs only tell you about a container that got far enough to produce them.

The system is a set of independent loops observing shared state, so components disagree briefly and constantly. That is normal operation rather than a fault, and several everyday behaviours are it showing through.

Traffic to a terminating pod. Removing a pod from endpoints and telling the kubelet to stop it are separate controllers acting independently. A pod can stop before every node's routing rules have been updated, so requests arrive at something shutting down. That is why a preStop hook that sleeps briefly before shutdown is standard practice: it holds the pod alive while the removal propagates.

Stale reads. A pod list may not include something created a moment ago.

Status lagging the spec. Covered in the first lesson, and the reason observedGeneration exists.

Two controllers disagreeing. Both act on their own view, and the result converges but not instantly, and not always monotonically.

The design implication is that anything running on Kubernetes should tolerate being started, stopped and moved at any time, without notice. Graceful shutdown handling, retries with backoff, and no assumption of stable identity are not best practices bolted on; they are the requirements of the environment.

An application that cannot survive being killed mid-request will be killed mid-request.

The benefits are genuine and so is the bill, and it is worth stating without hedging.

Operational surface. A cluster is a distributed system with its own consensus store, certificates, upgrades and failure modes. Running one is a job, and running one badly is worse than not running one.

Debugging depth. A request now traverses ingress, service routing, several controllers' decisions and a container runtime. A failure can be at any layer, and the layers have separate tools and separate mental models.

Conceptual load. This cursus covers the core and omits a great deal: RBAC, network policies, admission control, storage classes, custom resources. A team adopting Kubernetes is adopting all of it eventually.

Configuration volume. A trivial service needs a deployment, a service, an ingress, config, secrets and probes. The YAML grows, and templating tools appear to manage the YAML, and those need managing too.

Cost. Control plane, spare node capacity for rescheduling, and the engineering time above.

The honest position is that this is worth paying when the problem is genuinely one of many services, many machines, and continuous change, and it is a poor trade for a handful of services on a few machines.

The common failure is adopting it for a problem it does not have yet.

Several situations are better served by something simpler, and recognising them early saves a great deal.

A few services on a few machines. A container runtime with a compose file, or a managed platform, does this with a fraction of the concepts. The threshold is roughly where manual placement stops being tractable, and that is further away than enthusiasm suggests.

Genuinely stateless request handling with variable load. Serverless platforms handle scheduling, scaling and availability entirely, and there is nothing to operate.

A single stateful system. Running a database on Kubernetes is possible and well-supported by operators, and it is a substantial amount of machinery to manage one thing whose failure modes you now own twice. A managed database is usually the better trade.

No operational capacity. Kubernetes moves complexity rather than removing it, from bespoke deployment scripts into a standard system. A team without someone able to operate that system has taken on the complexity without the benefit.

The positive case is equally specific: many services, many machines, frequent change, and a team that will benefit from a common substrate. There the standardisation genuinely pays, because everything speaks one API and the tooling ecosystem is enormous.

The environment imposes requirements, and applications that do not meet them fail in ways that look like platform problems.

Handle SIGTERM. Stop accepting new work, finish what is in flight, exit. The pod gets a grace period and is then killed. This is the PID 1 issue from the containers cursus, and it is why deploys drop connections.

Start fast, and tolerate restarts. A pod may be moved at any time. Slow startup makes every rollout slow and every recovery slower.

Externalise all state. The filesystem is ephemeral and the pod has no stable identity. Anything that must survive goes to a volume or an external service.

Configure from the environment. One image, many environments, as the images lesson established.

Log to stdout, structured. The platform collects it.

Expose readiness honestly. Report not-ready while dependencies are unavailable or the process is overloaded, and traffic will be routed elsewhere.

Tolerate a changing world. Peers appear and disappear, addresses change, and calls fail transiently. Retries with backoff, timeouts, and no assumption of stable topology.

That list is essentially the twelve-factor discipline, and it is not coincidence: those constraints were derived from the same observation that a process should be disposable, which is the whole premise the platform is built on.

Four lessons, one idea. Kubernetes is a framework for control loops over declarative state, and container orchestration is the first application built on it. Every object is a controller with its own definition of correct, every relationship is a label match computed continuously, and every guarantee is eventual.

That design is unusually robust: interrupted controllers resume, missed events do not matter, and the system survives its own control plane failing.

And its costs are exactly the mirror of its benefits. A system with no imperative steps has no moment of completion. A system that continuously corrects drift will correct yours. A system where everything is loosely coupled by matching has no error when a match is wrong.

The transferable idea is worth more than the tool: reconciliation converts a sequence of fragile steps into a repeated comparison, and that trade, robustness for immediacy, is available in far more places than cluster orchestration.