Reconciliation: Declare the End State, Not the Steps

The lesson Containers From First Principles ends where a single host runs out: addresses that change on restart, no way to find a service, and no bridge spanning machines.

The obvious response is a program that runs the steps: start three copies here, register them there, watch for failures, restart what died. That is imperative orchestration, and it fails in a specific way. Every step assumes the state it expected, so an interruption halfway leaves the system somewhere the script cannot reason about, and recovery means enumerating partial states nobody anticipated.

Kubernetes takes the other approach. You submit a description of the desired end state, and never say how to reach it.

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          image: myapp@sha256:abc123

That says three copies of this image should be running. It does not say start, or check, or restart. Something else continuously makes it true, and that shift is the whole system.

A controller is a program running one loop forever.

loop:
    desired  = read the spec from the API
    actual   = observe the world
    if actual != desired:
        take one step toward desired

That is all of it. Every controller in the system has this shape, and there are many: one for deployments, one for replica sets, one for endpoints, one for nodes, one for jobs. Each watches a slice of the desired state and works on that difference alone.

The important property is that the loop never terminates and never assumes. It does not track what it did last time, and it does not need to. Each iteration re-observes reality and recomputes the difference from scratch.

So a controller interrupted at any point simply resumes. There is no partial state to recover, because there was no progress being tracked, only a comparison being re-run.

That is what makes the design robust rather than merely fashionable. Correctness does not depend on any action succeeding; it depends on the comparison being repeated.

The distinction that makes this work is borrowed from electronics and is worth stating precisely.

Edge-triggered means acting on transitions. A pod died, so react. This requires catching every event, because a missed one is a change nobody responds to.

Level-triggered means acting on the current value. Two pods exist, three are wanted, so create one. It does not matter how the count reached two, or whether any event announcing it was seen.

Kubernetes is level-triggered, and the consequences are large.

Missed events are harmless. A controller restarting after an outage does not replay history. It observes the world now and fixes the difference now.

Duplicate events are harmless. Processing the same notification twice recomputes the same difference and finds nothing left to do.

Ordering does not matter. There is no sequence to get wrong, because nothing depends on sequence.

That is why a Kubernetes cluster survives its own control plane being down. The workloads keep running, nothing is queued up, and when the controllers return they look at reality and continue.

Events in the system exist purely as an optimisation, to avoid polling. Correctness rests on the periodic re-observation, and events only make convergence faster.

Two architectural decisions are visible here, and both are what let the system be extended without being modified.

One writer. Only the API server touches etcd. Controllers, the scheduler and the kubelets never read or write storage directly; they go through the same API you do, with the same authentication and validation.

No component talks to another. The scheduler does not call the kubelet. It writes a node assignment into the API, and the kubelet, watching for pods assigned to it, notices. Every interaction is mediated by shared state rather than by messages between parties.

That is why adding a custom controller requires changing nothing. It watches the API like every built-in controller does, and it is indistinguishable from them.

The scheduler illustrates the pattern most clearly. It is not a special subsystem; it is a controller whose reconciliation is "pods with no node assigned should have one", and it can be replaced with your own.

The cost is that etcd is the single point everything depends on, and it holds consensus state using Raft, which the lesson Distributed Systems Fundamentals covers.

Every object carries two sections, and the split is the loop written into the data model.

spec is what you want. You write it; controllers only read it.

status is what is. Controllers write it; you only read it.

spec:
  replicas: 3            # you asked for three
status:
  replicas: 3
  readyReplicas: 2       # one is not ready yet
  observedGeneration: 7  # the spec version this status reflects

That last field matters more than it looks. observedGeneration says which version of the spec the status describes, which is how you tell whether a controller has caught up with a change you just made or is still reporting on the previous one.

Without it, reading a status after an update is ambiguous: healthy might mean the new version is fine, or that the old version was and nothing has happened yet. Nearly every "the deploy said it worked but the old code is running" confusion is this field being ignored.

The convention generalises. Any well-designed custom resource has the same split, and it is the reason a controller and a user never fight over a field: they write to disjoint parts of the object.

The loop demands one discipline from anything written into it, and violating it is the standard way custom controllers cause outages.

A reconcile function will be called many times for the same state. Watches deliver duplicate events, resyncs fire periodically, a controller restart re-processes everything it can see. There is no guarantee of exactly-once anything.

So the function must be idempotent: running it repeatedly against an already-correct world must do nothing.

The failure is subtle because it usually works. A reconcile that creates a resource without first checking whether it exists succeeds the first time. Under a duplicate event it creates a second. Under a restart it creates more. The bug appears as gradual resource multiplication, hours or days after deployment, with no error anywhere.

The discipline is to make every operation a statement about the end state rather than an action. Not create, but ensure-exists. Not scale-up-by-one, but ensure-count-is-three.

That is the same reframing as the whole system, applied one level down: describe the destination, not the step, and the code becomes safe to run at any time, any number of times.

The architecture's real payoff is that the mechanism is available to you on the same terms as to the built-in components.

A custom resource definition adds a new object type to the API. Kubernetes then stores, validates, versions and serves it exactly as it does a Deployment, and every existing tool works with it immediately because it is the same API.

A custom controller watches that type and reconciles it against reality. Pair the two and you have extended the system with a concept it never knew about.

The pattern is used heavily. A database operator defines a PostgresCluster type and a controller that provisions instances, configures replication, takes backups and handles failover. Declaring one is then the same kind of act as declaring a Deployment.

What makes this more than a plugin system is that the extension is indistinguishable from the core. There is no separate extension API, no reduced capability, and no different lifecycle. The built-in controllers are written against the same interfaces.

That is the strongest argument for the design. Kubernetes is less a container orchestrator than a framework for building control loops over declarative state, and container orchestration is the first application built on it.

The last two rows are where the cost lands, and they are the source of most operational confusion.

Submitting a change always succeeds, because all it did was write a document. Whether the world can be made to match is discovered later, by controllers, and reported in status. So a command returning successfully means nothing about whether the thing works, which is a genuinely different mental model from a deployment script that fails when a step fails.

The third row has a similar double edge. Automatic drift correction is exactly why a manual fix applied during an incident silently disappears: a controller observed a difference from the declared state and removed your change.

That is the system working correctly, and it is why the declaration, not the cluster, has to be the thing you edit.