Pods, Services, and the Label That Joins Them

The smallest deployable unit is a pod, not a container, and the difference is deliberate.

A pod is one or more containers that share namespaces: the same network namespace, so they share an IP and can reach each other on localhost, and optionally shared volumes. They are always scheduled onto the same node and started and stopped together.

So a pod is essentially the group of processes that would have been one machine's worth of tightly coupled work.

The reason for the extra layer is that some things genuinely belong together and should not be one image. A sidecar pattern puts a proxy, a log shipper or a credential refresher alongside the application: separate images, separate lifecycles in the build sense, but co-located and sharing the network at run time.

The rule for deciding is whether the pieces must scale together. An application and its log shipper do; an application and its database do not, and putting them in one pod means scaling the database whenever you scale the app.

The important property for everything that follows: a pod is mortal and never restarted. It is created, it runs, it dies. Something else creates a replacement, and the replacement is a different pod with a different name and a different IP.

Nobody creates pods directly in practice, because a bare pod that dies stays dead. The higher-level objects are controllers layered on each other, each reconciling the layer below.

ReplicaSet ensures a count. Its loop is: count pods matching my selector, create or delete until it equals the target. That alone gives self-healing, since a node failure reduces the count and the next iteration restores it.

Deployment manages ReplicaSets to provide updates. Changing the image does not modify the existing ReplicaSet; it creates a new one and shifts replicas from old to new gradually.

That indirection is the whole reason Deployment exists. Two ReplicaSets exist during a rollout, one scaling down and one scaling up, which is what makes a rollback a matter of reversing the direction rather than redeploying anything.

StatefulSet is the variant for workloads needing stable identity: pods get ordinal names that persist across restarts, their storage follows them, and they start and stop in order. Databases need this; stateless services do not.

DaemonSet ensures one pod per node, which is the shape of agents: log collectors, monitoring, network plugins.

Each is the same loop with a different definition of what correct means.

None of those controllers holds a list of the pods it owns. They hold a selector, and membership is computed by matching labels.

# The pods carry labels
metadata:
  labels:
    app: api
    tier: backend

# Everything else selects on them
selector:
  matchLabels:
    app: api

A ReplicaSet counts pods matching its selector. A Service routes to pods matching its selector. A network policy applies to pods matching its selector. There is no registration step anywhere, and no object contains a list of another.

This is a loose coupling that behaves exactly like a database join, computed continuously rather than stored.

It is powerful and it has one sharp edge worth knowing before you meet it. Because membership is computed rather than declared, a label change silently changes ownership. Relabel a pod so it no longer matches its ReplicaSet, and the ReplicaSet sees its count drop and creates a replacement, while the relabelled pod keeps running, owned by nothing.

Overlapping selectors are the same hazard: two controllers both matching a pod will fight over it, each reconciling toward its own idea of correct, with no error raised because neither is doing anything invalid.

A Service solves the problem the first step created: pods are mortal and their addresses change, so nothing can point at one.

A Service is a stable name and a stable virtual IP that never changes for the life of the Service, regardless of what happens to the pods behind it.

The implementation is worth knowing because it explains the debugging. That virtual IP is not assigned to any interface anywhere. Pinging it may well fail even when the Service works perfectly. It exists only as a rule on every node that rewrites the destination to a real pod address.

The list of real addresses is maintained by another controller, the endpoints controller, running the same loop: find pods matching the Service's selector, and keep the list current.

The last two boxes are the operational payoff. Only pods reporting ready are in the list. A pod still starting up, or one whose readiness probe is failing, is absent, so traffic is never sent to it. Removing a pod from service is therefore not a routing change; it is a pod becoming unready, and the endpoints controller noticing.

Two probes look similar and do opposite things, and confusing them causes outages rather than inconveniences.

Readiness asks whether this pod should receive traffic. Failing it removes the pod from the Service's endpoints. The pod keeps running and is simply not sent requests.

Liveness asks whether this pod is broken beyond recovery. Failing it restarts the container.

The distinction matters most under load. A service that is slow because it is overloaded should fail readiness, shedding traffic until it recovers. If it fails liveness instead, it gets killed, its capacity leaves the pool, load redistributes onto the remaining pods, which then also fail liveness. That is a cascading failure caused entirely by a misconfigured probe, and it is a well-known way to turn a slowdown into an outage.

The guidance that follows is specific. A liveness probe should check only that the process is fundamentally broken, such as a deadlock, and should be generous. It should not check dependencies: a database being down does not mean this pod needs restarting, and restarting every pod because a shared dependency is unavailable makes recovery slower.

A startup probe handles slow initialisation, disabling the other two until the application is up, so a slow start is not mistaken for a hang.

Updating an image is one field change, and what follows is several controllers reacting in sequence.

The Deployment controller notices its spec differs from the ReplicaSet it manages, and creates a new ReplicaSet with the new template, scaled to zero.

It then shifts, bounded by two settings. maxSurge caps how many pods may exist above the target; maxUnavailable caps how many may be missing. Within those bounds it scales the new ReplicaSet up and the old one down, one increment at a time.

Each new pod is created, starts, and is added to the Service's endpoints only when its readiness probe passes. Each old pod is removed from endpoints first, then given a grace period to finish in-flight work before being terminated.

The whole sequence is reconciliation, so it is interruptible. Changing the image again mid-rollout does not queue behind the first: the controller creates a third ReplicaSet and starts converging on that instead.

And because the old ReplicaSet still exists at zero replicas, rolling back is scaling it up again rather than rebuilding or re-pulling anything. That is why rollback is fast, and it is the concrete benefit of the extra layer of indirection.

A Service is reachable inside the cluster. External traffic needs one of several options, and they differ in cost and capability rather than in kind.

ClusterIP, the default, is internal only. This is correct for anything other services call, and it is what most Services should be.

NodePort opens the same high-numbered port on every node, forwarding to the Service. It works and it is crude: an arbitrary port number and no TLS or routing.

LoadBalancer asks the cloud provider for a real load balancer pointing at the nodes. It works well and provisions one load balancer per Service, which becomes expensive quickly.

Ingress is the usual answer for HTTP. One load balancer fronts an ingress controller, which routes by hostname and path to many Services, and terminates TLS centrally.

rules:
  - host: api.example.com
    http:
      paths:
        - path: /v1
          backend:
            service:
              name: api-v1

Note the shape: Ingress is itself just declared desired state, and an ingress controller reconciles it into whatever proxy configuration it manages. Even the edge is a control loop.

Read the middle column and the apparent sprawl resolves. There is one pattern, and each object is a controller with a different definition of correct.

The joining mechanism is equally uniform: labels and selectors, everywhere, with membership computed continuously rather than registered. That is what makes the objects composable and what makes a stray label change confusing.

What none of this has addressed is the physical question. Something must decide which node each pod runs on, and something must stop workloads from exhausting the machines they share.

That is scheduling and resource management, and it is where the largest and most common misconfigurations live.