Images: A Filesystem You Can Address by Hash

An image is not a disk image or an archive of a filesystem. It is an ordered list of layers, each of which is a set of filesystem changes: files added, modified or deleted relative to the layer beneath.

At run time the layers are stacked and presented as one directory tree by a union filesystem, which merges them so the container sees a normal root while the storage remains separate per layer.

The stacking rule is simple. A file present in several layers takes its content from the topmost one containing it. Deletion is recorded as a marker entry, conventionally called a whiteout, that hides a file from lower layers without removing it, because lower layers are immutable and shared.

That immutability is the point. A layer, once built, never changes, which is what makes it safe for many images and many running containers to share the same one on disk.

So an image is closer to a stack of patches with a merged view than to a snapshot, and almost everything else about container tooling follows from that representation.

Every layer in an image is read-only. Starting a container adds one thin writable layer on top, and that is where every change made while it runs goes.

The mechanism is copy-on-write. Reading a file finds it in whichever layer holds it. Writing one copies it up into the writable layer first, then modifies the copy. The original in the lower layer is untouched, which is what lets fifty containers share one image while each has an independent view.

Two consequences dominate practice.

Container data is ephemeral. Removing the container deletes its writable layer and everything written to it. This is the single most common surprise for people starting out, and it is a design decision rather than an oversight: it is what makes containers disposable and replaceable.

First writes to large files are slow. Modifying one byte of a two-gigabyte file copies the whole file up before the write proceeds. A database storing data in the writable layer pays this repeatedly, which is one of several reasons databases use volumes instead.

The third lesson covers where data should go. The rule to carry now is that anything which must outlive the container must not be written into it.

Each layer is identified by a digest, a hash of its contents, and an image is identified by the hash of a manifest listing those digests.

That is the same construction as the lesson The Object Database: Git Is a Content-Addressed Store, applied to filesystem layers instead of files, and it buys the same properties.

Deduplication is automatic. Twenty images built on the same base share one copy of each base layer on disk and pull it once, because identical content has an identical digest.

Transfers are incremental. Pulling an image downloads only the layers absent locally, which is why a rebuild of the top layer of a large image pushes and pulls quickly.

Identity is verifiable. A digest names exactly one bit-pattern, so referencing an image by digest is a guarantee about what runs.

That last point has real operational weight, because tags are mutable and digests are not. A tag is a pointer that can be moved to a different image at any time, so deploying myapp:v2.1 today and tomorrow may run different code with no record of the change.

Deploying by digest removes the ambiguity, and it is what supply-chain controls and signing rely on. Tags are for humans; digests are what should appear in anything reproducible.

A build file is a list of instructions, and each one that changes the filesystem produces a layer. The build cache keys each layer on the instruction plus the content it consumes, so an unchanged step is reused rather than re-run.

The last box is the rule that governs build times, and it is why instruction order is a performance decision rather than a stylistic one.

Read the ordering above. package.json is copied and dependencies installed before the application source is copied. Since dependency files change rarely and source changes constantly, the expensive install step stays cached across almost every build.

Invert those two and every one-line source edit re-runs the full dependency install, because the layer above it changed.

That single reordering is the most valuable thing to know about writing build files, and it generalises: order instructions from least to most frequently changing. The cache is a prefix match, so everything after the first change is rebuilt.

The most common image-size mistake follows directly from immutability, and it defeats the obvious fix.

RUN wget https://example.com/big.tar.gz    # layer: +500 MB
RUN tar xzf big.tar.gz                     # layer: +500 MB
RUN rm big.tar.gz                          # layer: +0 MB, still 1 GB total

Deleting the archive does not shrink the image. The rm adds a whiteout marker in a new layer that hides the file, and the earlier layer still contains it. The image carries a gigabyte to ship a half-gigabyte payload, and the deleted archive is still extractable from the image by anyone who has it.

The fix is to keep the file's entire lifetime inside a single layer:

RUN wget https://example.com/big.tar.gz \
 && tar xzf big.tar.gz \
 && rm big.tar.gz                          # one layer, +500 MB

The security version of this is more serious than the size version. A secret written and then deleted in a later instruction is still present in the earlier layer, and anyone who pulls the image can read it. Deleting a credential in a subsequent step is not remediation, exactly as rewriting Git history is not.

Both cases are the same principle: in a content-addressed, append-only store, removal is not an operation that exists.

Building software usually needs far more than running it: compilers, headers, test frameworks, package caches. Shipping all of that is wasteful and increases attack surface.

A multi-stage build uses several FROM instructions and copies only the results forward.

FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN go build -o /app ./cmd/server

FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

The final image contains one binary. The Go toolchain, the source, the module cache and every intermediate artefact stay in the build stage and are never part of what ships.

The security benefit exceeds the size benefit. A minimal final image has no shell, no package manager and no interpreter, so an attacker achieving execution inside it has very little to work with. Distroless and scratch bases take this furthest, containing only the application and its direct dependencies.

The cost is debuggability. No shell means no exec into a running container to look around, and the answer is ephemeral debug containers that attach a toolbox to the target's namespaces rather than putting one in every image.

That trade is worth making deliberately: a debugging shell present in production images is present for attackers too.

An image is often described as guaranteeing that what ran in testing runs in production. That is true of the image, and building the same image twice is a separate and much harder problem.

Several things make a build non-deterministic.

Unpinned base images. FROM node:20 resolves to whatever that tag points at today. It moves.

Package managers. apt-get install curl installs the current version from the current index, which changes daily.

Network fetches. Anything downloaded during a build depends on what the server returns at that moment.

Timestamps. File modification times embedded in layers differ between builds, changing digests even when content matches.

So two builds of the same source at different times routinely produce different images, and the difference is invisible until something breaks.

The mitigations are ordinary discipline: pin base images by digest rather than tag, pin package versions explicitly, vendor dependencies where practical, and use a lockfile for everything that has one.

The honest framing is that images give portability, meaning this artefact runs the same everywhere, and reproducibility is a separate property you have to work for. Conflating the two is how teams end up unable to rebuild the artefact currently in production.

Every row is a consequence of two facts: layers are immutable and additive, and they are named by content.

That is the same pair of properties as Git's object model, and the same practical rules fall out of both. Deletion is not removal. Identity is a checksum. Sharing is automatic because identical content is one object.

What the format does not settle is what actually runs the thing. An image is a filesystem plus some metadata, and something has to fetch it, unpack it, perform the namespace and cgroup sequence from the previous lesson, and supervise the result.

That machinery is more layered than the single docker command suggests, and it is the next lesson.