Why ML Systems Rot

In 2015 D. Sculley and nine co-authors at Google published Hidden Technical Debt in Machine Learning Systems at NIPS. Its most quoted observation is that the machine learning code is only a small fraction of a real-world system.

Around the box that trains a model sit configuration, data collection, feature extraction, data verification, resource management, process management, serving infrastructure, analysis tools and monitoring. Every one of those is ordinary engineering. Together they dwarf the model.

The paper's argument is not that this surrounding infrastructure is badly built. It is that machine learning erodes the abstraction boundaries software engineering depends on, so the usual techniques for keeping a large system maintainable work less well here. MLOps is the practice that grew up around exactly that problem.

The paper gives the effect a name: CACE, for Changing Anything Changes Everything.

A model learns weights jointly. Feed it features $x_1$ through $x_n$ and it finds a setting that works for that exact set. Now add a feature, remove one, or let one feature's distribution drift. The optimiser rebalances, and the learned importance of every other feature moves with it.

This is not a bug to be fixed. It is what joint optimisation means. But it destroys the property engineers rely on most: that you can reason about a component in isolation. There is no subset of an ML system you can change while holding the rest still, which is why a one-line change to a feature definition can require re-validating the whole model.

CACE has two named consequences in the paper.

Entanglement is the direct one: changing the learning algorithm changes which input signals it needs, which in turn forces changes to the infrastructure feeding it. The blast radius of a modelling decision reaches into systems the modeller does not own.

Correction cascades are the sneakier one. You have model A. It is nearly right for a new problem, so rather than retrain, someone learns a small model B that corrects A's output. Then C corrects B. Each layer is cheap and locally sensible. Collectively they create a stack where improving A can make the end-to-end result worse, because B and C were fitted to A's specific errors. Unpicking the stack later costs far more than training the right model once.

One feature edit does not stay local. It moves every weight, which moves the output, which reaches anything downstream that was fitted to the old behaviour.

A model writes predictions to a table, a topic or a file. Nothing stops another team reading them.

The paper calls these undeclared consumers, and they are the ML equivalent of what software engineering calls visibility debt. Because there is no record of who depends on a given output, there is no way to know what you break when you change it. Worse, a consumer can feed the prediction back into a system that eventually produces your training data, creating a hidden feedback loop where the model influences its own future inputs.

The fix is access control and declared interfaces, not diligence. If reading a model's output requires requesting it, the dependency graph exists. If the output sits in an open table, the graph is unknowable by construction.

Two system-level anti-patterns from the paper show up in almost every ML codebase.

Glue code is the supporting code written to get data into and out of a general-purpose package. Using an off-the-shelf library is cheap up front, and the glue that adapts your data to its expectations is where the mass ends up. The paper's suggested remedy is to wrap the black box in a common API so components stay swappable rather than letting the package's shape leak everywhere.

Pipeline jungles are what data preparation becomes when it grows by accretion: scrapes, joins and intermediate files added one at a time, each sensible on the day. Detecting an error in the resulting graph is hard, and recovering from one is harder. The paper is blunt that the fix is redesign, not more incremental additions.

Software engineering has good tools for code dependencies. A compiler and a linker will tell you what breaks. Nothing equivalent exists by default for data.

The paper's warning is about unstable and underutilised data dependencies. An unstable input is one whose meaning changes over time, often because it is itself the output of another model that someone else is improving. An underutilised input is one carrying so little weight that removing it would cost almost nothing, yet it stays because nobody knows it is safe to drop.

That table is the whole argument for data validation as a pipeline stage rather than an afterthought.

Google's own MLOps guidance lists the practical differences that follow from all this.

1. Team composition. The team includes data scientists or researchers who build models but may not be experienced software engineers.
2. Development is experimental. You try different features, algorithms and parameter settings, and the job is to find what works, then reproduce it.
3. Testing is more involved. Beyond unit and integration tests you need data validation, trained model quality evaluation, and model validation.
4. Deployment is not one artifact. You are deploying a multi-step pipeline that retrains and redeploys, not a single prediction service.
5. Production decays differently. Models fail from evolving data profiles rather than from bad code, so shipping without monitoring means failing without noticing.

Every practice in the rest of this path answers one of these five.

MLOps is DevOps applied to a system where the behaviour is learned from data rather than written down. That one difference is what generates the extra work.

A DevOps pipeline versions code. An MLOps pipeline has to version code, data and the trained model together, because reproducing a result requires all three. A DevOps test asserts on outputs. An MLOps test also has to assert on distributions, since the input can change while every line of code stays the same.

The rest of this path follows that thread: the maturity levels that describe how much of this is automated, the data and feature layer where skew is introduced, and the monitoring that decides when a model has stopped being worth serving.