The Three Levels of MLOps Automation

Google's MLOps guidance sorts teams into three levels. It is easy to read them as a scorecard and wrong to do so.

The levels do not measure model quality. A level 0 team can have a better model than a level 2 team. What they measure is how fast you can get a new idea reliably into production, and how fast you can react when the world moves.

That reframing matters because it tells you when to invest. If your data distribution is stable and one release a year is genuinely fine, level 0 is not a failing grade. If your data shifts weekly and you cannot retrain in under a month, the gap between those two numbers is the actual problem, and automation is how you close it.

Level 0 is where most teams start and many stay. Google's description is precise about what it means.

Every step is manual: data analysis, data preparation, model training and validation. The work lives in notebooks, and the handoff to production is a person passing an artifact to another team.

That handoff is the defining feature. Data scientists build the model, engineering deploys it, and the two groups are disconnected. Releases are infrequent, on the order of a couple of model versions per year. There is no CI, because there is nothing to integrate continuously. There is no CD, because deployment means deploying a prediction service rather than a pipeline. And crucially there is a lack of active performance monitoring, so nobody is watching the thing that will actually go wrong.

Level 0 does not fail on the day you ship. It fails months later, and the two causes are worth naming separately.

The first is decay. The model was fitted to a data profile that keeps evolving. Without monitoring, the accuracy curve bends downward and the first signal is a business metric moving, not an alert.

The second is irreproducibility. A notebook run six months ago used some version of some data with some parameter set. When you need to retrain, reconstructing that is archaeology. This is why the artifact you version has to be the pipeline rather than the model: a model file tells you what came out, and only the pipeline tells you how.

Both failures come from the same missing thing, which is an automated path from data to deployed model.

Level 1 changes what you deploy. You stop shipping a trained model and start shipping the pipeline that trains it, which then runs on its own in production.

The payoff is continuous training: the model retrains on fresh data automatically, driven by live triggers. To make that safe, two automated gates go in front of it.

• Data validation checks incoming data against an expected schema and value ranges, and stops the pipeline if it does not match.
• Model validation checks that the newly trained model actually beats the one currently serving, and stops the deployment if it does not.

Without both, automated retraining is just an automated way to push a worse model. Components are modularised and containerised so they can be reused across pipelines rather than copied.

Three components appear at level 1 that do not exist at level 0.

A feature store centralises feature definitions so experimentation, training and serving all read the same computation. That is the structural fix for training-serving skew, covered in the next lesson.

An ML metadata store records every pipeline execution: which code version, which data, which parameters, which artifacts came out. This is what makes a result reproducible and a regression debuggable.

A pipeline orchestrator runs the steps and decides when. Google lists the triggers explicitly: on a schedule, on demand, on new data availability, on model performance degradation, or on significant changes in the data distribution.

Those last two triggers only exist if you are monitoring, which is why monitoring is a prerequisite for automation rather than a nice addition to it.

At level 1 the pipeline runs itself, but a new pipeline is still deployed by hand. That is fine when you change the pipeline rarely. It is the bottleneck when your team is testing new architectures and feature engineering constantly.

Level 2 adds CI/CD for the pipeline code itself. A commit triggers a build, the components are tested and packaged, the artifacts are deployed through test and pre-production to production, and the pipeline then runs and registers its trained model.

What stays manual even at level 2 is worth noting, because it is easy to assume the goal is a system with no humans in it. Google explicitly leaves development and experimentation manual, and model analysis manual. The judgement about whether a model is good is not automated. The path from a judged-good idea to production is.

Google names six stages in an ML CI/CD pipeline. The last one loops back to the first, which is the whole point: monitoring is not the end of the process, it is the trigger for the next turn.

The words are borrowed from software but the test suites are not.

CI in an ML pipeline tests things a normal build never touches. Google lists unit testing the feature engineering logic, testing that the model converges rather than diverging in training, testing explicitly for NaN values produced by the implementation, and component plus integration tests across pipeline steps.

CD verifies compatibility with the target infrastructure, tests the prediction service API, and runs load tests to confirm queries per second and latency hold up.

# a pipeline CI job, roughly
steps:
  - unit_test_feature_logic
  - assert_no_nan_in_outputs
  - train_smoke_run_converges
  - integration_test_components
  - package_containers

The NaN check reads as trivial until it saves you. A silent NaN propagates through training and produces a model that loads, serves and is useless.

The honest answer is the lowest one that keeps up with your data.

Each level costs real engineering time, and the cost is permanent because the automation itself needs maintaining. Buying level 2 for a model retrained twice a year means maintaining CI/CD for a pipeline that almost never changes.

A usable test: measure how long it takes you to get a retrained model into production today, then ask how quickly your input data meaningfully changes. If the first number is smaller, you have enough automation. If it is larger, you are permanently serving a stale model and the gap tells you what to fix.

Most teams find the biggest single win is not level 2 but the monitoring at level 1, because it converts an invisible decay into a signal.