MLOps Engineer Interview Prep: Questions and a Mock Test

The framing that most reliably marks a strong candidate is that machine learning systems change along more axes than software does. The 2019 article Continuous Delivery for Machine Learning by Sato, Wider and Windheuser puts it directly: ML applications "are subject to change in three axis: the code itself, the model, and the data".

That is the whole difficulty in one sentence. Ordinary continuous delivery assumes that the same input to the same code produces the same output, so versioning the code is enough to reproduce a release. Here, an identical pipeline run against different data produces a different model, and an identical model against shifted data produces different behaviour. Reproducing a production incident therefore requires knowing which code, which data snapshot and which trained artefact were involved.

The article's definition of the practice is worth being able to paraphrase: "a software engineering approach in which a cross-functional team produces machine learning applications based on code, data, and models in small and safe increments that can be reproduced and reliably released at any time, in short adaptation cycles". Note that reproducibility is stated as a property of the release process rather than of the model's output, which is the right distinction to make when an interviewer points out that training is often non-deterministic.

The practical answer to that objection: pin seeds and library versions where you can, and where you cannot, version the artefact itself so the exact model that served traffic can always be retrieved and re-evaluated.

Google Cloud's MLOps guidance defines levels that interviewers use as a shared scale, so it is worth being able to place a team on it.

Level 0 is described as a manual, script-driven and interactive process, where every step including data analysis, data preparation, model training and validation is done by hand. The characteristic symptom is that only one person can produce the model, and the notebook that produced the current production version may no longer run.

Level 1 automates the pipeline itself, with the stated goal of continuous training so that new data produces a new model without a human running the steps. Crucially it requires more than a scheduler: automated data validation and automated model validation, plus pipeline triggers and metadata management. That combination is the point. Automated retraining without automated validation is a mechanism for shipping a regression on a schedule.

Level 2 adds continuous integration and delivery for the pipeline itself, so changes to the pipeline code are built, tested and deployed automatically.

The distinctions the guidance draws are quotable and useful. Continuous integration here is "not only about testing and validating code and components, but also testing and validating data, data schemas, and models". Continuous delivery delivers "a system (an ML training pipeline) that should automatically deploy another service (model prediction service)". Continuous training is "a new property, unique to ML systems".

A good answer to "where would you start" usually rejects jumping to level 2 and picks the one manual step that breaks most often.

Serving rounds cover the same ground as any service deployment plus a few model-specific concerns.

The first is the shape of inference. Batch scoring writes predictions ahead of time and is cheap and simple, but the prediction is as fresh as the last run. Online inference computes on request and needs a latency budget, autoscaling and a plan for the cold start of loading a large model into memory. Streaming sits between them. Choosing correctly from the freshness requirement, rather than defaulting to an endpoint, is the expected reasoning.

The second is rollout. The standard progression is shadow, then canary, then full. Shadow mode sends real traffic to the candidate model without its predictions affecting anyone, which surfaces skew, latency and error behaviour at zero user risk, and is more valuable here than in ordinary services because it can also compare predictions against the incumbent on identical inputs. Canary then exposes a slice of users and measures the business metric, since a model can be more accurate offline and worse in production.

The third is that health checking a model server has a trap. A liveness probe should establish that the process is stuck, not that a dependency is slow: Kubernetes documentation warns that "incorrect implementation of liveness probes can lead to cascading failures", including restarts under high load. For a model server whose startup includes loading several gigabytes of weights, a startup probe is what prevents the platform from killing a container that is merely still initialising.

This is the question that most reliably separates candidates who have operated a model from those who have deployed one.

Accuracy is usually not computable in real time, because ground truth arrives weeks later or never. So production monitoring is built from proxies. Input distribution monitoring compares recent feature values against a reference window and alerts on divergence. Prediction distribution monitoring watches the model's own output, which is cheap and catches a surprising amount, because a model whose positive rate doubles overnight is telling you something even if you do not yet know what. Feature health, nulls, ranges, cardinality, catches upstream pipeline breakage, which is the most common cause of a model appearing to degrade.

Be precise about drift types. Data drift is a change in the input distribution. Concept drift is a change in the relationship between inputs and target, which is worse because inputs look normal while the model becomes wrong. Only the former is detectable without labels.

The judgement half is what an alert means. A distribution shift is a prompt to investigate, not evidence of degradation: a legitimate product launch changes the input mix without harming the model, and retraining reflexively on every alert is how teams end up chasing noise.

CD4ML frames the goal as closing the loop, capturing production data to "adapt models based on learnings taken from real production data, creating a process of continuous improvement". The design consequence is that logging predictions and their eventual outcomes is part of the serving system, not an afterthought.

Testing rounds probe whether you can define a pass condition for a system that is probabilistic by design.

The reframing that works is to test everything that is deterministic normally, and to test the model behaviourally rather than by asserting an exact output. Deterministic parts include feature transformations, the serving contract, schema conformance and preprocessing, and these deserve ordinary unit tests. The model is then covered by threshold tests on a fixed evaluation set, comparison against the current production model rather than against an absolute number, and slice-based checks that catch a model whose aggregate improved while a segment that matters got worse.

Data tests are the other half, and are frequently the higher-value ones because pipeline breakage causes more incidents than modelling regressions. Schema and type conformance, null rates, range checks, cardinality, and volume checks that catch a source silently halving. Google's Rules of Machine Learning gives the specific prescription for the skew case: "the best way to make sure that you train like you serve is to save the set of features used at serving time, and then pipe those features to a log to use them at training time".

The governance question usually follows. What is recorded about a model that served a decision: the training data version, the code version, the hyperparameters, the evaluation results, who approved promotion and when. In regulated settings that lineage is the deliverable, and being able to say that a model registry exists to hold it, rather than to store files, is the right level of answer.