mlops

A model that has stopped working returns answers with the same confidence as one that still does. This lesson covers the difference between data drift and concept drift, what to monitor when labels arrive months late, the triggers that should start a retraining run, and the cases where retraining is the wrong response.

The most common way a good model fails in production is that the features it is served differ from the ones it was trained on. This lesson covers where that divergence comes from, why point-in-time correctness is harder than it looks, what a feature store actually solves, and when you do not need one.

Google's MLOps guidance describes three maturity levels, from a fully manual handoff to a pipeline that tests and deploys itself. This lesson covers what is automated at each level, the six stages of an ML CI/CD pipeline, and why level 2 is the wrong target for most teams.

A trained model is a small box inside a large system, and the system is what decays. This lesson covers the failure modes that are specific to machine learning: why changing one feature moves every weight, why hidden consumers break silently, and why data dependencies cost more than code dependencies.

An aggregate metric tells you a model is worse than you hoped and nothing about why. Evaluation that writes results back onto each sample turns a number into a set of images you can look at. This lesson covers the evaluation methods and their protocols, per-sample true and false positive counts, how a confusion matrix cell becomes a view, and where this workflow stops.

A view is a filtered, sorted or sliced window onto a dataset, built by chaining stages and computed lazily without touching the underlying data. This lesson covers what that buys, the distinction between filtering samples and filtering the labels inside them that catches nearly everyone, how views compose with the App and with evaluation, and when a view should be turned into a real subset.

Most computer vision projects are limited by their dataset rather than their architecture, and most teams cannot see their dataset. This lesson covers why a tool for inspecting visual data exists, the data model FiftyOne uses to make a dataset queryable rather than a folder of files, why it sits on a database, and where curation fits against annotation and experiment tracking.

Learn how to systematically evaluate Retrieval-Augmented Generation systems in production using RAGAS, TruLens, and Phoenix — covering golden sets, retrieval drift, embedding drift, and cost-aware eval scheduling.