Evaluation, and Closing the Loop

A detection model reports mean average precision of 0.61. That is the whole result, and it supports exactly one decision: whether to accept the model.

Everything you actually want to know is invisible in it. Which classes are failing. Whether the failures are missed objects or spurious ones. Whether they concentrate in a condition, a camera, a time of day. Whether the errors are the model's or the labels'. Whether the drop from the previous version is a real regression or noise on a small slice.

The usual response is to compute more aggregates: per-class scores, a confusion matrix, a precision-recall curve. Those help and they remain aggregates, so the next question is always which specific samples produced this number, and answering it means writing a script that joins predictions to ground truth and dumps images to a folder.

That script is the friction this lesson is about removing. If evaluation writes its results back onto each sample, then every aggregate has a view behind it, and the question which samples became a query rather than a task.

Evaluation is a method on a dataset or a view, with a variant per task type: classifications, detections, segmentations, regressions, and others including polygons.

Each compares a predictions field against a ground truth field on the same samples, which is possible because both live on the sample as typed label objects rather than in separate files.

Each produces aggregate statistics of the kind you expect: precision, recall and F1, per-class breakdowns, a confusion matrix, a classification report, and where applicable precision-recall and ROC curves. Regression evaluation produces error metrics instead.

Detection evaluation additionally takes a protocol, because there is no single agreed definition of a correct detection. The COCO protocol is the default, with Open Images and ActivityNet for video temporal detection also supported. They differ in how matches are made and how edge cases are treated, so a model's score is protocol-dependent.

That last point deserves emphasis, because it is a common source of confused comparisons. Two numbers computed under different protocols are not comparable, and a model whose published score was computed one way can look better or worse when you evaluate it another way, with nothing having changed about the model.

The design decision that changes the workflow is that evaluation, given a key to store results under, writes per-sample outcomes back onto the dataset.

Each sample gains counts of true positives, false positives and false negatives under that key, along with per-sample accuracy where the task defines one. For detection, individual predictions and ground truth objects are also marked with their outcome, so a specific box is recorded as a true positive, a false positive, or a ground truth object that was missed.

Those are ordinary fields, which means every technique from the views lesson applies to them.

Sort by false positive count descending and you have the samples where the model hallucinated most. Match on samples with zero true positives and at least one ground truth object and you have complete failures. Filter predictions to those marked false positive and the App shows only the spurious boxes, with the correct ones hidden.

That is the whole idea. An aggregate metric is a summary of a column that now exists, so every number in the report is one query away from the images behind it. The report stops being the end of the analysis and becomes the index into it.

The confusion matrix is the clearest illustration of what this buys, because in its usual form it is a table of numbers that raises questions it cannot answer.

A cell saying the model predicted truck 340 times when the ground truth was bus tells you a confusion exists and nothing about its nature. The explanations are quite different and the fix depends on which it is. The two classes may be genuinely ambiguous in your imagery. The annotation guideline may not have distinguished them consistently, so the ground truth is itself inconsistent. The model may be systematically failing on one visual subtype. Or a small number of near-duplicate scenes may account for most of the count.

When the matrix is interactive, clicking that cell produces the view of those 340 samples, and thirty seconds of looking usually distinguishes the four.

That is worth stating as a general principle rather than a feature. The value is not the plot; it is that the plot is backed by a query. Any summary statistic whose underlying samples can be retrieved is a starting point, and any summary statistic that cannot is a conclusion you have to accept.

Because evaluation runs on a view, evaluating a slice costs nothing beyond constructing the view, and this is where aggregate numbers most often mislead.

A model at 0.61 overall might be at 0.78 in daylight and 0.31 at night. It might be strong on the four common classes and useless on the three rare ones that motivated the project. It might have degraded on one camera after a lens change nobody recorded.

An overall metric weights by frequency, so a subgroup that is ten percent of the data can be catastrophically bad while moving the aggregate by a few points. If that subgroup is the one you care about, the headline number is actively misleading.

This is the same argument the catalogue makes about quantization and about model collapse, and it recurs because it is a property of averages rather than of any technique.

The practice that follows is to decide the slices before evaluating, from what you know about the deployment: conditions, sources, classes, object sizes, times. Evaluate each separately and report them as a table. The overall number goes at the bottom, not the top, because it is the least informative row.

The question that decides what to do next is whether a failure is the model's fault or the annotation's, and it is the question aggregate metrics can never answer.

A false positive has two readings. The model detected something that is not there. Or the model detected something that is there and the annotator missed it, in which case the model is right and the metric is wrong.

This matters because the responses are opposite. A genuine model failure suggests more training data of that kind, or a different architecture, or an accepted limitation. A labelling failure suggests fixing the labels, and if you respond to it by training harder you are teaching the model to reproduce the annotation error.

The workflow this path has been building answers the question by combining the previous lesson with this one. Take the samples the evaluation marked as failures, and cross-reference them against mistakenness. Failures on samples that also score high for suspected label errors are candidates for the second reading. Then look, because the distinction is visual and a human settles it in seconds.

That combination is the thing that is genuinely hard to do without tooling, and it is the strongest argument for having any.

Everything in this path assembles into one cycle, and the tool's design makes sense once you see which step it is trying to make cheap.

Start with data and labels. Train a model elsewhere. Load its predictions back onto the same samples. Evaluate, which writes per-sample outcomes.

Now the diagnosis step, which is where the loop usually breaks in practice. Query the failures, slice them, cross-reference against label-quality scores, and look. The output is a decision about what is actually wrong.

That decision routes three ways. Bad labels go to re-annotation, sent out as a view and returning into the same fields. Missing coverage goes to acquiring or selecting more data, prioritised by uniqueness or hardness. A genuine model limitation goes back to training.

Then the cycle repeats, and evaluation under a new key sits alongside the old one, so versions are comparable per sample rather than only in aggregate.

The claim the tool makes is narrow and worth stating plainly. It does not improve the model or the labels. It makes the diagnosis step fast enough that the loop actually turns, where otherwise teams skip diagnosis and change the model because that is the step with tooling.

Four things this workflow does not give you, worth knowing before adopting it.

It does not decide what matters. Every technique here surfaces candidates and ranks them, and someone still has to know that a missed pedestrian is worse than a missed bollard. The tool has no notion of what your errors cost.

It scales the looking, not the deciding. Reviewing the two hundred most suspicious labels is a person's afternoon. On a dataset with tens of thousands of suspect labels, ranking helps and does not remove the work.

It is another system to run. A database, a Python environment, an App, and datasets that live somewhere and need managing. For a project with a few thousand images and one person, a folder and a notebook may genuinely be less overhead, and the tool earns its place as scale and team size grow.

And it is largely visual-first. The data model handles images, video, point clouds and 3D, and a team working on tabular or text data is in the wrong tool.

The general shape is familiar from the rest of this catalogue. The instrument makes a measurement cheap. Deciding what to measure, and what to do about it, remains the skill.