The Case for Looking at Your Data

A computer vision project that is not working has a small number of possible causes, and the architecture is rarely the interesting one. Model code is largely a solved commodity: a strong detector or classifier is a few lines away, pretrained and well tested.

What differs between a system that works and one that does not is almost always the dataset. Classes that are confused because their examples genuinely overlap. Labels that are wrong in a consistent way because an annotation guideline was ambiguous. A rare but important condition with forty examples. Near-duplicate images inflating both the training set and the test set, so the evaluation is measuring memorisation.

None of those are visible in a loss curve. They are visible by looking at the data, which is the part almost nobody does at scale, because a dataset of a hundred thousand images is not something a person can page through.

That gap is what a curation tool addresses. Not training models faster, but making a dataset something you can interrogate: ask which samples the model fails on, which labels look wrong, which images are nearly identical, and get an answer in seconds rather than a script.

In most projects a dataset is a directory of images plus one or more annotation files, and that arrangement causes specific, recurring problems.

The labels and the images are separate. Answering a question that spans both, such as show me every image where the model predicted a person and the ground truth has none, means writing a script that joins two files.

The annotation format dictates the questions. COCO JSON, YOLO text files and Pascal VOC XML each store different things in different shapes, so a query that is easy in one is awkward in another, and moving between them loses information.

Derived information has nowhere to live. Model predictions, embeddings, quality scores and evaluation results are each written to their own file in their own format, and relating them means another script.

And nothing is visual. Every question is answered by a script producing numbers, when the question was about images.

The consequence is that inspecting a dataset costs enough effort that people skip it, form a hypothesis about what is wrong, and change the model instead. The purpose of the data model in the next step is to make that inspection cheap enough to actually do.

FiftyOne's data model is three concepts, and almost everything else follows from them.

A Dataset is an ordered collection of samples. It is the top-level object you load, query and pass around.

A Sample is one media item: an image, a video, a point cloud. It has exactly one required attribute, a filepath pointing at the media, and receives a unique identifier when it enters a dataset. Notice what this means: the media itself is not copied into the dataset. The sample references it, so building a dataset does not duplicate terabytes of images.

A Field is an attribute on a sample, functioning like a column. Fields hold primitives, so a sample can carry a string, a number, a date, a list or a dictionary. They also hold label objects, which is where the structure lives.

The important property is that fields are dynamic. There is no schema to declare in advance and no migration to run. Adding model predictions to a dataset means writing them to a new field, which is created on the spot. Adding an embedding, a quality score, or a note from a reviewer works the same way.

That is what lets everything about a sample live in one place.

The second design decision is that annotations are typed objects rather than whatever shape the source format happened to use.

The library defines classes for the kinds of label computer vision produces: Classification for a single label, Classifications for multi-label, Detections for bounding boxes, Polylines, Segmentation masks, Heatmaps, Keypoints, Cuboids, rotated boxes, temporal detections for video, 3D detections and polylines, Regression, and GeoLocation.

What this buys is that the format becomes an import detail. COCO, YOLO and Pascal VOC all load into the same Detections objects, so a query written once works regardless of where the annotations came from, and exporting to a different format is a conversion rather than a rewrite.

It also means ground truth and predictions are the same type in different fields. A sample can carry ground_truth detections and predictions detections side by side, and comparing them is comparing two fields of one object rather than joining two files.

And because a field can hold any label type, a sample can carry several at once: boxes, a segmentation mask, a scene-level classification and a geolocation, all on the same image, which is how real annotation projects actually accumulate.

A design decision that looks like an implementation detail and is not: datasets are persisted in MongoDB, a non-relational database, rather than held in memory.

Three consequences follow, and they are the reason the tool scales past a toy.

Dataset size is not bounded by RAM. A dataset of millions of samples is a database with millions of documents, and only the samples you are currently looking at are loaded. A design that held everything in Python objects would fail at a size real projects reach easily.

Queries execute in the database rather than in a loop. Filtering to samples matching a condition is a database query over indexed fields, not a scan through a list, which is what makes interactive exploration feel immediate.

And the schema is flexible, which is what makes dynamic fields possible. A document store does not require every document to have the same fields, so adding predictions to half a dataset is not a migration.

The cost is that a database has to be running. In ordinary use the library manages that for you, and it is worth knowing it is there, because it is the thing that occasionally needs attention and it explains why a dataset persists between sessions without being explicitly saved.

The structural claim is that a sample accumulates everything known about a media item, and the diagram is worth reading as a contrast with the folder-of-files arrangement.

At the centre is a sample, holding a filepath to the media on disk. Attached to it are fields, and the fields come from different places at different times.

Ground truth labels arrive at import, from whatever annotation format the project uses. Model predictions arrive later, written to their own field so the two coexist rather than one overwriting the other. Embeddings arrive from a model applied to the images. Evaluation results arrive when predictions are scored against ground truth, and are written back per sample. Quality scores arrive from the analysis methods a later lesson covers. And arbitrary metadata, such as which camera took the image or which annotator labelled it, can be attached at any point.

The payoff is that a question spanning several of these is one query rather than a join. Show me samples where the model was confident, the evaluation says false positive, and the image is a near-duplicate of something in training. Every term in that sentence is a field on the same object.

The library has a companion graphical application, and its role is worth being precise about, because a visualisation layer is easy to dismiss as convenience.

The App displays a dataset or a view of one: a grid of media with labels rendered on top, panels for statistics and embeddings, and controls for filtering.

What makes it more than a viewer is that it is bidirectional with the Python session. A view constructed in code appears in the App. A selection made by dragging in the App is available in Python as a view. So exploration alternates between the two: query to narrow, look to understand, select what looks interesting, and continue querying from that selection.

That loop is the actual product. The reason data problems go uninvestigated is not that people lack curiosity, it is that each cycle of hypothesis and check costs a script. Reducing that to seconds changes how many hypotheses get checked.

The honest caveat is that a person still has to look and still has to know what they are looking for. The tool makes the looking cheap. It does not tell you which question to ask, and the later lessons on the analysis methods are largely about narrowing that down.

A curation tool overlaps with several other categories, and knowing the boundaries prevents both duplicated effort and disappointed expectations.

It is not an annotation tool. Drawing boxes and managing an annotation workforce is a different product, and the integration model is a round trip: send a view out to an annotation platform, get the results back into the same fields. The tool is where you decide what needs annotating and where you check what came back.

It is not experiment tracking. Which hyperparameters produced which metric is a different question, tracked elsewhere. This is about what the model did on which samples, which is complementary.

It is not a training framework. Models are trained wherever you train them; predictions are loaded in afterwards. There is a model zoo for applying pretrained models to get predictions or embeddings quickly, which is a convenience rather than a training story.

And it is not a data warehouse. The media stays where it is, referenced by path.

The category it occupies is the inspection and decision layer between having data and training on it, and the argument for it is that this layer usually has no tooling at all.

Three things carry into the rest of this path.

The unit of work is the sample, and everything known about a media item lives on it as fields. Predictions, evaluations, embeddings and scores are not separate artefacts to be joined; they are columns on the same row.

Queries run in a database, which is why exploration is interactive at a scale where a script would not be. That property is what makes the next lesson's chained views useful rather than a syntax preference.

And the tool answers questions rather than asking them. It will show you the samples where a model failed, and it will not tell you that your rare class is the problem. Deciding what to investigate remains the skill.

The next lesson covers views, which is how a question becomes a query. Then the analysis methods, which narrow down what is worth asking about a dataset too large to inspect by hand. Then evaluation, which is where model results and data quality meet.

The underlying argument throughout is the one this lesson opened with: for most projects the dataset is the lever, and the reason it is not pulled is that it is hard to see.