The Brain: Computing Fields Worth Querying

Almost every method in this lesson works the same way underneath, and seeing that once makes the rest predictable.

Run the images through a trained model and take an intermediate representation, typically the vector just before the classification head. That vector is an embedding, and its useful property is that the model learned to place semantically similar images near each other, because that is what made classification possible.

So distance in embedding space approximates visual and semantic similarity. Two photographs of the same street corner land close together. A photograph and a diagram of the same object land further apart than the photographs but closer than either is to a picture of a dog.

Once a dataset has embeddings, a set of otherwise hard questions become geometry. Which images are unusual becomes which points are far from everything. Which images are redundant becomes which points are nearly coincident. Which images are similar to this one becomes nearest neighbours.

The embedding model matters and is often left at a default. A general-purpose model gives general-purpose similarity, which is usually what you want for exploration. For a domain where the meaningful differences are subtle, a model trained on that domain gives sharper structure.

Embeddings have hundreds or thousands of dimensions, which is unhelpful for looking at. The visualisation method reduces them to two so a dataset becomes a scatter plot.

Four reduction methods are available and they behave differently. UMAP is the default and tends to preserve both local neighbourhoods and some global structure, producing well-separated clusters. t-SNE preserves local neighbourhoods well and distorts global distances, so cluster separation on the plot should not be read as a real distance. PCA is linear, fast and deterministic, and shows the directions of largest variance rather than clusters. Manual lets you supply coordinates you computed yourself.

The resulting coordinates are written to the dataset as a field, which is the point: the plot is not an image, it is data. In the App it appears as an interactive panel where a lasso around a region of points selects those samples, and the corresponding images appear alongside.

That bidirectional link is what turns a scatter plot into a tool. A cluster sitting apart from everything is a question. Selecting it and looking at the images answers it, and frequently the answer is a data problem: one camera with different colour balance, a batch of images from a different source, a class the annotators handled inconsistently.

Two methods score every sample by how it sits relative to the rest of the dataset, and they are opposite ends of the same measurement.

Uniqueness measures how distinct an image is from all the others. It needs nothing but the raw images, which makes it the first thing worth running on an unlabelled corpus. High uniqueness means the image is unlike its neighbours; low uniqueness means many similar images exist.

Representativeness measures the reverse: how typical a sample is, how well it sits in a dense region of the distribution. It also needs only images or precomputed embeddings.

The workflows they support are genuinely different. Uniqueness answers what should be annotated first when you have a large unlabelled pool and a small annotation budget. Labelling a hundred near-identical images teaches a model very little; labelling a hundred distinct ones covers far more of the input space, so annotating in descending uniqueness order is a defensible prioritisation.

Representativeness answers what a typical example looks like, which is useful for understanding what a dataset actually contains, and for finding statistical outliers at the low end.

Run together, the two ends of uniqueness are both informative: the top is what to label, the bottom is what is redundant.

The most directly valuable method, and the one with the strictest requirements, looks for annotation errors.

Mistakenness needs both ground truth labels and model predictions including logits, meaning the full score distribution rather than only the winning class. The reason is that the signal it looks for is disagreement weighted by confidence.

The underlying idea is simple. When a model is confidently wrong about a sample, there are two explanations: the model is wrong, or the label is. Neither is certain from one example, and a model that is confidently wrong on a sample that resembles many correctly-labelled ones is suspicious in a way that random errors are not.

The method scores every sample and writes it back, so a view sorted by descending mistakenness is a review queue with the most likely errors first. For detection tasks it also flags probable missed objects and poorly localised boxes.

The honest caveat is that this is a heuristic and it finds candidates, not errors. Some high-scoring samples are genuinely hard examples the model should get wrong, and the review is a human job. What it changes is the economics: instead of auditing a random sample of a hundred thousand labels, you audit the two hundred most suspicious, and the error rate among those is far higher.

Hardness measures how difficult a sample is for a model, using predictions with confidence scores. It sounds similar to mistakenness and answers a different question, and the distinction decides which one to reach for.

Mistakenness asks whether the label is wrong. It requires ground truth, because it compares the label against the prediction.

Hardness asks whether the model finds the sample difficult, which is visible from the prediction alone. A confident prediction indicates an easy sample; a flat distribution across classes indicates the model is unsure. Ground truth is not required, which is what makes it usable on unlabelled data.

That is the workflow it supports. Given a large unlabelled pool and a model, hardness ranks which samples the model would most benefit from seeing labelled, because a sample it already handles confidently teaches it little. Annotating in descending hardness order is active learning in its simplest form.

Uniqueness and hardness are worth contrasting too. Uniqueness is a property of the data alone and available before any model exists. Hardness is a property of a particular model on that data. Early in a project you have only the first. Once a model exists, hardness is the sharper signal, because it targets what this model specifically lacks.

The methods differ mainly in what they require, and organising them that way is the fastest route to picking one.

With raw images only, uniqueness and representativeness are available, along with visualisation and similarity. This is the unlabelled-corpus situation, and the question these answer is what to annotate first.

Add model predictions and hardness becomes available. Still no ground truth needed, and the question sharpens from what is distinct to what this model finds difficult.

Add ground truth as well, plus prediction logits, and mistakenness becomes available. Now the question is which existing labels are wrong, which is the highest-value question once a dataset is labelled.

The diagram is worth reading as a progression through a project. Early on you have images and a budget, and uniqueness directs it. Once a first model exists, hardness directs the next round. Once labels accumulate, mistakenness finds the ones corrupting training.

Visualisation and similarity sit across all three stages, because they need only embeddings and answer questions at every phase.

The similarity method builds an index over embeddings so that nearest-neighbour queries are fast, and it unlocks two capabilities that are worth separating.

The first is image similarity. Select a sample and retrieve the most similar ones, which is how you turn one interesting example into the set of all examples like it. Found a failure case? Find the others. Found a mislabelled image? Find the batch it came from.

A special case of this is duplicate detection. Exact and near-duplicate samples are simply pairs at near-zero distance, and finding them matters more than it sounds, because duplicates spanning a train and test split make an evaluation measure memorisation.

The second capability is text search. With a model that embeds images and text into the same space, a natural-language query can be embedded and matched against image embeddings, so you can ask for images containing a red truck at night without any label saying so. That is a substantial change in how a large unlabelled corpus can be explored.

The index has backend options, from a local scikit-learn implementation for modest datasets to dedicated vector databases for large ones, which is the same tradeoff the catalogue's vector database path covers in depth.

These methods produce numbers, and numbers invite more confidence than they deserve. Four cautions are worth carrying.

Every score depends on the embedding model. Uniqueness computed with a general-purpose model measures unusual to that model, which may not match unusual for your task. If a domain's meaningful variation is invisible to the embedding, the scores will be about lighting and composition instead.

The scores are relative to the dataset. A sample's uniqueness is a statement about its neighbours, so adding data changes it. They are not absolute properties and do not transfer between datasets.

They are heuristics, and mistakenness in particular produces candidates rather than findings. Treating a high score as an error and relabelling without looking will introduce errors of its own.

And the visualisation is a projection. Two dimensions cannot faithfully represent a thousand, so apparent clusters can be artefacts and apparent distances can mislead, especially with t-SNE where global geometry is not preserved.

The reliable use of all of them is the same: as a ranking that decides where a human looks first. As a ranking they are very good, and that is the whole claim being made.