What Bias Means, and Where It Enters

Most unproductive arguments about bias are two people using the word differently and neither noticing.

Statistical bias is the difference between an estimator's expected value and the true value. A biased estimate is systematically off. This is a technical property with no moral content, and reducing it is ordinary modelling work.

Societal bias is prejudice or unfair treatment of a group. This is a moral and legal concept, and it is what most people mean outside a technical setting.

Disparate outcome is a measured difference in how a system performs or decides across groups. This is an observation, and it is neither of the first two by itself. A difference can exist without prejudice and without a statistical error.

The confusion is consequential. A team told their model is biased may hear a technical claim and respond by improving calibration. The person raising it may have meant the model disadvantages a group. Both proceed satisfied and nothing was addressed.

The discipline this lesson recommends: use disparity for the measurement, and reserve any claim about unfairness for a separate, explicitly argued step. A measured disparity is a finding. Whether it is unfair depends on why it exists and what the system is for, which measurement alone cannot settle.

This step is uncomfortable and skipping it produces audits that cannot survive challenge.

A measured difference between groups can arise from several causes, and they carry different implications.

The system is worse at predicting for one group, because it saw less data about them or because the features are less informative for them. This is a defect, and it is the clearest case for action.

The system is accurate for both groups and the underlying rates genuinely differ. A model predicting equipment failure will show different rates across sites with different equipment ages, and that difference is the model working.

The system is accurate and the underlying rates differ because of historical injustice. The measurement is correct about the world and the world reflects something the organisation may not want to perpetuate. Reproducing it accurately is not a technical error and may still be the wrong thing to do.

And the label itself encodes a decision rather than an outcome, which the next steps cover.

The practical implication is that an audit reporting a disparity has not finished. It has to say which of these it believes is operating, on what evidence. An audit that reports a number and asserts unfairness will be dismissed by anyone who wants to dismiss it, and an audit that reports a number and asserts nothing is useless.

Disparity enters a system at identifiable points, and knowing which one is operating determines what can be done about it.

Historical data records past decisions, and where those decisions were themselves unequal, the pattern is in the training set.

Sampling determines who is represented. A model trained mostly on one population performs worse on others, and the deficit is invisible in aggregate accuracy.

Label definition is the most consequential and least examined. What counts as success is a choice, and it often encodes a prior decision rather than an outcome.

Feature selection determines what the model can use, including proxies for attributes you excluded.

Objective choice sets what the model optimises, which distributes errors between groups.

And deployment context decides who is actually subject to it, and where the threshold sits.

The last three are made by the team, this quarter, and are therefore the most tractable. The first two are inherited.

The most important step in this lesson, and the one teams almost never examine, is what the model is being trained to predict.

Models rarely predict the thing anyone actually cares about. They predict a recorded proxy for it, and the gap between the two is where a great deal of disparity lives.

A hiring model does not predict who would perform well. It predicts who was hired, or who received a good performance rating, both of which are records of past human decisions. If those decisions were uneven, the model learns to reproduce them and its accuracy is a measure of how faithfully it does so.

A model predicting re-offending predicts re-arrest, which measures policing patterns as much as behaviour. Two populations with identical underlying behaviour and different policing intensity produce different labels.

A model predicting good customer may predict spending, which correlates with income, which correlates with a great deal else.

So the question to ask of any model is: what does this label actually record, and what decisions or measurement processes generated it? Where the answer is a past human decision, the model is a decision-imitator rather than an outcome-predictor, and every disparity in those past decisions is in the training signal.

That framing changes the remedy. No amount of post-processing fixes a label that encodes the wrong thing.

The instinctive fix is to remove the protected attribute from the model. It does not work, and understanding why is essential to everything that follows.

In a dataset of any richness, protected attributes are recoverable from combinations of other features. Postcode correlates with ethnicity in most countries with residential segregation. Name correlates with ethnicity and gender. Purchase history, browsing patterns, the device used, the time of day someone applies, the school attended, gaps in employment history: each carries some signal, and a model with enough features reconstructs what you removed.

This is called proxy discrimination, and the insurance and credit sectors identified it long before machine learning, which is why their regulatory frameworks address it directly.

So removing the attribute achieves two things, both bad. The disparity persists, because the model found it elsewhere. And you have destroyed your ability to measure it, because you no longer hold the attribute you would need to check.

That second consequence is the serious one. Fairness-through-unawareness produces a system that is exactly as unequal as before and can no longer be audited, while feeling more responsible to the team that implemented it.

The correct posture is the opposite: retain the attribute for measurement, control its use in the model deliberately, and be able to demonstrate what the system does across groups. Which requires holding data many organisations are nervous about holding, and that is the subject of the next lesson.

A system can post excellent overall numbers while failing a subgroup badly, and the arithmetic makes this easy rather than exceptional.

If a group is ten percent of the population and the model performs poorly on them, the aggregate moves by a small amount that looks like noise. A model at ninety-four percent overall might be ninety-seven percent on the majority and seventy percent on that group, and nobody reading the headline number would suspect it.

Three consequences.

Aggregate accuracy is not evidence of anything about subgroups, and reporting it as though it were is the most common failure in model evaluation.

The smaller the group, the better it hides, which means the populations most likely to be underserved are the ones the metrics are least likely to reveal.

And optimisation makes it worse. A training process minimising average loss will trade accuracy on a small group for accuracy on a large one, because that trade improves the objective. The model is doing what it was asked.

The remedy is disaggregated reporting as a default rather than an investigation. Performance by group, in the standard evaluation output, every time, with small groups reported separately rather than pooled into other.

And it needs to be by group rather than only overall, which requires knowing the groups, which returns to the measurement-data problem.

Measuring one attribute at a time can show acceptable results on every attribute and conceal a serious failure at their intersection.

The mechanism is straightforward. A model may perform adequately across gender and adequately across ethnicity while performing badly for women of a particular ethnicity, because that intersection is a small share of both marginal groups and its poor performance is diluted in each.

The concept comes from legal scholarship on discrimination, where Kimberlé Crenshaw's work established that treating categories separately misses harms specific to their combination. The statistical version is the same observation.

The practical difficulty is real. Intersections multiply quickly: four attributes with a few levels each produce dozens of cells, most of them too small for a reliable estimate, and testing many cells invites false positives.

So the workable approach is targeted rather than exhaustive. Identify in advance the intersections most likely to matter for this system and this domain, based on knowledge of who the system affects rather than on combinatorics. Report those specifically, with confidence intervals wide enough to be honest about small samples. And state which intersections you did not test, because an audit that silently omits them is claiming more coverage than it has.

The general rule: marginal results across single attributes are a screen, not a clearance.

Before measuring anything, four decisions determine whether the result will be actionable or arguable.

Which groups. Named explicitly, chosen because the system affects them and because law or context makes them relevant, not because the data was convenient. Record why each was included and, more importantly, why any obvious candidate was excluded.

What comparison. Disparity is relative, so state the reference: compared with the overall population, with the best-performing group, or against an external benchmark. The choice changes the numbers substantially and it should be made deliberately.

What would count as a problem, set before results are visible. The same discipline the release gate lesson applies, and for the same reason: a threshold chosen after seeing a disparity will be chosen to accommodate it.

And what happens if a problem is found. If the answer is a discussion, you are running an investigation rather than an audit.

One further framing that improves every audit: state the causal hypothesis you will test. Not merely is there a disparity but if there is one, is it because the model is less accurate for this group, because the label encodes past decisions, or because base rates differ. Naming the candidates in advance is what turns a measurement into an explanation, and only an explanation supports a remedy.