Why the Standard Metrics Mislead

Almost every paper in this field reports two numbers, and understanding exactly what they measure explains most of the field's evaluation problems.

Average Displacement Error (ADE) is the mean Euclidean distance between the predicted and observed positions, averaged over every timestep in the prediction horizon and every person.

Final Displacement Error (FDE) is the distance between predicted and observed position at the final timestep only.

They are reported in metres, they are trivially interpretable, and they have been the field's currency since the beginning.

They are also unimodal by construction. Both compare one predicted trajectory to the one observed trajectory, which embeds the assumption that a single correct future exists.

The first lesson established that this assumption is false in exactly the cases that matter. Everything that follows in this lesson is a consequence of measuring a multimodal problem with a unimodal ruler.

Return to the person approaching a T-junction, who will turn left or right with equal probability.

Suppose the truth was a left turn. Compare two predictions.

A committed prediction of a right turn is confidently wrong. Its displacement error is large, roughly the full separation between the two paths.

A hedged prediction going straight ahead into the wall is physically absurd. Its displacement error is roughly half that separation, because straight ahead sits between the two options.

So the metric scores the impossible trajectory better than the plausible-but-wrong one.

This is not an edge case, it is the systematic incentive. A model minimising expected displacement error learns to predict conditional means, and the observable signature is exactly what practitioners report: predictions that are too straight, too slow, too centred, and unwilling to commit to a turn.

The deeper point is that this is not a modelling failure to be fixed by a better architecture. The metric is asking for the conditional mean, and the model is supplying it correctly.

Generative models produce many samples, so the field adopted a convention for scoring them: generate N trajectories, usually 20, and report the error of the best one against the ground truth.

The justification is reasonable on its face. A multimodal predictor should be credited for covering the true outcome among its hypotheses, and penalising it for the samples that went the other way would defeat the purpose of modelling multiple modes.

The problem is what this fails to measure.

Best-of-N tells you whether the model can produce a good trajectory. It says nothing about whether the model knows which of its samples is likely. It ignores the other nineteen entirely, however implausible they are.

A model producing wildly diverse, largely nonsensical trajectories can score well, because with twenty scattered guesses one may land near the truth. A well-calibrated model producing twenty tightly clustered sensible predictions can score worse.

This matters for deployment specifically. A planner cannot act on twenty hypotheses without knowing their probabilities, and it certainly cannot act on the knowledge that one of them was retrospectively correct.

Comparability suffers too: results at N of 20 and N of 1 are not comparable, and the choice of N is not always prominently reported.

Three predictions, ranked by displacement error against a left-turn ground truth. The ranking inverts what a downstream planner would want, because the metric measures distance to one observed path rather than plausibility.

Beyond ranking, the metric is silent on several properties that determine whether a prediction is usable.

Collisions. Nothing checks whether predicted trajectories pass through each other. A model can predict two people occupying the same point and score perfectly well, since each is individually close to its ground truth.

Physical plausibility. Nothing checks acceleration, speed, or turning rate. Predictions can be superhuman and remain unpenalised.

Scene compliance. Nothing checks whether trajectories pass through walls or off pavements.

Interaction quality. This is the important one for the field's stated purpose. Since interactive scenes are a minority of any realistic dataset, a model can fail every interaction while scoring well on average, because the easy majority dominates.

The response from Alahi's lab is explicit. The TrajNet++ evaluation framework is described as going beyond standard distance-based metrics, introducing novel metrics measuring a model's capability to emulate pedestrian behaviour in crowds, and evaluating whether a model outputs socially acceptable trajectories rather than merely close ones.

Collision rate between predicted trajectories is the most direct such metric, and it is one displacement error is structurally incapable of capturing.

Metrics are only half the difficulty. The datasets have their own issues, and they compound.

The classic benchmarks are small. The datasets that anchored the field for years contain limited scenes recorded from fixed overhead cameras in a few locations. Reported improvements of a few centimetres on datasets of that size are within noise, and are frequently reported as advances anyway.

Scenes are not annotated for difficulty by default. Without categorisation, a benchmark's average is dominated by trivial linear walking, which is why interaction-centric indexing was a necessary contribution rather than a refinement.

Observations are clean. Benchmark positions are curated and complete. A deployed system receives noisy detections with occlusions, identity switches between frames, and missing observations. A model trained and evaluated on perfect inputs has never encountered the conditions it will operate in.

Behaviour is location-specific. Norms differ by country and by context, so a model tuned on a handful of European campus scenes has learned those conventions specifically.

The combined effect is that benchmark leaderboard position is a weak predictor of deployed performance, which is the gap the last lesson addresses.

What a defensible evaluation reports, given all of the above.

Two habits do most of the work.

Always report constant velocity on your own split. It costs nothing and it immediately reveals whether an improvement is real. A method beating it by a small margin on an averaged metric has not demonstrated social intelligence.

Report per-category results. If a model's advantage disappears once trivial scenes are excluded, the advantage was in modelling straight-line walking, and saying so plainly is more useful than an aggregate that conceals it.

The questions that separate a real contribution from a benchmark artefact, applicable to your own work as much as anyone else's.

Was constant velocity reported? If not, that omission is itself informative.

What is N? A best-of-20 result compared against a single-prediction baseline is not a comparison.

Is the improvement larger than seed variance? Single-run results differing by centimetres on small datasets are not evidence.

Does the gain survive restriction to interactive scenes? This is the question the field's own benchmarking work was designed to make answerable.

Are collisions or plausibility reported at all? Their absence means the model may be producing physically impossible trajectories.

The underlying lesson generalises well past trajectory forecasting. A metric is a specification of what you are asking for, and displacement error asks for proximity to one observed path. Optimise it hard enough and you get exactly that: a model that hedges toward averages, never commits, and cannot be trusted to produce a trajectory a person would actually walk.

What a deployed system needs is different, and the final lesson takes that up.