Robustness and Deployment

The first deployment surprise is that these models are not robust to inputs barely different from the ones they handle well.

The VITA lab has a research line on analysing and robustifying trajectory prediction models, and the general finding across this literature is uncomfortable: small, carefully chosen perturbations to an observed trajectory can produce large changes in the prediction.

The mechanism is the standard adversarial one. These are learned functions in high-dimensional input spaces, and such functions generally have directions of high sensitivity that no ordinary test set explores.

What makes it operationally serious here is that an adversary is not required. Real perception is noisy. Detections jitter by a few centimetres between frames. A tracker occasionally swaps identities between nearby people. Occlusions produce gaps that must be interpolated.

So the perturbations that adversarial analysis constructs deliberately are approximated by ordinary sensor noise accidentally. A model whose predictions swing under small input changes will behave erratically in deployment even with nobody attacking it, and benchmark evaluation on clean curated positions never reveals this.

Benchmarks hand you clean trajectories. A deployed system produces them itself, and every stage of that pipeline introduces error the model never saw during training.

Detection misses people who are partly occluded, distant, unusually posed, or dressed unlike the training distribution.

Tracking must associate detections across frames. In dense crowds it swaps identities, which corrupts a trajectory's history in the worst possible way: the input now describes two different people spliced together.

Coordinate transformation projects image detections into the ground plane, and calibration error grows with distance.

Latency means the most recent observation is already old by the time a prediction is produced.

The combined effect is that the trajectory reaching the predictor is shorter, noisier, and occasionally simply wrong compared to any benchmark input.

Two mitigations are standard, and they pull in different directions. Train on realistic noise, injecting jitter, dropouts, and identity switches during training so the model has encountered them. And use richer cues cautiously: body pose improves prediction and fails exactly when conditions are poor, so a model depending on it must degrade gracefully when it is unavailable.

The second gap is that human behaviour is not one distribution.

Location. Which side people pass on, how closely they crowd, how they treat crossings: all vary by country and city. A model trained on European campus footage has learned those specific conventions.

Context. A train station at rush hour, a quiet park, a shopping street, and a stadium exit are different behavioural regimes. Density, purposefulness, and group structure all differ.

Time and weather. People move faster in rain, cluster under shelter, and behave differently at night.

Population. Children move unpredictably. People with mobility aids move differently. Someone carrying a large object has different clearances.

That last row carries a safety implication worth stating directly. A model trained predominantly on typical adult pedestrians will predict atypical pedestrians worse, and those groups are frequently the most vulnerable in a traffic setting.

So the fairness concern here is not abstract. Evaluating only aggregate error hides systematically worse performance on exactly the people for whom a prediction failure is most dangerous, which makes per-group evaluation a safety requirement rather than a reporting nicety.

Here is the problem unique to deployment, and it has no analogue in benchmark evaluation.

Benchmarks are recorded footage. The pedestrians in them were not reacting to your robot, because your robot was not there.

Once deployed, your machine is part of the scene. People see it, form expectations about it, and adjust. That breaks a hidden assumption: the model was trained on how people behave around other people, and it is being applied to how people behave around a robot.

The effect is not small and it is not stable. Early in a deployment people are curious and cautious around an unfamiliar machine. Later they learn its behaviour and exploit it. If a delivery robot always yields, pedestrians learn to walk straight at it, and the model's predictions of them become systematically wrong in a way that grows over time.

The deeper consequence is that prediction and planning are coupled. Your planned action changes what people do, which changes what you should have predicted. Treating prediction as a fixed upstream module ignores the loop.

The frozen robot problem is the visible symptom: an overly cautious machine that always yields becomes both useless and, eventually, predictably exploitable.

Four distinct gaps separate a benchmark number from field performance, and they compound. Each is invisible to standard evaluation, which is why leaderboard position predicts deployed behaviour poorly.

Given that predictions will be wrong, the design question shifts from accuracy to how wrong-ness is handled.

The naive integration takes the single most likely trajectory and plans around it as though it were fact. This inherits every failure mode above with no mitigation.

Better approaches share one property: they treat the prediction as a distribution.

Probabilistic occupancy. Rather than a trajectory, predict the probability that each region is occupied at each future time, then plan to keep collision probability below a threshold. This degrades gracefully, since an uncertain prediction produces a diffuse occupancy field and mild caution rather than confident error.

Multi-hypothesis planning. Verify the plan against several sampled futures, requiring safety across all of them rather than the most likely one.

Reachability-based fallback. Alongside the learned prediction, compute the set of positions a person could reach under physical limits, and ensure a safe fallback exists within it. This provides a guarantee that does not depend on the model being right.

Interaction-aware planning. Model the coupling explicitly, planning an action while accounting for how it will change others' behaviour.

The common principle: a prediction that is confidently wrong must not be able to cause a collision, which means calibrated uncertainty is worth more than a lower average error.

What to verify before a prediction model controls anything that moves.

Two rows carry the most weight.

The population breakdown is an ethical requirement, not a nice-to-have. If the model predicts children or wheelchair users worse than average adults, the aggregate metric is concealing exactly the failure that matters most.

The model-independent fallback is what makes the system defensible. Learned prediction improves normal-case behaviour; it should never be the only thing preventing a collision, because everything in this lesson describes ways it will occasionally be wrong.

Four claims, at the strength the evidence supports.

The problem is jointly determined and multimodal. Each person's future depends on everyone else's, and at decision points several futures are genuinely possible. Both properties defeat the naive formulation of one independent prediction per person.

Learned interaction models work and the mechanism is understood. Social pooling shares hidden states among neighbours, handling variable crowd size and permutation invariance, and produces collision avoidance and group movement that were never programmed. Attention and graph methods refine which neighbours matter.

Standard evaluation systematically misleads. Displacement error rewards hedged predictions over plausible ones, ignores collisions and physical plausibility, and best-of-N reporting credits diversity without calibration. Constant velocity remains competitive, and interaction-centric categorisation exists because averages conceal where models fail.

Deployment introduces gaps benchmarks cannot measure. Perception noise, distribution shift, adversarial sensitivity, and the feedback loop from your own robot all degrade field performance below benchmark numbers.

The durable principle: predict distributions, evaluate on what you actually need, and never let a learned prediction be the only thing standing between your machine and a person.