Predicting Where People Will Walk

Any machine that moves through space shared with people has to answer one question continuously: where will they be in the next few seconds?

A self-driving car approaching a crossing must know whether the pedestrian on the kerb is about to step out. A delivery robot on a pavement must decide which side to pass on. A warehouse vehicle must plan around workers who do not stop for it.

Without prediction, only two behaviours are available, and both fail. Treat everyone as static, and you plan into collisions. Treat everyone as potentially anywhere, and the reachable set explodes until the machine freezes, unable to justify any motion. That failure has a name in the field, the frozen robot problem, and it is what makes prediction load-bearing rather than optional.

Alexandre Alahi leads the Visual Intelligence for Transportation laboratory at EPFL, whose research centres on understanding and predicting human social behaviour from multimodal visual data, at the intersection of computer vision, machine learning, and robotics applied to transportation and mobility.

The framing that lab uses is socially-aware AI: artificial intelligence augmented with social intelligence, on the argument that this is what safe deployment requires.

The standard formulation is deliberately narrow, which is both its strength and a limitation to keep in view.

You observe the positions of every person in a scene over a short history, typically a few seconds sampled at a fixed rate. You must output their positions over a prediction horizon, commonly the next four to five seconds.

Two features of this setup drive everything that follows.

The output is joint, not per-person. You are not making N independent forecasts. Each person's future depends on what everyone else does, and everyone else is deciding simultaneously by the same logic. The prediction problem is coupled.

The horizon is short but not trivially so. Over half a second, momentum dominates and a straight-line extrapolation is nearly right. Over five seconds, people change speed, turn, avoid each other, and reach destinations. That is where the difficulty lives, and it is why benchmark horizons sit in that range.

What makes the task genuinely hard is not the physics of walking. It is that the correct answer is not a single trajectory.

Consider a person approaching a T-junction in a corridor. They will turn left or right. Both are entirely plausible, and nothing in the observed history determines which.

The true distribution over futures is multimodal: it has separate peaks at genuinely different outcomes, with low probability between them.

Now observe what a model trained to minimise mean squared error does. Minimising expected squared error means predicting the conditional mean. The mean of a left turn and a right turn is straight ahead, into the wall.

This is not a training failure. The model is correctly optimising the loss it was given, and the loss is wrong for the task. The behaviour is systematic, and it shows up as a recognisable pathology: predicted trajectories that hedge toward the average, are too straight, too slow, and avoid committing to any decision.

The implication runs deeper than a modelling detail. Any evaluation that compares one predicted trajectory to one observed trajectory rewards this hedging, because the average is closer to both outcomes than either outcome is to the other.

That single observation drives most of the methodological argument in this field, and the evaluation lesson returns to it.

The second source of difficulty is that pedestrian motion is governed by rules nobody states.

The Social LSTM paper opens on exactly this point: humans navigate complex crowded environments according to social conventions, respecting personal space, yielding right of way, and avoiding collisions.

These conventions are specific and consequential for modelling.

Personal space is not a fixed radius. It is elongated in the direction of travel, since you need more clearance ahead than beside, and it scales with speed.

Avoidance is anticipatory and mutual. People adjust well before a collision would occur, and both parties usually adjust, each expecting the other to do so.

Groups move as units. People walking together maintain formation and are collectively avoided, so treating them as independent individuals mispredicts both their motion and everyone else's response.

Conventions are local. Which side you pass on differs by country, and density norms differ by culture and context.

None of this is derivable from physics. It is learned social behaviour, which is precisely why the field moved from hand-designed rules to learned models.

The recursion is the structural difficulty. Each person predicts the others and acts on that prediction, so a model that forecasts each person independently is solving a different and easier problem than the one that actually occurs.

Before any sophisticated model, there is a baseline that must be beaten, and its persistence is the most instructive fact in the field.

Constant velocity takes the person's current velocity and extrapolates in a straight line. Two lines of code, no learning, no parameters.

def constant_velocity(observed, horizon):
    v = observed[-1] - observed[-2]          # current velocity
    return [observed[-1] + v * (t + 1) for t in range(horizon)]

This baseline is competitive on standard benchmarks, and on some it is very hard to beat.

The explanation is that most pedestrian motion is genuinely uneventful. People walk in roughly straight lines at roughly constant speed for most of any given interval. Interactions requiring avoidance are a minority of the data, so a model that predicts the boring majority well scores well on an average metric.

Work from Alahi's lab makes the diagnostic point explicitly: the high error of constant velocity and Kalman-filter baselines can be attributed to their not modelling social interactions. That statement identifies where the error concentrates, and therefore where a benchmark averaging over all scenes hides the difference.

So a headline improvement over constant velocity may reflect better modelling of ordinary walking rather than any social intelligence at all.

It follows that instances are not equally difficult, and separating them is essential to measuring progress.

A benchmark averaging over a realistic distribution of scenes is dominated by the top rows, so it measures mostly the easy cases.

This is exactly the motivation for interaction-centric benchmarking. The TrajNet++ benchmark, developed by Parth Kothari, Sven Kreiss, and Alexandre Alahi, is described as a large-scale interaction-centric trajectory forecasting benchmark comprising explicit agent-agent scenarios, with proper indexing of trajectories through a hierarchy of trajectory categorisation.

The categorisation is the substantive contribution: separate the scenes where interaction matters from the ones where it does not, then report performance per category rather than an average that conceals which you handle.

Collecting what the problem demands gives the specification the next lessons build against.

Model interactions. Each person's future depends on the others, so the architecture must share information between agents rather than predicting independently.

Represent multiple futures. A single output trajectory is the wrong object when the truth is multimodal, so the model must produce a distribution or a diverse set of samples.

Handle a variable number of agents. Scenes contain two people or forty, with no fixed ordering, so the architecture must be permutation-invariant and size-agnostic.

Run in real time. A prediction useful for a vehicle must be produced many times per second, alongside perception and planning, on hardware in the vehicle.

Those four pull against each other. Full joint modelling of all agents is combinatorially expensive. Representing many futures multiplies the output. Real-time budgets constrain both.

Every architecture in this field is a particular compromise among them, and the next lesson traces how those compromises evolved from hand-designed force models to learned interaction modules.