Modelling Interaction: From Social Forces to Social Pooling

The first serious attempt at pedestrian interaction predates deep learning and remains a useful reference point.

The social force model, introduced by Dirk Helbing and Peter Molnar in 1995, treats a pedestrian as a particle subject to forces. Three components suffice for the basic version.

An attractive force pulls toward the person's destination at their preferred walking speed. A repulsive force pushes away from other pedestrians, growing as they get closer. A further repulsive force pushes away from walls and obstacles.

Sum the forces, integrate, and trajectories emerge.

The model's achievement was showing that collective phenomena arise from simple local rules. Simulated crowds spontaneously form lanes in bidirectional corridors, and show oscillating flow at bottlenecks, both of which are observed in real crowds. Nobody programmed lane formation; it emerged.

Its limitations are equally clear. The force functions are hand-designed, with parameters requiring tuning per scenario. It is inherently reactive rather than anticipatory, responding to current positions rather than predicted ones. And real avoidance is not symmetric and physical: people negotiate, using gaze and small early adjustments.

The paper that moved the field to learned interaction is "Social LSTM: Human Trajectory Prediction in Crowded Spaces" by Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese, at CVPR 2016, pages 961 to 971.

The starting point is natural. Give each person their own recurrent network, an LSTM, which reads their observed positions and predicts their future. That models individual motion well and models interaction not at all, since the networks are independent.

The contribution is how they are connected. As the paper describes it, unlike a conventional LSTM, information is shared between multiple LSTMs through a new pooling layer that pools the hidden representation from LSTMs corresponding to neighbouring trajectories, capturing interactions within that neighbourhood.

The mechanism deserves precision. Each person's LSTM has a hidden state summarising their motion so far. At each timestep, for each person, the hidden states of nearby people are gathered and combined into a single vector, which is fed into that person's LSTM alongside their own position.

So each person's prediction is conditioned on a learned summary of what their neighbours are doing.

The pooling layer solves three problems at once, which is why the design persisted long after LSTMs were replaced.

Variable neighbour count. A scene has any number of people. Pooling aggregates an arbitrary set into a fixed-size vector, so the network's input dimension does not depend on crowd size.

Permutation invariance. The neighbours have no meaningful ordering. Pooling by summing or maxing over a spatial grid gives the same result regardless of the order they are processed in, which is the correct symmetry for the problem.

Learned rather than specified interaction. The social force model requires you to write down how repulsion behaves. Pooling learns what to extract from neighbours' states directly from data, so conventions that vary by culture or context are learned rather than assumed.

The reported result was a substantial improvement over prior forecasting methods, and the paper demonstrated that the predicted trajectories exhibit social behaviours including collision avoidance and group movement.

That last point matters more than the error number. Those behaviours were not built in. They emerged from training on real crowd data, which is the same emergence the social force model showed, now learned instead of designed.

The structural idea is one recurrent network per person, coupled at every timestep through a pooling operation over neighbours. That template survived the replacement of every individual component.

Social LSTM addressed interaction. It did not address multimodality: it still produced one trajectory per person, which the previous lesson showed is the wrong object.

The response was to make prediction generative, and two families emerged.

Adversarial approaches. Social GAN, from Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi, pairs a trajectory generator with a discriminator judging whether a trajectory looks socially plausible. Sampling different noise vectors yields different futures. The training signal is realism rather than proximity to one observed path, which is exactly the fix multimodality demands.

Latent variable approaches. Condition prediction on a sampled latent variable, typically via a conditional variational autoencoder. Different samples give different futures, with an explicit probabilistic interpretation.

Both introduce a real difficulty. The generator can achieve a good loss by producing near-identical samples, a form of mode collapse, so it must be pushed toward diversity explicitly. The common device is a variety loss: draw several samples and backpropagate through the best one, which rewards having at least one good sample rather than making the average good.

That objective interacts with evaluation in a way the next lesson examines closely.

Pooling has a weakness that becomes obvious once stated: it treats all neighbours within a radius as equally relevant, and they are not.

A person walking directly toward you matters enormously. A person the same distance away walking parallel to you matters little. Someone behind you matters barely at all. Distance is a poor proxy for relevance.

Attention fixes this by learning the weights. Each person attends to others with learned coefficients depending on relative position, relative velocity, and encoded state. The model decides who matters rather than being told that proximity implies relevance.

Graph neural networks express the same idea structurally. Represent the scene as a graph with people as nodes and interactions as edges, then pass messages along edges. Attention becomes learned edge weights, and multiple rounds of message passing let influence propagate: A affects B, which affects C, capturing indirect effects that single-round pooling misses.

Work from Alahi's lab has pursued interpretability alongside accuracy, including approaches building interpretable social anchors for forecasting in crowds. That direction matters for deployment: a safety-critical system benefits from an answer to "which agent caused this prediction?"

Everything so far uses trajectories only: sequences of two-dimensional positions. People carry far more signal than that.

Body pose is strongly predictive. A pedestrian at a kerb who has turned their head toward traffic is behaving differently from one who has not. Foot placement and torso orientation indicate intended direction before the position changes.

Scene context constrains everything. People walk on pavements, avoid walls, use crossings, and head toward doors. A model unaware of the physical layout will predict paths through buildings.

Gaze and gesture carry the negotiation. Much pedestrian avoidance is settled by brief eye contact establishing who yields.

The VITA lab's Social-Transmotion line pursues exactly this, using richer cues than positions alone, and related work has explored pre-trained models for human motion prediction across modalities.

The engineering tension is worth stating plainly. Richer inputs improve prediction and increase fragility: pose estimation fails at distance, in poor light, and under occlusion. A model depending on pose degrades when pose is unavailable, which in a deployed system is often precisely when conditions are difficult.

The families and what each buys.

Two gotchas that recur across the literature.

First, added complexity does not reliably buy accuracy on standard benchmarks. The persistence of the constant velocity baseline is the standing evidence, and much of the gap between reported and real improvement comes from benchmarks dominated by non-interactive scenes.

Second, generative models are evaluated in a way that flatters them. Reporting the best of many samples measures whether the model can produce a good trajectory, not whether it knows which one is likely. A model producing wildly diverse nonsense can score well if one sample happens to land nearby.

That evaluation question is serious enough to warrant its own lesson.