NeMo-map: Neural Implicit Flow Fields for Spatio-Temporal Motion Mapping

AI summary

Overview

• Research area: Robotics / mobile robotics, specifically maps of dynamics (MoDs) for modeling human motion patterns in crowded environments.
• Technical level: Intermediate. The paper assumes familiarity with probabilistic mixture models (Gaussian mixtures, wrapped distributions) and neural implicit representations, but its core idea — replacing a discrete grid with a continuous learned function — is easy to grasp.
• Scope: The paper proposes NeMo-map, a continuous spatio-temporal map of dynamics that maps coordinates to Semi-Wrapped Gaussian Mixture Model parameters, and evaluates it against CLiFF-map, Online CLiFF-map, and STeF-map on the ATC and ETH/UCY datasets.

What This Paper Is About

Maps of Dynamics (MoDs) summarize where and how people typically move in a given environment, so robots can predict motion beyond their sensor range. Existing MoDs — most notably CLiFF-map and STeF-map — must discretize the environment into grid cells and fit a local model per cell, which loses information, requires interpolation, and makes offline construction costly. NeMo-map replaces the grid with a single neural network that learns a continuous function from a spatio-temporal coordinate to the parameters of a Semi-Wrapped Gaussian Mixture Model, so motion statistics can be queried at any location and any time.

Key Contributions

1. A continuous map of dynamics representation. NeMo-map is described as the first-of-its-kind continuous spatio-temporal MoD, built on implicit neural representations rather than discretized grid cells. It removes the need for spatial discretization and imputation in unevenly sampled regions.
2. A coordinate-to-distribution formulation. The model learns a function Φθ that maps a spatio-temporal query (x, y, t) to the full set of SWGMM parameters — weights, means, and covariances — producing multimodal velocity predictions (speed and orientation jointly) at arbitrary coordinates.
3. Feature-conditioned architecture. Spatial information is retrieved from a learnable grid by bilinear interpolation, and temporal information is encoded with a SIREN network, with the raw coordinates and both feature streams concatenated and passed through an MLP.
4. Empirical validation on real-world data. Experiments on two public datasets (ATC and ETH/UCY) show lower negative log-likelihood than CLiFF-map, Online CLiFF-map, and STeF-map, plus improved performance on long-term human motion prediction and practical map-building times (under 20 minutes for a full day of ATC data).

Main Findings

• Best accuracy on ATC: NeMo-map achieves an NLL of 0.775 ± 2.052, versus 1.527 ± 4.156 for Online CLiFF-map, 1.964 ± 4.953 for CLiFF-map, and 5.576 ± 9.314 for STeF-map. The NLL reduction relative to NeMo-map is +0.752 (95% CI [0.749, 0.755]) for Online CLiFF-map, +1.189 ([1.185, 1.192]) for CLiFF-map, and +4.801 ([4.794, 4.809]) for STeF-map. Paired t-tests give p < 0.001.
• Best accuracy on ETH/UCY: Across the ETH, HOTEL, UNIV, and ZARA scenes, NeMo-map reports NLL of -0.384 ± 2.051, -0.838 ± 4.043, 0.404 ± 1.902, and -0.342 ± 2.152 respectively. CLiFF-map reports 0.112 ± 4.005, 0.701 ± 4.533, 0.518 ± 2.125, and 0.068 ± 4.265. Reductions versus NeMo-map for CLiFF-map are +0.496, +1.539, +0.114, and +0.410; for Online CLiFF-map they are +0.470, +2.079, +0.173, and +0.528; for STeF-map they are +2.699, +4.187, +10.528, and +3.126.
• Efficient map building: NeMo-map trains on a full day of data in under 20 minutes (19.26 minutes on the ATC dataset), compared with 1831 minutes for CLiFF-map, 23.859 minutes for Online CLiFF-map, and 0.815 minutes for STeF-map. NeMo-map's inference time is reported as 1.363 × 10⁻⁶ seconds, versus 1.914 × 10⁻³ seconds for both CLiFF-map variants and 5.665 × 10⁻⁵ seconds for STeF-map.
• Stronger downstream prediction: On the ATC long-term human motion prediction task with a 60 s horizon, NeMo-map achieves ADE 5.08 ± 4.28 and FDE 10.97 ± 9.92, better than CLiFF-LHMP (5.45 ± 4.54 / 11.45 ± 10.20), STeF-LHMP (5.88 ± 5.55 / 12.24 ± 12.16), TUTR (12.10 ± 8.20 / 27.26 ± 19.23), and MID (21.20 ± 5.28 / 44.47 ± 10.91). Paired t-tests give p < 0.01.
• Deep learning predictors can be infeasible: The paper reports that trajectories predicted by TUTR and MID are unfeasible in the sense of crossing walls, while MoD-guided predictions follow human flow and implicitly respect the environment structure.
• SIREN temporal encoding is best: Ablation on the ATC dataset gives NLL of 0.775 ± 2.052 for SIREN, versus 0.857 ± 2.113 for a temporal grid (+0.082, 95% CI [0.081, 0.083]) and 0.838 ± 2.105 for Fourier features (+0.063, 95% CI [0.062, 0.064]).
• Qualitative multimodality and temporal variation: In the ATC central open area, NeMo-map preserves both intersecting horizontal and vertical flows, while CLiFF-map cuts off the middle section of the vertical flow. In the east corridor, NeMo-map captures flows shifting from left/upwards in the morning to right/downwards in the evening, at 09:00, 12:00, 15:00, and 18:00, without being given an explicit map.
• Advantage over STeF-map's representation: STeF-map uses an 8-bin discretized orientation histogram and omits speed, whereas SWGMM-based methods (NeMo-map and CLiFF-map) jointly model speed and orientation and represent orientation continuously.

Methodology in Plain English

The starting point is that human motion at a location is naturally described by a probability distribution over velocity, where velocity has two parts: speed (a positive number) and orientation (an angle on a circle). The authors use a Semi-Wrapped Gaussian Mixture Model (SWGMM) to capture this, which can represent several distinct motion modes at once and allows correlation between speed and orientation. This can be visualized as a distribution on a cylinder, with orientation wrapped around the circular axis and speed running along the vertical axis.

The novelty is in how those distributions are produced. Instead of dividing the environment into a grid and fitting a mixture model per cell — as CLiFF-map does — the authors train a neural network that takes any query (x, y, t) and outputs the SWGMM parameters directly. The network input combines three things: the raw spatial and temporal coordinates, a spatial feature obtained by bilinearly interpolating a learnable grid at (x, y), and a temporal feature produced by a SIREN network (which uses periodic sine activations). These are concatenated and passed through an MLP.

The network outputs 6J raw values per query for J mixture components. These are converted into valid distribution parameters: weights through a softmax, mean speed through a max with zero, mean orientation taken modulo 2π, variances through an exponentiated clamped value, and a correlation coefficient through 0.99·tanh. Training minimizes the negative log-likelihood of observed motion samples under the predicted mixture density, with the semi-wrapped normal wrapping the angular component by summing over winding numbers k in {-1, 0, 1}.

In the experiments, J is set to 3, coordinates are normalized to [-1, 1], the spatial MLP uses hidden sizes [128, 64] with ReLU activations, and the two-layer SIREN uses ω₀⁽¹⁾ = 30 in the first layer and ω₀⁽ʰ⁾ = 1 in the hidden layer. Because ATC shows strong daily regularity, timestamps are converted to time-of-day; for ETH/UCY, the frame index is used instead, since no such periodic structure is present.

Evaluation is done by querying the map at each test sample's (x, y, t), computing the likelihood of the observed motion under the predicted distribution, and reporting negative log-likelihood. Baselines are CLiFF-map (grid resolution 1 m, convergence precision 1e-5 for mean shift and EM with max 100 iterations, trained separately per hour), Online CLiFF-map (stochastic EM, one map per hour), and STeF-map (k_stef = 8, model orders set to 2).

Datasets: ATC was collected in a shopping mall in Japan with 3D range sensors, recording pedestrian trajectories between 9:00 and 21:00 over 92 days, covering approximately 900 m². The first four days (2012 Oct 24, 2012 Oct 28, 2012 Oct 31, and 2012 Nov 04) are used, with Oct 24 for training and the other three days for evaluation; observations are downsampled from over 10 Hz to 1 Hz, giving 717,875 training samples and 5,114,478 test samples. ETH/UCY uses four scenes: ETH and HOTEL from the ETH dataset, and UNIV and ZARA from the UCY dataset.

Why This Matters

Impact on research. The paper reframes maps of dynamics as a continuous learned function rather than a collection of per-cell fits, which removes grid-resolution tuning, boundary discontinuities, and the memory cost of storing a distribution for every cell. It also connects MoD research to the broader neural implicit representation literature, showing that these representations can serve as compact, differentiable priors for spatial robotics tasks.

Real-world applications:

• Socially aware robot navigation in crowded spaces such as malls, airports, and campuses, where knowing the expected flow outside sensor range enables less reactive planning.
• Long-term human motion prediction for service robots and automated vehicles, where the map acts as a prior that keeps predicted trajectories consistent with corridors and obstacles.
• Localization and state estimation in dynamic environments, since the map encodes expected motion rather than static geometry alone.
• Site-specific behavior modeling for building or facility management, based on aggregate statistics of pedestrian flow.

Industry relevance. The reported training time of 19.26 minutes on a full day of ATC data — versus 1831 minutes for building CLiFF-map — and the very small reported inference time make continuous MoDs plausible for deployment workflows, although the paper does not claim real-time operation or deployment on robot hardware.

Future Directions

1. Online updating. The authors plan to add online update mechanisms so the representation can adapt continuously to evolving crowd behavior, which the current model does not support.
2. Handling sharp discontinuities. The paper acknowledges that the method struggles with sharp spatial or temporal discontinuities, such as temporary barriers or sudden event-driven changes, and that adapting to these requires additional observations — a limitation it shares with existing MoD approaches.
3. Hyperparameter sensitivity. Ablations on the SWGMM component number J and the spatial feature grid resolution H × W are described as being in Appendix E, but results are not reported in the paper content available here.
4. Broader downstream evaluation. The downstream task validation is limited to long-term human motion prediction on ATC; extending it to navigation and localization in dynamic environments is stated as motivation but not demonstrated.

Target Audience

Robotics researchers working on human-aware navigation, motion prediction, and environment modeling will find the main contribution directly relevant, as will researchers interested in neural implicit representations applied to spatial statistics rather than graphics or geometry. Practitioners building maps of dynamics for service or mobile robots benefit from the efficiency and accuracy comparisons against CLiFF-map and STeF-map. Readers without a background in probabilistic mixture models or circular statistics may need to consult the referenced SWGMM and CLiFF-map literature to follow the methodology fully.

Authors’ abstract

Safe and efficient robot operation in complex human environments can benefit from good models of site-specific motion patterns. Maps of Dynamics (MoDs) provide such models by encoding statistical motion patterns in a map, but existing representations use discrete spatial sampling and typically require costly offline construction. We propose a continuous spatio-temporal MoD representation based on implicit neural functions that directly map coordinates to the parameters of a Semi-Wrapped Gaussian Mixture Model. This removes the need for discretization and imputation for unevenly sampled regions, enabling smooth generalization across both space and time. Evaluated on two public datasets with real-world people tracking data, our method achieves better accuracy of motion representation and smoother velocity distributions in sparse regions while still being computationally efficient, compared to available baselines. The proposed approach demonstrates a powerful and efficient way of modeling complex human motion patterns and high performance in the trajectory prediction downstream task. Project code is available at https://github.com/test-bai-cpu/nemo-map