Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction

AI summary

Overview

Research area: Robotics / autonomous driving motion forecasting — specifically trajectory prediction from incomplete, variable-length historical observations.

Technical level: Advanced. The paper assumes familiarity with vectorized scene encoders, transformers (self/cross-attention), Mamba state-space models, feature distillation, and multimodal trajectory prediction metrics.

Scope: The paper proposes a Progressive Retrospective Framework (PRF) that cascades small "retrospective units" to progressively turn features from short, incomplete trajectory observations into features equivalent to those from complete standard-length observations, improving prediction without training a separate model per observation length.

What This Paper Is About

Existing trajectory prediction models are trained and tuned on idealized, fixed-length histories, but real driving data often delivers short or interrupted histories — for example, when a vehicle first enters the ego vehicle's sensor range or is re-detected after occlusion. Performance of state-of-the-art models such as DeMo degrades sharply as the number of observed timesteps shrinks. The paper's goal is a single model that stays accurate across many observation lengths, rather than mapping short observations to a canonical representation in one difficult step or training a separate model per length.

Key Contributions

1. Progressive Retrospective Framework (PRF): a cascade of retrospective units that progressively aligns features from variable-length observations with those from complete observations, placed between the encoder and decoder so it is plug-and-play with existing prediction models.
2. Retrospective Distillation Module (RDM) and Retrospective Prediction Module (RPM): each unit pairs RDM, which distills features of an incomplete trajectory toward its previous history timesteps using a residual-based strategy (a gating branch and a residual branch), with RPM, which reconstructs the omitted timesteps from the distilled features using decoupled queries (anchor-free mode queries for multimodal proposals and anchor-based state queries for refinement).
3. Rolling-Start Training Strategy (RSTS): a training scheme that generates multiple training samples from a single sequence by shifting the observation start point, improving data efficiency — for Argoverse 2 with T_o = 50, T_f = 60, and ΔT = 10, one sequence yields 4 samples for decoder training and {4, 3, 2, 1} samples for the units {Φ⁴, Φ³, Φ², Φ¹}.
4. Extensive validation on Argoverse 2 and Argoverse 1: demonstrating improved variable-length prediction with both QCNet and DeMo backbones, and state-of-the-art results on standard benchmarks.

Main Findings

• Degradation without PRF is large: On Argoverse 2, DeMo-Ori (original DeMo evaluated on variable-length inputs) goes from 0.861/1.533 mADE₆/mFDE₆ at 10 observed timesteps to 0.658/1.278 at 50, an average gap (AVG–Δ50) of 0.066/0.093. QCNet-Ori shows a gap of 0.063/0.100.
• Isolated Training (IT) gives only marginal gains at high cost: QCNet-IT achieves an AVG–Δ50 of 0.007/0.034 on Argoverse 2, and DeMo-IT gives 0.006/0.021, but the paper states this requires training and maintaining multiple models across observation lengths, incurring substantial computational and memory overhead.
• PRF improves variable-length prediction with a single model: DeMo-PRF reaches 0.617/1.183 at 10, 0.603/1.155 at 20, 0.598/1.143 at 30, 0.599/1.145 at 40, and 0.596/1.142 at 50 on Argoverse 2, with AVG–Δ50 of 0.008/0.015. QCNet-PRF gives 0.727/1.213 at 10 through 0.702/1.166 at 50, with AVG–Δ50 of 0.010/0.016.
• Argoverse 1 shows the same pattern: DeMo-PRF reaches 0.602/0.952 at 5 timesteps, 0.567/0.901 at 10, 0.565/0.904 at 15, and 0.568/0.909 at 20, with AVG–Δ20 of 0.010/0.010, versus DeMo-Ori's AVG–Δ20 of 0.083/0.119.
• PRF outperforms related variable-length methods: DTO, FLN, LaKD, and CLLS are all outperformed across observation lengths with both backbones.
• Best reported single-model result on the Argoverse 2 leaderboard: DeMo-PRF achieves b-mFDE₆ 1.81, mADE₆ 0.60, mFDE₆ 1.14, MR₆ 0.13, mADE₁ 1.49, and mFDE₁ 3.72, ahead of DeMo+ReMo (1.84/0.61/1.17/0.13/1.49/3.74) and other listed methods such as SmartRefine (1.86/0.63/1.23/0.15/1.65/4.17).
• On the Argoverse 1 leaderboard, PRF is best on mADE₆ and mFDE₆: DeMo-PRF reports b-mFDE₆ 1.73, mADE₆ 0.70, mFDE₆ 1.03, MR₆ 0.11. Tamba lists a lower b-mFDE₆ of 1.67, and QCNet lists 1.69.
• Ablations isolate each module's contribution: Adding RDM to the DeMo backbone improves 10-timestep results from 0.876/1.455 to 0.655/1.257; adding RPM further gives 0.652/1.241; adding RSTS gives the final 0.617/1.183.
• Progressive distillation beats one-shot distillation: Direct distillation from lengths {10, 20, 30, 40} to 50 yields 0.663/1.275 at length 10 versus PRF's 0.652/1.241, with the paper reporting larger gains for shorter observations. A t-SNE visualization of 10→50 distilled features versus native 50-step features supports better alignment for the progressive strategy.
• Mamba is the best temporal model in RPM: Comparing GRU (0.662/1.286 at length 10), Attention (0.653/1.261), and Mamba (0.652/1.241), Mamba performs best.
• Modest inference cost: On Argoverse 2, inference time and FLOPs are 0.268 s / 1.651 G at length 10, 0.236 s / 1.581 G at 20, 0.203 s / 1.513 G at 30, 0.172 s / 1.443 G at 40, and 0.140 s / 1.375 G at 50. Each additional retrospective stage adds about 0.07 G FLOPs and 0.03 s of latency. The paper states RDM and RSTS are used only during training, and RPM is disabled at inference, so RPM adds no inference cost.

Methodology in Plain English

The problem is framed as follows: given standard observations of length T_o, the paper defines shorter inputs X^v with length T_v = T_o − v·ΔT, where v ranges over {1, 2, …, τ} and τ = T_o/ΔT − 1. Instead of forcing one network step to jump from a short history to a full-length representation, PRF breaks the gap into τ small steps. Each retrospective unit Φ^v is responsible for recovering only the ΔT timesteps that separate observation length T_v from T_{v−1}. A shared encoder extracts features for all observation lengths, and the cascade of units runs between the encoder and decoder.

Inside each unit, the Retrospective Distillation Module (RDM) first conditions agent features on the encoded map via cross-attention, then runs two parallel branches: a logit branch that produces an element-wise gating vector through a sigmoid, and a residual branch that learns the feature of the omitted timesteps. The two are fused as a gated shortcut plus residual, which the paper says preserves reliable components, imputes the missing ones, and keeps gradients flowing.

The Retrospective Prediction Module (RPM) then reconstructs the omitted trajectory segment from the distilled feature, providing implicit supervision for distillation. It uses mode queries to produce coarse multimodal proposals and state queries — refined by cross-attention and Mamba sequence modeling — to refine those proposals as anchors. Because RPM always recovers a fixed ΔT timesteps regardless of observation length, one RPM is shared across all units, and it is batch-processed during training and disabled at inference.

Training uses the Rolling-Start Training Strategy: because each unit targets a specific observation length, a single sequence with a shifted start point produces training windows for several units at once. The total objective combines the decoder losses (smooth-L1 regression plus cross-entropy classification, following QCNet and DeMo settings), losses on mode and state queries in RPM, and a smooth-L1 distillation loss on RDM features.

Experiments use Argoverse 2 (250,000 scenarios from six cities; 11 s sequences sampled at 10 Hz; first 5 s history, subsequent 6 s prediction) and Argoverse 1 (324,557 scenarios from Miami and Pittsburgh; 5 s sequences at 10 Hz; first 2 s history, remaining 3 s prediction). For Argoverse 2, T_o = 50, T_f = 60, ΔT = 10, giving observation lengths {10, 20, 30, 40, 50}; for Argoverse 1, T_o = 20, T_f = 30, ΔT = 5, giving {5, 10, 15, 20}. Models are trained end-to-end for 60 epochs with batch size 16 using Adam at an initial learning rate of 0.003 and weight decay of 0.01, implemented in PyTorch on 8 Nvidia RTX 4090 GPUs. Metrics are mADE_K, mFDE_K, b-mFDE_K, and MR_K for K = 1 and K = 6.

Why This Matters

Impact on research. The paper reframes variable-length trajectory prediction as a progressive alignment problem rather than a one-shot feature mapping problem, and shows that decomposing a large information gap into a cascade of small ones measurably helps. Because PRF sits between encoder and decoder, it can be attached to existing architectures — the paper demonstrates this with two different backbones, QCNet and DeMo — which lowers the barrier for other groups to build on it.

Real-world applications (as motivated by the paper's scenarios):

• Autonomous driving perception and planning: handling vehicles that newly enter the ego vehicle's perception range.
• Occlusion and tracking recovery: handling agents re-detected after being lost due to occlusion or tracking errors.
• Safety-critical maneuver planning: the paper notes that degraded prediction can propagate to downstream planning and control, increasing risk of unsafe maneuvers and collisions.
• Fleet-scale deployment: avoiding the cost of maintaining a separate prediction model per observation length, as Isolated Training requires.

Industry relevance. Companies working on autonomous driving and ADAS need predictors that behave consistently under partial observations caused by sensor range limits, occlusion, and tracking failures. A single model that covers many observation lengths — with roughly 0.07 G FLOPs and 0.03 s added per extra retrospective stage, and 0.140 s / 1.375 G at the standard length of 50 — is a practical alternative to multi-model pipelines. The authors provide code at https://github.com/zhouhao94/PRF.

Future Directions

• Reducing inference overhead of the cascade: the paper reports that inference cost grows almost linearly as observation length decreases, so compressing or partially bypassing the retrospective stages is a natural next step.
• Extending beyond the fixed omission interval: PRF is built around a constant ΔT (10 for Argoverse 2, 5 for Argoverse 1), and the paper notes that observation lengths outside the admissible sets are truncated to the nearest shorter length. Handling irregular, non-uniformly missing history is left open.
• Broader integration with downstream planning: the paper motivates the work by the risk that prediction degradation propagates to planning and control, but does not report joint planning or closed-loop safety evaluations.
• Transferring the progressive retrospective idea to other domains: the framework is presented in the context of traffic agents; applying the same cascade-and-distill scheme to other sequence prediction settings with truncated histories is an untested extension.

Target Audience

Researchers and engineers working on trajectory prediction and motion forecasting for autonomous driving, particularly those dealing with real-world incomplete or variable-length observations. It is also relevant to practitioners who want a plug-and-play module for existing transformer-based prediction backbones, and to readers interested in knowledge distillation and progressive feature alignment more broadly. Readers without background in attention-based trajectory models, distillation losses, or multimodal forecasting metrics such as mADE_K, mFDE_K, b-mFDE_K, and MR_K will find the method section demanding.

Authors’ abstract

Trajectory prediction is critical for autonomous driving, enabling safe and efficient planning in dense, dynamic traffic. Most existing methods optimize prediction accuracy under fixed-length observations. However, real-world driving often yields variable-length, incomplete observations, posing a challenge to these methods. A common strategy is to directly map features from incomplete observations to those from complete ones. This one-shot mapping, however, struggles to learn accurate representations for short trajectories due to significant information gaps. To address this issue, we propose a Progressive Retrospective Framework (PRF), which gradually aligns features from incomplete observations with those from complete ones via a cascade of retrospective units. Each unit consists of a Retrospective Distillation Module (RDM) and a Retrospective Prediction Module (RPM), where RDM distills features and RPM recovers previous timesteps using the distilled features. Moreover, we propose a Rolling-Start Training Strategy (RSTS) that enhances data efficiency during PRF training. PRF is plug-and-play with existing methods. Extensive experiments on datasets Argoverse 2 and Argoverse 1 demonstrate the effectiveness of PRF. Code is available at https://github.com/zhouhao94/PRF.