MC-DeTra: Motion-Consistent Joint Object Detection and Socially-Aware Trajectory Forecasting in Bird's-Eye-View Images

AI summary

Overview

Research area: computer vision for autonomous driving, specifically unified (joint) object detection and multi-modal trajectory forecasting from LiDAR and HD maps in bird’s-eye-view (BEV). Technical level: Advanced. Scope: a documented open reimplementation of DeTra plus train-only motion-consistency auxiliaries that improve forecasting of moving vehicles without changing inference.

What This Paper Is About

Self-driving stacks must both detect nearby vehicles and predict where they will go. Cascaded pipelines (detect, then track, then forecast) lose sensor context and let early errors snowball; unified models try to do both from one shared BEV representation. DeTra is the strongest LiDAR-based joint model of this type, but it has no public code, and even strong models still struggle on moving actors, whose errors matter most for safety. MC-DeTra reimplements DeTra and adds three training-only losses—past-motion reconstruction, surrounding-traffic occupancy, and heading–motion consistency—plus a gradient-norm diagnostic so those losses do not fight the shared backbone. At test time the extras are dropped, so latency stays the same.

Key Contributions

1. MC-DeTra training recipe. Two annotation-derived auxiliaries (Past Reconstruction and Occupancy Auxiliary) plus a heading-consistency regularizer, with weights chosen by measuring each term’s gradient at the shared LiDAR trunk.
2. Open DeTra reimplementation. Documented approximations (bilinear deformable attention, lightweight GraphSAGE map encoder, axis-aligned gIoU, no intensity), matching hyperparameters where possible, plus a strict detection-conditioned forecasting protocol on Waymo.
3. Controlled ablation from one checkpoint. Every row is fine-tuned from the same initializer under the same schedule, so gains are not confounded by detector strength or evaluation choices.
4. Released artifacts. Code, configs, and evaluation tooling at https://github.com/diuzhevVlad/MC-DeTra.

Main Findings

• Detection is preserved. Every auxiliary configuration stays within ±0.16 AP@0.7 of a no-auxiliary fine-tune; the full model is essentially tied (71.11 vs. 71.00). The ~1.2 AP@0.7 lift over the raw initializer comes from extra training, not from the new losses.
• Calibrated full model leads on displacement metrics. MC-DeTra (PR/OA/HC weights 0.05 / 0.30 / 0.25) achieves the best dynamic minFDE₆ (1.9915), minADE₆, and brier-minFDE₆—about 2.4% better dynamic minFDE₆ than the initializer and 2.9% better than plain fine-tuning. Controller A (adaptive PR/OA weights) wins miss rate.
• Occupancy is the strongest lever. Raising OA weight from 0.20 to 0.30 is the largest single gain. At the shared trunk, OA contributes ~35% of the forecast gradient; Past Reconstruction ~6.4%; Heading Consistency only ~0.02% even at weight 0.25.
• Heading consistency is metric-specific and easily over-weighted. At 0.25 it helps ADE and brier-FDE while staying nearly neutral on FDE/MR; at 0.50 or as a large solo loss it hurts. A proposed trajectory–occupancy consistency term was gradient-starved at all stable weights and was disabled.
• Gains are small, distributed, and inference-free. Over ~135k co-detected dynamic vehicles, minFDE₆ falls for 54.6% and rises for 45.4%, with near-balanced large-error tails. Deployed latency matches the plain reimplementation (360.7 ms on RTX 3080 Laptop, 292.4 ms on A100, 104.9 ms on H200) and stays inside a 2 Hz budget.
• Caveats the authors state. Gains are modest and concentrated on vehicles that are actually moving; the reimplementation still trails published DeTra (shorter training, simpler map encoder); results are single-run; auxiliary weights were chosen on the same validation protocol used for reporting.

Methodology in Plain English

The base model turns a LiDAR sweep and an HD map into a BEV feature grid, proposes actor boxes with a CenterNet-style head, then refines a volume of trajectories (current pose plus 6 future modes over 5 seconds) with transformer blocks that attend to LiDAR, map tokens, other actors, and alternative futures. The first pose of each trajectory is the detection.

MC-DeTra does not change that architecture at test time. During training only, it attaches three extra heads/losses to representations that already exist:

• Past Reconstruction (PR). From the same refined query used for the current box, a small GRU predicts where the actor just came from. Reconstructing history forces the query to encode short-term dynamics that also help predict the future.
• Occupancy Auxiliary (OA). A BEV head predicts occupancy and flow of surrounding traffic. That grounds the shared backbone in “where mass is and how it is moving”—the social context of nearby vehicles—without replacing trajectory outputs with occupancy.
• Heading Consistency (HC). For actors predicted to move more than 0.5 m, the detected box yaw is pulled toward the direction of the model’s own best-mode forecast. The future waypoints are already pinned by a strong Laplace loss, so in practice the box rotates to match the motion rather than the motion being bent toward the box.

Because all losses share one backbone, a term that looks important on paper can still be gradient-starved. The authors measure each weighted gradient at the LiDAR trunk as a fraction of the forecast gradient, then set static weights (OA highest, PR smaller, HC tiny) or run a simple controller that retargets PR and OA toward those fractions. All auxiliaries are removed at inference.

Evaluation is deliberately strict: vehicle-only, 80% recall, rotated-IoU 0.5 true-positive gate, and a stationary/dynamic split so parked cars cannot inflate scores. Every ablation starts from one DeTra-reproduction checkpoint so comparisons are like-for-like.

Why This Matters

Joint detection–forecasting is a live research direction because cascaded stacks compound errors and throw away sensor context. This paper’s distinctive move is not a new backbone, but train-only inductive biases plus a diagnostic that shows which auxiliary actually moves the shared representation. That is reusable for any shared-backbone multi-task driver, not only DeTra.

Real-world applications:

• Onboard prediction for robotaxis and AV stacks that already rasterize LiDAR+maps into BEV and must stay inside a ~2 Hz budget.
• Safety-critical scoring of moving actors, where static-vehicle metrics hide the errors that cause collisions.
• Multi-task training of perception–prediction networks (and related camera-only stacks such as UniAD-style models) where extra losses fight over one trunk.
• Reproducible baselines for labs that could not previously extend DeTra because no code existed.

Industry relevance is immediate for teams shipping LiDAR (or LiDAR+map) planners: the method adds no latency, preserves detection AP, and targets the moving-actor regime that product metrics often underweight. The gradient-norm recipe is also a practical tool for anyone balancing detection, forecasting, occupancy, and consistency terms without weeks of grid search.

Future Directions

• Match the original DeTra map encoder (connected lane graph) and train for the full 240k steps so the reproduction is no longer the limiting factor.
• Make scene-level trajectory–occupancy consistency actually contribute gradient instead of being starved, so decoded modes stay consistent with a dense occupancy field.
• Extend beyond vehicles to pedestrians and cyclists, where social context and heading–motion alignment behave differently.
• Test whether the gradient-norm controller helps more when there are many competing auxiliaries that are expensive to tune by hand; also run multi-seed training and held-out weight selection so the small reported deltas can be trusted statistically.

Target Audience

Researchers and engineers working on autonomous-vehicle perception and prediction, especially anyone building or training joint detector–forecasters, BEV transformers, or multi-task models with competing losses. Also useful for practitioners who need a reproducible DeTra-style baseline and a concrete recipe for adding occupancy/past-motion supervision without touching inference. Less suitable as a first introduction to detection or forecasting; the paper assumes familiarity with BEV pipelines, DETR-style refinement, and Waymo-style metrics.

Authors’ abstract

Unified models for object detection and trajectory forecasting aim to merge perception and prediction for autonomous driving, refining actor trajectories directly over shared bird's-eye-view (BEV) images rasterized from LiDAR and high-definition maps. Their accuracy on dynamic, moving actors, however, remains the hardest part of the task, and the strongest such model, DeTra, has no public implementation. We contribute an openly released DeTra reimplementation with documented approximations, and on top of it MC-DeTra: a family of motion-consistency mechanisms that add supervision through two annotation-derived auxiliary signals -- each actor's observed past motion and the occupancy of the surrounding traffic that forms its social context -- and one inter-output consistency constraint that aligns an actor's predicted heading with its predicted direction of motion. Every proposed loss is train-only and inference-safe: it shapes the shared BEV representation during training and is removed at test time, adding no inference latency. On the Waymo Open Dataset, evaluated under a strict, detection-conditioned forecasting protocol, MC-DeTra improves dynamic, socially-situated trajectory forecasting while preserving or improving detection accuracy; a gradient-based loss-calibration analysis exposes how the auxiliary objectives compete at the shared backbone, and our ablation identifies which signals contribute most. We release code, configurations, and evaluation tooling at https://github.com/diuzhevVlad/MC-DeTra.