DriveMotion: A Large-Scale Multi-Source Benchmark for Driver Motion Sequence Modeling and Forecasting

AI summary

Overview

Research area: Computer vision and human motion forecasting, applied to in-cabin driver monitoring.

Technical level: Intermediate. The framework is understandable without deep domain expertise, but the evaluation metrics (MPJPE, Part-State F1) and pose-estimation terminology assume some background in computer vision or sequence modeling.

Scope: This paper introduces DriveMotion, a 393-hour, 360-driver multi-source benchmark of continuous in-cabin whole-body motion sequences, along with two evaluation protocols for forecasting how driver movement evolves over the next several seconds.

What This Paper Is About

Driver-centric datasets typically treat driver understanding as a recognition problem: classify a predefined behavior from a short labeled video clip. Human motion forecasting, meanwhile, has mature benchmarks, but they are built almost entirely from laboratory motion capture or outdoor pedestrians, not seated, partially occluded drivers filmed from heterogeneous cabin cameras. DriveMotion fills that gap by assembling long, continuous in-cabin motion sequences into a unified representation and defining a standardized task for predicting future driver movement, including around the moments when drivers actually do something, such as initiating a turn or a lane change.

Key Contributions

1. A multi-source in-cabin motion corpus. 393 hours of 133-keypoint COCO-WholeBody motion sequences at 10 Hz from 360 drivers, integrating long naturalistic fleet recordings with synchronized CAN signals and road-facing video, a curated set of public in-cabin videos chosen for camera-view and body-visibility variation, and the AIDE dataset for behavior and emotion annotations. All sources pass through one extraction pipeline and carry per-joint validity masks, so occluded or unobservable joints are marked rather than imputed.

2. A driver motion forecasting benchmark with dynamics-anchored evaluation. The paper identifies vehicle maneuvers offline from CAN signals and places evaluation windows around them (pre-maneuver, post-maneuver, and matched stable-driving controls), while never giving CAN to the model at inference. This counters the problem that uniformly sampled windows are dominated by near-stillness, where simply repeating the last pose is a strong baseline.

3. A robustness protocol under observation shift. A second protocol trains on the fleet source alone versus the full corpus and evaluates on held-out web creators, with breakdowns by annotated viewpoint and visibility level, to test whether models generalize across differently recorded driver imagery.

4. Reproducible infrastructure. Identity-disjoint splits grouped by recording device and creator channel, fixed hashed evaluation subsets, frozen state thresholds, a full curation ledger, reference implementations for persistence, GRU, MLP, Transformer, CVAE, diffusion, motion-token, and LLM-style baselines, and skeleton-only video releases that avoid distributing identity-preserving crops.

Main Findings

• Driver motion is predictable mainly in short, motion-active bursts. Under uniform sampling, a persistence baseline (repeat the last observed pose) reaches 10.2 MPJPE@4s and the best learned model 9.0, only a 12% margin. On the motion-active quartile of the same test set the margin roughly doubles, and on the quiet half every model ties with persistence. The natural distribution is dominated by near-stillness.

• Maneuvers are the informative window. Arm motion in pre-maneuver windows is 3.4 times larger than in stable-driving controls drawn from the same routes, and head motion rises seconds before maneuver onset, consistent with anticipatory visual checking. Evaluated on these anchored windows, learned models reduce error over persistence by 13–15% (macro MPJPE 7.75 down to 6.62–6.75).

• Maneuver-enriched training substantially improves behavioral prediction. Part-State F1@2s rises from 0.215 for the zero-motion reference to 0.309 for the enriched Transformer, a 44% improvement, with the gains concentrated in the pre- and post-maneuver strata. The dose–response study shows F1 improving monotonically as the fraction of anchored training windows increases, at a modest coordinate-accuracy cost.

• Training-source diversity matters more than model size for cross-view transfer. A Transformer trained only on fleet data reaches 19.68 MPJPE on held-out web drivers, 45% worse than repeating the last pose. Training on the full corpus drops this to 12.18, a 38% reduction and the only configuration that beats persistence on web footage, while barely changing in-domain fleet performance. The same architecture, schedule, and parameter budget are used in both cases, so the entire effect comes from training composition. The largest gain is on front views, the viewpoint farthest from the fleet camera.

• Scaling plateaus early. Going from 7.6M to 44.5M parameters improves every evaluation stratum; 133M holds those values without extending them. Above roughly 45M parameters, capacity is no longer the limiting factor on this corpus.

• Exterior scene features do not help coordinates but do help behavior. Adding frozen exterior embeddings leaves MPJPE essentially unchanged while improving Part-State F1 from 0.292 to 0.309 and reducing the freeze rate from 0.51 to 0.40, concentrated under maneuver-enriched training.

• Current forecasters under-express the motion they detect. A monotone per-part amplitude recalibration fitted only on anchored training windows raises Transformer-L macro Part-State F1 from 0.287 to 0.517 without touching the predicted coordinates. A linear probe on frozen forecaster features reaches 0.52–0.58 per-part F1, and a simple persistence-of-amplitude prior beats every trajectory-derived predictor. The interpretation is that models often identify upcoming motion but under-estimate its amplitude when decoding continuous trajectories.

• Anticipatory signal exists for some maneuvers and not others. A linear probe on 8 seconds of observed pose predicts left turns and hard braking at roughly 2.8 and 2.9 times prevalence AUPRC, and deceleration and acceleration at 1.5–1.9 times, but lane-change precursors are not separable from chance using pose alone. A from-scratch classifier with matched capacity performs comparably, so the signal lives in the pose dynamics rather than in forecaster-specific features.

• Head-pose evaluation requires hardware-grounded labels. Vision-derived head pose on web footage produces spurious excursions in 97% of windows, versus a 6.1% rate for the hardware reference, so head metrics are scored only where device-derived labels exist.

• Extracted motion passed five audits. Of 1,395 raw fleet routes, 12 were removed because the extracted skeleton disagreed with the device face-position channel and 54 more failed extraction or quality gates. On a 3,000-window sample, 98.2% of frame-to-frame trunk-keypoint transitions fall below 0.25 shoulder widths, 95.6% of arm transitions do, and core-joint validity averages 95.0%.

Methodology in Plain English

The authors did not simply repackage existing datasets. They funneled three very different video sources through a single extraction pipeline so that everything ends up as the same kind of object: a 133-slot body keypoint sequence sampled ten times a second, with confidence values, a mask marking which joints are actually visible, head pose, optional vehicle signals like speed and steering, and metadata about the source, camera view, and visibility.

The three sources play distinct roles. Long naturalistic fleet recordings supply temporal continuity and vehicle-dynamics signals. Curated public in-cabin videos supply variation in where the camera sits and how much of the driver is visible. AIDE supplies semantic labels that let the researchers stratify errors by driver activity. Because the camera view and body visibility vary so much, the pipeline marks unobservable joints explicitly instead of guessing them, so evaluation can be conditioned on partial observability rather than measuring imputation artifacts.

The most consequential design decision concerns how to pick test windows. In ordinary driving, drivers sit still for long stretches, so random sampling produces mostly trivial cases where the persistence baseline wins. The researchers instead identify maneuvers offline from CAN signals and build evaluation windows around them: a pre-maneuver window spanning the transition into execution, a post-maneuver window covering the four seconds after onset, and matched stable-driving controls sampled from the same routes. The CAN data is used only to select windows; it is never available to the model at prediction time. Splits are fixed before events are mined, so anchoring cannot leak information across the train-test boundary.

The forecasting task itself is straightforward: given 8 seconds of observed motion, predict the next 4 seconds of keypoints and head pose. Models receive the motion history, optionally with frozen exterior scene features from the road-facing camera covering the observation window only. They are scored on geometric accuracy (mean per-joint error at 4 seconds over 23 points that are observable across all camera placements) and on behavioral accuracy (whether the head, torso, and each arm will be still, move a little, or move a lot at 2 seconds), plus a freeze rate that flags forecasts collapsing toward stillness.

For the generalization study, the researchers compare a model trained only on fleet data against the same architecture trained on the full multi-source corpus, evaluating both on held-out web creators with breakdowns by viewpoint and visibility level.

Why This Matters

Impact on research. Driver monitoring research has largely been organized around clip-level recognition, and human motion forecasting has largely been organized around laboratory or outdoor capture. DriveMotion sits at the intersection, giving forecasting researchers a realistic in-cabin domain with hard partial observability, and giving driver-monitoring researchers a continuous prediction task rather than a classification task. The paper's most provocative result for the field is that behavioral accuracy and geometric accuracy favor different models, and that trajectory decoding discards a large amount of predictive information that is demonstrably present in model features. That reframes what the open problem is: not just better forecasts, but better expression of what forecasters already know.

Real-world applications:

• Takeover readiness in automated driving. Predicting whether a driver's hands, head, or torso are about to move gives a leading indicator of whether the driver is preparing to retake control, which is exactly the signal conditional-automation handover systems need.

• Distraction and attention monitoring. Anticipatory head and arm motion around maneuvers can distinguish a driver who is checking mirrors and preparing to change lanes from one who is not engaged with the driving task.

• Crash-adjacent behavior analysis. Anchoring evaluation to braking, acceleration, and turn events creates a natural frame for studying driver response timing and pre-event posture.

• Driver-state modeling for policy and insurance. Continuous behavioral signals around maneuvers could inform risk-aware monitoring and selective-sensing systems that only sample cameras when something is likely to happen.

Industry relevance. Automotive OEMs, fleet operators, and driver-monitoring-system vendors all depend on camera-based cabin sensing, and most current production systems are built on recognition heads rather than forecasting. A standardized continuous forecasting benchmark, with identity-disjoint splits and fixed evaluation subsets that can be reproduced exactly, gives vendors a common yardstick for a capability they are beginning to ship. The finding that training-source diversity drives cross-view transfer far more than model scaling is directly actionable for anyone assembling cabin-camera training data, since collecting more footage from more viewpoints is cheaper than scaling models.

Future Directions

• Closing the gap between ranked and decoded motion states. The paper shows that forecaster features contain far more future motion-state information than decoded trajectories express, and that a simple per-part amplitude recalibration recovers much of it. Whether a principled decoding objective, a better training loss, or a different output parameterization can close this gap without a post-hoc calibration step remains open.

• Extending intent anticipation beyond what pose alone supports. Left turns and hard braking are predictable from pre-maneuver pose, but lane-change precursors are not. The paper leaves open whether lane-change signal resides in head pose, gaze, hand position at the wheel, or requires exterior context, and what that implies for pose-only forecasters.

• Richer exterior and scene encoding. The released exterior features are deliberately simple frozen ResNet-50 embeddings, and the authors note stronger scene encoders remain untested. Given that these features improve behavioral prediction and reduce freeze rate without helping coordinates, more expressive context representations are an obvious next target.

• Growing the corpus and addressing remaining gaps. The collection recipe is designed to be extendable, with the fleet still recording, the web pipeline able to ingest new public videos, and further datasets re-mappable as AIDE was. The paper also names structural limitations worth addressing: no metric 3D ground truth, no synchronized multi-angle captures of the same driver, inconsistent head-pose provenance across sources, and a web viewpoint mixture shaped by creator recording habits rather than deployment cameras. Policy-gradient fine-tuning of the token model with the Part-State metric as reward did not beat its supervised baseline and is reported as an open direction.

Target Audience

Researchers working on human motion forecasting who want to move beyond laboratory and pedestrian benchmarks; computer vision and machine learning practitioners building driver-monitoring or in-cabin sensing systems; automotive and fleet-safety engineers who need a reproducible way to compare continuous driver-behavior models; and benchmark designers interested in the general problem of evaluation on data where the dominant behavior is inactivity, which is a methodological contribution that transfers well beyond driving.

Authors’ abstract

Driver motion can provide cues to ongoing behavior, attention, and near-term driving intent. However, most existing driver-centric datasets focus on recognizing predefined driver behaviors from short video clips, while human motion forecasting benchmarks largely target motion outside the vehicle. We introduce DriveMotion, a multi-source benchmark for continuous driver motion forecasting. DriveMotion contains 393 hours of 133-keypoint motion sequences at 10 Hz from 360 drivers, integrating naturalistic driving data, curated public in-cabin videos, and the AIDE dataset into a unified representation with per-joint validity masks and synchronized driving context. Naturalistic driving contains long periods of limited body movement, making uniformly sampled evaluation dominated by persistence and less sensitive to brief but behaviorally meaningful motion. To address this, we use dynamics-anchored evaluation, placing forecasting windows around vehicle maneuvers identified offline from CAN signals without providing CAN to the model at inference. Arm motion in pre-maneuver windows is 3.4x greater than in route-matched stable-driving controls. On these anchored windows, learned models reduce forecasting error over persistence by up to 15%, while maneuver-enriched training improves forecast-derived Part-State F1 by 44% over the zero-motion reference. Training on the full multi-source corpus further reduces forecasting error on held-out web drivers by 38% compared with BATON-only training. DriveMotion provides identity-disjoint splits, fixed evaluation subsets, and reference implementations for reproducible evaluation of continuous driver motion forecasting. The dataset and benchmark are available at https://huggingface.co/datasets/HenryYHW/DriveMotion