Valeo Near-Field: a novel dataset for pedestrian intent detection

AI summary

Overview

Research area: Computer vision for intelligent vehicles, specifically multimodal pedestrian perception, 3D pose estimation, localization, and intent prediction in the near field of a vehicle.

Technical level: Intermediate. The paper is a dataset and benchmark contribution; understanding the baseline results requires some familiarity with computer vision metrics (MPJPE, ADE, accuracy) and pose estimation pipelines, but the core ideas are accessible.

Scope: This paper introduces the Valeo Near-Field (VNF) dataset, a synchronized multimodal recording of pedestrians interacting with a stationary ego-vehicle, along with baseline benchmarks for 3D localization, pose estimation, and body height estimation.

What This Paper Is About

Existing pedestrian datasets are built mainly around street crossings, pinhole cameras, and laboratory-controlled 3D pose capture, leaving a gap for the specific case of a pedestrian approaching a parked car with an intent toward it (for example, opening a door or trunk). The authors argue that without synchronized, multimodal, near-field data captured in realistic conditions, algorithms for predicting pedestrian intent around vehicles cannot be properly developed or benchmarked. The paper's goal is to release such a dataset and provide baseline results so researchers have a shared reference point.

Key Contributions

1. A new multimodal dataset (VNF): 300 sequences, each roughly one minute at 30 fps (about 540,000 frames), combining four fisheye surround-view cameras, four LiDAR sensors, ultrasonic sensors, and an MVN Awinda motion capture suit capturing 23 body joints, collected with 13 participants indoors and outdoors.

2. A synchronization and alignment protocol: Because the motion suit lacks timestamps compatible with the vehicle sensors, the authors designed a physical calibration gesture (sequential raising of left arm, right arm, then both arms) to align motion suit data with camera and LiDAR streams in time.

3. Ground-truth 3D annotation strategy: Manual LiDAR-based annotation of the pedestrian's 3D position and orientation at 3 fps is used to correct the open-loop drift accumulated by the motion suit, with all data translated into vehicle coordinates (origin at the rear axle center).

4. A benchmark suite with baseline results: A multi-stage pipeline (YOLOX for detection, a fisheye-aware fine-tuned ViTPose for 2D skeletons, and transformer-based multi-view 3D uplift/localization) evaluated across 5-meter distance zones, with metrics for 3D pose, localization, and a newly introduced Body Height Error.

A public test set of 51 sequences from 12 participants (42 outdoor, 9 indoor scenes) is released, with faces and license plates blurred.

Main Findings

• Localization accuracy is counterintuitively lower at very close range: Accuracy in the 5–10 m zone (0.86) is higher than in the 0–5 m zone (0.62), even though Average Distance Error stays consistent at 0.32 m. The authors attribute this to very close pedestrians often appearing in only one camera view, and to open vehicle doors blocking the view or invalidating calibration because the left and right cameras are mounted on those doors.

• 3D pose error follows distance, except up close: MPJPE is 201 mm in the 0–5 m zone, drops to 180 mm at 5–10 m, then rises to 183 mm (10–15 m) and 199 mm (15–20 m). Smaller pedestrian appearance at distance explains the far-end increase; missing joints outside the camera field of view likely explains the near-end value.

• The pipeline generalizes across indoor/outdoor and gender: MPJPE figures are similar for indoor and outdoor scenes and for male versus female participants, suggesting the baseline is not tuned to a single condition.

• Height estimation is systematically biased upward: The 3D pose estimator predicts pedestrians roughly 7% taller than their measured ground truth. Height error is lowest at 5–15 m (about 10 cm) and worse at the extremes, and errors are larger for female participants, which the authors connect to them being shorter on average in this dataset.

• Near-field occlusion is treated as a first-class evaluation problem: The dataset deliberately keeps partially or fully occluded pedestrians in the evaluation, requiring pose completion or tracking rather than discarding hard cases, which departs from typical object detection evaluation practice.

Methodology in Plain English

The authors mounted a full sensor suite on a stationary BMW G11 S7: four fisheye parking cameras providing 360-degree coverage, four LiDAR scanners positioned to minimize body-generated occlusions at close range, ultrasonic sensors for very short distances, and a wearable motion capture suit worn by each pedestrian.

Because the motion suit and the vehicle sensors could not be time-synced through timestamps, each recording begins with a scripted arm-raising gesture that can be recognized in both the camera footage and the suit data, establishing a common time origin. Each participant then walks from roughly 15–20 meters away and performs either an intent scenario (opening a door, sitting inside, opening the trunk) or a non-intent scenario (walking past as a passerby), with one to three pedestrians per sequence.

The motion suit registers position in an open-loop manner, so positional error accumulates over a sequence. To correct this, the researchers manually annotated the pedestrian's 3D position and orientation from top-view LiDAR scans at 3 fps and aligned the suit data to those annotations, then expressed everything in vehicle coordinates. A sanity check projects the corrected 3D skeleton back onto the fisheye images.

For the baseline, they run a four-stage inference pipeline: detect pedestrians, estimate 2D skeletons with a fisheye-geometry-aware model, uplift those skeletons to 3D, associate detections across camera views by minimizing pairwise 3D distances, and finally refine 3D locations with a transformer model that accounts for fisheye multi-view geometry. No model is retrained on VNF; results come from direct inference.

Why This Matters

Impact on research: The dataset fills a specific gap — near-field, stationary-vehicle interaction captured with fisheye geometry, multi-view overlap, and motion-capture-grade 3D ground truth. It also raises the bar for evaluation by insisting that occluded pedestrians still be scored rather than dropped, which forces progress on pose completion and tracking rather than easy-case performance.

Real-world applications:

• Automatically opening doors or the trunk when an authorized pedestrian approaches, while avoiding collisions with the moving door itself.
• Adjusting seats, mirrors, and cabin settings to match an approaching driver's body dimensions.
• Near-field collision avoidance and speed control when a pedestrian is within a few meters of the vehicle.
• Detecting pedestrian intent at crosswalks and parking lots to modulate ADAS behavior.

Industry relevance: The work originates from Valeo's research division with academic collaborators at Université Paris-Saclay and CEA List, and it targets the kind of embedded, hardware-constrained perception that production ADAS and autonomous driving systems must run. The benchmark explicitly tracks accuracy, efficiency, and scalability on embedded systems, which aligns the dataset with industrial deployment constraints rather than purely academic accuracy races.

Future Directions

• Extending to dynamic scenarios: The dataset covers a stationary ego-vehicle only. Capturing interactions where the vehicle is moving would broaden applicability and is explicitly named as future work.
• Reducing motion suit drift: The open-loop localization of the suit requires manual LiDAR correction. Better closed-loop or sensor-fused capture could remove this manual step and improve positional ground truth.
• Adding environmental diversity: Weather, lighting, and season variation are not extensively covered, which limits how well models trained on this data generalize to real deployment conditions.
• End-to-end joint optimization: The authors suggest moving beyond the current multi-stage baseline toward pipelines that jointly optimize detection, pose, localization, and intent prediction, and combining this with synthetic data augmentation or self-supervised learning.
• Refining handling of open-door frames: The authors leave it to dataset users to decide how to handle frames where open doors block the view or break camera calibration.

Target Audience

Researchers and engineers working on pedestrian perception, 3D human pose estimation, multi-view localization, and intent or trajectory prediction for intelligent vehicles. The paper is most useful to those who need realistic multimodal near-field data with 3D ground truth, and to practitioners building embedded ADAS perception pipelines who want a benchmark that reflects occlusion, fisheye distortion, and hardware constraints. Students and newcomers to autonomous driving perception can also use it as a concrete reference for how a multimodal dataset is designed, synchronized, and benchmarked.

Authors’ abstract

This paper presents a novel dataset aimed at detecting pedestrians' intentions as they approach an ego-vehicle. The dataset comprises synchronized multi-modal data, including fisheye camera feeds, lidar laser scans, ultrasonic sensor readings, and motion capture-based 3D body poses, collected across diverse real-world scenarios. Key contributions include detailed annotations of 3D body joint positions synchronized with fisheye camera images, as well as accurate 3D pedestrian positions extracted from lidar data, facilitating robust benchmarking for perception algorithms. We release a portion of the dataset along with a comprehensive benchmark suite, featuring evaluation metrics for accuracy, efficiency, and scalability on embedded systems. By addressing real-world challenges such as sensor occlusions, dynamic environments, and hardware constraints, this dataset offers a unique resource for developing and evaluating state-of-the-art algorithms in pedestrian detection, 3D pose estimation and 4D trajectory and intention prediction. Additionally, we provide baseline performance metrics using custom neural network architectures and suggest future research directions to encourage the adoption and enhancement of the dataset. This work aims to serve as a foundation for researchers seeking to advance the capabilities of intelligent vehicles in near-field scenarios.