PEDESTRIAN: An Egocentric Vision Dataset for Obstacle Detection on Pavements

AI summary

Overview

• Research area: Computer vision / egocentric (first-person) vision for pedestrian safety, applied to urban sidewalk obstacle detection.
• Technical level: Intermediate. The dataset construction is easy to follow; interpreting the benchmark results requires some familiarity with deep learning classification models, transfer learning, and layer freezing.
• Scope: The paper introduces a new publicly released egocentric video/image dataset of sidewalk obstacles and reports baseline classification results from fine-tuning 16 pretrained deep learning architectures on it.

What This Paper Is About

Sidewalks in cities are frequently blocked by objects that force pedestrians to step into unsafe space, but few datasets exist that focus specifically on these obstacles from a walker's point of view. The authors build and release the PEDESTRIAN dataset, captured with smartphone cameras at chest height to mimic a pedestrian's viewpoint, covering 29 obstacle types recorded in Nicosia, Cyprus. They then fine-tune a range of deep learning models on the data to establish benchmark performance for obstacle detection and recognition.

Key Contributions

1. A new egocentric dataset for pedestrian safety. The PEDESTRIAN dataset contains 340 videos of 29 different obstacle types found on urban sidewalks, collected using smartphone cameras from a pedestrian's point of view. This expands on the authors' previous work, where the dataset included 9 obstacle types.
2. A structured obstacle taxonomy. The 29 obstacle types are organized into 3 high-level categories (Physical Condition, Infrastructure, Temporary) and 8 subcategories (Construction, Damage, General Interest, Layout, Other, Street Decor, Street Furniture, Vehicles).
3. A balanced benchmark subset. A subset of 14500 images (29 obstacle types × 500 images each) was extracted to balance the data in terms of images per obstacle type for fair model comparison.
4. A benchmark of 16 deep learning architectures. The authors fine-tuned 16 pretrained model architectures at all three taxonomy levels, both with frozen and unfrozen layers, performing 480 fine-tuning experiments in total to set baseline results.

Main Findings

• Dataset scale and composition: The dataset contains 340 videos totaling 2332 seconds and 82120 frames/images. Video duration varies from 1s to 17s. All videos are Full HD 1080 × 1920 pixels in portrait orientation.
• Three lighting conditions: Data was captured under daytime with sunlight (117 videos, 829s, 24884 frames), night (116 videos, 784s, 33624 frames) and cloudy day (107 videos, 718s, 23612 frames).
• Two capture devices: Videos were captured with a Xiaomi Mi Mix 3 (60 FPS) and an Apple iPhone 7 (30 FPS), to prevent models from overfitting to a single device. The iPhone 7 contributed 275 videos (1927s, 57837 frames) and the Mi Mix 3 contributed 65 videos (405s, 24283 frames).
• Category-level breakdown: Infrastructure accounted for 153 videos (1029s, 35810 frames), Temporary for 128 videos (885s, 31159 frames) and Physical Condition for 59 videos (417s, 15151 frames).
• Obstacle type classification is near-saturated: At the Obstacle type level, all models produced accuracies over 99%. ConvNeXt-Small reached 99.96% with frozen layers, while EfficientNetV2-S led with 99.89% without layer freezing. The lowest reported values were ResNet-152 at 99.52% (unfrozen layers) and ResNet-18 at 99.33% (frozen layers).
• Layer freezing has a level-dependent effect: Freezing layers made negligible difference at the Obstacle type level, but for the Category and Subcategory levels leaving layers unfrozen generally yielded better results.
• Category level best results: EfficientNet-B0 achieved the highest accuracy of 99.90% with unfrozen layers, while ConvNeXt-Large topped the frozen-layer setting with 99.61%.
• Subcategory level best results: EfficientNetV2-S reached the highest unfrozen accuracy at 99.92%, with ConvNeXt-Large best among frozen-layer models at 99.86%.
• Smaller models struggle more on coarser levels: Models with fewer parameters were generally less adept at Category and Subcategory classification. The authors attribute this partly to the balanced subset being balanced only for Obstacle type, not for Category or Subcategory, and partly to the greater feature abstraction required at those levels.
• Model size range tested: The smallest architecture was MobileNetV3-Small at 2.5M parameters and the largest was ConvNeXt-Large at 197.8M parameters. All used ImageNet-1K pretrained weights available through PyTorch.

Methodology in Plain English

The authors first studied pedestrian safety problems in modern cities and identified 29 kinds of obstacles commonly found on sidewalks, grouping them into 3 categories and 8 subcategories. They then recorded 340 videos, each showing one obstacle type, using two smartphone cameras held at chest height to simulate a wearable camera's viewpoint, and covering three lighting conditions. Individual frames were extracted from the videos to form the dataset, and a privacy step blurred faces, car license plates, and signs to comply with GDPR.

For benchmarking, they extracted a balanced subset of 500 images per obstacle type (14500 images total); 500 was chosen because the smallest available pool for any obstacle type was Crowded Pavement with 694 frames. A random 70% / 20% / 10% split into training, validation and testing was used, with training for 30 epochs, batches of 32 images, the Adam optimizer, and a learning rate of 0.001. Each of the 16 architectures was fine-tuned to classify at all three taxonomy levels, either freezing all layers except the last or not freezing any layers, and every combination was repeated 5 times with different random seeds. That gives 16 architectures × 3 taxonomy levels × 2 freezing settings × 5 repetitions = 480 fine-tuning experiments.

Why This Matters

• Impact on research: The paper addresses a gap identified in prior reviews of pedestrian safety models — that relatively few egocentric datasets focus specifically on pedestrian obstacle detection. It provides the dataset and reproducible benchmark code, including PyTorch Dataset class implementations, so results can be replicated.
• Real-world applications:
  • Smartphone applications that warn pedestrians in real time about obstacles in their path.
  • Assistive systems for people using mobility aids, including those with mobility impairments and the elderly, as explicitly noted by the authors.
  • Reporting tools that let pedestrians flag hazards and transmit their geographic location to authorities.
  • Safety alerts for pedestrians who are distracted by mobile phone use.
• Industry relevance: The dataset targets mobile hardware constraints, which explains the inclusion of compact architectures such as MobileNetV2 (3.5M parameters) and MobileNetV3-Small (2.5M parameters) alongside larger models. This makes the benchmark relevant to developers of on-device vision applications.
• Accessibility of the resource: The full dataset, including videos and the balanced benchmark subset, is distributed as a 5.9 GB zip archive through Zenodo (DOI: 10.5281/zenodo.10907945).
• Positioning relative to prior datasets: The paper compares PEDESTRIAN against datasets such as EPIC-Kitchens, Ego4D, Charades-Ego, EGTEA Gaze+, UT Egocentric, EgoObjects, EgoGesture, TerraFirma, DR(eye)Ve, EGO-CH and TREK-150, noting that only a few target pedestrian safety and none exclusively target obstacles obstructing sidewalks.

Future Directions

• Geographic expansion: Collecting images from other cities and countries, which the authors name as a main future goal, is intended to improve the robustness of obstacle detection models. They plan to explore synergies with partners in other countries.
• Additional data modalities: Incorporating audio and sensor data into the dataset to enable more comprehensive obstacle detection solutions.
• Federated learning: Investigating federated learning to enable collaborative model improvement while addressing privacy concerns.
• Interdisciplinary collaboration: Working with urban planners to gain insight into practical implementations of these technologies.

Target Audience

This paper is most useful for computer vision and mobile computing researchers who need a first-person dataset for training or evaluating obstacle detection models, and for developers building pedestrian-facing safety applications on smartphones. It also serves urban planners and accessibility researchers interested in how visual data can characterize sidewalk hazards, and deep learning practitioners looking for a benchmark that compares many architecture families (MobileNet, EfficientNet, GoogLeNet, DenseNet, ResNet, ConvNeXt) under a consistent fine-tuning protocol.

Authors’ abstract

Walking has always been a primary mode of transportation and is recognized as an essential activity for maintaining good health. Despite the need for safe walking conditions in urban environments, sidewalks are frequently obstructed by various obstacles that hinder free pedestrian movement. Any object obstructing a pedestrian's path can pose a safety hazard. The advancement of pervasive computing and egocentric vision techniques offers the potential to design systems that can automatically detect such obstacles in real time, thereby enhancing pedestrian safety. The development of effective and efficient identification algorithms relies on the availability of comprehensive and well-balanced datasets of egocentric data. In this work, we introduce the PEDESTRIAN dataset, comprising egocentric data for 29 different obstacles commonly found on urban sidewalks. A total of 340 videos were collected using mobile phone cameras, capturing a pedestrian's point of view. Additionally, we present the results of a series of experiments that involved training several state-of-the-art deep learning algorithms using the proposed dataset, which can be used as a benchmark for obstacle detection and recognition tasks. The dataset can be used for training pavement obstacle detectors to enhance the safety of pedestrians in urban areas.