mmWalk: Towards Multi-modal Multi-view Walking Assistance

AI summary

Overview

Research area: Computer vision for accessibility — multi-modal datasets, vision-language models (VLMs), and assistive navigation for people with blindness or low vision (BLV), with connections to embodied AI and robotics.

Technical level: Intermediate. The paper's core is a dataset and benchmark rather than a new architecture, so no deep math is required, but familiarity with VLMs, visual question answering (VQA), and benchmarking metrics helps.

Scope: The paper introduces mmWalk, a large simulated multi-view, multi-modal walking dataset with an accompanying VQA benchmark for BLV-safe outdoor navigation, and benchmarks current VLMs against it.

What This Paper Is About

People who are blind or have low vision face serious risks when navigating outdoors — crossing roads, uneven pavement, narrow paths, and obstacles cause falls and injuries, with surveys cited showing over 63% have experienced an injury-causing incident. Existing datasets for assistive vision focus on indoor scenes, object recognition, or generic VQA, and they rarely capture safety-critical hazards from multiple viewpoints. This paper builds a large synthetic dataset that combines multi-view (walker, guide dog, drone), multi-modal (RGB, depth, semantic), panoramic, sequential walking data with explicit annotations of BLV-relevant corner cases and navigational landmarks, plus a VQA benchmark to measure how well today's VLMs handle these tasks.

Key Contributions

1. The mmWalk dataset: A multi-view, multi-modal walking dataset collected in the CARLA simulator, containing 120 manually controlled trajectories across 7 urban scenarios and 5 weather conditions, totaling 62,167 synchronized frames and over 559K panoramic images in RGB, depth, and semantic segmentation modalities. Each frame carries contextual metadata including trajectory descriptions, action labels, corner cases, and landmarks.

2. BLV-specific annotation scheme: The dataset explicitly encodes 8 outdoor corner cases (cross road in danger, cross road, uneven road, barrier, narrow path, entrance locating, high obstacles, deadend) drawn from prior literature, plus 18 navigational landmarks selected from the ATmaps survey of landmarks important to blind pedestrians.

3. The mmWalkVQA benchmark: A pipeline and resulting benchmark of 69,391 visual question-answer triplets spanning 9 categories and 3 difficulty levels, generated with GPT-4o from multi-view RGB plus BLV-friendly clockwise spatial descriptions derived from depth and semantic data, then automatically and manually filtered.

4. A benchmark of state-of-the-art VLMs with generalization validation: Systematic zero-shot, few-shot, and fine-tuned evaluation of six open-source VLMs, including a demonstration that a model fine-tuned on mmWalk transfers to the real-world EgoTextVQA dataset.

Main Findings

• Current VLMs struggle substantially. Across all nine VQA categories, the best zero-shot model (InternVL2-8B) reached only 41.35 normalized points, and the best 3-shot model (LLaVA-Next-7B) reached 43.71. Risk assessment and navigational landmark tasks in particular exposed weaknesses.

• Spatial reasoning (M1) is the hardest category. The M1 Spatial task required combining all input modalities and produced the lowest scores for both zero-shot and 3-shot models, indicating broad failures in spatial understanding.

• Fine-tuning on mmWalk helps considerably. InternVL2-8B improved from 41.35 to 55.21 (+13.86 points) after fine-tuning, outperforming every zero-shot and few-shot configuration — evidence that the dataset provides useful training signal.

• Scene complexity drives difficulty. Large open areas (Plaza, Parking) yielded higher scores; intricate settings like Corner, Busstop, and Gasstation produced lower scores across models.

• Full multi-view input is generally best, but drone views can hurt risk assessment. Ablating input views showed walker+dog+drone outperformed single- or dual-view configurations on average. However, for the H1 risk assessment task, drone imagery was the least effective, likely because an elevated perspective misses ground-level hazards like uneven surfaces. The paper notes some tasks were answered equally well from any single view.

• LLM-as-a-judge correlates well with human ratings. A human study of 3,575 sampled VQA responses found an average Spearman's rho of 0.864 between human and GPT-4o scores, rising to 0.924 for the fine-tuned model, supporting the reliability of the automatic evaluation pipeline.

• Fine-tuned models generalize to real outdoor data. On EgoTextVQA, the mmWalk-finetuned InternVL2-8B improved the overall score from 18.5/1.4 (frame input) to 21.55/1.92, with gains in Location, Description, Intention Reasoning, and Others categories.

Methodology in Plain English

The researchers built a simulated environment using CARLA, an open-source autonomous driving simulator, and manually drove three virtual "sensor carriers" — a pedestrian walker, a guide dog, and a drone — along 120 curated routes covering seven scenario types and five weather conditions. For each moment in time, they captured panoramic images (constructed from cubemaps into equirectangular format) in RGB, depth, and semantic segmentation, along with pedestrian action labels and IMU data.

To create the question-answer benchmark, they sampled 7,570 frames and fed GPT-4o the RGB images from all three views. Rather than passing raw depth and semantic maps to the language model, they converted those into strings describing object positions in a clockwise order around the pedestrian, mirroring how a person might describe surroundings to someone who cannot see. Combined with trajectory metadata and a few hand-written example pairs per question category, GPT-4o generated candidate VQA triples. These were filtered automatically by keyword screening for evasive answers, spot-checked manually, and split by trajectory rather than random frame so that test scenarios do not leak into training.

For evaluation, they selected six open-source VLMs in the 7-8B parameter range and scored their answers using GPT-4o-mini as an automated judge on a 1-5 scale, later normalized to 0-100. They ran zero-shot, 3-shot, and fine-tuned settings, and cross-checked the automated judge against human annotators on a subset. Finally, they tested whether mmWalk fine-tuning transfers to EgoTextVQA, a real-world outdoor VQA dataset.

Why This Matters

The paper targets a demographic of over 2.2 billion people affected by vision impairment, and it presents evidence that today's best open-source VLMs are far from reliable on the safety-critical questions that matter most to that population, such as "is this path hazardous?" This reframes walking assistance as a distinct and under-served benchmark problem rather than a byproduct of general VQA or autonomous driving research.

Real-world applications:

• Smartphone or wearable navigation aids that warn BLV users about uneven ground, narrow paths, or approaching roads.
• Guide-dog-supporting systems that fuse dog-mounted and user-mounted camera views for obstacle awareness.
• Drone-assisted navigation for BLV pedestrians in complex urban environments, where an aerial view supplements ground-level perception.
• Landmark-based wayfinding tools that identify entrances, poles, and traffic lights to help users reach destinations.

Industry relevance: The findings bear on developers of assistive technology, VLM builders focused on embodied and egocentric reasoning, robotics and autonomous systems teams, and accessibility standards bodies. The paper is also notable for releasing data and code openly, which supports reproducible benchmarking of inclusive AI.

Future Directions

• Closing the sim-to-real gap. The authors acknowledge that synthetic-only data may introduce bias and fail to capture the full diversity of real-world BLV experience. Collecting real-world data and adapting the VQA generation pipeline to it is a natural next step.

• Better evaluation metrics. LLM-as-a-judge proved reliable on average but can carry its own biases; the paper calls for investigating those biases and developing more stable, less model-dependent metrics.

• Exploiting underused modalities. IMU data, sequential frames, and semantic labels are present in mmWalk but only lightly leveraged. Using them more fully could unlock tasks beyond VQA, such as image captioning and embodied AI training.

• Improving multi-view fusion and risk reasoning. The finding that elevated drone views miss ground-level hazards, and that spatial questions are consistently hardest, suggests open questions about how to weight and fuse heterogeneous viewpoints for safety-critical judgment.

Target Audience

Researchers and practitioners in assistive computer vision, accessibility technology, and multimodal learning benefit most from this paper, particularly those working on VQA benchmarks, egocentric or panoramic perception, and embodied AI. VLM developers looking for challenging evaluation suites in safety-sensitive domains, and engineers building navigation aids for BLV users, will find the dataset, benchmark, and failure analysis directly actionable.

Authors’ abstract

Walking assistance in extreme or complex environments remains a significant challenge for people with blindness or low vision (BLV), largely due to the lack of a holistic scene understanding. Motivated by the real-world needs of the BLV community, we build mmWalk, a simulated multi-modal dataset that integrates multi-view sensor and accessibility-oriented features for outdoor safe navigation. Our dataset comprises 120 manually controlled, scenario-categorized walking trajectories with 62k synchronized frames. It contains over 559k panoramic images across RGB, depth, and semantic modalities. Furthermore, to emphasize real-world relevance, each trajectory involves outdoor corner cases and accessibility-specific landmarks for BLV users. Additionally, we generate mmWalkVQA, a VQA benchmark with over 69k visual question-answer triplets across 9 categories tailored for safe and informed walking assistance. We evaluate state-of-the-art Vision-Language Models (VLMs) using zero- and few-shot settings and found they struggle with our risk assessment and navigational tasks. We validate our mmWalk-finetuned model on real-world datasets and show the effectiveness of our dataset for advancing multi-modal walking assistance.