Research
WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation
Overview - Research area: Computer vision and multimodal AI — specifically pixel-grounded Large Vision–Language Models (LVLMs) applied to accessibility-aware pedestrian navigation. - **Technica
- arXiv
- 2603.10703
- Published
- 2026-03-11
- Authors
- Rafi Ibn Sultan, Hui Zhu, Xiangyu Zhou, Chengyin Li, Prashant Khanduri, Marco Brocanelli, Dongxiao Zhu
AI summary
## Overview - **Research area:** Computer vision and multimodal AI — specifically pixel-grounded Large Vision–Language Models (LVLMs) applied to accessibility-aware pedestrian navigation. - **Technical level:** Advanced. The paper assumes familiarity with vision-language architectures, segmentation decoders (SAM), contrastive losses (InfoNCE), and LoRA fine-tuning. - **Scope:** The paper introduces a new task (Grounded Navigation Guide), a new model (WalkGPT), and a new dataset (PAVE), and evaluates them on grounded conversation, segmentation, depth estimation, referring expression segmentation, hallucination, and ablations. ## What This Paper Is About Pedestrians — especially people with mobility challenges — need guidance that describes not just *what* is in a scene but *where* it is and *how far away* it is, so they can tell an accessible sidewalk from a harmful obstacle such as a stair, parked vehicle, or temporary obstruction. Existing LVLMs can describe images but lack explicit spatial grounding, which causes object hallucinations and unreliable depth reasoning, and existing spatial-aware models typically require the user to supply visual cues or anchor points, which a walking pedestrian cannot do. WalkGPT addresses this by unifying language reasoning and segmentation inside a single architecture: given a pedestrian-view image and a navigation query, it produces a conversational response containing segmentation masks for accessible and harmful features plus relative depth information, without any user-provided cues. ## Key Contributions 1. **WalkGPT**, described as the first-of-its-kind LVLM for pedestrian accessibility via grounded spatial reasoning, targeting the newly defined task of *Grounded Navigation Guide*. 2. **Two new architectural modules** — the Multi-Scale Query Projector (MSQP) and the Calibrated Text Projector (CTP), the latter guided by a proposed Region Alignment Loss — with structured token supervision to align visual and language representations for pixel-level grounding. 3. **PAVE** (Pedestrian Accessibility and Visual-grounded Evaluation), a large-scale VQA dataset built from 41k pedestrian-view image–question–answer triplets carrying accessibility-aware questions and depth-grounded answers. 4. **Strong empirical results**, including state-of-the-art performance on grounded navigation guidance, plus lower hallucination and higher object coverage than non-grounded LVLMs. ## Main Findings - **Baselines fail without fine-tuning:** All zero-shot grounded LVLMs (GLAMM, LISA, PixelLM, GSVA, OMG-LLaVA, Sa2VA) scored very low across text and segmentation metrics — for example, LISA at 0.97 CIDEr and 1.50 mIoU — and depth metrics are listed as N/A because they produced no depth estimations even with added `` tokens. - **Fine-tuning helps but is not enough:** Fine-tuned variants improved substantially (Sa2VA-FT reached 38.82 CIDEr; PixelLM-FT reached 18.10 mIoU and 39.00 Depth Acc.), but WalkGPT still led. - **Best grounded navigation results:** WalkGPT (13B) reached 43.01 METEOR, 20.16 mIoU, 32.71 Recall, 48.95 Depth Acc. and 70.66 AbsRel. WalkGPT (7B) reached 41.97 CIDEr and 42.36 METEOR. The paper reports the 13B model improving mIoU by more than 10% over PixelLM-FT (20.16 vs. 18.10) and raising depth accuracy by over 25% (48.95 vs. 39.00). - **Generalization to referring expression segmentation:** Without RES-specific training, WalkGPT reached 76.2% on refCOCO-val and 72.6% on refCOCOg-val, outperforming LISA and PixelLM by up to about 3–4%. - **Vision-only segmentation models also struggle on PAVE:** U-Net (16.85 mIoU), nnU-Net (18.55), and Swin-UNETR (20.60) all fell under roughly 21% mIoU, and WalkGPT attained a comparable 20.16 mIoU with higher Recall (32.71). - **Hallucinations reduced, coverage increased:** WalkGPT recorded CHAIR_i of 18.49 and Cover of 83.66, versus LLaVA-1.5 (22.16 / 33.04), LLaVA 1.6 Mistral (23.56 / 38.83), and Qwen-VL-Chat (26.78 / 31.42). - **Every component matters (ablation):** Replacing MSQP with an MLP dropped results to 39.50 METEOR / 17.40 mIoU / 43.39 Depth Acc.; removing multi-scale aggregation gave 41.60 / 19.30 / 44.70; reducing MSQP queries to Q=8 gave 38.10 / 16.20 / 45.33; replacing CTP with a linear mapping gave 40.70 / 18.60 / 47.98; removing the contrastive loss gave 41.00 / 18.90 / 47.00; freezing the LLM gave 35.20 / 17.80 / 40.21. - **The `` token is specifically responsible for depth:** Removing it left segmentation nearly unchanged (20.16 to 20.01 mIoU) but dropped Depth Acc. from 48.95 to 38.77. - **Known failure mode:** In a presented case study, strong road reflections on a building façade were misinterpreted as physical obstacles, producing incorrect guidance even though the path was fully accessible. Motion blur, noisy surfaces, and class imbalance cause similar errors. ## Methodology in Plain English WalkGPT reuses a single shared SAM ViT-H pixel encoder for both conversation and mask prediction, and the language model is initialized from pretrained checkpoints (a 13B version and a 7B version) and adapted with LoRA. Two components carry most of the design novelty. The **Multi-Scale Query Projector (MSQP)** takes the pixel encoder's features and pools them at several resolutions — native, pooled by 2, pooled by 4, and a global mean — then lets small sets of learnable queries cross-attend to each scale with two cross-attention layers. This keeps both local detail and whole-scene context in a compact set of image tokens before the LLM sees them. The **Calibrated Text Projector (CTP)** handles the reverse direction: when the model emits a `` token in its answer, CTP expands that token's hidden state into multiple calibrated sub-embeddings before passing it to the pixel decoder, so each segmentation prompt carries richer semantics than a plain linear projection would preserve. A **Region Alignment Loss** addresses the information loss from squeezing a 4096-dimensional LLM embedding down to the 256-dimensional visual space. Each `` token cross-attends to the visual features, the top-K most-attended regions become pseudo-targets, and an InfoNCE contrastive loss pulls the projected embedding toward its matching region and away from unrelated ones, with the pixel encoder frozen. For data, the authors built **PAVE** from the real-image subset of SANPO, using its human-annotated masks and depth maps and excluding synthetic imagery. For each feature mask they compute the minimum pixel depth as its closest visible distance, and pass the RGB frame, detected features, distances, and accessibility labels (e.g., sidewalk accessible; vehicle or stair harmful) to GPT-5-nano, which generates the question and a structured answer. Outputs use four structured tokens: `` for an overall accessibility judgment, `` to ground referenced objects in the dialogue, `` to prompt the segmentation decoder, and `` to state object-level depth in text. Training combines cross-entropy on the conversation, Dice plus cross-entropy for segmentation, and the contrastive alignment term, weighted as α₁ = 0.1, α₂ = 0.05 (Dice) and 0.35 (BCE), and α₃ = 0.3, with InfoNCE temperature 0.07 and top-8 hard negatives. Pretraining optimizes only MSQP on ADE20K and the RefCOCO family; fine-tuning then jointly trains MSQP, CTP, the pixel decoder, and LoRA on PAVE. Training used 10 epochs, batch size 16 with gradient accumulation 10 (effective batch 160), AdamW at 2×10⁻⁴, bf16 precision, 448×448 images, and 54 optimization steps per epoch. ## Why This Matters **Impact on research.** WalkGPT argues that pedestrian navigation is an underexplored domain, since most automated navigation work targets vehicles. By unifying conversation, pixel-level segmentation, and depth description in one next-token-prediction process — and by releasing a 41k-triplet benchmark — it gives the field a task definition and an evaluation target where one previously did not exist. **Real-world applications:** - Assistive navigation for people with mobility challenges, flagging stairs, uneven terrain, parked vehicles, and temporary obstructions from a first-person view. - Accessible route planning and wayfinding tools that need to explain *why* a path is or is not usable, not just flag it. - Perception modules for assistive wearables or phone-based navigation apps using head-mounted pedestrian-view imagery. - A grounding and hallucination-reduction reference point for other safety-critical multimodal systems that must not invent objects. **Industry relevance.** The paper's hallucination results (CHAIR_i 18.49, Cover 83.66 versus 22.16–26.78 and 31.42–38.83 for non-grounded LVLMs) speak directly to deployment concerns: a guidance system that describes objects that are not there can issue hazardous instructions. The use of LoRA fine-tuning, a shared SAM encoder, and depth emerging from language supervision rather than a dedicated depth head also point toward relatively lightweight adaptation paths for product teams. ## Future Directions 1. **Improved depth estimation.** The authors explicitly list this as planned future work, and their own ablation shows depth accuracy is the metric most sensitive to removing the `` supervision (48.95 to 38.77). 2. **Cross-domain generalization.** The paper states future work will evaluate the model on additional navigation and grounding datasets. 3. **Reducing dataset artifacts and ambiguity.** The limitations section notes WalkGPT may still be affected by dataset artifacts that introduce ambiguity. 4. **Handling reflections, motion blur, and class imbalance.** The failure case study shows single-view images where reflective surfaces distort depth cues, and the authors note similar issues from blur, noisy surfaces, and severe class imbalance — conditions the PAVE scenes contain and that remain open problems. ## Target Audience Researchers and practitioners working on grounded vision-language models, referring expression segmentation, and spatial reasoning will get the most from this paper, as will accessibility and assistive-technology engineers who need pedestrian-level, first-person scene understanding. It is also relevant to dataset builders interested in how the PAVE annotation pipeline combines sensor-derived depth with LLM-generated question-answer pairs and verification. Readers without background in LVLM architecture, segmentation decoders, or contrastive learning will find the methods section demanding.
Authors’ abstract
Ensuring accessible pedestrian navigation requires reasoning about both semantic and spatial aspects of complex urban scenes, a challenge that existing Large Vision-Language Models (LVLMs) struggle to meet. Although these models can describe visual content, their lack of explicit grounding leads to object hallucinations and unreliable depth reasoning, limiting their usefulness for accessibility guidance. We introduce WalkGPT, a pixel-grounded LVLM for the new task of Grounded Navigation Guide, unifying language reasoning and segmentation within a single architecture for depth-aware accessibility guidance. Given a pedestrian-view image and a navigation query, WalkGPT generates a conversational response with segmentation masks that delineate accessible and harmful features, along with relative depth estimation. The model incorporates a Multi-Scale Query Projector (MSQP) that shapes the final image tokens by aggregating them along text tokens across spatial hierarchies, and a Calibrated Text Projector (CTP), guided by a proposed Region Alignment Loss, that maps language embeddings into segmentation-aware representations. These components enable fine-grained grounding and depth inference without user-provided cues or anchor points, allowing the model to generate complete and realistic navigation guidance. We also introduce PAVE, a large-scale benchmark of 41k pedestrian-view images paired with accessibility-aware questions and depth-grounded answers. Experiments show that WalkGPT achieves strong grounded reasoning and segmentation performance. The source code and dataset are available on the \href{https://sites.google.com/view/walkgpt-26/home}{project website}.