Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

AI summary

Overview

• Research area: Embodied AI and human-robot interaction, specifically proxemic (personal-space) risk assessment from egocentric robot camera images using vision-language models (VLMs); the paper sits at the intersection of computer vision, spatial grounding, prompt engineering, and parameter-efficient fine-tuning.
• Technical level: Intermediate. Readers should be comfortable with VLM terminology, LoRA/QLoRA fine-tuning, classification metrics (F1, recall), and IoU-based localization, but the paper's arguments are explained in fairly accessible prose.
• One-sentence scope: The paper evaluates three open-weight VLMs (InternVL3.5-4B, Qwen2.5-VL-3B-Instruct, SmolVLM2-2.2B-Instruct) on a new 1,243-image egocentric proxemic-danger dataset, comparing three prompt strategies and two rounds of QLoRA fine-tuning against a stratified random baseline.

What This Paper Is About

Mobile robots sharing space with people must judge how dangerous the current scene is — whether a person is too close and a collision or discomfort is likely. This paper asks whether general-purpose vision-language models can make that judgment from a single first-person robot image, without depth sensors or explicit geometric reasoning. It builds a four-level proxemic danger dataset from JRDB and tests whether prompting and lightweight fine-tuning make any of three open-weight VLMs useful for the task, including whether their predictions are actually grounded in the right person in the scene.

Key Contributions

1. A new proxemic-risk dataset. 1,243 egocentric robot images derived from JRDB, manually labeled into four danger levels following Hall's proxemic zones: High (n=222, 17.9%), Moderate (n=407, 32.7%), Low (n=299, 24.1%), and Minimum (n=315, 25.3%). It is split into indoor (n=713; good lighting n=505, poor lighting n=208) and outdoor (n=530), and the authors state they will release it as an open-source artifact for reproducibility.
2. A systematic evaluation grid. Three open-weight VLMs (Qwen2.5-VL-3B-Instruct, InternVL3.5-4B, SmolVLM2-2.2B-Instruct) × three prompt strategies (Simple, Moderate, Advanced) × three checkpoints (before, after, after_2 fine-tuning), scored against a random baseline (weighted F1 = 0.25).
3. A joint classification-and-grounding analysis. Besides accuracy, weighted F1, and high-danger recall, the paper measures mean IoU, the ratio of IoU > 0.5, and the ratio of IoU = 0 for predicted person bounding boxes, directly testing whether a correct danger label corresponds to looking at the right person.
4. The finding that classification success does not imply spatial grounding. Correctly and incorrectly classified frames show essentially the same IoU (0.204 versus 0.215), and Qwen's true positives in the high-danger class (IoU = 0.488) barely exceed its false negatives (IoU = 0.464).

Main Findings

• All models sit near the random baseline on overall classification. Weighted F1 ranges from 0.21 to 0.24 across configurations versus the random baseline of 0.25. Accuracy ranges from 0.254 to 0.304.
• Qwen is the only model that detects high danger at all. High-danger recall for Qwen is 0.395 (before), 0.432 (after), and 0.440 (after_2), versus 0.026/0.033 for InternVL and 0.016/0.069/0.055 for SmolVLM.
• Fine-tuning produces little overall gain. The change in weighted F1 is less than 0.02 in all cases, and it slightly worsens localization for every model.
• Prompt complexity trades overall accuracy for high-danger recall. The Simple prompt gives the best weighted F1 (0.303) and mean IoU (0.232) but the lowest high-danger recall (0.122). The Advanced prompt reverses this: weighted F1 0.249, mean IoU 0.196, but high-danger recall 0.316. The Moderate prompt sits between on recall (0.196) with weighted F1 0.263 and mean IoU 0.110, and yields the lowest ratio of IoU = 0 (0.416). The authors frame this inversion as a possible "format tax."
• Only Qwen localizes people usefully. Before fine-tuning, Qwen reaches mean IoU 0.459 with IoU > 0.5 on 0.545 of predictions; after fine-tuning this drops to 0.413 (IoU > 0.5 on 0.481) and after_2 to 0.419 (0.482). InternVL stays near mean IoU 0.12–0.13, and SmolVLM at 0.046–0.061 with IoU = 0 on 0.612–0.680 of predictions.
• Per-class quality is uneven in the two best Qwen configurations. With after_2 and the Simple prompt, per-class F1 is 0.278 (high), 0.287 (moderate), 0.146 (low), and 0.440 (minimum). With after and the Advanced prompt, high-danger recall reaches 0.790 but precision is 0.182, and F1 collapses for moderate (0.150), low (0.038), and minimum (0.135).
• No configuration optimizes both goals. The paper's two candidate deployments are (1) Qwen + Simple prompt + second fine-tuning round for balanced overall performance, reported in the discussion with weighted F1 = 0.290, and (2) Qwen + Advanced prompt + first fine-tuning round for high-danger recall of 0.79 with poor precision and weak performance elsewhere. InternVL and SmolVLM cannot reliably detect high-danger situations even after fine-tuning.

Methodology in Plain English

The authors took egocentric images captured by a mobile robot from the JRDB dataset and manually sorted 1,243 of them into four proxemic danger levels based on how close people are to the robot, from Minimum (no risk) to High (a person inside close personal space, where inaction means unavoidable collision or discomfort). Annotation was manual rather than threshold-based, which the authors acknowledge may introduce inaccuracies.

They then ran three open-weight VLMs — Qwen2.5-VL-3B-Instruct (3B parameters), InternVL3.5-4B (4B), and SmolVLM2-2.2B-Instruct (2.2B) — on every image using three prompts. The prompts shared the same proxemic-zone definitions and the same JSON output format (distance zone, distance estimate, bounding box, danger level), but differed in complexity: a brief Simple description, a Moderate version with instructions and output constraints, and an Advanced version with an internal chain of reasoning over visual cues. Only the danger level was scored for classification; the bounding box was scored separately for grounding.

For adaptation they used QLoRA in two stages on a balanced training set of 200 images (50 per danger class), drawn from videos different from those used to build the test set to avoid leakage. Stage 1 adapted only the attention projections (q_proj, v_proj); Stage 2 extended adaptation to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, and lm_head. Both used r = 8, alpha = 16, dropout = 0.05, and no offset adaptation. This produced three checkpoints per model: before, after, and after_2. Results were measured with accuracy, weighted F1 (robust to class imbalance), high-danger-class recall (the safety-critical metric, since false negatives are costliest), and IoU-based grounding metrics against the ground-truth person location.

Why This Matters

Impact on research. The paper tests a specific and underexplored question: whether general visual-linguistic understanding can substitute for or complement geometric safety systems in embodied navigation. Its most distinctive result is negative and structural — a model can output a correct "high danger" label while its bounding box points somewhere unrelated, meaning benchmark accuracy alone cannot certify that a VLM has understood the scene. That undercuts the assumption that classification performance is evidence of spatial understanding, and it motivates joint evaluation of labels and grounding rather than labels alone.

Real-world applications.

• Warehouse and logistics robots operating alongside pickers in aisles, where crowded conditions are the high-danger case and missed detections are the costliest errors.
• Service and hospitality robots (hotel delivery, airport guidance, food service) that must respect personal space and avoid causing discomfort.
• Assistive and healthcare robots moving through corridors and patient rooms where intrusion into intimate or personal zones is unacceptable.
• Sidewalk and campus delivery robots that must judge outdoor crowd density, the outdoor portion of this dataset (n=530).

Industry relevance. The models tested are all open-weight and small (2.2B–4B parameters), sized for local deployment on robot hardware, and the adaptation method used (QLoRA) is deliberately lightweight. The paper's framing of deployability — requiring a reliable danger trigger with high recall but usable precision — reflects how a safety module would actually be integrated into a control loop, and its conclusion that even the best high-recall configuration has unproven practical deployability is a caution for anyone planning to ship a VLM-based safety layer.

Future Directions

• Scale up the fine-tuning data. Fine-tuning used only 200 images, 50 per class; the authors explicitly identify the small dataset as a likely limit on the learning signal and call for larger training datasets.
• Compare against larger, API-scale models. Future work should test whether the observed limits come from model architecture or from model scale, which this study cannot separate given its 2.2B–4.4B parameter range.
• Jointly improve classification and grounding. Since correct labels did not track correct localization, an open question is whether the two can be improved together rather than traded against each other.
• Diagnose and mitigate degradation in small models. Additional fine-tuning hurt SmolVLM (2.2B), which the authors attribute to catastrophic forgetting under limited capacity; whether this can be avoided through different adaptation strategies remains unresolved.

Target Audience

Robotics and embodied-AI researchers working on safe navigation and human-robot interaction; VLM researchers interested in spatial grounding and the gap between classification accuracy and genuine scene understanding; practitioners evaluating small open-weight models for on-robot deployment; and students or engineers interested in prompt engineering and QLoRA fine-tuning as applied to a concrete safety task. The paper's negative results and its explicit documentation of dataset composition and training configuration make it useful both to those planning similar benchmarks and to those deciding whether a VLM is ready to serve as a proxemic safety trigger.

Authors’ abstract

Assessing proxemic danger from a robot's egocentric perspective is critical for safe embodied navigation in human environments and requires both visual and contextual reasoning. We evaluate three opensource vision-language models (VLMs) (\textit{InternVL}, \textit{Qwen-VL}, and \textit{SmolVLM}) on the classification of egocentric robot images into four danger levels, comparing three prompting strategies and two rounds of QLoRA fine-tuning against a stratified random baseline. Without fine-tuning, all models perform near the baseline, while fine-tuning yields only modest overall improvements. However, \textit{Qwen-VL} with an advanced prompt achieves substantially higher recall for high-danger cases than the other models. An analysis of person localization further shows that correct danger classification does not correspond to better spatial grounding, indicating that a model may produce a useful safety label without attending to the relevant region of the scene. These results show that current VLMs remain limited in fine-grained proxemic reasoning and spatial grounding, although targeted prompting and fine-tuning can improve high-danger detection in selected models.