From the Laboratory to Real-World Application: Evaluating Zero-Shot Scene Interpretation on Edge Devices for Mobile Robotics

AI summary

Overview

Research area: Computer Vision / Vision-Language Models applied to Mobile Robotics, specifically zero-shot scene interpretation and human action recognition on edge hardware.

Technical level: Intermediate. The paper assumes familiarity with VLMs, LLMs, zero-shot segmentation, and common evaluation metrics, but its central argument is practical and application-oriented rather than mathematically dense.

Scope: An evaluation of whether a small vision-language model (SmolVLM2) can perform zero-shot scene and action description on a mobile robot's edge device across real-world indoor, campus, and city video data.

What This Paper Is About

Video understanding and visual commonsense reasoning would let mobile robots go beyond detecting objects to interpreting what people are doing and why. Large VLMs can do this, but their computational cost makes them impractical for edge devices, and running them via cloud servers conflicts with privacy and with the reality that network access cannot be guaranteed for a mobile platform. This paper asks whether a small VLM deployed locally on an edge device can produce usable zero-shot scene descriptions in real cityscape, on-campus, and indoor scenarios, and it examines the practical obstacles that appear when moving such a model from the lab into real deployment.

Key Contributions

1. An integrated edge-plus-cloud architecture for zero-shot video interpretation. The pipeline pairs a small local VLM for scene description with local object detection and tracking, and optionally queries larger cloud-based foundation models for further analysis, keeping raw personalized data on the local models to preserve privacy.

2. A semantically guided segmentation workflow. The VLM's generated description is decomposed into nouns, which then serve as text prompts for zero-shot detection and segmentation using Grounded DINO with SAM, letting the pipeline highlight only the elements relevant to the description instead of segmenting a fixed class list.

3. A real-world multi-domain evaluation of SmolVLM2 on edge hardware. The model is evaluated on 234 minutes of video collected in Trier, Germany, split into Campus Indoor, Campus Outdoor, and City domains, with manually annotated five-second clips.

4. An analysis of biases, latency, and the inadequacy of automated evaluation metrics. The paper documents specific model failure modes, the end-to-end information delay, and how poorly BERTScore and Sentence Similarity track human judgment in this setting.

Main Findings

• Overall zero-shot correctness was 65.4%. Across all generated descriptions, 65.4% were judged correct by human experts.

• Performance varied substantially by domain. City scored highest at 79.6% correct, while Campus Indoor scored lowest at 53.3%. The paper attributes the indoor gap to a more diverse and complex activity set, citing examples such as putting dishes into a dishwasher, handing an apple to another person, picking up scissors, writing on a whiteboard, and climbing a ladder, versus more simplistic outdoor actions such as walking on a sidewalk or driving down a road.

• Agents were described more reliably than actions or objects. The Agent sub-category reached 93.7% correctness, compared with 83.2% for Object and 78.9% for Action.

• Automated similarity metrics correlate weakly with human judgment. BERTScore produced correlation coefficients from 0.229 to 0.483 across domains, lower than Sentence Similarity for every domain. Agreement with manual evaluation was 67.1% for BERTScore versus 73.6% for Sentence Similarity. Boxplots show heavy overlap between quartiles, especially for BERTScore.

• The model has identifiable action and object biases. It used "sitting down" to describe both sitting down and standing up. When a whiteboard was present, the model tended to fold it into the action regardless of relevance, producing descriptions like "A person is standing on a ladder and is writing on a whiteboard" instead of the ground truth "A person is climbing a ladder."

• There is a meaningful information delay. Descriptions refer to the past five seconds, and inference adds another one to three seconds depending on scene complexity and system load, so the delay between an action occurring and the agent receiving information can reach eight seconds.

• Manual annotation of a "main action" is ambiguous in open domains. In City and, to a lesser extent, Campus Outdoor, the same scene can be legitimately described in multiple ways, which forced the authors to adopt a rule where descriptions count as correct if no single critical action is present.

Methodology in Plain English

The researchers built a two-part system. On the robot's edge device, a small VLM called SmolVLM2 (2.2 billion parameters) looks at a sequence of images and writes a text description of what is happening. Also running locally are separate models for detecting and tracking objects. If more analysis is needed, the edge system can send its own generated outputs to larger cloud-based foundation models, but the raw imagery stays local, which keeps personal data private.

To turn the text description into something visually useful, the pipeline extracts the nouns from the VLM's sentence and feeds them as prompts into a zero-shot detection and segmentation stack (Grounded DINO with SAM). If the model says "A woman is crossing the street at a crosswalk," the nouns woman, street, and crosswalk drive the segmentation, so the output highlights exactly the elements the description mentions rather than a fixed list of classes like cars, buildings, trees, and sky.

For evaluation, the team recorded 234 minutes of video in Trier, Germany, across three domains: Campus Indoor with 107 minutes, Campus Outdoor with 74 minutes, and City with 53 minutes. The footage was cut into five-second clips, each manually annotated with a description of the main action. Human experts then judged each VLM description against the annotation in three sub-categories — Action, Agent, and Object — with the whole description marked Correct only if all three sub-categories were right. Person characteristics such as gender were not scored, so "a man" and "a woman" were treated as equivalent. Alongside the human evaluation, the authors computed BERTScore and STSB DistilRoBERTa Sentence Similarity, then measured how well those automated scores matched the human verdicts using the point-biserial correlation coefficient and a threshold-based match percentage, where the threshold was set by optimization over the generated metric values.

Why This Matters

The work provides evidence about where the real gap lies in deploying VLM-based scene understanding on robots: not only in accuracy, but in latency, domain-dependent bias, and the absence of trustworthy automatic evaluation. It argues that small VLMs are genuinely usable zero-shot on unseen domains, while also showing that their errors are systematic rather than random — which is important, because systematic errors can be anticipated and filtered with domain prior knowledge, whereas random errors cannot.

Real-world applications the paper points to:

• Service robots in homes and hospitals, where understanding subtle human context, such as recognizing that a person has finished a meal and initiating cleanup, improves efficiency and comfort alongside safety.
• Autonomous vehicles and urban mobility, which must interpret complex visual cues in city environments; the paper specifically suggests using extracted information to build dynamic cooperative maps that flag points of interest like heavy pedestrian traffic or streets where cars emerge from hidden driveways.
• Assistive and documentation tasks, particularly non-time-critical assistance tasks, or documenting incidents in more time-critical environments where the eight-second delay matters less.
• Search and rescue with legged robots, as in the cited VLM-Predictive Control work, where robots traverse collapsed buildings, climb rubble, crawl through gaps, and back out of dead ends using semantic knowledge rather than external support.

Industry relevance: The paper speaks directly to robotics teams deciding between cloud inference and on-device models. Its architecture is explicitly privacy-preserving by design, since personalized data never leaves the local models, and it uses commodity-class hardware — an NVIDIA Jetson AGX edge device with stereo cameras, LiDAR, and a GNSS/INS module. The finding that automated metrics are unreliable for open-vocabulary description also matters commercially, because it implies that validating such systems in the field still requires human labor, which has direct cost implications for deployment and monitoring pipelines.

Future Directions

• Better metrics for open-vocabulary tasks. The authors identify this as a primary need, noting that existing probabilistic alternatives are either restricted to question-answer setups or to narrow tasks such as scene representation with exhaustive label sets, and are therefore not yet applicable here.
• Reliable automated evaluation of real-world domains to reduce manual annotation effort. The current evaluation depends heavily on human experts, which the paper frames as a bottleneck worth solving.
• Domain-specific bias characterization and output filtering. Since the model folds irrelevant objects such as whiteboards into descriptions and confuses related actions, the paper suggests that carefully evaluating biases per domain and filtering generated output using prior knowledge could stabilize predictions.
• Rethinking how the gained information is used given the latency. The paper raises the option of changing the evaluated time interval to reduce delay, but notes the underlying problem remains, leaving open the question of how cognitive agents should structure their behavior around multi-second-old scene descriptions.

Target Audience

This paper is most useful to robotics and computer vision engineers evaluating small vision-language models for on-device deployment, particularly those working on mobile platforms where connectivity, privacy, or latency rule out cloud inference. It also benefits researchers studying vision-language evaluation, since it provides a concrete real-world case where popular automated metrics diverge from human judgment, and it is relevant to practitioners designing human-robot interaction and assistive systems who need realistic expectations about what a small VLM can and cannot reliably describe.

Authors’ abstract

Video Understanding, Scene Interpretation and Commonsense Reasoning are highly challenging tasks enabling the interpretation of visual information, allowing agents to perceive, interact with and make rational decisions in its environment. Large Language Models (LLMs) and Visual Language Models (VLMs) have shown remarkable advancements in these areas in recent years, enabling domain-specific applications as well as zero-shot open vocabulary tasks, combining multiple domains. However, the required computational complexity poses challenges for their application on edge devices and in the context of Mobile Robotics, especially considering the trade-off between accuracy and inference time. In this paper, we investigate the capabilities of state-of-the-art VLMs for the task of Scene Interpretation and Action Recognition, with special regard to small VLMs capable of being deployed to edge devices in the context of Mobile Robotics. The proposed pipeline is evaluated on a diverse dataset consisting of various real-world cityscape, on-campus and indoor scenarios. The experimental evaluation discusses the potential of these small models on edge devices, with particular emphasis on challenges, weaknesses, inherent model biases and the application of the gained information. Supplementary material is provided via the following repository: https://datahub.rz.rptu.de/hstr-csrl-public/publications/scene-interpretation-on-edge-devices/