Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task

AI summary

Overview

Research area: Computer Vision, specifically Video Question Answering (VideoQA), multimodal large language models (MLLMs), and tool-augmented LLM agents.

Technical level: Advanced. The paper assumes familiarity with MLLMs (GPT-4o, Qwen-VL, InternVL), agent frameworks such as ReAct, object detection models (YOLO, Grounding DINO), and benchmark-driven evaluation.

One-sentence scope: The paper builds a 22-tool Video Toolkit and a Spatiotemporal Reasoning Framework (STAR) that alternately schedules temporal and spatial tools so an MLLM can progressively localize the question-relevant 3D Region of Interest in a video.

What This Paper Is About

VideoQA asks a model to answer questions about dynamic video content, which requires reasoning over both the spatial layout inside frames and the causal ordering of events over time. Existing MLLMs struggle to do both at once, often process large numbers of frames inefficiently, and tool-augmented LLMs tend to use tools along only one dimension (spatial or temporal) with poorly controlled invocation order. The paper's goal is to give an MLLM a large, balanced toolkit plus an explicit scheduling rule so it can narrow a video down step by step instead of answering in one shot.

Key Contributions

1. A comprehensive, plug-and-play Video Toolkit containing 22 different tools that covers a wide range of functionalities. The toolkit is organized into three categories—spatial tools, temporal tools, and general-purpose tools—and all tools are plug-and-play to keep the toolkit extensible.

2. The Spatiotemporal Reasoning Framework (STAR), a training-free, user-friendly, extensible agentic framework in which the LLM alternately invokes temporal and spatial tools. STAR adapts to intermediate tool results and progressively narrows down essential information, producing spatiotemporal-interleaved toolchains that localize the 3D Region of Interest (3D RoI).

3. A diagnosis and fix for the "Toolchain Shortcut" problem, where an LLM Planner takes shortcuts by favoring single-step tools (for example, directly calling a video-language model) instead of decomposing the problem into a longer step-by-step toolchain. STAR imposes an alternating temporal/spatial constraint and allows general tools only as a last resort.

4. Extensive evaluation across four VideoQA benchmarks, showing that augmenting GPT-4o with the lightweight Video Toolkit yields an 8.2% gain on VideoMME and 4.6% on LongVideoBench, outperforming leading 7B Video-LLMs and demonstrating stronger scalability as input frame counts grow.

Main Findings

• VideoMME accuracy gain: STAR reaches 70.0 overall accuracy on VideoMME using 30.2 frames and 15.8 seconds of runtime, an 8.2% improvement over GPT-4o reproduced with 32 input frames (61.8 overall, under 30 seconds). STAR's sub-scores are Short 78.9, Medium 68.3, and Long 62.9, compared with 68.3 / 60.7 / 56.3 for GPT-4o at 32 frames.

• Comparison against open-source Video-LLMs on VideoMME: STAR outperforms all open-source Video-LLMs around the 7B scale, achieving a 3.7% improvement over the best-performing InternVL3-8B (66.3). STAR approaches the performance of open-source Video-LLMs around the 72B scale; Qwen2-VL 72B scores 71.2 overall using 2 fps / 768 frames and 6–8 minutes of runtime, which STAR reduces to 15.8 seconds.

• VideoMME context on proprietary models and frame selection: GPT-4o at 1 fps / 384 frames scores 71.9 overall with over 10 minutes of runtime; Gemini 1.5 Pro scores 75.0 at 0.5 / 1 fps with over 10 minutes; Claude 3.5 Sonnet scores 60.9 at 20 frames; GPT-4v scores 60.0 at 10 frames; GPT-4o + T* scores 64.1 at 32 frames. The paper states that Gemini's and GPT-4o's higher scores partly reflect their use of subtitle information, which STAR does not use.

• LongVideoBench gain: On the LongVideoBench validation set, STAR reaches 57.2 overall using 29.6 frames and 15.3 seconds, a 4.6% improvement over GPT-4o reproduced with 32 frames (52.6, under 30 seconds). STAR's bucket scores are 63.6 (8s–15s), 68.0 (15s–60s), 56.8 (180s–600s), and 52.3 (900s–3600s). GPT-4o at 32 frames scores 60.7 / 62.4 / 50.1 / 49.0, and Gemini 1.5 Pro at 256 frames scores 64.0 overall. (The caption of Figure 2(a) describes a "5%" gain on LongVideoBench, while the abstract, contributions, and results section state 4.6%.)

• Long-video advantage over 7B Video-LLMs: On LongVideoBench, STAR outperforms Qwen2.5-VL-7B (53.7 overall, 1 fps / 512 frames, 1–3 min runtime) and InternVL3-8B (48.9 overall, 256 frames), with the margin described as particularly large in the long (180–600 s) and extra-long (900–3600 s) buckets.

• NExT-QA test set: STAR achieves 82.1 overall using only 7.2 frames, with 81.1 on causal, 81.5 on temporal, and 86.3 on descriptive questions—over 80% on all three categories. This is a 1.2% improvement over the best baseline T* (80.4 overall, 8 frames) and exceeds Qwen2.5-VL-7B (80.9 overall, 2 fps), AKeyS (78.1 overall, 7.6 frames), InternVL3-8B (75.7 overall, 8 frames), VideoTree (75.6 overall, 63.2 frames), VidF4 (74.1 overall, 8 frames), LVNet (72.9 overall, 12 frames), and VideoAgent (71.3 overall, 8.2 frames).

• NExT-QA validation set, lightweight variant: STAR-mini, using GPT-3.5-turbo-0125 as planner, scores 62.0 overall versus 55.7 for DoraemonGPT—a 6.3% improvement—while using 0.6 fps / 22.6 frames versus 28.7 fps / 1144.4 frames, and 5.4 LLM calls versus 8.5 (a reduction of 3.1). STAR-mini's sub-scores are 62.8 causal, 55.1 temporal, and 73.7 descriptive. DoraemonGPT scores 54.7 / 50.4 / 70.3.

• Ablation on toolchain strategies (VideoMME): STAR achieves 70.0 accuracy with 30.2 frames, toolchain length 8.7, and 6.3 tools. Spatiotemporal Disentanglement is second best at 68.6 accuracy with 40.6 frames, toolchain length 5.6, and 3.4 tools (STAR improves by 1.4 accuracy while processing 10.4 fewer frames). In-Context Learning scores 63.2 with 50.1 frames and toolchain length 5.4; No Constraints scores 61.2 with 112.6 frames and toolchain length 2.9; Prompting is lowest at 60.4 with 98.7 frames and toolchain length 3.6. STAR's improvements over Spatiotemporal Disentanglement are 3.1 in toolchain length and 2.9 in number of tools.

• Toolchain Shortcut is real: The unconstrained planner produces overly short toolchains (length 2.9) and repetitive tool usage, leading to excessive frame processing (112.6 frames) and high computational cost. Prompting alone gives only marginal improvement over no constraints.

• Balanced tool use: Figure 3 visualizes the percentage distribution of tool usage on VideoMME, showing tools are utilized in a well-balanced way with no tool excessively favored or underused.

• Scalability: As the number of input frames increases, STAR is reported to consistently achieve the highest accuracy on EgoSchema; the truncated content does not report the numeric EgoSchema values.

• Efficiency of the full framework: STAR runs on one NVIDIA RTX 4090 GPU, while STAR-mini can run on a personal computer such as a Mac. STAR uses tools based on open-source models with up to 3B parameters (e.g., QwenVL-2.5-3B) plus GPT-4o-2024-08-06 APIs; STAR-mini excludes tools larger than 500M parameters, its largest tool being a 500M-parameter BLIP, and uses GPT-3.5-turbo-0125.

Methodology in Plain English

The authors start from the observation that video questions decompose naturally into two dimensions: when something happens (temporal) and where in the frame it happens (spatial). They build a toolkit around that split.

Temporal tools handle time. The core one is a Frame Selector, which is itself an LLM: it looks at frame IDs and their textual descriptions and picks the frames worth keeping, or trims a segment of video. There are three variants of the Frame Selector, including an AKeyS-inspired version that considers task goals and scene transitions, and a T*-inspired version that arranges frames into an image grid for an MLLM to choose from. The toolkit also includes a temporal grounding tool built on a roughly 4B-parameter Grounded-VideoLLM that predicts the time span of a described event, a temporal referring tool that describes what happens within a given time span, a video trimmer, and an action localization tool that uses GPT-4o over an image grid.

Spatial tools handle the contents of frames. Two object detectors are provided, based on YOLO and Grounding DINO. Detected bounding boxes are used in three ways: converted into text descriptions fed back to the LLM; enlarged via a Patch Zoomer; or overlaid with visual markers following Set-of-Mark prompting before being passed back to the MLLM. Other spatial tools include image captioners at three scales (BLIP2 at about 1B parameters, LLaVA at about 7B, and GPT-4o), an image QA tool, an OCR text detector, a semantic segmentation tool based on Grounded-SAM, and a relevant patch zoomer following Octotools.

General and dual-purpose tools include Google Search, an object identifier inspired by T*, action recognition based on MMAction, image grid QA, multiple image QA, a Python code generator, an object tracker with person re-identification based on Ultralytics, a video summarizer using Qwen2.5-VL-3B, a text summarizer, and a video QA tool also using Qwen2.5-VL-3B. Following Octotools, each tool is wrapped in a standardized "tool card."

On top of the toolkit sits the STAR framework. The LLM Planner keeps a Visible Frame Dictionary whose keys are visible frame indices and whose values are the information gathered about each frame. It is initialized with a sparse, uniformly sampled set of frames: for videos longer than 16 seconds, 16 initial frames are extracted uniformly, and for videos of 16 seconds or less, frames are extracted at 1 fps. Temporal tools choose which frame indices to process and update the dictionary; spatial tools process those frames and write back their findings. At every step the planner decides whether the dictionary already contains enough information to answer.

The key design choice is the ordering constraint. In the authors' initial, unconstrained design, following the ReAct pattern of observation-then-thought, the planner would take shortcuts—jumping straight to a general-purpose tool like a video-language model rather than decomposing the problem. STAR forbids this. Temporal and spatial tools must alternate, and general tools are invoked only as a last resort. If the first tool is temporal, the planner draws from the temporal toolset at every odd step and the spatial toolset at every even step, with the reverse if the first tool is spatial. The authors argue this interleaving matters because temporal narrowing informs spatial narrowing and vice versa; decoupling them causes accuracy and efficiency to drop.

The framework is training-free and is evaluated with multiple-choice QA accuracy as the performance metric, plus number of processed frames and runtime as efficiency metrics. Experiments cover VideoMME (a test-only set of 2,700 multiple-choice examples spanning short videos under 2 minutes, medium videos of 4–15 minutes, and long videos of 30–60 minutes), NExT-QA (5,440 videos, roughly 52,000 human-annotated QA pairs, with 34K training, 5K validation, and 9K testing samples and videos averaging about 0.7 minutes), LongVideoBench (validation split of 1,337 questions, videos averaging around 8 minutes), and EgoSchema (over 5,000 multiple-choice pairs where human performance reaches 76% and current Video-LLMs fall below 70%). Subtitle information is deliberately excluded so only video content is used.

Why This Matters

Impact on research. The paper reframes VideoQA from "feed the model more frames" to "schedule the right tools in the right order." It shows that a lightweight, plug-and-play toolkit of models under 3B parameters can lift a strong proprietary MLLM by 8.2% on VideoMME and 4.6% on LongVideoBench while using only around 30 frames and about 15 seconds, instead of 6–8 minutes for a 72B Video-LLM. Naming and analyzing the Toolchain Shortcut problem gives the tool-learning community a concrete failure mode to design against, and the alternating temporal/spatial constraint is a simple, reusable recipe. The Visible Frame Dictionary and tool-card abstraction are also reusable infrastructure ideas.

Real-world applications.

• Video surveillance and traffic analysis: The paper's own example is that a Video-LLM struggles to count unique individuals in heavy pedestrian traffic, while object tracking plus person re-identification tools substantially improve accuracy—directly relevant to monitoring and counting tasks.

• Long-form media and content archives: LongVideoBench-style "needle-in-a-haystack" queries over hour-long videos map onto searching lecture recordings, sports footage, film, or broadcast archives for a specific moment or detail.

• Accessibility and assistive video description: OCR, captioning, and summarization tools chained together can extract text, describe scenes, and produce summaries for users who cannot watch a video directly.

• Egocentric and embodied assistants: EgoSchema-style egocentric video QA relates to wearable cameras and robots that need to recall what happened when and where.

Industry relevance. The result that a 3B-and-under tool stack plus a 4090 GPU—or an entirely sub-500M stack (STAR-mini, largest tool a 500M-parameter BLIP) on a Mac—can rival much

Authors’ abstract

Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the video. Our STAR framework enhances GPT-4o using lightweight tools, achieving an 8.2% gain on VideoMME and 4.6% on LongVideoBench. We believe that our proposed Video Toolkit and STAR framework make an important step towards building autonomous and intelligent video analysis assistants. The code is publicly available at https://github.com/fansunqi/VideoTool.