Optimizing Multimodal LLMs for Egocentric Video Understanding: A Solution for the HD-EPIC VQA Challenge

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Model (MLLM) optimization for egocentric (first-person) long-video visual question answering (VQA).

Technical level: Intermediate. The paper assumes familiarity with MLLM fine-tuning, prompt engineering, and VQA benchmarks, but its four-part pipeline is described in largely procedural terms.

Scope: The paper presents a full pipeline — input pre-processing, domain-specific fine-tuning of Qwen2.5-VL-7B-Instruct, a Temporal Chain-of-Thought prompting strategy, and answer cleaning/ensembling — that reaches 41.6% average accuracy on the HD-EPIC VQA benchmark.

What This Paper Is About

Multimodal large language models perform well on general visual question answering but struggle on the HD-EPIC VQA benchmark, which tests long egocentric kitchen videos with ambiguous queries, non-standardized answer choices, and questions that demand reasoning over long time spans. The authors set out to close that gap not by building a new model, but by optimizing every stage of the pipeline around an existing open-source model, Qwen2.5-VL-7B-Instruct. The goal is a system that answers the benchmark's 30 question prototypes more reliably than both larger general MLLMs and previous baselines.

Key Contributions

1. Input pre-processing framework. A multi-faceted strategy for question intent recognition and clarification, including classification of each query into one of 4 visual-context types (a single static image, multiple static images, a single temporally bounded video clip, or multiple distinct video segments), regular-expression-based prompt refinement, and standardization of multiple-choice option formatting.

2. Domain-specific fine-tuning. Fine-tuning of Qwen2.5-VL-7B-Instruct on a diverse collection of egocentric kitchen video datasets — EPIC-KITCHENS, CMU-MMAC, EGTEA Gaze+ of EgoProceL subsets, YOUCOOK2, VISOR, and selected portions of Ego4D relevant to object interaction and procedural understanding — with only the LLM components trained and the vision tower and MLP projector frozen.

3. Temporal Chain-of-Thought (T-CoT) prompting. A two-stage reasoning process for temporal questions in long videos, comprising explicit temporal cue exploitation, focused temporal windowing, multi-video synchronization, and hierarchical chunk-based processing for long videos.

4. Robust post-processing. An answer cleaning module that parses raw textual output into a single-letter choice and converts it to a zero-based index, plus an ensembling strategy that samples five semantically equivalent prompts per question and decides the final answer by majority voting.

Main Findings

• Overall accuracy: The full pipeline achieves 41.6% average accuracy on HD-EPIC VQA, compared to 37.6% for Gemini Pro, 36.9% for Qwen2.5 VL 32B Instruct, 33.5% for Qwen2.5 VL 7B Instruct, 32.4% for LLaVA-Video, 29.3% for LongVA, and 27.4% for VideoLlama 2 (as reported in the paper's per-category table).

• Per-category strengths: The system leads on Recipe (64.8), Ingredient (43.3), Action (42.0), 3D (40.9), and Motion (29.9). It trails Gemini Pro on Nutrition (37.0 vs. 34.7 — the proposed system is higher here), Motion (29.9 vs. 20.8), and Gaze (33.0 vs. 28.7), but scores lower than Gemini Pro on Nutrition (37.0 vs. 34.7 is higher; the paper reports Gemini Pro at 34.7 and the proposed system at 37.0) and on Gaze where Gemini Pro is 28.7 and the proposed system is 33.0.

• Pre-processing effect: Option-formatting changes alone produced measurable gains — converting numeric to alphabetic enumerators gave +1.6%, inter-option spacing +1.8%, semicolons +2.0%, and newline delineation for each option +2.4%. Targeted refinements together yield a +3.5% accuracy improvement. Removing pre-processing drops the average from 41.6% to 38.1%.

• T-CoT effect: The T-CoT strategy yields a +3.0% overall accuracy improvement across all tasks compared to direct VQA with only initial pre-processing. Removing fine-tuning and T-CoT together drops the average to 37.5%.

• Answer cleaning effect: The cleaning module improves accuracy by +0.8% over evaluating raw outputs. Removing post-processing entirely drops the average to 40.0%, though the Recipe category rises to 65.0 from 64.8 without it.

• Persistent temporal weakness: Despite fine-tuning, the model achieved 26% on Multi-Step Localization (vs. 22% pre-tuning, and Gemini Pro's 88%), 25% on Step Localization (vs. Gemini Pro's 70%), and 28% on Rough Step Localization (vs. Gemini Pro's 74%). The paper attributes this partly to training sequence lengths of 8192/32768 and a cap of 768 analyzed frames (total video tokens ≤ 24576), which is insufficient for videos longer than 12 minutes at 1 FPS (>720 frames).

• Scale does not help within this pipeline: Directly scaling from the 7B to the 32B variant offered no proportional performance gains; the authors attribute this to larger models' increased verbosity or uncertainty when constrained, and to extended T-CoT generations introducing noise (an overabundance of detailed short video segments proved less effective than fewer, longer ones).

• Efficiency cost: The multi-stage architecture incurs latency, which the authors flag as a critical performance-efficiency trade-off, particularly for real-time applications.

Methodology in Plain English

The authors treat the problem as a pipeline-engineering problem rather than a modeling problem. They start with Qwen2.5-VL-7B-Instruct and modify what goes in and what comes out.

First, they clean up the questions. Each question is classified by what visual material it needs — one still image, several still images, one video clip, or several separate clips — and then rewritten into a clearer, viewpoint-grounded form using pattern matching to extract entities, time markers, and relationships. The answer choices are reformatted so the model parses them more reliably.

Second, they fine-tune the language-model portion of Qwen2.5-VL-7B-Instruct on egocentric kitchen video datasets, keeping the vision components frozen. Training used a learning rate of 2×10^-7, batch size 2, 1 gradient accumulation step, 1 epoch, bfloat16 precision, the AdamW optimizer, a maximum sequence length of 131072 tokens, and video processing with a maximum of 768 frames per sample (minimum 4) with dynamic total pixel adjustment between 3136 and 846720.

Third, because fine-tuning alone left temporal questions weak, they add a two-stage reasoning procedure. Rather than asking the model a hard temporal question directly, they first extract or summarize the relevant context — resolving bounding boxes into object names, cutting the video to roughly ±10 seconds around a key moment, concatenating multiple clips onto a re-normalized timeline, or chunking long videos into non-overlapping 10-minute segments (under 768 frames) and generating a short narration for each, which is then prepended to the question.

Finally, they clean the model's raw text down to a single letter for automated scoring, and they run five semantically equivalent paraphrases of each question and take the majority vote.

Why This Matters

The paper shows that careful pipeline engineering — not just a bigger model — can move the needle on hard egocentric video benchmarks, and that directly scaling a model from 7B to 32B did not. It also documents concretely where current MLLMs still fail: long-range temporal reasoning, step localization, and ingredient ordering over long kitchen videos, where a proprietary model like Gemini Pro remains far ahead on several prototypes.

Real-world applications:

• Assistive and coaching systems for cooking, assembly, or other procedural tasks where a first-person camera watches a user and answers questions about what they just did or should do next.
• Automated documentation and compliance in kitchens, labs, or manufacturing, where long egocentric recordings need to be queried about object locations, movements, and step order.
• Wearable and AR assistants that need to answer "where is X?" or "what did I just do?" from a single viewpoint in real time.
• Dataset and benchmark tooling for egocentric video research, since the pre-processing and post-processing modules address ambiguity in question and option formatting that is common across benchmarks.

Industry relevance: the work is directly applicable to teams building wearable, robotics, or smart-home assistants that must reason over long first-person video with limited compute — and the latency trade-off the authors flag is exactly the practical constraint those teams face.

Future Directions

• Targeted fine-tuning of larger models. The authors posit that targeted fine-tuning of 32B or 72B models is a more promising path than scaling within the current pipeline, since the 32B variant delivered no proportional gains and produced noisier T-CoT output.
• Improving long-range temporal reasoning. The large gaps on Multi-Step Localization, Step Localization, Rough Step Localization, and Ingredients Order — and the frame/token caps the authors identify as a cause — point to a need for better long-video representations.
• Optimizing the latency-accuracy trade-off. The multi-stage architecture's added latency is described as critical for real-time applications, making efficiency optimization an explicit priority.
• Closing the gap to human-level cognition. The authors highlight deep reasoning and robust long-term memory for extended videos as the crucial open research trajectory.

Target Audience

Researchers and practitioners working on multimodal LLMs, egocentric and long-video understanding, and video question answering — particularly those participating in or benchmarking against HD-EPIC VQA. It is also useful for engineers who need a practical, reproducible recipe for squeezing better accuracy out of an existing open-source MLLM through input/output engineering rather than new model architecture. Readers looking for novel model design or theoretical analysis will find this is an engineering and ablation study instead.

Authors’ abstract

Multimodal Large Language Models (MLLMs) struggle with complex video QA benchmarks like HD-EPIC VQA due to ambiguous queries/options, poor long-range temporal reasoning, and non-standardized outputs. We propose a framework integrating query/choice pre-processing, domain-specific Qwen2.5-VL fine-tuning, a novel Temporal Chain-of-Thought (T-CoT) prompting for multi-step reasoning, and robust post-processing. This system achieves 41.6% accuracy on HD-EPIC VQA, highlighting the need for holistic pipeline optimization in demanding video understanding. Our code, fine-tuned models are available at https://github.com/YoungSeng/Egocentric-Co-Pilot.