Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?

AI summary

Overview

Research area: Computer vision and multi-modal large language models (LLMs), specifically streaming/online video understanding for interactive, goal-directed task assistance.

Technical level: Intermediate. The paper assumes familiarity with vision-language model components (vision encoders, Q-Formers, LLM backbones) and standard evaluation metrics (precision, recall, F1, BERT, ROUGE-L), though the problem framing itself is accessible.

Scope: The paper introduces a new cooking-domain dataset and benchmark for live, interactive step-by-step coaching, and proposes LiveMamba, a streaming multi-modal LLM baseline for delivering instructions and mistake feedback in real time.

What This Paper Is About

Multi-modal LLMs can converse about images and videos, but they are largely turn-based: they answer when prompted rather than reacting to events as they unfold in a live video stream. Real coaching requires three things at once — giving the next instruction, detecting when the user has completed it, and catching mistakes quickly enough to alert the user. The authors build a benchmark that captures these conditions (using cooking videos that contain real user errors) and a lightweight streaming model designed to meet them.

Key Contributions

1. Qualcomm Interactive Cooking dataset and benchmark, built by extending the CaptainCook4D dataset with timed instruction and feedback messages. Instruction messages describe the next recipe step; feedback messages acknowledge successful completion or flag mistakes. The authors state these messages are sufficient to independently guide a user to complete a given recipe.
2. LiveMamba, a light-weight multi-modal LLM for interactive cooking guidance, built on a Mamba-130M language backbone and an InternViT-300M-448px-V2_5 vision head with a Q-Former adapter.
3. A streaming evaluation protocol and metrics (Instruction Completion Accuracy, mistake detection precision/recall/F1, and feedback fluency via BERT and ROUGE-L) for measuring live guidance quality, plus a turn-based evaluation variant.
4. Empirical evaluation of state-of-the-art multi-modal LLMs on the benchmark, showing they struggle, with LiveMamba establishing a strong baseline.

Main Findings

• Zero-shot models perform poorly in the streaming setting. On the main set, LLaVA-NeXT reached IC-Acc 1.4, Video-ChatGPT 1.6, VideoChat2 1.6, Video-LLaVA 2.0, and VideoLLaMA3-7B 1.8, all with mistake precision, recall and F1 of 0.00 and BERT and ROUGE-L scores of 0.000. Gemini-2.5-Flash had the highest IC-Acc at 23.1 (the text states it "can recognize 18.9% of instructions being successfully completed"), followed by Qwen2.5-VL-7B at 18.9. No zero-shot approach performed well on mistake detection.
• Fine-tuning on the new dataset helps substantially in the streaming setup. LiveMamba (Ours) reached IC-Acc 31.5 and mistake F1 0.13 on the main set, compared with 7.6 IC-Acc and 0.01 F1 for a fine-tuned Videollm-online.
• Both augmentation schemes matter. Removing instruction completion augmentation dropped IC-Acc from 14.3 to 7.8. Removing counterfactual mistake augmentation dropped mistake F1 from 0.10 to 0.05, and the paper notes a significant jump in feedback fluency when it is added.
• External re-planning helps on the advanced planning set. LiveMamba (Ours) reached IC-Acc 12.6, mistake F1 0.19, BERT 0.941 and ROUGE-L 0.927, versus IC-Acc 10.9, F1 0.16, BERT 0.912 and ROUGE-L 0.901 without the re-planner. The authors note mismatch in instruction delivery remains highly challenging in this set.
• Turn-based evaluation gives higher numbers across the board. LiveMamba reached IC-Acc 51.0 and mistake precision/recall/F1 of 0.22/0.17/0.19, versus Qwen2.5-VL-7B at IC-Acc 38.9 and 0.11/0.04/0.06, Qwen2-VL-7B at 19.4 and 0.06/0.46/0.11, and VideoLLaMA3-7B at 17.8 and 0.08/0.61/0.15.
• Mistake detection is hard for specific reasons. The paper attributes weak performance to the fine-grained nature of mistakes (for example, 1 teaspoon versus 1 tablespoon of sugar, or spilling flour on the counter), the diversity of possible mistakes, and the need to detect them the moment they occur.
• Efficiency on a consumer H100 GPU. LiveMamba has an average real-time factor of 4, processing at 8.1 frames per second against input arriving at 2 frames per second. Time to first token averages 1.1 seconds; re-planning with Qwen3-32B takes 6.1 seconds on average.
• LiveMamba encodes more tokens per frame than a comparable streaming baseline. The Mamba-130M backbone allows 32 embedding tokens per frame versus 10 for Videollm-online, at similar memory cost.

Methodology in Plain English

The authors start from CaptainCook4D, a dataset of 384 egocentric cooking videos with graph-structured recipes and annotated action segments, including seven categories of mistakes. They remove noisy mistake annotations and add timestamps for mistakes across all categories except order error and missing steps, since those require more complex reasoning about prior and future steps. From this they build two versions: a Main Set, where the step-by-step plan follows the order actions appear in the video, and an Advanced Planning Set, where the plan comes from the recipe graph and the model must handle users who skip ahead or work out of order. The benchmark's test splits come from CaptainCook4D's original video recording splits.

For the model, a video frame is encoded by an InternViT-300M-448px-V2_5 head, compressed by a Q-Former into fewer tokens, and passed to a Mamba-130M language backbone. Two special tokens control the interaction: <vision> lets the model request the next frame, and <response> lets it speak when appropriate, giving it a "when-to-say" ability instead of turn-based prompting. When the user goes off-plan, LiveMamba calls an external re-planner (Qwen3-32B) that receives the initial plan, the completed steps, and the model's feedback, then selects the next optimal step.

Training happens in two stages. Pre-training tunes only the Q-Former adapter on image and video datasets for object grounding (LVIS, VISOR annotations from EPIC-KITCHENS) and fine-grained action understanding (SSv2, EPIC-KITCHENS and Ego4D narration). Fine-tuning trains both the adapter and the language backbone, with three augmentations: temporal jittering of instruction start timestamps by ±K seconds to absorb accumulated autoregressive error, with K = 30 found to work well; instruction completion augmentation, converting EPIC-KITCHENS and Ego4D Goal-Step action descriptions into instructions with a Qwen2.5-8B model; and counterfactual mistake augmentation, generating plausible grounded counterfactual action descriptions and swapping instructions between recipe steps so the model learns to recognize divergent mistakes and trigger re-planning.

Evaluation uses Instruction Completion Accuracy (a prediction counts as correct if it falls within a 30-second window centered on the ground-truth completion time, roughly spanning the last ~25% of one step and the first ~25% of the next), mistake detection precision/recall/F1 over the same window, and BERT and ROUGE-L scores for feedback fluency on true positive detections. Zero-shot models had to be prompted at 5-second intervals, and Videollm-online's narrations were piped to a helper LLM (Phi-3-mini-4k-Instruct) to make completion and mistake judgments.

Why This Matters

The paper identifies a capability gap: models that describe video well are not the same as models that can coach someone through a task in real time. By releasing a benchmark where correctness depends on when a model speaks, not just what it says, it gives the field a way to measure progress on live, situated assistance rather than post-hoc narration or question answering.

Potential real-world applications:

• Hands-free cooking assistance that watches a user's technique and warns about measurement, temperature, or timing errors.
• Guidance on compute-constrained devices such as mobile phones or smart glasses, which the paper explicitly names as deployment targets motivating LiveMamba's efficiency focus.
• Assistive coaching in physical tasks more broadly, given the paper's comparison to object manipulation and toy assembly datasets in related work.
• Remote expert assistance, where the paper cites tele-operated feedback settings (HoloAssist, WTAG) as prior but insufficiently self-contained alternatives.

Industry relevance: The work originates from Qualcomm AI Research, and its design priorities — a 130M-parameter language backbone, a 4x real-time factor, 1.1-second first-token latency, and on-device inference without server round-trips — reflect the constraints of shipping interactive assistants on edge hardware.

Future Directions

• Detecting subtle and divergent mistakes remains open. The authors state that robustly handling scenarios involving order errors or missed steps in the advanced planning set is challenging for all state of the art open-source models.
• Curation of high-quality mistake data is flagged as promising. The paper calls the jump from counterfactual mistake augmentation a sign that this direction deserves further research.
• Expanding beyond cooking. The work is focused on the cooking domain through Qualcomm Interactive Cooking, leaving generalization to other procedural domains unaddressed.
• Closing the gap between streaming and turn-based performance. IC-Acc was far higher in turn-based evaluation (51.0 for LiveMamba) than in streaming evaluation (31.5), and the paper attributes streaming difficulty to error propagation across steps.

Target Audience

Researchers and engineers working on streaming video understanding, multi-modal LLM agents, and egocentric activity recognition will find the benchmark and metrics most directly useful. It is also relevant to practitioners building on-device or edge assistants, and to anyone studying how to evaluate interactive, goal-directed AI systems where timing is part of correctness. Readers looking for a treatment of non-cooking domains, or of fully reactive setups with a compliant user, should note the paper's own statement that fully reactive setups are not possible with offline datasets and that this work stays within cooking.

Authors’ abstract

Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.