Improving Proactive AI Assistance with Hierarchical Procedural Understanding

AI summary

Overview

Research area: Machine learning for proactive AI assistance — specifically vision-language models (VLMs) that watch a continuous first-person video stream and decide what guidance to give, when to give it, and at what level of detail.

Technical level: Advanced. The paper assumes familiarity with vision-language model fine-tuning, streaming/online video settings, supervised fine-tuning objectives with masked losses, and proactive assistance metrics (sF1, gF1, PQS).

One-sentence scope: The paper builds a hierarchically annotated training set and benchmark (the ProactiveCoach suite) plus a fine-tuning method that teaches VLMs to give phase-, step-, and action-level proactive guidance and to switch guidance granularity on request.

What This Paper Is About

Proactive assistants must observe a user's ongoing task and speak up at the right moment, but existing datasets either only detect objects/events or only give guidance at one fixed level of detail. Fixed-granularity guidance makes it hard to tell whether a guided event is finished, hard to convey the broader purpose of the current step, and impossible to switch to a coarser or finer level when the user asks. The authors address this by constructing hierarchically structured guidance data, fine-tuning VLMs on it, and adding a lightweight router so the displayed guidance level can change without retraining the guide model.

Key Contributions

1. ProactiveCoach-Instruct (training) and ProactiveCoachBench (evaluation). A dataset of 11,119 samples built from 7,232 first-person videos across eight source datasets, with guidance structured at phase, step, and action levels plus explicit guidance timing. Splits are 10,007 samples for training and 1,112 for evaluation.

2. ProactiveCoach, a hierarchical guidance learning method. A single VLM is fine-tuned to jointly produce phase-, step-, and action-level guidance, with each guidance sentence paired with its level and a hierarchical ID encoding the parent–child structure.

3. An adaptive guidance system. The fine-tuned guide model generates multi-level guidance, and a lightweight guidance router selects which level to display based on the user's request — allowing level changes without additional fine-tuning of the guide model.

4. Empirical demonstration across four backbones. Hierarchical supervision improves overall performance over fixed-granularity (step-only) supervision with gains up to 9.6%p, and the adaptive system beats an in-context adaptation baseline by 57.1%p across four guidance-level transitions.

Main Findings

• Hierarchical supervision beats step-only supervision on every backbone. On ProactiveCoachBench step-level guidance, Qwen3-VL-4B improves from 49.32 to 58.96 Avg. with multi-level supervision; Qwen3-VL-8B from 53.07 to 59.37; Qwen3.5-4B from 50.06 to 59.30; Qwen3.5-9B from 51.18 to 58.81. The headline figure reported in the abstract and introduction is gains of up to 9.6%p.

• Adding intermediate levels helps step-level guidance, and all three levels help most. For the ablation on supervision levels, Step-only scores 50.06 Avg., Phase-Step 53.60, Step-Action 57.33, and Phase-Step-Action 59.30. The authors attribute the action-level benefit to evidence about what a step is made of and how far along it is, and the phase-level benefit to context about the step's purpose and expected outcome.

• Prompting alone does not produce reliable multi-level guidance. Zero-shot Qwen3.5-4B under a multi-level prompt scores 29.32 Avg. at phase, 26.99 at step, and 21.74 at action, versus 62.94, 59.30, and 53.44 for the fine-tuned model.

• The adaptive guidance system outperforms in-context adaptation. Across the four transitions Step→Phase, Step→Action, Phase→Step, and Action→Step, the fine-tuned system scores 85.54, 57.93, 66.56, and 73.70 Avg., versus baseline scores of 9.46, 18.65, 8.49, and 18.70. The paper reports a 57.1%p average overall improvement across the four transitions.

• Strong results against diverse baselines on ProactiveCoachBench. On the phase level, the best reported configuration (Qwen3-VL-8B with fine-tuning) reaches sF1 54.63, gF1 76.21, PQS 61.37, Avg. 64.07. On the step level it reaches 50.24 / 72.02 / 55.83 / 59.37, and on the action level 50.07 / 64.12 / 44.77 / 52.99. Baselines include Gemini 3 Flash Preview, Claude Haiku 4.5, GPT-5.4 mini, ProAssist-8B, VideoLLM-Online, MMDuet2, VideoChat3-4B, VideoChat-R1.5-7B, InternVL3-8B, VideoLLaMA3-7B, LLaVA-Video-7B, and Qwen3-VL-32B.

• EgoProactive validation results follow the same pattern. The best reported configuration reaches sF1 35.28, gF1 63.06, PQS 48.55, Avg. 48.96 at the step level. The authors note that some base models show high sF1 but very low gF1 (they respond when they should stay silent), while others show low sF1 but high PQS (they stay silent when a response is required).

• Human review of annotations scores 4.27 overall. Fifty examples sampled from ProactiveCoachBench were rated on video alignment, hierarchical coverage, goal relevance, and guidance quality, with scores above 4.0 on most criteria.

• Dataset coverage exceeds prior procedural assistance datasets on hierarchy. The comparison table lists 7.2K videos and 1.5K hours for this work, against 3.4K/0.5K for ProAssist, 2.5K/0.2K for GuideMe, 29.3K videos for Pro²Bench, 0.6K for StreamPro-Bench, 3.4K/0.3K for HiREST, 3.5K/0.1K for GUIDE, 4.5K for Ego4D-HCap, and 1.2M/128K for Action100M.

Methodology in Plain English

The researchers define a three-level procedure hierarchy: a phase is a high-level unit achieving an intermediate goal through multiple steps, a step is a procedural unit achieving a specific outcome through multiple actions, and an action is an observable fine-grained event and the finest guidance level. Each guidance sentence is linked to its event span and its guidance time, and guidance timing is defined independently at each level — the next guidance at a level is given right after the previously guided event at that level ends. For each video chunk, multi-level guidance is combined into a single <response> in phase, step, and action order; a chunk with no new guidance at any level is labeled <silent>.

Data construction has three stages. First, procedural data curation: GPT-4.1-mini scores videos for goal clarity and key-event coverage and retains suitable ones, then Gemini 2.5 Flash performs video-grounded recaptioning to fix captions that do not match the video, add missing ones, and align each caption to a temporal segment. Second, procedure hierarchy construction: GPT-4.1-mini filters for goal-relevant annotations, consolidates them into actions, groups actions into steps and steps into phases, sets parent time spans to cover children, and iteratively checks and refines the hierarchy for consistency. Third, proactive guidance generation: GPT-4.1-mini generates guidance at each level, and rule-based guide timing assignment places the next guidance at the frame immediately after the current guided event ends, either at a single level, at multiple levels simultaneously, or as <silent>.

Training uses supervised fine-tuning with teacher forcing, providing previous ground-truth outputs as history. Because <silent> is more frequent than <response>, the loss is computed only on selected chunks: all <response> chunks, each <silent> chunk immediately following a <response> chunk, and randomly sampled additional <silent> chunks matched to that total.

Experimental setup. Backbones are Qwen3-VL-4B/8B and Qwen3.5-4B/9B, trained for three epochs on eight B200 GPUs. Video is sampled at 1 FPS and grouped into 2-second chunks, with a decision at each chunk; a causal block-sparse attention mask restricts each token to frames from the current and preceding 10 seconds while preserving access to the instruction and all previous response and silence tokens.

Evaluation uses predefined anchors. Response Anchors use all ground-truth guidance timestamps; Silent Anchors use a randomly sampled timestamp with no ground-truth guidance from the central 30–70% of each phase, step, or action segment. On EgoProactive the original anchors are used. All models receive the most recent 10 seconds of video plus the complete ground-truth dialogue history, which prevents earlier model errors from compounding. Metrics are sF1 (semantic match and coverage of guidance), gF1 (correctness of respond-versus-silent decisions), and PQS (joint scoring of decisions and guidance quality), with GPT-5.2 used to judge guidance content quality.

Why This Matters

Impact on research. The work argues that "what to say" and "when to say it" cannot be evaluated well at a single fixed granularity, and supplies both supervision and an evaluation protocol designed around hierarchical progress. It shifts proactive assistance benchmarks from detection- or step-only setups toward multi-level procedural understanding and explicit guidance-level adaptation.

Real-world applications.

• Smart-glasses assistants that recognize what the user is doing and offer guidance at the moment it is needed.
• Humanoid assistant robots that monitor task progress and provide physical assistance when help is required.
• Cooking, assembly, and repair coaching, where the system tracks whether the current step is finished through its constituent actions.
• Adaptation to expertise: a novice may want action-level detail while an expert wants phase-level summaries, and the requested level can change mid-task.

Industry relevance. Proactiveness — deciding whether to speak or stay silent in a continuous stream — is a central product requirement for always-on assistants. The paper's router design shows that granularity control can be separated from guidance generation and handled by a lightweight component, which means adapting guidance level to a user does not require retraining the underlying model.

Future Directions

• Beyond the four evaluated transitions. The adaptive system is evaluated on Step→Phase, Step→Action, Phase→Step, and Action→Step; other transitions and repeated switching within a task remain untested.
• Stronger and larger backbones. Experiments cover Qwen3-VL-4B/8B and Qwen3.5-4B/9B, with Qwen3-VL-32B evaluated only as a baseline rather than a fine-tuning target.
• Improving what prompting alone can do. Zero-shot multi-level guidance scored far below fine-tuning, leaving open whether better prompting, in-context examples, or other interventions could narrow the gap without retraining.
• Broadening source data and domains. The dataset draws on eight egocentric video datasets and covers assembly, repair, electronics, food and beverage preparation, and household and gardening activities; generalization to other procedural domains and non-egocentric viewpoints is not established.

Target Audience

Researchers and practitioners working on proactive or streaming multimodal assistants, egocentric and procedural video understanding, and VLM fine-tuning. It is also relevant to product and applied teams designing always-on assistants that must decide when to interrupt and how much detail to give. Readers need prior familiarity with vision-language models and standard evaluation metrics for guidance quality, since the paper assumes this background.

Authors’ abstract

Proactive AI assistants continuously observe a user's activity and decide whether to provide new guidance or remain silent. They should provide appropriate guidance for the task, determine when to provide the next guidance based on task progress, and adjust the guidance level to the user's expertise and needs. Supporting these capabilities requires training and evaluation data that reflect procedural structure and capture how guidance should adapt to task progress and user needs. However, existing datasets either focus on detection-based proactive understanding or provide procedural guidance at a fixed granularity. Fixed-granularity guidance provides limited information about fine-grained progress and broader procedural context, making it difficult to determine completion and adapt guidance granularity. To address these limitations, we introduce the ProactiveCoach suite, comprising ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned VLMs with an adaptive guidance system. ProactiveCoach-Instruct provides hierarchically structured guidance at the phase, step, and action levels for learning task progress and procedural context. ProactiveCoachBench evaluates whether models provide appropriate guidance at the right time across different guidance levels and adapt when the requested level changes. We fine-tune pretrained VLMs on ProactiveCoach-Instruct and demonstrate its effectiveness across backbones. Compared with fixed-granularity supervision, hierarchical supervision improves overall performance across backbones by up to 9.6%p. We further build an adaptive guidance system by combining our fine-tuned model with a lightweight guidance router. Without additional fine-tuning, our system outperforms the in-context adaptation baseline by 57.1%p across four guidance-level transitions. Our project page is available at https://jinsuby.github.io/ProactiveCoach/.