Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision

AI summary

Overview

Research area: Egocentric (first-person) video understanding, streaming video-language models, proactive wearable assistance, and agentic synthetic-data generation for computer vision.

Technical level: Intermediate. The paper assumes familiarity with vision-language model fine-tuning (LoRA, token-level cross-entropy), classification metrics (macro-F1, G-mean), and agentic tool-calling pipelines, though the core idea is explained plainly.

Scope: A competition system description for the EgoProactive track of the ECCV 2026 Wearable AI Challenge, covering a single-token reformulation of the decide-to-interrupt problem, an agent-generated supervision corpus, model-selection protocol, and vocabulary pruning to meet a parameter limit.

What This Paper Is About

A proactive wearable assistant watching a user perform a procedural task through egocentric video must decide, after every eight-second chunk, whether to speak up with guidance or stay silent. The official task supplies the user's opening query and each video chunk, and scores submissions with macro-F1 over the interrupt/silent decision. The paper describes a system that ranked first in the large-model division and second in the ≤2B division of that challenge, and argues that visually grounded synthetic supervision matters more than supervision volume.

Key Contributions

1. A single-token "verbalizer" reformulation. Instead of generating the literal strings $interrupt$ plus an utterance or $silent$, the model predicts one token, yes or no, and the decision is read off the renormalised softmax probability over those two logits and thresholded at τ. This decouples the speak/stay-silent choice from utterance generation and improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation.
2. An agentic, visually grounded annotation pipeline built on the open-source Ambient video research agent, which inspected 234 of 235 clips and produced 13,730 training rows at roughly $0.05 per clip, including explicit silent intervals as positive labels.
3. A measurement protocol for a release-set-only regime, combining a 210/490 development/held-out split of the released videos with a cross-domain benchmark of 140 HoloAssist videos used to rank candidate models.
4. A provably lossless vocabulary prune that brings a 2.2132B-parameter model down to 1.9977B, meeting the ≤2B division limit with 544/544 identical chunk predictions and a G-mean of 0.7291 before and after.

Main Findings

• The competition result. The 4.54B entry took first place in the large division at 0.7179 macro-F1 on the hidden test set, and the pruned 1.9977B entry placed second in the ≤2B division at 0.6866, 0.0045 behind the winner.
• Generation failed in a specific way. The generative formulation reached interrupt precision 1.000 at recall 0.138, firing on 146 of 1,058 true positives. Macro-F1 still rewarded this with 0.4517 and G-mean with 0.4004. The verbalizer reached 0.7008 macro-F1 and 0.7007 G-mean.
• Conversation history is required. Without history the verbalizer always interrupts (macro-F1 0.3521, G-mean 0.0000), because it cannot tell whether assistance was already given. The prompt carries the user query and the four most recent dialogue turns.
• Visual grounding beats annotation volume. A narration-derived corpus was four times larger (953 clips, 59,359 chunks) and ten times cheaper (~$0.005 versus ~$0.05 per clip) than the agent corpus (234 clips, 13,730 rows), yet a 2B model trained only on narration reached 0.49 transfer, below the 0.59 obtained from an unrelated real corpus (HoloAssist), with an in-domain score of 0.52. The teacher saw rich text while the student saw sparse frames at 0.83 fps.
• The synthetic corpus helps transfer but hurts the in-domain benchmark. Adding the agent corpus raised cross-domain G-mean from 0.557 to 0.577 (+0.021) while lowering in-domain G-mean from 0.659 to 0.617 (−0.042); the gain was concentrated in interrupt recall (+0.064 F1). Adding a second corpus lowered cross-domain performance from 0.590 to 0.548.
• In-domain evaluation inverted the ranking. The intervention that mattered was a clear loss on the in-domain benchmark while being the selection criterion that led to the winning submission on the cross-domain benchmark.
• Bigger backbones did not help. Under the same recipe, the 27B model scored 0.002 below the 4B model (0.6989 versus 0.7007), and the 2B model stayed within 0.01 of it (0.6914). On the official leaderboard, a 4.54B model beat 27B and 28.9B entries.
• Agent policy design mattered measurably. Error analysis found the unconstrained agent omitted setup-phase cues, lagged action onsets, and produced redundant cues during repeated motions (20 cues for roughly five semantic steps). Allowing a ±1 s matching tolerance raised G-mean by approximately 0.05, and the final policy capped output at one cue per 8 s interval.
• The dense policy matched the real base rate without being told it. Agent annotation produced a 53% interrupt rate against the real set's 54.3%, within 1.3 points, while an identical pipeline with a looser policy landed at 11%.
• Vocabulary pruning was exact. Parameters fell from 2.2132B to 1.9977B and vocabulary from 248,320 to 143,084, with 544/544 identical chunk predictions, agreement on the interrupt probability of 0.00e+00, and zero byte-fallback incidence across all 700 released videos.
• Not reported: the paper does not report any metric on the quality or content of the generated utterances, since the metric never scores them, and it does not report test-set results under G-mean (test scores are macro-F1 only).

Methodology in Plain English

The starting point was a straightforward framing: fine-tune a vision-language model to output the literal command word followed by a short spoken sentence. That approach collapsed into a safe corner, speaking rarely and only when very confident. The authors changed the output format instead of the model: the model now emits exactly one token, yes or no, and the team reads the decision from the two logits' renormalised probabilities and a threshold. Because the threshold sits outside training, the operating point can be tuned after the fact without retraining, and a decision costs one forward pass with no decoding loop. Labels are masked everywhere except the final assistant turn's decision token and end-of-turn marker.

Each decision sees all frames observed so far, uniformly subsampled to at most 32 frames and resized to a maximum side of 512 px, plus the user's query and the four most recent turns. The emitted decision re-enters the history for the next chunk, which is what stops the model from interrupting every time.

Because labelled data were limited to the released validation set of 700 videos, 135 tasks and 1,947 scored decision points (54.3% requiring intervention), the team generated supervision. Their agent, built on Ambient, pairs a DeepSeek-V4-Flash-0731 orchestrator with a Qwen3.6-27B vision model and three tools: a coarse whole-video description, a question-answering tool for segments up to five minutes, and a detailed window describer. Any detail that becomes a label is confirmed with a window-level call first, using up to 12 inspection windows per clip. The output is a structured event list with timestamps, decision types, utterances, visible context, timing rationale, confidence, plus explicit silent intervals with reasons; events are then shifted by −0.5 s and binned into eight-second chunks.

They compared this against a much cheaper route that reads dense human narration text with an LLM and writes a per-chunk script with no vision pass at all. The narration route was scaled much further and transferred worse.

For the ≤2B division, they pruned the 2.2132B Qwen3.5-2B model contiguously: token IDs in [0, 143000) keep their original index, with an explicit set of higher vision, video and special tokens appended, so no re-indexing is possible; embedding rows are cloned exactly rather than re-initialised.

Training used Qwen3.5-4B and Qwen3.5-2B backbones with LoRA of rank 32 and α = 64, dropout 0.05, applied to all seven attention and MLP projections, a learning rate of 1×10⁻⁴ with a cosine schedule and 0.03 warmup ratio, batch size 1 with 8-step gradient accumulation, and bf16 with gradient checkpointing. The mix was the released videos seen twice plus the full agent-generated corpus. Model selection used G-mean rather than macro-F1, since macro-F1 awards about a third of the achievable score to always-silent or always-interrupt policies whereas G-mean assigns both zero.

Why This Matters

The paper's distinctive claim is methodological rather than architectural: it shows that on a timing decision, reformulating the output space is worth far more than scaling the backbone, and that synthetic supervision is only useful when it is grounded in evidence the student model can also perceive. It also documents a case where the in-domain benchmark actively pointed the wrong way, which is directly relevant to anyone training on a small released dataset and evaluating on it.

Real-world applications:

• Wearable and AR coaching assistants that must decide when to speak during cooking, repair, assembly or exercise without nagging the user.
• Assistive technology for hands-busy or low-vision users, where an interjection at the wrong moment is worse than none.
• Automated quality control and training review for industrial procedural tasks, where intervention timing signals whether a worker is on track.
• Cost-efficient data annotation, using tool-calling video agents to turn unlabelled footage into structured supervision at roughly $0.05 per clip.

Industry relevance: the results argue against simply buying a bigger model, showing a 4.54B system beating 27B and 28.9B entries across independent teams, and they give a concrete recipe for shipping a compliant sub-2B model via lossless vocabulary pruning. The emphasis on cross-domain rather than in-domain model selection is a practical lesson for teams with small labelled corpora and hidden test sets.

Future Directions

• Fix the evaluation protocol. The authors state they would establish a cross-domain evaluation set before training, and use it alongside in-domain results to separate genuine transfer from increased fit to the released data.
• Close the grounding gap. The narration-versus-agent comparison identifies visual grounding as the deciding factor, but no method is offered for measuring grounding directly or for repairing weakly grounded teachers.
• Revisit scale under better supervision. Since capacity scaling gave at most 0.01 G-mean in-domain and the 27B model trailed the 4B model by 0.002, it remains open whether larger backbones help once supervision and data composition improve.
• Explore mixture and diversity effects. Replacing 6.6k of 13.7k procedural examples with household and sightseeing footage dropped cross-domain G-mean from 0.590 to 0.548, and Ego4D, HoloAssist and Ego-Exo4D-derived synthetic data plus temporal augmentation gave no consistent improvement, leaving the question of which additional sources would transfer.

Target Audience

Researchers and engineers working on egocentric or streaming video-language models, proactive and always-on assistants, and wearable AI; competition participants in the ECCV 2026 Wearable AI Challenge; and practitioners interested in agentic annotation pipelines, parameter-budget compliance via vocabulary pruning, or model-selection protocols when labelled data are scarce.

Authors’ abstract

We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either $interrupt$<utterance> or $silent$, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.