Ambient @ EgoLongQA 2026: Distilling Long-Video perception into a Sub-2B Model

AI summary

Overview

Research area: Computer vision — long-form egocentric video question answering (VideoQA), knowledge distillation, and model compression.

Technical level: Advanced.

Scope: A single-paper system description of a 2B-parameter vision-language model that answers four-way multiple-choice questions about ten-minute egocentric videos in one greedy forward pass, built by distilling the perception module of a larger agentic pipeline.

What This Paper Is About

The EgoLongQA track of the Wearable-AI Challenge (ECCV 2026) asks models to answer multiple-choice questions about long egocentric videos, typically around ten minutes long, where questions are often two-hop and compound: they reference an anchor event and then ask about something that happened relative to it. The ≤2B parameter division constrains the entire multimodal checkpoint, vision tower included, which rules out running a tool-using agent at that size. The paper's goal is to show that a strong agent's accuracy can be moved into a small single-pass student model by distilling the right component of the agent rather than imitating the agent as a whole.

Key Contributions

1. Component-level distillation. Rather than imitating the agentic pipeline, the authors distill only its junior perception module into a 2B student, because the junior's input/output contract is exactly what the student receives at inference (zero train/test mismatch), whereas the senior orchestrator's multi-turn tool trajectory cannot be expressed in a single forward pass.

2. Admissibility by vocabulary pruning. The 2B backbone is 2.2132 B parameters, over the divisional limit. Pruning the multilingual embedding table from 248,320 to 143,469 rows takes the checkpoint to 1.9985 B with zero measured logit change on retained rows and byte-identical generation on 70 of 70 held-out items.

3. A prior-corrected evaluation protocol. Because supervised distillation transfers the benchmark's own answer prior into the student, the authors build training and held-out sets with options permuted to a uniform gold distribution, isolating perception from prior on identical content.

4. A 2B entry that placed first in the ≤2B division with 0.8279 on the organisers' held-out test set, with the test score matching the validation figure to within 1.4 pp.

Main Findings

• First place in the ≤2B division. The entry scored 0.8279 on the official held-out test set, a margin of 1.27 pp over the runner-up (smartkaist, Qwen-Atomic, 1.99 B, 0.8152).

• Distillation is the dominant lever. It moves the 2B backbone from 27.1% to 81.4% on the held-out questions — a 3× improvement. Adding synthetic data on top of distillation did not change that held-out figure (81.4% either way, 3,879 rows), though it improved out-of-distribution robustness.

• Junior passes carry most of the accuracy. Under the teacher's gated cascade, junior unanimous samples (senior skipped) accounted for 60–66% of samples at 91–95% accuracy, while the 34–41% escalated to the senior with tools scored only 63–67%. Two thirds of the validation set was already solved at over 90% by junior passes alone; the senior only ever saw the residual hard subset.

• Synthetic data teaches the task but does not absorb the prior. Training on 943 generated questions alone moved the base model from 27.1% to 54.3% on unseen videos. On 500 human-written egocentric questions with near-uniform gold (gold C-rate 27%), the synthetic-only model predicted C 26% of the time, versus 40% for the trace-distilled model and 19% for the 2B base.

• Vocabulary pruning is lossless here. The embedding table is 23% of the backbone (508.6 M of 2.2132 B, vocabulary 248,320 × 2048). After pruning to 143,469 rows the model is 1.9985 B, a margin of 1.50 M under the limit, with a measured logit difference of exactly 0 on all retained rows.

• Resolution transfer is cheap. Training at max_pixels = 50176 (≈64 vision tokens per frame) and inferring at 331776 (≈423 tokens per frame) is worth +5.8 points over training and inferring at the low setting, while cutting training cost roughly 6×.

• One epoch is enough. Epoch 1 outperformed epoch 2 in all five distillation runs. On the option-shuffled model, a second epoch costs 4.3 points on the debiased metric while raising the skewed one, because it re-absorbs the prior that shuffling removed.

• Latency collapses. The teacher pipeline runs at roughly 393 s/sample; the student runs at roughly 32 s/sample, replacing a three-vote, gated, senior-agent pipeline with one greedy generation. The 2B student reaches 89% of the accuracy of the large agentic pipeline using 1.1% of its parameters.

• Reinforcement learning did not help. GRPO with verifiable reward over the SFT-distilled model produced no reproducible held-out gain. The authors suspect the cause is a lack of prompts that are both in-distribution and unseen — roughly 25 videos, which they call insufficient.

• More frames help bigger models and hurt small ones. At ≤2B, going from 100 to 400 frames dropped full-700 accuracy from 64.3% to 61.5% (−4.4 pp), while at 35B the same change (Qwen3.6-35B-A3B in the agentic pipeline) improved accuracy by +11.2 pp.

• All teachers were open-weight. No closed model was used anywhere in the pipeline, at a measured cost of about 1.4 points against the best closed junior the authors evaluated. The distillation set is 2,936 oracle-filtered traces over 605 videos, with a median of five distinct descriptions per question.

Methodology in Plain English

The authors start from an existing agentic pipeline called Ambient, which samples 100 uniform frames, runs a junior perception module three times, and only if the three passes disagree calls in a larger senior model that can use clip-retrieval tools to look at parts of the video in higher fidelity. The senior's tool loop is the source of the pipeline's power but cannot be squeezed into a single forward pass.

Their key decision is to distill only the junior. Teacher models run the junior contract over the training videos and emit a timestamped free-text description followed by a structured <video_description> and <answer><choice>X</choice><reason>…</reason> block — verbatim the agent's existing "preempt-answer" tool contract. Only traces where the emitted choice matched ground truth and all three sampled junior passes agreed are kept, yielding 2,936 traces over 605 videos. The teacher pool is entirely open-weight, dominated by google/gemma-4-31b-it (42.0% of traces) and qwen/qwen3.5-122b-a10b (40.1%).

Separately, they use the Ambient agent to generate 943 audited multiple-choice questions over 408 Ego4D videos. The generation pipeline rejects questions answerable without the video (an adversarial filter), checks the evidence is actually visible (frame-grounded verification, which rejected 397 items), and balances the answer letters, since raw generator output is 42% "A". Net yield is 26%, with 1,319 rejected overall.

The student is a Qwen3.5-family 2B vision-language model that consumes 100 uniformly sampled frames scaled to a maximum dimension of 768 along with the question and four options. Training is LoRA (r=16, α=32) for one epoch with a cosine schedule, batch size 1 with gradient accumulation 8, and a 3× token weight on the <choice> span; dropping that weight to 1.0 was worse on every evaluation.

To meet the parameter cap, they keep token ids [0, 143000) unchanged, append a small set of explicitly retained rare tokens, relocate the 33 added and special tokens, and copy every embedding row exactly. The transformer, vision tower, projector and tokenizer are untouched, and the model code remaps tokenizer ids before the embedding lookup.

Evaluation is done on a held-out set of videos with the same questions but options permuted to a uniform gold distribution, because the validation gold distribution is severely skewed (always answering C scores 63.4%). All experiments are measured against this debiased held-out set.

Why This Matters

The paper argues that for agentic systems, the useful distillation target may be a single component rather than the whole agent: the junior perception module's contract is directly imitable, so a small model can inherit most of the pipeline's accuracy at a fraction of the parameters and roughly a twelfth of the per-sample latency. The paper also documents that supervised distillation imports the benchmark's answer prior into the student, and offers a permutation-based evaluation as a way to separate perception from prior on identical content.

Real-world applications:

• Wearable and smart-glasses assistants that need to answer questions about the last ten minutes of egocentric video on-device, where a ≤2B checkpoint and ~32 s/sample inference are plausible constraints.

• Always-on memory and recall tools for field work, inspection, or training, where a user asks about an event and then a follow-up about something relative to it.

• Assistive technology for people who need retrospective summaries of what happened during a recorded span of activity.

• Edge deployment generally, since the vocabulary-pruning technique demonstrates how to fit a near-limit multimodal checkpoint under a hard parameter budget without changing the logits the task depends on.

Industry relevance: The result is a recipe for turning an expensive multi-model agent into a shippable single-pass model, which matters for cost, latency, and on-device deployment. The finding that reinforcement learning gave no reproducible gain over supervised distillation — and the diagnosis that this stems from too few prompts that are both in-distribution and unseen (roughly 25 videos) — is a concrete data-curation constraint for teams attempting post-training on scarce video QA data.

Future Directions

• Sourcing prompts that are both in-distribution and unseen. The paper attributes the failure of GRPO to a shortage of such prompts (roughly 25 videos) and explicitly frames this as the constraint that decoupled-clipping and reward-normalization fixes (DAPO, Dr. GRPO) do not address.

• Understanding the frame-budget reversal. More frames helped the 35B model (+11.2 pp from 100 to 400 frames) and hurt the ≤2B model (−4.4 pp over the same change); the paper reports the effect but does not explain the mechanism.

• Closing the gap to the teacher. The student reaches 89% of the large agentic pipeline's accuracy, but the paper does not report a route for the remaining 11%, nor whether a 27B distillation (which reached 85.7% from the same 2,936 traces) suggests headroom from larger students.

• Generalizing the pruning argument. The authors document an asymmetry that makes vocabulary pruning harder for generative models than for classifiers, but do not report results for other architectures or vocabularies; the paper does not report inference hardware, memory footprint, or energy figures.

Target Audience

This paper is most useful to applied researchers and engineers building efficient multimodal systems — particularly those working on video-language models, knowledge distillation, agentic pipelines, and on-device or edge deployment under strict parameter budgets. It is also relevant to challenge participants and practitioners who need a concrete, reproducible account of how a large tool-using agent can be collapsed into a single small model, and to anyone designing evaluation sets for benchmarks with skewed answer distributions. Readers without a background in vision-language models or distillation will find the method section dense, though the design rationale in the introduction and the negative results section are broadly readable.

Authors’ abstract

We describe our entry to the EgoLongQA track of the Wearable-AI Challenge in ECCV 2026, which placed first in the <=2B parameter division with 0.8279 on the held-out test set. Our system is a single 2B vision-language model that answers multiple-choice questions about ten-minute egocentric videos in one greedy forward pass; It is obtained by distilling the junior perception module of a tool-using agentic pipeline, not the agent itself into a small student, using teacher traces filtered to those that answered correctly. it reaches 89% of the accuracy of the large agentic pipeline using 1.1% of its parameters. This raises a 27.1% base model to 81.4% on our held-out questions. The 2B backbone has 2.2132B parameters and therefore over the divisional limit, to make the entry admissable we prune the multilingual embedding table from 248,320 to 143,469 rows, reaching 1.9985B with provably identical logits on retained rows.