Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

AI summary

Overview

Research area: Agentic video understanding, automated agent design, and code-level self-improvement of language-model agents.

Technical level: Advanced. The paper assumes familiarity with multimodal LLM agents, tool-use harnesses, evolution/search-over-programs methods, and video QA benchmarks.

Scope: One sentence: Video-RSI lets a frozen-language-model video agent rewrite its own executable "harness" — the code controlling what it observes and how it uses observations — by revisiting training videos to diagnose failures and accepting revisions that improve answer accuracy relative to visual frame cost.

What This Paper Is About

Video understanding agents decide where to look in a long video and which evidence to trust, and those decisions are made by an executable harness rather than by the model's weights. The problem the authors identify is that an execution trace only records what the current harness happened to observe, so a failure could come from a sampling gap, a misperception, or a failure to use evidence already collected — three explanations that imply very different code fixes, with nothing in the trace to tell them apart. The goal is to make the agent's own language model serve as an "Editor" that gathers new evidence from the original training videos, diagnoses the failure, revises the harness code, and keeps only revisions that improve accuracy without unbounded increases in visual cost.

Key Contributions

1. Video-RSI framework. A framework for video harness self-improvement in which the same frozen language model acts as both Solver of video questions and Editor of the executable harness, with model weights never updated.

2. An improvement loop linking investigation to revision and selection. The loop connects Active Video Investigation (revisiting training videos to test competing failure explanations) to diagnosis-guided code revision, and then to a cost-aware gate that decides which candidates persist.

3. Evaluation across four video understanding benchmarks plus dynamics analysis. The authors report accuracy and frame-usage results on MLVU test, LongVideoBench long val, Video-MME Long, and EgoSchema, analyze investigation and evolution dynamics over 20 revision attempts, and compare active investigation against trace-only revision.

4. Evidence that investigation beats trace-only revision and that cost-aware selection beats accuracy-only selection, as shown in the ablation table and in the retained-versus-rejected revision counts.

Main Findings

• Accuracy improves while frame usage drops relative to the initial harness. The final evolved harness improves accuracy while using 10.8–44.1% fewer frames than the initial harness H_S0 across the four benchmarks, according to the analysis of evolved harnesses.

• Video-RSI beats both reproduced baselines in accuracy on all four benchmarks. With the harness evolved on LVBench and frozen for evaluation, Video-RSI reports MLVU 72.9 accuracy at 61.1 frames, LongVideoBench 71.1 at 41.1, Video-MME 80.0 at 21.6, and EgoSchema 78.2 at 42.4. The reproduced comparisons are VideoSeek (MLVU 68.1 / 83.9, LongVideoBench 70.9 / 71.9, Video-MME 78.7 / 17.8, EgoSchema 73.2 / 72.3) and VideoHarness-RSI (MLVU 64.1 / 83.2, LongVideoBench 68.4 / 92.1, Video-MME 69.7 / 92.8, EgoSchema 78.0 / 66.8).

• The accuracy–cost trade-off varies by benchmark rather than being a uniform saving. Against VideoSeek, Video-RSI gains 4.8 percentage points on MLVU while using 27.2% fewer frames; on LongVideoBench the main benefit is substantially fewer frames at similar accuracy; on EgoSchema both accuracy and frame usage improve; on Video-MME higher accuracy comes with a moderate increase in frame usage. Video-RSI uses lower frame usage than VideoSeek in all comparisons except Video-MME.

• Active investigation outperforms trace-only revision. The trajectory-only variant, restricted to existing execution traces, scores MLVU 64.7 at 60.3 frames, LongVideoBench 67.2 at 43.1, Video-MME 73.7 at 28.5, and EgoSchema 76.6 at 51.8. Active investigation yields higher final accuracy on every benchmark; on MLVU the gain reaches 8.2 percentage points at similar inference-time frame usage, and on the other three benchmarks higher accuracy accompanies fewer frames. The investigation run retained five of twenty revisions versus three for trajectory-only revision.

• Cost-aware selection outperforms an accuracy-only gate. The accuracy-only gate reaches MLVU 67.1 at 81.9 frames, LongVideoBench 69.5 at 66.4, Video-MME 74.9 at 43.1, and EgoSchema 77.4 at 60.5 — improving accuracy over the initial harness on all four benchmarks but increasing frame usage on three. Video-RSI beats it in accuracy while using fewer frames everywhere, and on Video-MME uses roughly half as many frames.

• Evolution dynamics are non-monotonic and split between early efficiency and later accuracy. On the development set, the first accepted revision produces most of the net frame-usage reduction, while later revisions steadily improve accuracy with smaller, nonmonotonic cost changes. On MLVU, the first accepted revision lowers accuracy despite improving both development metrics, accuracy recovers with the next revision, and the final two accepted revisions account for most of the net MLVU accuracy gain. From H_S2 onward, MLVU accuracy rises together with frame usage, yet the final harness remains more accurate and uses fewer frames than H_S0.

• Retained revisions fall into three patterns of change. Structured perception appears in H_S2 (goal-directed observations with structured findings, image references, and explicit uncertainty, with image references mapped to frame timestamps); evidence reuse through relation graphs appears in H_S4 (extracting entities, events, and relations from accumulated visual, transcript, and OCR observations, and returning evidence gaps for eligible relational or temporal questions, with graph queries processing no new frames); and selective observation routines appear in H_S3 (a Global-64 route expanding the initial 32-frame overview for selected whole-video questions) and H_S5 (an occurrence tool that defines counting units, verifies candidate events, and consolidates overlapping evidence to avoid duplicate counts).

Methodology in Plain English

The researchers keep every model frozen and treat the agent's surrounding program — the harness — as the thing to improve. The harness covers prompt construction, tool implementation, observation processing, memory, and execution control.

1. Run the current harness on training questions and collect the answer, execution trace, and resource record.
2. Investigate failures actively. Rather than trusting the trace alone, the Editor picks representative failures and issues diagnostic queries that vary the temporal region, sampling density, modality, or perceptual question — for instance, sampling more densely to test whether a short event fell between sampled frames. It can consult video observations, speech transcripts, on-screen text, and reruns of the unchanged incumbent, and perception requests omit the gold answer. Evidence that contradicts an explanation redirects the next query. The harness itself is not modified during this phase.
3. Turn the diagnosis into a code revision. The Editor specifies when the change applies and which successful behaviors to preserve, and may debug the candidate through training-side trials distinct from the investigation.
4. Apply a cost-aware gate. The candidate and incumbent are compared on the same private, video-disjoint selection set under matched conditions. The gate accepts a candidate if accuracy rises and mean frame count stays within (1 + α) of the incumbent, or if accuracy drops by at most ε while frame count falls to at most (1 − β) of the incumbent. With α = 0.1, β = 0.2, and ε = 1/|D_g|, at most one fewer correct answer is allowed in the cost-reduction branch. The Editor receives only aggregate accuracy and frame counts; per-example gate outcomes stay private.
5. Iterate. Accepted revisions become the incumbent for the next attempt, and the final accepted harness is frozen for held-out evaluation.

Setup details: evolution used 216 LVBench questions for investigation and revision plus a fixed 72-question, video-disjoint development set for selection; 20 revision attempts were run. The Solver and Editor use DeepSeek-V4-Pro, visual observations use Qwen3.6-Plus, and the initial harness H_S0 supports global and local video inspection, transcripts, OCR, and observation memory. Frame usage counts all frames processed by successful visual-model calls across each complete execution, averaged over questions, with offline Editor observations excluded. Baselines were split between reproduced implementations (VideoSeek and VideoHarness-RSI, both run with DeepSeek-V4-Pro) and results taken from original publications.

Why This Matters

Impact on research. The work argues that improving the executable layer around a frozen model is a viable route to better video understanding, without training or fine-tuning. It sharpens a distinction that prior self-improvement work leaves blurred: the evidence an Editor has when proposing a change. By showing that trace-only revision underperforms active re-investigation of the source video, and that an accuracy-only acceptance rule loses efficiency, it separates two design questions — what to change, and which changes to keep — and supplies concrete mechanisms for each.

Real-world applications.

• Long-video question answering and video search assistants, where relevant moments are brief, temporally distant, or surrounded by visually similar distractors.
• Surveillance and monitoring review, where reducing frames processed per query lowers the cost of scanning hours of footage.
• Accessibility and media indexing over lecture recordings, meeting archives, or broadcast footage with subtitles, OCR, and transcripts already available.
• Agentic pipelines that must answer over large video corpora under a fixed inference budget, where the accuracy–cost gate provides an explicit operating point.

Industry relevance. The approach improves deployed agent behavior while holding model weights and model endpoints fixed, which fits settings where a vendor serves frozen models and developers control only the orchestration code. The cost-aware gate makes the trade between answer quality and visual observation a configured parameter (α, β, ε) rather than an afterthought. The reported savings are framed explicitly as visual frames saved rather than total model computation, so adopters should read them as a vision-processing cost signal rather than a full compute-bill estimate.

Future Directions

• Transferability of evolved harnesses. The harness is evolved on LVBench questions and frozen before evaluation on MLVU, LongVideoBench, Video-MME, and EgoSchema. Whether a harness evolved on one corpus transfers to substantially different video domains is not reported.
• Cost accounting beyond frames. The paper measures visual frames rather than total model computation. Extending the gate to tokens, latency, or financial cost is an open question the analysis of evolved harnesses explicitly leaves unaddressed.
• Why development gains do not transfer step by step. The MLVU trajectory shows the first accepted revision improving both development metrics while lowering MLVU accuracy. Understanding and predicting such mismatches between the private selection set and the evaluated benchmark remains unresolved.
• Scaling the evolution budget and the Editor's evidence. All results use 20 revision attempts and retained only a handful of them (five of twenty under active investigation). How accuracy and cost behave at larger attempt budgets, and what richer investigation interfaces would add, are not reported.

Target Audience

Researchers and engineers working on multimodal LLM agents, long-video question answering, and automated agent or program search. It is most useful to readers already comfortable with tool-using video agents and benchmark reporting on Video-MME, LongVideoBench, EgoSchema, and MLVU, and to practitioners who control the orchestration code around a frozen model and want a concrete recipe for improving accuracy per frame of video processed. Readers looking for model-training or fine-tuning methods will find this paper explicitly outside that line of work.

Authors’ abstract

Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution. Code is available at https://github.com/bingjunluo/Video-RSI .