VideoJev: System One Model for Video Understanding and Interaction

AI summary

Overview

Research area: Computer vision, specifically video understanding and multimodal large language models (MLLMs), with a focus on agentic decision-making.

Technical level: Advanced. The paper assumes familiarity with MLLM architectures, autoregressive decoding, temporal grounding metrics, mixture-of-experts models, and calibration measures.

Scope: The paper introduces VideoJev, a family of two models (a dense 4B and a mixture-of-experts 35B-A3B) that convert video understanding and interaction into structured candidate-scoring decisions, and evaluates them across three temporal grounding and four multiple-choice video QA benchmarks.

What This Paper Is About

Visual agents need to turn video observations into decisions — locating evidence, verifying whether a step occurred, or choosing between possible actions — sometimes repeatedly over the same video. Existing video MLLMs express these decisions by generating text token by token (timestamps or answer letters), which is semantically flexible but slow, while discriminative video models are fast but task-specific. VideoJev's goal is to combine the semantic flexibility of pretrained MLLMs with direct, non-autoregressive prediction by replacing answer generation with a shared head that scores temporal intervals and textual options against the same video evidence.

Key Contributions

1. A unified video System One model family. VideoJev casts temporal grounding and multiple-choice video QA as candidate-scoring tasks over a shared head, so both tasks use one language-conditioned decision interface instead of autoregressive generation.

2. Reusable visual computation across queries. The video prefix is encoded and prefilled once, then reused through two modes: Cached inference for queries arriving over time, and Joint inference for scoring multiple queries in a single causal forward pass. The same concatenated query layout is used during training.

3. Two model scales with controlled evaluation. VideoJev-4B (dense, from Qwen3.5-4B) and VideoJev-35B-A3B (MoE, from Qwen3.6-35B-A3B) are evaluated on seven benchmarks, with measured latency and a controlled autoregressive fine-tuning comparison using identical training data and settings.

4. Ablations isolating the source of gains. Experiments separate architectural benefit from data, test joint versus cached inference, test joint versus single-task training, and measure the value of Soft IoU supervision, final-MLP removal, and two-stage versus single-stage adaptation.

Main Findings

• Grounding improvements over backbones: VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of 2.8–3.1x. VideoJev-4B reaches 65.3 mean mIoU (6.7 points above Qwen3.5-4B and 3.3 points above TimeLens2-4B); VideoJev-35B-A3B reaches 67.7 mean mIoU.

• Precise localization under a strict criterion: Against TimeLens2-4B, VideoJev-4B raises R1@0.7 by 2.8–4.8 points across the three grounding datasets. Scaling from 4B to 35B-A3B adds 2.4 mean mIoU points with only a 22% increase in measured latency.

• Video QA improvements over Instruct backbones: Four-benchmark mean scores improve by 8.0 points (4B) and 5.7 points (35B-A3B) over their Instruct backbones. Gains extend to every QA benchmark. Per-benchmark figures for VideoJev-4B are LVBench 67.9, Video-MME 75.0, LongVideoBench 73.6, MLVU 80.6 (mean 74.3); for VideoJev-35B-A3B they are LVBench 73.4, Video-MME 81.8, LongVideoBench 77.1, MLVU 85.9 (mean 79.5).

• Competitive with explicit reasoning at far lower cost: VideoJev-4B is within 0.6 points of Qwen3.5-4B Thinking in mean QA score, while VideoJev-35B-A3B exceeds Qwen3.6-35B-A3B Thinking by 0.7 points. Measured speedups over these Thinking baselines are 129x and 100x. Both outperform their Thinking counterparts on LVBench and match or improve on LongVideoBench; Thinking retains an edge on Video-MME and MLVU.

• Architectural gain beyond data alone: Fine-tuned with an autoregressive Qwen3.5-4B under identical post-training data and single-stage setup, VideoJev-4B scores 1.3 points higher in mean grounding mIoU and 1.9 points higher in mean QA score.

• Joint inference slightly helps rather than harms quality: With the same checkpoint per size, Joint inference improves mean grounding/QA scores by 0.8/0.5 points at 4B and 0.3/0.7 points at 35B-A3B, improving in 13 of 14 model–benchmark comparisons, with only a 0.1-point decrease on QVHighlights at 35B-A3B.

• Joint training of grounding and QA is mutually beneficial: Joint training improves all seven benchmarks, raising mean grounding mIoU by 0.7 points and mean QA score by 1.3 points, with the largest QA gain on LongVideoBench (+1.9 points).

• Soft IoU supervision matters most for grounding: Adding Soft IoU raises mean grounding mIoU from 60.7 to 65.3 (+4.6 points) and mean QA score from 73.9 to 74.3 (+0.4 points), with improvements on all seven benchmarks.

• Median-free readout works fine: Removing the final MLP raises mean grounding and QA scores by 0.2 points each, so the simpler pre-MLP readout is adopted.

• Two-stage versus single-stage is near parity: Two-stage adaptation scores slightly higher on four of seven benchmarks; single-stage leads by only 0.1 point in mean grounding mIoU and 0.4 points in mean QA, mainly due to MLVU. Two-stage updates the backbone on only 70% of videos.

• Multi-query cost stays small: Joint inference adds only 55 ms for grounding and 31 ms for QA as query count grows from one to ten. At ten queries, forward times are 0.39 s and 1.64 s versus 0.85 s and 2.17 s under Cached inference, while independently measured Charades mIoU and LVBench accuracy improve by 1.5 and 0.8 points.

• Generative baselines scale worse: The autoregressive Direct QA baseline drops from 60.0% accuracy under Cached inference to 58.7% under Joint inference, below VideoJev's 67.9% at comparable latency (1.67 s versus 1.64 s). Qwen3.5-4B Instruct takes 5.98 s for ten grounding queries at 52.3 mIoU, versus VideoJev's 0.39 s and 60.1 mIoU; Qwen Thinking takes 1,393 s for ten QA queries at 66.4% accuracy.

• Confidence separates reliable from unreliable answers, but is miscalibrated: On LVBench, the 35B-A3B model reaches 95.8% accuracy on its 591 predictions with confidence at least 99%, compared with 73.4% overall. However, four-benchmark mean expected calibration error (ten equal-width bins) is 8.65 and 8.13 percentage points for 4B and 35B-A3B; VideoJev receives no calibration-specific training, and its probabilities tend to overestimate correctness.

Methodology in Plain English

The researchers start from an off-the-shelf video-capable MLLM. Instead of letting the model generate an answer word by word, they remove the final MLP and swap the language-model output head for a candidate scorer. Two new special tokens, <|option_end|> and <|decide|>, mark the end of each candidate and of the whole candidate list; both are initialized from the pretrained <|im_end|> embedding. Hidden states at these tokens are projected to 512-dimensional vectors, normalized to unit length, and compared by a dot product scaled by 10 with a learned bias.

The input is arranged as: video tokens first, then the task instruction, the language query, and a list of candidates. For temporal grounding, candidates are timestamp texts copied from the video prefix, each identifying a frame-pair interval; the backbone's twofold temporal compression defines one candidate interval per frame pair. Scores are trained with multi-label binary cross-entropy (does this interval belong to the event?) plus a duration-weighted Soft IoU loss (does the set of positive intervals cover the event?). At inference, a fixed threshold of 0.25 is subtracted from the logits and Kadane's algorithm finds the contiguous range with the largest sum, yielding a single event span in O(K) time.

For multiple-choice video QA, candidates are the answer texts, trained with softmax cross-entropy. To reduce position shortcuts, an answer-independent permutation is applied to eligible training questions (questions that refer to option positions or earlier turns keep their original order). At inference the highest-probability option is selected.

Training uses two stages. Stage one freezes the backbone and new marker embeddings and trains only the head on 30% of the videos, giving the scorer a stable feature space. Stage two adapts the language backbone, marker embeddings, and head on the remaining 70%. The visual encoder and merger stay frozen throughout. Training data is TimeLens-100K and TimeLens2-93K for temporal membership, plus an internally curated corpus of roughly 240K multiple-choice questions over 19K videos spanning gaming, sports, cooking, music and dance, outdoor activities, and crafts, with four or five options per question and videos from 4 minutes to nearly 2 hours. Queries from the same video are grouped around a shared prefix. Training ran on 64 NVIDIA H200 GPUs with BF16, a global batch size of 64 packed sequences, 2 fps video sampling, a maximum sequence length of 144K tokens, an 80% visual-token budget, AdamW with weight decay 0.01, and gradient clipping at 1.0; the two stages run for 160 and 375 optimizer updates, using DeepSpeed ZeRO-2 for 4B and ZeRO-3 for 35B-A3B.

Evaluation covers grounding on the TimeLens-Bench refined annotations of Charades-STA, ActivityNet Captions, and QVHighlights, and QA on LVBench test (1,549 questions), Video-MME test (2,700 questions), LongVideoBench validation (1,337 questions), and the seven multiple-choice tasks of MLVU development (2,174 questions), with no subtitles or audio. Latency was measured on the full Charades test set (1,313 videos; 3,363 queries) and the full LVBench test set (103 videos; 1,549 questions) on H200 GPUs at batch size 1.

Why This Matters

Impact on research: The paper extends the System One paradigm from images to video, where decisions must respect temporal structure and where the same expensive visual context is queried repeatedly. It shows that direct candidate scoring can match or exceed explicit reasoning on QA and substantially improve temporal grounding, and it offers two inference modes that make repeated judgments cheap. The controlled autoregressive comparison argues that the benefit comes from the decision architecture, not just from the training data. It also documents a real weakness — miscalibrated confidence — that future work must address if these decisions are to be trusted in agent workflows.

Real-world applications:

• Procedural monitoring: checking whether an ingredient has been added, or a step completed, before choosing the next instruction in cooking or assembly guidance.
• Interactive assistants: answering many follow-up questions about the same video stream without re-encoding the video for each question.
• Embodied agents and computer-use systems: evaluating candidate actions or states against visual evidence before acting.
• Long-video retrieval and verification: locating a described event in hours-long footage and confirming whether a stated condition holds.

Industry relevance: The efficiency profile is the headline for deployment. Grounding speedups of 2.8–3.1x over the backbones and QA speedups of 129x and 100x over Thinking-mode baselines, achieved while remaining competitive in accuracy, matter for any product that must respond in a timely way during an ongoing video interaction. The 256K-token context budget and 2,048-frame cap at 2 fps define a concrete operating envelope for long-video services. The paper notes that VideoJev's probabilities overestimate correctness, so production systems would currently need calibration or escalation policies before accepting its decisions automatically.

Future Directions

• Extending the candidate interface to more tasks. The authors propose action recognition, event detection, scene-state classification, and visual condition verification as further decision tasks that could use the same scoring interface, reducing reliance on task-specific models and prediction heads.

• Open-ended agent workflows. A higher-level System Two component could decompose open-ended goals into visual checks, propose candidate events, actions, or states for VideoJev to evaluate, and revise candidates when observations are ambiguous or unexpected. The authors frame this as "System Two organizes the decision space, while System One evaluates it against visual evidence."

• Streaming and memory management. Streaming inputs will require incremental visual encoding, efficient memory management, and timely updates to the decision context — the paper's Cached inference is a step in this direction but is not a full streaming solution.

• Calibration, uncertainty, and selective prediction. Since predicted probabilities tend to overestimate correctness (four-benchmark mean ECE of 8.65 and 8.13 percentage points), the authors suggest calibration-aware training, uncertainty estimation, and selective prediction to decide when to accept a System One decision or defer to higher-level reasoning. Temperature scaling on an independent calibration set is named as one possible remedy.

Target Audience

Researchers and practitioners working on video-language models, multimodal agents, and efficient inference. It is most useful for readers already comfortable with MLLM internals (hidden states, output heads, causal attention, MoE routing) and with temporal grounding metrics such as mIoU and R1@IoU thresholds. Engineers building real-time or interactive video systems will find the latency measurements and the Cached/Joint inference distinction directly relevant, while those focused on agent architectures will find the System One framing and the proposed System Two/System One division of labor the most transferable ideas.

Authors’ abstract

Visual agents must turn video observations into timely decisions to track progress and choose what to do next. Building on the emerging System One paradigm, we introduce VideoJev, a family of models that cast video understanding and interaction as structured decision tasks. We adapt pretrained multimodal large language models (MLLMs) by replacing autoregressive answer decoding with a shared head that scores temporal intervals and textual options against video evidence. These scores yield event spans or answer choices, unifying temporal grounding and multiple-choice video question answering. This combines the semantic flexibility of MLLMs with efficient decisions that agents can use directly. As an interaction unfolds, new goals and intermediate results prompt further queries about the same visual history. VideoJev therefore reuses visual computation through Cached inference for queries as they arrive and Joint inference for multiple queries in a single forward pass. Across three temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of $2.8$-$3.1\times$. On four video QA benchmarks, both achieve performance comparable to their backbones in Thinking mode, with measured inference speedups exceeding $100\times$. These results offer a unified, reusable decision interface for composing structured visual judgments into complex, open-ended workflows.