VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

AI summary

Overview

Research area: Computer vision and multimodal retrieval — specifically two-stage video–text search, where a fast first-stage retriever proposes candidates and a heavier second-stage model reranks them.

Technical level: Intermediate. The paper assumes familiarity with dual encoders, cross-encoders, and contrastive/InfoNCE training, but the core idea (a small reusable per-video cache plus a lightweight joint scorer) is explainable without deep background.

Scope in one sentence: VEDJE is a cached joint reranker for video–text retrieval that compresses features within each sampled frame, keeps the frames' token groups separate in a small on-disk cache, and trains that cache with a feature-change prediction objective so a 33M-parameter scorer can rerank candidates without ever running a video encoder at query time.

What This Paper Is About

Two-stage video search is limited by the verification step: video–text dual encoders compare a precomputed video vector against a query without any token-level interaction, so they struggle to separate videos with similar scenes but different actions or moments. Joint rerankers fix this but are expensive for video, because stored features and joint scoring sequences both grow with the number of sampled frames, and re-encoding candidates at query time is slow (multimodal-LLM rerankers put multi-billion-parameter decoders directly on the query path).

VEDJE's goal is to make joint video reranking deployable by building the video cache offline, before queries are known, keeping it small enough to store at index scale, and still giving a lightweight joint encoder enough evidence to rerank accurately.

Key Contributions

1. Compact frame-indexed video caches. Building on the image-caching recipe of Edje, VEDJE compresses features within each sampled frame into a small bank of learned tokens, but concatenates those banks in temporal order rather than pooling them, so the joint scorer sees evidence from across the sequence under a fixed storage budget.

2. Feature-change supervision for compression. A training-only predictor learns to estimate differences between frozen features at different sampled times (future-delta prediction). The predictor is discarded after training, so the auxiliary signal improves retrieval from a tight cache without adding any query-time computation.

3. Accuracy and efficiency evaluation across four benchmarks. VEDJE improves R@1 in both retrieval directions over matched first-stage retrievers on MSR-VTT, MSVD, DiDeMo, and ActivityNet, including over a fine-tuned VideoCLIP-XL first stage, with complete-query latency measured end to end.

Main Findings

• Gains over every matched first stage on MSR-VTT. VideoPrism: 50.1 → 54.6 T2V R@1 (+4.5) and 49.8 → 54.6 V2T (+4.8). PE-Core-B: 47.6 → 53.9 (+6.3) and 47.3 → 53.9 (+6.6). Zero-shot VideoCLIP-XL: 50.1 → 56.5 (+6.4) and 49.9 → 57.1 (+7.2).

• Gains survive first-stage fine-tuning. With a fine-tuned VideoCLIP-XL first stage, reranking raises T2V R@1 from 56.2 to 59.8 and V2T R@1 from 55.1 to 58.8; Figure 1 reports this at roughly 157M total online parameters (33M cached reranker included).

• Fourfold cache reduction costs little T2V recall. With feature-change supervision, shrinking the cache from 64 tokens / 48 KiB to 16 tokens / 12 KiB per video moves T2V R@1 from 54.6 to 54.4 (within 0.2 points) while V2T falls from 54.6 to 53.1.

• The 48 KiB cache is 128.5× smaller than the full frame-and-patch representation of the same VideoPrism features.

• Feature-change supervision helps most when the cache is tightest. T2V gain from the delta objective rises from 0.6 points at 64 tokens to 1.9 points at 16 tokens; V2T shows the larger gain at 64 tokens (+1.0 versus +0.5 at 16 tokens).

• Delta targets beat reconstruction. At 64 tokens, current-frame reconstruction reaches 54.2 T2V / 53.9 V2T R@1, versus 54.6 in both directions for the feature-change target.

• Keeping frames separate beats pooling at equal storage. With a 64-token cache, mean pooling gives 51.7 / 51.6 R@1, attention pooling 52.0 / 51.6, and separate frame groups 54.0 / 53.6 before auxiliary training and 54.6 / 54.6 with it — a 2.0-point improvement over attention pooling before auxiliary training, in both directions.

• Joint scoring matters more than the scoring rule alone. On the same cache and candidates, trained MaxSim with the residual prior reaches 51.9 T2V R@1 versus 54.6 for the joint encoder. A prior-only model reaches 50.1 and a joint-only model 43.8, versus 54.6 for the full model.

• A frozen CLIP backbone is competitive with fine-tuned baselines. VEDJE-CLIP reaches 49.8 T2V / 49.6 V2T R@1 on CLIP ViT-B/16, versus EERCF's 49.9 / 47.8 (fine-tuned) — comparable T2V and 1.8 points higher V2T with the visual backbone frozen. Using EERCF's released ViT-B/32 candidate generator with 50 candidates, VEDJE lifts 42.8 to 49.1 T2V R@1.

• Complete-query cost is close to first-stage-only at small candidate counts. On one NVIDIA L40S with batch size 1, 20 candidates take 5.79 ms (64 tokens) and 5.82 ms (16 tokens), versus roughly 4.3 ms for first-stage retrieval alone; the gap widens with candidates (at 100: 7.83 ms versus 6.33 ms).

• Cache residency affects latency. At 100 candidates the 64-token model needs 7.8 ms with warm NVMe reads and 11.1 ms with cold reads; at 20 candidates the 64- and 16-token caches contribute 0.983 MB and 0.246 MB of video payload respectively.

Methodology in Plain English

The design splits the work into three stages.

Offline, once per video. A frozen visual encoder (VideoPrism-Base in the default configuration, at 288×288 with 16 sampled frames) produces patch features for each sampled frame. A small shared two-layer transformer decoder uses a handful of learned "cache queries" to attend to each frame's patches and write out a few tokens per frame — four by default, for 64 tokens total. Those per-frame token groups are stored side by side in temporal order, so the cache is a stack of small evidence blocks, one per sampled time, rather than one collapsed clip vector or a full patch grid.

Training the cache to be informative. Because the cache is written before anyone knows what will be asked, it has to preserve whatever later queries might reference. The authors add a training-only predictor that, from a frame's compressed tokens, tries to predict how the frozen patch features change at a future sampled time. The target is the difference between two feature slices, which cancels whatever is identical at both times and focuses the objective on change. This loss is added to the standard matching, contrastive, and masked-language objectives at equal weight. The predictor is thrown away afterwards, so query time is unaffected.

Online, per query. The first-stage retriever returns candidates and a coarse score. A compact 12-layer MiniLM model (~33M parameters) reads the query tokens and cached video tokens together in one sequence with learned positional embeddings, producing a pairwise representation. The scalar first-stage score is embedded and added as a residual prior in representation space, and a small nonlinear head turns the result into the final score. Crucially, no visual encoder runs at query time, and only the retrieved candidates' caches are read.

Why This Matters

Impact on research. The paper reframes video reranking as a cache-design problem rather than a query-time compute problem, and provides matched-budget controls showing that how cached evidence is organized (separate frame groups versus pooling) matters as much as how much is stored. It also supplies a cheap, annotation-free auxiliary objective for training compressive caches, and an explicit storage accounting formula (2·T·M·d bytes per video) that makes cache budgets comparable across systems.

Real-world applications:

• Large-scale video search over archives or user-generated libraries, where per-video storage and per-query latency both bind.
• Moment- and action-sensitive search (e.g. finding the clip where a specific state change occurs, not just a similar-looking scene).
• Video-to-text retrieval for captioning, subtitling, or metadata workflows over an already-indexed corpus.
• Retrieval-augmented pipelines where a first-stage index exists and only a cheap reranker can be afforded on the query path.

Industry relevance. A 12 KiB per-video cache that is written once and reused is operationally attractive: reranker updates do not require re-encoding the corpus (only a backbone or compressor change does), and the measured 5.79 ms complete-query cost at 20 candidates on a single L40S is in the range of interactive deployment. Keeping multi-billion-parameter decoders off the query path is the central selling point relative to MLLM rerankers.

Future Directions

• Query-aware cache access. The cache is query-blind: it is written once and serves every later query about that video. Selectively reading or expanding parts of it at query time is proposed as a natural extension.
• Removing the re-indexing constraint. The cache depends on a fixed visual backbone and compressor, so updating either requires re-indexing the entire corpus.
• Jointly training the two stages. First and second stages are currently trained separately; closing that loop is flagged as future work.
• Beyond English web video. The four benchmarks use English captions and web-sourced videos; the paper states these evaluations do not establish performance on multilingual video retrieval. The appendix additionally examines target formulation, prediction offset (horizons 2, 3, 4), loss-stack contributions, and a reranker-backbone control.

Target Audience

Researchers and engineers working on multimodal retrieval, video search infrastructure, or efficient inference — particularly those who already have a dual-encoder first stage and are deciding whether joint reranking is affordable. It is also useful for readers interested in compression objectives for reusable feature caches, and for practitioners who need concrete storage, latency, and recall trade-off tables rather than a new state-of-the-art claim.

Authors’ abstract

Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.