Adaptive Evidential Learning for Temporal-Semantic Robustness in Moment Retrieval

AI summary

Overview

Research area: Computer Vision / video-language understanding, specifically moment retrieval (MR) — locating the temporal segment in an untrimmed video that matches a natural language query — with a focus on evidential (uncertainty-aware) learning.

Technical level: Advanced. The paper assumes familiarity with Deep Evidential Regression (DER), Normal-Inverse-Gamma (NIG) distributions, DETR-style set prediction, and cross-attention architectures.

Scope (one sentence): The paper introduces DEMR (Debiased Evidential Learning for Moment Retrieval), a framework that adds a Reflective Flipped Fusion block, a query reconstruction task, and a Geom-regularizer to an evidential regression baseline in order to fix modality imbalance and counterintuitive uncertainty estimation in moment retrieval.

What This Paper Is About

Moment retrieval models must find the start and end times of the video clip that matches a text query, but most existing methods are deterministic — they produce a single answer with no way to express doubt — and they struggle when the query is ambiguous or the relevant frames are hard to align. The authors first build an uncertainty-aware baseline by plugging Deep Evidential Regression into moment retrieval, then discover that this baseline has two problems: it over-relies on visual features and under-uses text, and its uncertainty regularizer assigns high uncertainty to accurate predictions instead of hard ones. The goal of DEMR is to correct both biases so that the model's uncertainty actually tracks how difficult a moment is, improving retrieval accuracy and robustness.

Key Contributions

1. The authors introduce an uncertainty-aware moment retrieval baseline built on Deep Evidential Regression, then develop DEMR on top of it to target challenging and ambiguous moments.
2. They propose a Reflective Flipped Fusion (RFF) block with dual branches that alternate the roles of video and text features as queries and keys/values in a shared cross-attention module, plus an auxiliary query reconstruction (QR) task that masks query nouns and reconstructs them to strengthen text sensitivity.
3. They introduce a Geom-regularizer that replaces the heuristic vanilla DER regularizer, enforcing the principle that accurate predictions should carry high evidence and inaccurate predictions low evidence, and they show its gradient depends on both error and evidence rather than error alone.
4. They evaluate on Charades-STA, QVHighlights, TACoS, and the debiased ActivityNet-CD and Charades-CD benchmarks, reporting gains in effectiveness, robustness, and interpretability; code is released at https://github.com/KaijingOfficial/DEMR.

Main Findings

• Modality imbalance is measurable and reducible. On the QVHighlights validation split, the baseline showed high uncertainty sensitivity to visual noise (Var_vis = 9.17) and low sensitivity to textual noise (Var_text = 0.85), a gap of Δ_Var = 8.32. Adding the RFF block moved this to 8.63 / 1.60 / 7.03, adding QR to 4.89 / 3.91 / 0.98, and the full model to 4.85 / 5.54 / 0.69.

• Each component improves retrieval. In the same ablation, R1@0.5 rose from 61.1 (baseline) to 62.4 (+ RFF block), 63.8 (+ QR), and 65.0 (full model).

• The Geom-regularizer calibrates uncertainty. Scatter plots of uncertainty against normalized error show that without DER the model has unreliable inference, NLL-only training is overconfident (low uncertainty at all error levels), and the vanilla regularizer is counterintuitive (higher uncertainty at lower error). The Geom-regularizer instead produces higher uncertainty at higher error.

• Main benchmark results (Table 1). DEMR reaches 65.0 R1@0.5, 49.4 R1@0.7, and 43.0 mAP on QVHighlights; 37.3 R1@0.5, 19.4 R1@0.7, and 33.9 mIoU on TACoS; and 60.2 R1@0.5, 38.0 R1@0.7, and 51.6 mIoU on Charades-STA. For comparison, CG-DETR reports 67.4 / 52.1 / 42.9 on QVHighlights, 39.5 / 23.4 / 37.4 on TACoS, and 58.4 / 36.3 / 50.1 on Charades-STA.

• Competitive with MLLM-based methods. On the QVHighlights test set with a BLIP-2 backbone, DEMR reports 76.36 R1@0.5, 62.91 R1@0.7, 56.82 mAP@0.75, and 52.32 mAP, versus Mr. BLIP at 74.77 / 60.51 / 53.38 / 51.37 and LLaVA-MR at 76.59 / 61.48 / 54.40 (mAP not reported).

• Mask ratio matters for the QR task. Masking a single noun performed best (MR-full-mAP-key 36.93, R1@0.5 64.06, R1@0.7 43.61), outperforming 0% masking (32.75 / 57.23 / 37.11), 25% (33.40 / 61.74 / 39.17), 50% (32.43 / 59.55 / 37.13), 75% (31.80 / 59.23 / 34.26), and masking all nouns (31.71 / 58.52 / 34.45).

• Hyperparameter sensitivity. mAP peaks at λ_geom = 10⁻², which the authors adopt; performance degrades when λ_der exceeds 1×10⁻², so they set λ_der = 1×10⁻³. Increasing QR epochs from 0 to 50 boosts mAP by +1.5%, with the optimal QR learning rate reported as 1×10⁻⁴.

• Temporal bias robustness. On Charades-CD, DEMR's i.i.d. scores are 71.10 / 62.20 / 43.29 (R1@0.3 / 0.5 / 0.7) and o.o.d. scores are 67.81 / 52.46 / 30.97, a gap of 3.29 / 9.74 / 12.32. The paper states this narrows the IID–OOD gap to 3.29% versus 12.00% for CM-NAT on R1@0.3, and that DEMR surpasses MomentDiff and MomentDETR by 3.37% and 10.47% respectively on Charades-CD R1@0.3.

• Uncertainty rises in out-of-distribution temporal regions. Compared with no DER constraints, NLL-only (overconfident), and vanilla regularization (no OOD sensitivity), the Geom-regularizer substantially increases epistemic uncertainty in temporal OOD regions of QVHighlights.

Methodology in Plain English

Step 1 — an evidential baseline. The authors take a standard moment retrieval setup and attach a Deep Evidential Regression head. Instead of predicting one boundary time per clip, the head predicts the parameters of a Normal-Inverse-Gamma distribution. From those parameters the model derives a point prediction, an aleatoric uncertainty (irreducible noise in the data, β/(α−1)), and an epistemic uncertainty (the model's own lack of knowledge, β/(υ(α−1))). Training combines a negative log-likelihood loss with DER's heuristic regularizer, which multiplies the prediction error Δ by the evidence Φ.

Step 2 — fixing modality imbalance. Two additions address the baseline's over-reliance on vision. The Reflective Flipped Fusion block runs video and text through a shared cross-attention module in alternating directions, then refines each branch with self-attention, repeated across four layers. The query reconstruction task masks one noun in the query during the alignment stage and asks the model to recover it using both the remaining query tokens and the video; the QR head is then frozen and its loss removed for the rest of training and inference.

Step 3 — fixing counterintuitive uncertainty. The authors inspect the gradient of the vanilla regularizer with respect to evidence, finding it equals −Δ — it depends only on error, so the model has no signal telling it when evidence has been suppressed enough. Because low-error samples dominate batches near convergence, their evidence gets over-suppressed while high-error samples are under-penalized. The Geom-regularizer normalizes both error and evidence and pushes the pair (Δ̄, Φ̄) toward the line Φ̄ + Δ̄ = 1, giving a gradient of −2(Δ̄ + Φ̄ − 1) that depends on both quantities.

Step 4 — training and evaluation. Training is two-stage: QR alignment (spaCy extracts nouns; the paper reports 30 epochs at a 1e-5 learning rate), then DEMR boundary prediction with gradients from the line regularizer detached. Backbones are frozen CLIP ViT-B/32 and SlowFast ResNet-50, NMS with threshold 0.7 is applied at evaluation, and experiments run on four Tesla V100 GPUs.

Why This Matters

The paper's central claim is that moment retrieval models should know when they are likely wrong, not just produce a boundary. It contributes the first extension of evidential regression to moment retrieval that the authors are aware of, and it diagnoses a structural flaw in DER's regularizer that is relevant beyond this task — the same heuristic is used in other regression domains cited in the paper.

The uncertainty calibration results also connect to dataset bias research: the paper visualizes the joint distribution of ground-truth start and end timestamps in QVHighlights and shows that underrepresented temporal regions correspond to higher epistemic uncertainty under DEMR. This suggests uncertainty estimates could be used as a diagnostic for data bias rather than only as an inference-time signal.

Real-world applications implied by the setting:

• Video search and editing tools, where a user types a description and the system returns a clip and can flag low-confidence results for review.
• Video surveillance or compliance review, where an operator searches for a described event and needs to know when the system's answer should be double-checked.
• Media asset management and archival retrieval, where sparse or ambiguous metadata means queries are often vague.
• Content moderation, where uncertain matches could be routed to a human rather than acted on automatically.

Industry relevance: the method is designed to be attached to frozen pretrained backbones and evaluated with standard metrics, which lowers the cost of adoption relative to training a multimodal large language model from scratch. The paper explicitly frames uncertainty-aware modeling as a step toward more trustworthy AI decisions in video understanding, and notes the limitation that current performance is constrained by data quality and scale.

Future Directions

• Integration with MLLM backbones. The authors state that future work will integrate DEMR with advanced multimodal large language models to expand its application and improve reliability in video tasks — something they expect could yield substantial improvement given that DEMR is already competitive without dense clip-word guidance or dense frame sampling.
• Scaling and data quality. The conclusion notes that current performance is constrained by data quality and scale, raising the question of how DEMR behaves on larger, more carefully annotated video corpora.
• Extending the regularizer analysis. The paper shows the vanilla DER regularizer's gradient depends only on error, and proposes the Geom-regularizer as a fix; whether a similar line-based constraint helps other regression tasks that use DER is left open.
• Choosing the QR mask ratio automatically. The mask ratio study shows a sharp difference between masking one noun and masking more, including all nouns; how to set or learn the best mask ratio per query is not resolved.

Target Audience

This paper is best suited to computer vision and video-language researchers who already work on temporal grounding or moment retrieval and want to add uncertainty quantification to an existing model, and to machine learning researchers interested in the failure modes of Deep Evidential Regression's regularizer. Practitioners building video search or review systems who need calibrated confidence signals will also find the ablation and calibration analyses useful. Readers without a background in evidential learning or attention-based detection architectures will need to consult the cited DER and DETR references first.

Note: the truncated content provided does not report specific dataset sizes or per-dataset training-set counts; those are stated to be in the supplementary material.

Authors’ abstract

In the domain of moment retrieval, accurately identifying temporal segments within videos based on natural language queries remains challenging. Traditional methods often employ pre-trained models that struggle with fine-grained information and deterministic reasoning, leading to difficulties in aligning with complex or ambiguous moments. To overcome these limitations, we explore Deep Evidential Regression (DER) to construct a vanilla Evidential baseline. However, this approach encounters two major issues: the inability to effectively handle modality imbalance and the structural differences in DER's heuristic uncertainty regularizer, which adversely affect uncertainty estimation. This misalignment results in high uncertainty being incorrectly associated with accurate samples rather than challenging ones. Our observations indicate that existing methods lack the adaptability required for complex video scenarios. In response, we propose Debiased Evidential Learning for Moment Retrieval (DEMR), a novel framework that incorporates a Reflective Flipped Fusion (RFF) block for cross-modal alignment and a query reconstruction task to enhance text sensitivity, thereby reducing bias in uncertainty estimation. Additionally, we introduce a Geom-regularizer to refine uncertainty predictions, enabling adaptive alignment with difficult moments and improving retrieval accuracy. Extensive testing on standard datasets and debiased datasets ActivityNet-CD and Charades-CD demonstrates significant enhancements in effectiveness, robustness, and interpretability, positioning our approach as a promising solution for temporal-semantic robustness in moment retrieval. The code is publicly available at https://github.com/KaijingOfficial/DEMR.