Conformal Coverage Guarantees for Any Video Temporal Grounder

AI summary

Overview

• Research area: Computer vision — video temporal grounding (locating the moment in a video that answers a language query) combined with conformal prediction and distribution-free uncertainty quantification.
• Technical level: Advanced. The paper proves coverage, efficiency, and conditional-coverage results, and assumes familiarity with split conformal prediction, exchangeability, quantile calibration, and risk control.
• Scope in one sentence: The paper introduces Cover, a post-hoc, model-agnostic wrapper that turns any video temporal grounder — a trained localizer or a black-box video–language model — into one that outputs a temporal region containing the true moment with probability at least 1 − α, and it validates this across three benchmarks and five grounders.

What This Paper Is About

Event boundaries in continuous video are ambiguous: when the same query–video pair is re-annotated, independent annotators mark moments that overlap by less than half on a large fraction of samples. The ground truth for temporal grounding is therefore a distribution over intervals, yet every grounder returns a single interval with no statement of reliability, so at deployment a wrong interval is indistinguishable from a right one. The paper's goal is to change the output object: instead of a point interval, a grounder should emit a temporal region carrying a coverage guarantee, and that guarantee should be obtainable without retraining the grounder or accessing its internals.

Key Contributions

1. A region, not a point, for a task whose ground truth is a distribution. The first distribution-free uncertainty quantification for video temporal grounding: a post-hoc wrapper that attaches a finite-sample coverage guarantee to any grounder, trained or black-box, by calibrating a single scalar on held-out labels, with no retraining and no white-box access.

2. Temporal nonconformity scores for both kinds of grounder. A two-sided boundary-widening score for grounders that emit an interval, and a super-level-set score for grounders that expose a relevance signal, the latter returning regions that spend temporal budget only where the grounder places relevance. A two-parameter per-boundary variant calibrates each side separately where boundary error is asymmetric.

3. Theory about what the region costs, not only that it covers. An efficiency identity with asymptotic length-optimality of the calibrated widening, length-conditional coverage of the normalized score under a scale model, and an explicit coverage bound under the within-video dependence that breaks exchangeability. Standard coverage, tightness, and risk-control results appear in the Appendix.

4. Certification as measurement. Across three benchmarks and five grounders, realized coverage tracks the target, and calibration exposes what point metrics hide: an evidential grounder built for uncertainty quantification realizes only 0.660 against its own nominal 0.80 target while Cover restores a valid guarantee on the identical predictions, a VLM's event onsets are essentially free of error while all of its boundary error sits on offsets, and hand-picked margins swing realized coverage by up to 0.904.

Main Findings

• Coverage validity holds for most dataset–grounder pairs. Realized coverage tracks the target closely over the grid 1 − α ∈ {0.50, 0.60, 0.70, 0.80, 0.90, 0.95}. ActivityNet-Captions realizes 0.895 ± 0.003 (n_cal = 7811) and Charades-STA 0.900 ± 0.011 (n_cal = 1500) at the 0.90 target.

• One systematic exception. QD-DETR on QVHighlights runs below target at 0.879 ± 0.016, under the k/(n+1) floor the Appendix Proposition 12 predicts for i.i.d. calibration rows. A video-level permutation test (2000 permutations, p ∈ [0.08, 0.52]) rules out the frozen split as the cause; the gap is grounder-specific and traces to conditional coverage.

• Point metrics hide subgroup failure. QD-DETR undercovers QVHighlights' Long and Multi-window strata by 0.148 and 0.127, against Qwen's 0.008 and 0.046 (Appendix Table A10). The dataset's small, query-clustered pool — 625 rows over 190 videos — lets that subgroup failure move the marginal average.

• Fixed margins are unreliable; calibration is not. Across six hand-picked constants, realized coverage ranges from 0.096 to 1.000. Cover's calibrated λ realizes coverage within approximately 0.02 of the requested target on every (grounder, dataset) pair, while no uncalibrated fixed margin achieves this across datasets or models.

• No score family dominates uniformly. For QD-DETR on Charades-STA the two interval families cross between targets: sec is shorter at 0.80 (20.1s vs. 21.1s) while norm is shorter at 0.90 (24.4s vs. 24.9s). On ActivityNet-Captions sec is shorter at both targets with no crossing. On QVHighlights, sls is shortest at both tested targets (91.5s and 111.6s vs. 101–103s and 128–129s for the interval families), but the fuller α-grid shows sls losing to both interval families at looser targets (1 − α ≤ 0.70) and only overtaking them from roughly 0.80 upward.

• Super-level-set regions can be disconnected. The connected-components count averages between 1.25 and 1.51 across the grid on QVHighlights, matching the disconnected behavior the method permits.

• One parameter or two is a property of the grounder. For QD-DETR the single parameter is the right choice: the two-parameter variant is longer in 9 of 12 combinations. For Qwen the ordering inverts: two parameters are shorter in 9 of 12 combinations, by −3.4% on average and up to −9.1% on QVHighlights at 1 − α = 0.80.

• A VLM knows when events start, not when they end. For Qwen, λ_start calibrates to zero or near zero in 7 of 12 rows and never exceeds 5.44, while λ_end is never below 3.31 and reaches 115.4 (Appendix Table A3). The single-parameter score applies an end-driven λ to an already-adequate start side, which is why the two-parameter saving is largest on QVHighlights and smallest on Charades-STA.

• Refusals inflate black-box regions. Qwen's refusal rate is 10.4% on Charades-STA, 2.9% on ActivityNet-Captions, and 2.6% on QVHighlights. Marginal length is measurably larger than non-refused length wherever refusal is non-trivial (Charades-STA: 25.4s marginal vs. 24.5s non-refused for norm), so refusal-conditional is the fair basis for comparing efficiency against QD-DETR.

• Risk control, conditional coverage, transfer, and calibration size behave as predicted. Risk control keeps the miss rate within budget (δ = 0.1) across all configurations, with interval length increasing by only 1.4–14.8% and shrinking at the expected O(n^{-1/2}) rate. Mondrian calibration reduces the worst-stratum deviation by more than 2.5× on QD-DETR, with benefit diminishing when the calibration pool is too small to populate each stratum. Duration-aware reweighting reduces the cross-dataset coverage gap from 18.8 percentage points to 3.6, though it cannot compensate for target regions unsupported by the source data. Calibration uncertainty shrinks from ±0.042 to ±0.007 at the expected O(n^{-1/2}) rate, with usable guarantees from as few as 20 calibration videos.

• Evidential uncertainty does not deliver its nominal level. An evidential grounder purpose-built for uncertainty quantification realizes only 0.660 against its own nominal 0.80 target, while Cover restores a valid guarantee on the identical predictions.

Methodology in Plain English

The researchers leave the grounder completely alone. They take a held-out set of video–query pairs with ground-truth intervals, run the grounder on each one, and measure how badly it misses: for interval-emitting grounders, the largest per-side overshoot (normalized by the predicted interval length, or in plain seconds); for grounders that expose a relevance signal, how far the signal must be lowered so it stays above threshold across the entire true moment. They then sort these miss amounts and take the empirical quantile at the level the user requests, with a finite-sample correction. At test time, every prediction is widened by that single calibrated quantity. Because the calibration set and the test point are exchangeable, the resulting region contains the true moment with probability at least 1 − α. The only added computation is sorting scalars. Variants cover the case where the two boundaries have very different error (calibrate each side separately), the case where the user wants the expected miss rate bounded rather than full coverage (risk control), the case where a subpopulation such as short events needs its own guarantee (stratified/Mondrian calibration), and the case where the test distribution shifts (weighted calibration).

Experiments use Charades-STA, ActivityNet-Captions, and QVHighlights, each split at the video level into a calibration pool and a test set (40/60, frozen at seed 42), disjoint from anything used to build the grounder. Coverage variability is estimated by resampling within the frozen calibration pool (50 resamples at 50% of its videos). Two grounder types are wrapped: a trained DETR-style localizer (QD-DETR via Lighthouse) and a black-box video–language model prompted for interval retrieval (Qwen2.5-VL-7B-Instruct). Because Lighthouse caps usable video length, videos longer than 150 seconds are dropped, which affects only ActivityNet-Captions and removes 34.5% of its original pool. Where QVHighlights ground truth spans multiple intervals, the envelope interval — earliest start and latest end — is scored. Section G of the Appendix adds three further grounders, including SRAM, a training-based grounder that reports boundary uncertainty.

Why This Matters

Impact on research. The paper moves uncertainty quantification for temporal grounding from architecture-specific, retraining-dependent estimates to a post-hoc, distribution-free guarantee with a user-chosen error rate. It supplies the theory the new output object needs — how large the certified region becomes, when coverage survives conditioning on event length, and how it degrades when moments from one video break exchangeability — and it shows that a purpose-built evidential grounder's own uncertainty can be miscalibrated (0.660 against a 0.80 nominal target) while the wrapper repairs it on identical predictions.

Real-world applications:

• Retrieval indexes and video search: a region with a stated guarantee tells the system how far to widen its search instead of trusting a single interval.
• Video editing tools: editors can be shown a certified span of footage guaranteed to contain the event, with the width made explicit.
• Evidence-seeking agents: an agent has a principled signal for when to defer or expand its search rather than treating every confident interval as correct.
• Annotation-adjacent quality control: since annotators disagree heavily, a calibrated region quantifies how much of that disagreement the system has absorbed.

Industry relevance. The wrapper requires no retraining and no white-box access, so it applies unchanged to a trained localizer and to a model served behind a text-only API — the deployment mode most commercial video–language systems actually use. The calibration cost is sorting n scalars, and the calibration-size study puts the practical floor at roughly 20 videos, so adoption does not demand a large new labeling effort.

Future Directions

• Closing the conditional-coverage gap. Exact attribute-free conditional coverage is unattainable distribution-free; stratified calibration and score normalization each recover part of it, but the paper leaves open how far each can be pushed, and notes that a user who cares about a specific subpopulation should calibrate on it.

• Handling non-exchangeability beyond bounded dependence. The cross-dataset study finds reweighting repairs one transfer direction fully and the other not at all, a positivity limit no reweighting scheme can fix. Within a long video, clips and queries are temporally dependent, and the beyond-exchangeability literature offers adaptive remedies the paper does not claim as solved.

• Improving efficiency without a strong base grounder. The wrapper inherits the grounder's quality rather than making a weak grounder strong, and the super-level-set score is only as good as the relevance signal it thresholds. Better relevance signals would translate directly into shorter certified regions.

• Extending certification to downstream decisions. The paper includes a downstream-utility study connecting certified width to a deployment rule (Appendix Table A15 and A16), and where partial overlap suffices, a miss-rate or IoU-based risk applies unchanged — leaving room for more task-specific risk objectives.

Target Audience

Researchers and practitioners working on video–language understanding, video temporal grounding, and moment retrieval; applied scientists who need calibrated uncertainty around interval predictions; and engineers deploying trained localizers or black-box video–language models who want a coverage guarantee without retraining or model access. Readers need a working knowledge of conformal prediction or statistical learning theory to follow the analysis section, though the problem motivation and empirical results are accessible to a broader computer vision audience.

Authors’ abstract

Event boundaries in continuous video are ambiguous: re-annotate the same query-video pair and independent annotators mark moments that overlap by less than half on a large fraction of samples. The ground truth for video temporal grounding is therefore a distribution over intervals, yet every grounder returns a single interval with no statement of reliability, so at deployment a wrong interval is indistinguishable from a right one. COVER changes the output object: a post-hoc, model-agnostic wrapper that turns any grounder, a trained localizer or a black-box video--language model, into one that emits a temporal region containing the true moment with probability at least $1-α$, by calibrating the quantile of a temporal nonconformity score on held-out labels and widening the base prediction by that amount. The guarantee is finite-sample and distribution-free under exchangeability, and requires neither retraining nor white-box access. We give two score families, a two-sided boundary-widening score for grounders that emit an interval and a super-level-set score for grounders that emit a relevance signal, and develop theory specific to grounding that bounds how large the certified region becomes, when coverage survives conditioning on event length, and how it degrades when moments from one video break exchangeability. Across three benchmarks and five grounders, realized coverage tracks the target, and calibration exposes what point metrics hide.