Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks

AI summary

Overview

Research area: Computer vision — video inference, temporal consistency, and corruption robustness for frame-based (single-image) neural networks.

Technical level: Intermediate. The paper combines a formal loss analysis with practical adapter design; readers need some familiarity with temporal smoothing (exponential moving averages), feature-space activations, and standard vision benchmarks.

Scope: The paper proposes a class of lightweight, causally-applied "stabilization adapters" plus a unified training loss that makes pre-trained image models temporally stable and robust on video without modifying the base network's weights.

Authors and affiliation: Matthew Dutson, Nathan Labiosa, Yin Li, and Mohit Gupta (University of Wisconsin–Madison). arXiv:2512.03014v1 [cs.CV], 02 Dec 2025, licensed CC BY 4.0.

What This Paper Is About

Video is often processed frame-wise: each frame is passed independently through an image-based model, so outputs flicker from frame to frame and can fail when inputs contain time-varying corruptions (noise, compression artifacts, weather). The goal is to make these pre-trained, single-image models produce stable and robust video predictions without retraining or altering the original architectures. The authors frame stability and robustness as a single optimization problem and derive the conditions under which a stability penalty helps rather than hurts.

Key Contributions

1. A class of stabilization adapters that can be inserted into virtually any architecture, generating control signals from recent spatiotemporal context to modulate both intermediate features and model outputs. Adapters are lightweight, modular, causal (outputs at time t depend only on times ≤ t), and leave the original model parameters untouched.

2. A unified accuracy-stability-robustness loss (denoted U_c) that combines an extended corruption-robustness term (R_c, extending Hendrycks and Dietterich's definition to arbitrary metrics and time series of duration τ) with a corruption-stability term (S_c), weighted by a constant λ.

3. A theoretical analysis identifying two bounds on λ: the oracle bound λ < 1/2, below which the ground truth is the global minimizer of the loss in prediction space, and the collapse bound λ > τ − 1, above which the global minimizer is exact repetition of the initial prediction. The authors recommend training with λ < 0.5.

4. A resource-efficient training procedure that learns only the adapter parameters Δφ while the base network weights φ₀ stay frozen, evaluated across denoising, image enhancement, monocular depth, semantic segmentation, and adverse-weather robustness.

Main Findings

• Stability and robustness are linked. The authors define stability as the negative expected difference between adjacent predictions (Equation 1), show that the same input corruptions causing instability also reduce accuracy, and treat robustness as a natural extension of temporal consistency.

• Static feature-space smoothing is not enough. For static stabilizers (output fixed, simple fixed, simple learned), increasing stabilization reduces quality through over-smoothing; there is no benefit to stabilizing in feature space for these variants.

• Controlled and spatial stabilizers improve both axes. For image enhancement with HDRNet, controlled and spatial stabilizers have a region where both PSNR and stability beat the base model, and spatial fusion yields roughly 2 dB improvement for the high-intensity effect. Figure 4 reports that the spatial-fusion stabilizer reduces frame-to-frame variation by up to approximately 35 percent while exceeding base-model quality.

• Naive feature stabilization can backfire on denoising. With NAFNet, the "simple fixed" stabilizer worsens both PSNR and instability, because the network predicts an uncorrelated iid noise residual that should not be smoothed. Learned and controlled stabilizers avoid this, improving both stability and PSNR when λ ≤ 0.4.

• Collapse is easy to reach. Setting λ = 8 (> τ − 1) produced prediction collapse in both the enhancement and denoising experiments, with instability below 10⁻³ (a constant prediction), confirming the practical relevance of the collapse bound.

• Extreme-noise exception. Under extreme noise (σ = 0.6), the spatial-fusion stabilizer was about 6 dB worse than other methods; this gap appeared only when evaluating long sequences (hundreds of frames), shrank as τ decreased, and could be at least partially mitigated by increasing τ during training (Appendix F.6).

• Corruption robustness (Table 1) improves substantially. Across patch drop, elastic distortion, frame drop, JPEG artifacts, and impulse noise, stabilizers sharply reduced instability for enhancement, denoising, and depth. Examples: patch drop enhancement PSNR rose from 17.43 to 31.39 with instability falling from 164.6 to 30.36; impulse-noise enhancement PSNR rose from 14.28 to 27.37 with instability falling from 217.8 to 30.90; depth instability fell from 9.89 to 4.73 for patch drop and from 14.17 to 4.91 for frame drop. The only notable exception was reduced PSNR for denoising with dropped frames (33.42 to 27.34), which the authors note PSNR does not capture perceptually.

• Adverse weather (Table 2, RobustSpring). Training stabilizers with a frozen base model gave large gains over the unstabilized model — rain PSNR 21.43 → 28.63 and instability 151.76 → 57.88; snow PSNR 18.62 → 31.34 and instability 262.48 → 59.31. Fine-tuning the base model gave better single-image quality but higher instability. Combining fine-tuning with stabilization gave the best overall results (rain PSNR 32.61, SSIM 0.938, instability 58.30; snow PSNR 35.20, SSIM 0.956, instability 58.98).

• Causal, streaming-compatible by construction. All stabilizers use only current and past inputs, which the authors identify as critical for latency-sensitive streaming video.

Methodology in Plain English

The starting point is a pre-trained image model whose weights are frozen. The researchers insert small "stabilizer" modules at selected layers and at the model output. The simplest stabilizer is an exponential moving average (EMA): each value becomes a weighted blend of its current value and its own previous stabilized value, controlled by a decay parameter β.

Because a single fixed decay cannot adapt to changing scenes, they add a stabilization controller network: a shared backbone g that compares the current and previous frames, plus one small head h_i per stabilized layer that predicts per-value decay rates from the frames, the current unstabilized features, and the previous stabilized and unstabilized features. A further spatial fusion variant lets the head predict a spatially varying, softmax-normalized decay kernel so features from a local neighborhood in the previous frame can be translated into the current one, which helps under motion.

Training uses the unified loss: match the target, plus a λ-weighted penalty on frame-to-frame prediction differences. The authors analyze this loss to find where stabilization is safe (λ < 1/2) and where it forces the model to simply repeat itself (λ > τ − 1). They train stabilizers with backpropagation through time on short video snippets (τ = 8 in the enhancement and denoising setups), using an L2 distance for δ.

Experiments cover four task families and their base models: denoising (NAFNet, with AWGN at σ = 0.1, 0.2, and 0.6 on the NFS dataset and additional DAVIS results in Appendix F.4), image enhancement (HDRNet reproducing the local Laplacian operator at moderate σ = 0.4, α = 0.5 and strong σ = 0.4, α = 0.25 settings on NFS, scaled to a short edge of 360), monocular depth (Depth Anything v2, with stabilizers on DepthAnythingReassembleLayer, DepthAnythingFeatureFusionLayer, and the output, trained on 50 simulated indoor scenes from VisionSim rendered at 50 FPS), and semantic segmentation (DeepLabv3+ in Appendix F.2). HDRNet and NAFNet were fine-tuned before stabilizer training; Depth Anything was deliberately not fine-tuned on the small dataset to avoid overfitting. NFS training used 80 of 100 videos with 20 for validation; the VisionSim depth set used 40 of 50 scenes for training with 10 for validation; RobustSpring used 8 of 10 sequences for training with 2 for validation.

Why This Matters

Impact on research. The paper reframes temporal consistency and corruption robustness as two views of one objective, backed by provable bounds on the stability weight (λ < 1/2 for the oracle bound, λ > τ − 1 for collapse). This gives a principled recipe for a problem that is usually handled with task-specific video architectures trained on large video datasets, and it explicitly positions general adapters as a complement to — not a replacement for — specialized video models.

Real-world applications:

• Video denoising and image enhancement, where flicker directly degrades perceptual quality.
• Autonomous or assisted driving in inclement weather, where the authors motivate robustness with the example of erratic monocular depth estimates affecting a collision-avoidance system.
• Latency-sensitive streaming video pipelines, since the adapters are causal and depend only on current and past frames.
• Deployment on sensor feeds with transient artifacts such as compression, noise, dropped frames, or dropped patches.

Industry relevance. The adapters are lightweight, modular, and require no changes to the original network, so a company can stabilize an already-deployed or costly-to-retrain model — such as Depth Anything v2, which the authors highlight for the scale of data used in its training — with minimal additional compute and no large-scale video dataset. The paper's own guidance is that joint fine-tuning plus stabilization is the best choice for small-to-medium models, while stabilizer-only training remains reasonable when retraining a large base model is infeasible.

Future Directions

• Broaden the theory beyond norm-based distances. The oracle and collapse bounds assume δ can be expressed as a norm on the prediction space, which excludes many multi-component or more sophisticated losses; such losses may still work but would require more careful tuning of λ.
• Develop alternative distance metrics. All experiments used a simple Euclidean norm for δ; the authors suggest a variant of the Wasserstein metric may perform better for 2D outputs and feature maps by accounting for spatial tensor structure.
• Close the sim-to-real gap for depth. Stabilizers trained for Depth Anything showed sim-to-real difficulty; the authors conjecture that real video contains "baseline corruptions" (sensor noise, compression artifacts, optical phenomena) absent from simulated video.
• Resolve the extreme-noise, long-sequence behavior of spatial fusion. Its roughly 6 dB deficit appeared only over hundreds of frames and only shrank as τ decreased; the paper's partial mitigation (increasing τ during training) suggests a training-window question that remains open.

Target Audience

Researchers and engineers working on video inference, temporal consistency, video denoising or enhancement, and robustness to natural corruptions. It is especially relevant to practitioners who want to stabilize existing, pre-trained image models — including large ones that are impractical to retrain — and to readers interested in the theory of balancing accuracy, stability, and robustness in a single training objective. Readers looking for task-specific video architectures or new state-of-the-art benchmark results should note that the authors explicitly state they expect specialized architectures to outperform this general approach on benchmarks.

Authors’ abstract

When applied sequentially to video, frame-based networks often exhibit temporal inconsistency - for example, outputs that flicker between frames. This problem is amplified when the network inputs contain time-varying corruptions. In this work, we introduce a general approach for adapting frame-based models for stable and robust inference on video. We describe a class of stability adapters that can be inserted into virtually any architecture and a resource-efficient training process that can be performed with a frozen base network. We introduce a unified conceptual framework for describing temporal stability and corruption robustness, centered on a proposed accuracy-stability-robustness loss. By analyzing the theoretical properties of this loss, we identify the conditions where it produces well-behaved stabilizer training. Our experiments validate our approach on several vision tasks including denoising (NAFNet), image enhancement (HDRNet), monocular depth (Depth Anything v2), and semantic segmentation (DeepLabv3+). Our method improves temporal stability and robustness against a range of image corruptions (including compression artifacts, noise, and adverse weather), while preserving or improving the quality of predictions.