Generating the Past, Present and Future from a Motion-Blurred Image

AI summary

Overview

Research area: Computer vision — computational photography, motion deblurring, and generative video modeling.

Technical level: Intermediate (requires familiarity with diffusion models and basic image-formation concepts, but the core idea is accessible).

Scope: This paper presents a method that repurposes a large pre-trained video diffusion model to turn a single motion-blurred photograph into a full video sequence spanning the moment of capture — and, uniquely, plausible frames from just before and just after that moment.

What This Paper Is About

When a camera or subject moves during an exposure, the resulting motion blur is usually treated as damage to be removed. This paper argues the opposite: blur is a compressed record of motion, and a sufficiently powerful generative model can "decode" it. The goal is to produce a physically plausible video that explains the blur, rather than a single sharp still image, and to extend that video backward and forward in time beyond the original exposure window.

Key Contributions

1. Reformulating motion deblurring as conditional video generation. Instead of training a network to map a blurry image to a sharp one, the authors reframe the problem as sampling from a video distribution conditioned jointly on the blurred image and on the desired exposure windows of the output frames.

2. Exposure-interval conditioning. A new mechanism that encodes the start and end time of each output frame's exposure — and applies it relative to the input image's own exposure — giving the user direct control over which temporal slice (past, present, or future) the model generates.

3. State-of-the-art results on standard benchmarks. The method substantially outperforms prior work on GoPro, B-AIST++ and A-BIST++ style evaluations in PSNR, SSIM, LPIPS, FVD, and optical-flow end-point error.

4. Demonstrated downstream applications. The generated videos are coherent enough to support off-the-shelf tracking, structure-from-motion (4D reconstruction), and 3D human head pose estimation — including for historical black-and-white photographs up to 80+ years old.

Main Findings

• Large gains on the "present" (during-exposure) task: On the GoPro dataset, the method reaches 30.01 dB PSNR versus 26.54 dB for MotionETR and 25.23 dB for Jin et al. FVD drops to 21.46, compared with 94.90 and 235.53 for the baselines. Optical-flow error (EPE) is 0.39 versus 1.46 and 3.38.

• Consistent improvement on the B-AIST++ dance dataset: 27.37 dB PSNR (versus 26.69 for Animation from Blur) and FVD of 37.16 versus 138.27. Crucially, baseline comparisons are restricted to methods that address only the present task, since none of them handle past/future extrapolation.

• Past and future prediction works, but is less accurate. Given a blur synthesized from 7 frames, the model predicts a 13-frame sequence (3 before, 7 during, 3 after). Quality is evenly high across the 7 middle frames — suggesting every instant inside the blur is equally well constrained — and stays in the 20–30 dB PSNR range for frames outside the exposure window.

• Motion ambiguity is handled explicitly at the patch level. Because a video and its time-reversed version can both explain the same blur, the authors introduce bidirectional patch-based metrics that pick whichever temporal direction (forward or reversed) is locally best, capturing spatially varying ambiguity that frame-level metrics miss.

• The model captures real-world motion priors, and is still multimodal. Sampling multiple outputs for a blurred face yields both left-to-right and right-to-left motion; sampling for a moving taxi consistently yields forward motion — evidence that the model retains commonsense priors from pre-training while still representing genuine ambiguity.

• Historical and in-the-wild generalization. The model recovers plausible dynamics from WWII Normandy landing photos, a 1971 Muhammad Ali boxing photo, and a 1998 photo of astronaut John Glenn. It also succeeds on dancers, concerts, sports, city intersections, and deforming cloth — scenes where small-dataset baselines break down.

• Output video is geometrically coherent. Videos are lifted into dynamic 3D (4D) point clouds with camera trajectories via MegaSaM, and into a single 3D head model from a blurred portrait — meaning the generated frames are consistent across time in a way that supports multi-view reasoning.

Methodology in Plain English

The researchers start from CogVideoX-2B, a publicly available 2-billion-parameter text-to-video diffusion transformer. Rather than train a deblurring network from scratch, they fine-tune this existing model — which has already absorbed vast amounts of real-world motion and appearance from internet-scale video data — to accept a new kind of conditioning.

The input is a motion-blurred image and a set of requested exposure intervals. The blurry image is compressed into the model's latent space. Each output frame is also represented in latent form, but each latent frame is tagged with numeric start/end times for its exposure. To make this meaningful, the authors normalize the input image's exposure to the interval [−0.5, 0.5]; any requested frame's exposure is then expressed relative to that. So an interval inside [−0.5, 0.5] means "show me the present," an interval ending before −0.5 means "show me the past," and one starting after 0.5 means "show me the future."

Those exposure times are encoded with a sinusoidal position encoding (the same style used in transformers to represent positions) and added into the latent frames, alongside standard spatial and temporal position encodings. The model then denoises the latent video as usual.

Training data is built by pulling high-frame-rate clips from five public datasets (GoPro at 240 FPS, Adobe240, REDS, iPhone240, Sports240 — 694 clips averaging 507 frames each). Because source videos often have gaps between frame exposures, the authors upsample everything to 1920 FPS with a frame interpolator before averaging groups of frames to simulate physically accurate blur, always doing the averaging in linear sRGB rather than gamma-corrected space. Fine-tuning took 10 days on 16 NVIDIA L40 GPUs at batch size 64 for 20,000 iterations. At run time, 50 denoising steps with classifier-free guidance produce a video in about two minutes.

Why This Matters

Impact on research: The paper represents a shift in how motion blur is treated — from a corruption to be inverted into a signal to be interpreted. It joins a growing line of work showing that large pre-trained generative models act as general-purpose priors for classic inverse problems, and it is one of the first to demonstrate extrapolation (past/future) rather than just restoration (present). The exposure-interval conditioning scheme is a reusable idea for any task where a model must reason about when light arrived at the sensor.

Real-world applications:

• Photo restoration and archiving — bringing museum and news-archive photographs to life, including monochrome images from decades ago.
• Consumer photography — recovering a usable clip, or at least a sharp frame, from an otherwise ruined handheld shot.
• Visual effects and post-production — generating plausible pre- and post-roll motion from a single frame, useful for editorial and matchmove work.
• Robotics and autonomous systems — inferring motion direction and 3D structure from blur-dominated frames captured under fast motion or low light, feeding downstream tracking and pose estimation.
• Forensics and sports analysis — extracting sub-exposure motion detail from single captures where the moment has already passed.

Industry relevance: The method builds entirely on an open-source base model and fine-tunes it, meaning the barrier to productizing this capability is far lower than if a new architecture had to be trained from scratch. Since output videos are good enough to be consumed by existing tracking, SfM, and face-modeling pipelines, the technique could slot into existing VFX, archival, and computational-photography toolchains as a preprocessing step.

Future Directions

• Quantifying how far in time the model can extrapolate. The paper shows a 3-frames-before / 3-frames-after window works, but does not characterize where past/future predictions stop being reliable. Establishing this boundary is an open question.

• Handling ambiguity better. The metrics acknowledge that a blur admits many valid videos, but the evaluation still selects the best local temporal ordering post hoc. Developing protocols that evaluate a distribution of plausible outputs — rather than one chosen sample — would be more faithful to the problem.

• Scaling to stronger base models. The work uses a 2B-parameter model; whether larger, more recent video diffusion engines improve extrapolation quality or reduce the visible degradation outside the exposure window remains untested.

• Real-world capture without simulated blur. All training blur is synthesized from high-FPS video via frame averaging. The authors show in-the-wild generalization, but paired real blur/sharp training data — or a hybrid training scheme — could close the remaining gap between synthetic and physical blur.

• Tighter integration with 3D reconstruction. The 4D and head-pose results use off-the-shelf tools applied after generation. Jointly optimizing generation and geometry could yield stronger spatial consistency for challenging non-rigid scenes.

Target Audience

This paper is most valuable to computer vision and graphics researchers working on computational photography, deblurring, or video generation, and to machine learning engineers interested in adapting large pre-trained diffusion models to new conditioning signals. It will also interest archivists, VFX artists, and practitioners in sports or forensic analysis who deal with single-frame captures of fast motion. Readers need a working understanding of diffusion models and camera imaging to follow the technical sections, but the conceptual contribution — using large pre-trained models as priors for physics-based inverse problems — is broadly accessible.

Authors’ abstract

We seek to answer the question: what can a motion-blurred image reveal about a scene's past, present, and future? Although motion blur obscures image details and degrades visual quality, it also encodes information about scene and camera motion during an exposure. Previous techniques leverage this information to estimate a sharp image from an input blurry one, or to predict a sequence of video frames showing what might have occurred at the moment of image capture. However, they rely on handcrafted priors or network architectures to resolve ambiguities in this inverse problem, and do not incorporate image and video priors on large-scale datasets. As such, existing methods struggle to reproduce complex scene dynamics and do not attempt to recover what occurred before or after an image was taken. Here, we introduce a new technique that repurposes a pre-trained video diffusion model trained on internet-scale datasets to recover videos revealing complex scene dynamics during the moment of capture and what might have occurred immediately into the past or future. Our approach is robust and versatile; it outperforms previous methods for this task, generalizes to challenging in-the-wild images, and supports downstream tasks such as recovering camera trajectories, object motion, and dynamic 3D scene structure. Code and data are available at https://blur2vid.github.io