Learning to Refocus with Video Diffusion Models

AI summary

Overview

Research area: Computational photography and generative computer vision, specifically post-capture image refocusing using video diffusion models.

Technical level: Intermediate. The paper assumes some familiarity with diffusion models, latent spaces, and classifier-free guidance, but the core idea (turning a focus sweep into a "video") is intuitive.

Scope: The paper proposes a method that takes a single defocused photograph and generates a full, perceptually realistic focal stack — a sequence of images at progressively shifted focus planes — in one pass, along with a new 1,637-scene smartphone focal stack dataset.

What This Paper Is About

Autofocus systems frequently lock onto the wrong subject, and photographers often want to change the plane of focus after capturing an image. Prior solutions demand special hardware (light field cameras), multiple captures, or depth maps, and existing software approaches mostly produce flat, all-in-focus images rather than natural defocus effects. This paper's goal is to let anyone re-focus an ordinary single photo interactively, using generative modeling to "hallucinate" the missing focal planes in a way that looks like real optical blur.

Key Contributions

1. A large-scale real-world focal stack dataset. 1,637 scenes (1,474 train / 163 test) captured on an iPhone 12 with a custom five-phone rig, each scene containing nine aligned focus steps plus an all-in-focus reference, all shot in RAW across diverse controlled and in-the-wild environments. This is larger than the previous largest public real-world set (510 stacks on a Google Pixel 3).

2. A refocusing method built on video diffusion. The authors reframe focal stack prediction as a temporally coherent video generation problem, using Stable Video Diffusion to reconstruct the entire stack from a single defocused input in one sampling pass — rather than refocusing one frame at a time as RefocusGAN does.

3. A position-dependent modification to classifier-free guidance. Instead of replicating the conditioning frame's latent across all frames (as stock SVD does), the method places the conditioning latent only at its matching focal position and zeros the rest. This lets a single image condition the stack at any position, which is essential for flexible refocusing.

4. Demonstrated downstream capabilities. The reconstructed stacks enable all-in-focus synthesis, depth-of-field editing, and incidental reduction of slight motion blur, and the model generalizes to film cameras, DSLRs, and other smartphones without retraining.

Main Findings

• Large focus shifts are where the method wins. Across LPIPS and FID on the paper's dataset, the diffusion approach is the strongest method when refocusing from the extreme positions (focal position 1 or 9) — the regime that matters most when users want to jump from a foreground subject to the background.

• Small focus shifts favor conventional deblurring. At intermediate positions (notably position 5, where many scenes are already sharp), pixel-domain baselines like NAFNet outperform the diffusion model, because VAE latent compression discards high-frequency detail. The authors report their method is 2–4 dB worse in PSNR than deblurring networks, a trade-off they accept in exchange for perceptual realism.

• Non-generative baselines regress to the mean. When asked to produce a plausible refocus over a large distance, deterministic deblurring networks average multiple possible solutions and blur the result, while the generative model synthesizes sharp, realistic detail.

• The position-dependent conditioning is not optional. An ablation that reuses SVD's original motion-profile-ID conditioning fails entirely, simply reproducing the input image in every frame.

• Human raters strongly prefer the output. In a user study restricted to refocusing distances greater than four focal positions, 20 users evaluating 20 scenes voted for this method 88.25% of the time versus 11.75% for the best baseline (NAFNet).

• Video diffusion models already understand focus pulls. Prompting an off-the-shelf generative video tool with something like "a focus pull of a bug on a leaf" produces focal-stack-like sequences, which motivated using these priors rather than training from scratch.

• Robustness and generalization hold up. The model tolerates gamma shifts (0.5 and 2.0) and additive Gaussian noise (σ = 0.05), and refocuses images from other phones, film cameras, and DSLRs while preserving original color profiles.

• Known failure modes. The model breaks down on extreme bokeh from large-aperture DSLR lenses, which is outside the iPhone training distribution, and applying it frame-by-frame to video introduces temporal artifacts because consistency is not enforced.

Methodology in Plain English

The central insight is that a focal stack — the same scene photographed at several focus distances — looks a lot like a short video where the camera performs a focus pull. So instead of building a specialized refocusing network, the authors take a pretrained video diffusion model (Stable Video Diffusion) and teach it that the "motion" between frames is a change in focus.

Training data comes from nine-step focus sweeps on an iPhone. Because adjusting focus physically changes the lens geometry, the images shift and distort slightly — a phenomenon called focal breathing — so the authors convert RAW files, correct lens distortion, and rescale each frame to align with the sharpest-field-of-view frame. They also generate an all-in-focus reference per scene using Helicon Focus.

The key technical change is in how the input image is fed to the model. Standard SVD conditions on a first frame by copying that frame's latent into every position of the conditioning tensor, which tells the model nothing about where in the sequence the input sits. The authors instead put the input latent only at its true index and fill the rest with zeros, then randomly choose that index each training batch so the model learns to refocus in either direction. At inference, the model denoises the whole stack in a single sampling pass with a guidance weight of 1.5, meaning one input photo in, nine refocused frames out.

Why This Matters

Research impact: This is an early demonstration that video diffusion priors transfer to optical phenomena — that a model trained on motion can be repurposed for focus. It also establishes a substantially larger real-world benchmark for refocusing than previously existed, which gives the field a fairer testing ground than synthetic focal stacks.

Real-world applications:

• Smartphone photo editing. A "focus slider" in a camera app that works on already-captured photos, no depth sensor or dual capture required.
• Rescue shots from failed autofocus. Users can recover images where the camera locked onto the wrong subject, including scenes with faces or thin structures.
• Computational bokeh and depth-of-field styling. Generating all-in-focus images or selectively controlling which parts of a scene stay sharp for creative effect.
• Mild motion-blur cleanup. Because the training data contains no motion blur, the model tends to generate blur-free frames, incidentally sharpening moving subjects during refocusing.

Industry relevance: The work comes from Adobe researchers in collaboration with York University and was published at SIGGRAPH Asia 2025, with code and data released publicly. It fits directly into commercial photo-editing pipelines and mobile camera stacks, where post-capture flexibility is a competitive feature and where manufacturers want software solutions instead of additional lens hardware.

Future Directions

• Recover high-frequency detail at small focus changes. The authors attribute their weakness at intermediate focal positions to VAE latent compression and suggest pixel-space diffusion models, improved autoencoders, or hybrid latent-pixel architectures as fixes.

• Extend to large-aperture and varied optics. Training on DSLR and large-sensor data could teach the model the exaggerated bokeh it currently fails on, and aperture size or focal length could become explicit conditioning signals for controllable defocus.

• Enforce temporal consistency for video. The current model refocuses still images; applying it per-frame to video causes flicker, so a temporally aware variant is an open problem.

• Remove the dependence on focal-position metadata. The method currently needs to know (or guess) which focus plane the input corresponds to; more robust automatic inference, or sampling across positions and selecting the best output, would make it usable on arbitrary photos at scale.

Target Audience

Researchers and graduate students in computational photography, computational imaging, and generative vision; graphics and imaging engineers at camera and editing-software companies; and practitioners building post-capture editing tools who want to understand both the promise and the current limits of diffusion-based refocusing — particularly the trade-off between perceptual realism and pixel-accurate reconstruction.

Authors’ abstract

Focus is a cornerstone of photography, yet autofocus systems often fail to capture the intended subject, and users frequently wish to adjust focus after capture. We introduce a novel method for realistic post-capture refocusing using video diffusion models. From a single defocused image, our approach generates a perceptually accurate focal stack, represented as a video sequence, enabling interactive refocusing and unlocking a range of downstream applications. We release a large-scale focal stack dataset acquired under diverse real-world smartphone conditions to support this work and future research. Our method consistently outperforms existing approaches in both perceptual quality and robustness across challenging scenarios, paving the way for more advanced focus-editing capabilities in everyday photography. Code and data are available at www.learn2refocus.github.io