MODEST: Multi-Optics Depth-of-Field Stereo Dataset

AI summary

Overview

• Research area: Computer vision / computational photography — specifically depth-of-field (DoF) rendering, defocus deblurring, and stereo camera calibration, presented as a dataset and benchmarking paper (arXiv:2511.20853, cs.CV).
• Technical level: Intermediate. The optical concepts (aperture, focal length, circle of confusion, bokeh) and the dataset construction are explained accessibly, while the evaluation section assumes familiarity with PSNR, SSIM, LPIPS and camera calibration terminology.
• Scope (one sentence): The paper introduces MODEST, an ultra-high-resolution (5472×3648 px, 20MP) stereo DSLR dataset of 20,000 images across 10 scenes, 10 focal lengths (28–70mm) and 5 apertures (f/2.8–f/22.0), plus calibration sets, and uses it to benchmark four state-of-the-art shallow DoF rendering methods and three defocus deblurring methods.

What This Paper Is About

Training and evaluating models that render shallow depth of field or remove defocus blur is held back by the lack of large, full-frame, high-fidelity datasets of real photographs. Real optical blur depends jointly on focal length, aperture, sensor size, focus distance, lens characteristics and scene geometry, yet most public datasets fix one focal length or a narrow set of camera settings, so models never learn how blur changes when the optics change.

MODEST addresses this by capturing the same 10 carefully curated scenes under a systematic grid of 50 optical configurations (10 focal lengths × 5 apertures) with a synchronized stereo pair of Canon EOS 6D cameras, and by shipping intrinsic and extrinsic calibration images so the geometry can be recomputed by future algorithms.

Key Contributions

1. A new ultra-high-resolution multi-optics stereo DSLR dataset. MODEST contains 20,000 images across 10 scenes; each scene is captured at 10 focal lengths and 5 apertures per focal length, at multiple stereo viewpoints, at 20MP full-frame resolution (5472×3648 px), with scenes deliberately containing reflective and semi-transparent surfaces, point lights, fine details, and multi-scale optical illusions. It also provides a global monocular intrinsics calibration set and per-scene, per-focal-length stereo extrinsics calibration sets.

2. The first systematic benchmarking of SOTA DoF and defocus deblurring methods across changing focal parameters on real, non-synthetic complex scenes. The authors demonstrate specific limitations: DoF methods fail to reproduce cat-eye vignetting bokeh and non-uniform intensity bokeh, do not clearly distinguish the focus region, produce similar blur across a wide depth range, and give inconsistent results on nearly identical inputs; defocus deblurring methods struggle to sharpen details at wide apertures and show tiling artefacts.

3. Evidence that SOTA DoF methods have non-intuitive tunable parameters. BokehMe, Dr.Bokeh and BokehDiff show insensitivity to focal length (the optimal blur-strength parameter does not change with the input image's focal length), while Bokehlicious's F-parameter follows a nonlinear trend with input focal length at constant aperture.

4. Public release of the dataset, calibration files, and data processing/evaluation tools for non-commercial, purely academic research use (https://modest-dataset.netlify.app/), to support reproducible benchmarking.

Main Findings

• Dataset scale and structure: 20,000 images, 10 scenes, 5472×3648 px (20MP), focal lengths of 28–70mm and apertures from f/22.0 to f/2.8, giving 50 distinct optical configurations. Each focal length yields roughly 100 stereo image pairs (20 pairs per aperture across five aperture settings), i.e. 2000 images per scene. At least two shots are captured per viewpoint at identical optical settings (multi-capture). The stated depth range is 0.5 m–10 m.

• Capture hardware: Two identical Canon EOS 6D full-frame DSLR cameras with 28–70mm zoom lenses, remotely and synchronously triggered, on a balanced height-adjustable tripod rig with an adjustable baseline up to a maximum of 30 cm. Identical camera/lens models are used deliberately to isolate make-dependent optical variation between the left and right views.

• Calibration quality: The global intrinsics set uses a 16×12 ChArUco pattern with 43mm squares, rigidly attached with less than 1mm surface deviation; 60–100 images per focal length calibrate pinhole intrinsics and k1, k2, p1, p2 distortion per camera. Stereo extrinsics are calibrated per focal length per scene using a 7×5 checkerboard. The provided ChArUco images achieve RMS reprojection errors of less than 0.09px per camera.

• Shallow DoF benchmark setup: Four models — BokehMe, Dr.Bokeh, BokehDiff and Bokehlicious — were evaluated zero-shot on four scenes (S1, S3, S5, S8) at five focal lengths (fl28, fl36, fl45, fl60, fl70). The task is to synthesize an f/2.8 shallow-DoF image from the corresponding sharp f/22.0 input. 20 images per focal length × 5 focal lengths × 4 scenes = 400 images total.

• Shallow DoF numeric results (4-scene average): Bokehlicious performs best on all three metrics (PSNR 28.01, SSIM 0.90, LPIPS 0.22), followed by Dr.Bokeh (PSNR 27.20, SSIM 0.89, LPIPS 0.26), then BokehMe (PSNR 26.68, SSIM 0.88, LPIPS 0.28). BokehDiff performs worst in PSNR and SSIM (PSNR 26.20, SSIM 0.84, LPIPS 0.27) and is never the optimal method on any of the four scenes.

• Human perceptual ranking: 10 reviewers produced 530 total rankings (53 different images each across five focal lengths). Mean Opinion Rank: BokehMe 2.32, Bokehlicious 2.35, Dr.Bokeh 2.54, BokehDiff 2.80 (lower is better). BokehMe and Bokehlicious outputs were favored most; BokehDiff was least favored — showing numeric metrics and perceptual quality can disagree.

• Inference time on A100-40GB GPU: BokehMe 17.7+12.2 s, Dr.Bokeh 120.4 s, BokehDiff 23.1+2e-6 s, Bokehlicious 17.8 s.

• Stereo depth helps DoF rendering: Using stereo depth predicted by FoundationStereo as input on Scene 1, every method improved over its monocular Scene 1 result — for example Bokehlicious rose to PSNR 28.29 / SSIM 0.90 / LPIPS 0.21 (from 27.53 / 0.91 / 0.20 in the monocular Scene 1 setting).

• Performance degrades with focal length: With aperture fixed, DoF performance drops as focal length increases from fl28 to fl70, even though each result used parameters tuned at that focal length. The authors attribute this to models failing to scale the Circle of Confusion under changing magnification and field of view.

• Defocus deblurring benchmark: Restormer, NRKNet and ViTDeblur were evaluated across five focal lengths on the same four scenes. Four-scene averages: NRKNet PSNR 27.13 / SSIM 0.87 / LPIPS 0.10 at 0.35 s per image; ViTDeblur PSNR 27.09 / SSIM 0.88 / LPIPS 0.08 at 12.91 s; Restormer PSNR 26.94 / SSIM 0.88 / LPIPS 0.08 at 6.36 s.

• Motion deblurring methods fail on defocus: Two motion-deblurring methods stress-tested on Scene 1 performed poorly: EVSSM reached an average PSNR of 19.34 (SSIM 0.76, LPIPS 0.16) and FFTformer 20.03 (SSIM 0.77, LPIPS 0.19).

• Depth estimators fail on optical illusions: Depth Anything, Unidepth V2 and Depth Pro all struggle to estimate depth accurately for the optical illusion patterns in MODEST.

• Tuning cost is high: For BokehMe, grid search over blur-strength K ∈ [40,60], normalized focus plane disparity ∈ [0.15,0.55] and defocus scale ∈ [10,30] at each of 5 focal lengths took about 63 hours. For Dr.Bokeh, grid traversal over K ∈ [15,35] and normalized focus plane disparity ∈ [0.2,0.6] took about 50 hours.

• Likely source of failure: The authors suggest the limitations stem from training on small-resolution, limited-optics datasets such as EBB! and BLB, which lack explicit optical structure, leading models to learn appearance-blur correlations rather than optical principles.

Methodology in Plain English

The team built a stereo rig from two matching full-frame Canon EOS 6D cameras with 28–70mm zoom lenses, triggered together so left and right frames are captured at the same instant, on a tripod whose baseline can be adjusted up to 30 cm. They chose 10 scenes that are deliberately hard: reflective spheres and ceramic jars, semi-transparent glass doors, patterned and illusion-printed surfaces, small toys and fine details, flowers, and light-emitting sources, arranged with occlusions and ambiguous background depths.

For each scene they fixed the composition and then swept the optics: 10 focal lengths from 28mm to 70mm, and at each focal length 5 apertures (f/2.8, f/5.0, f/9.0, f/16.0, f/22.0), capturing about 20 stereo pairs per aperture. They also took at least two shots per viewpoint at identical settings so robustness to nearly identical inputs can be tested. The very wide apertures provide physically blurred images that serve as ground truth, while f/22.0 provides the sharp, all-in-focus input.

Separately, they captured calibration data: a global set using a 16×12 ChArUco board (43mm squares) at near/far distances, view angles from 0–180 degrees and up/level/down pitch, giving 60–100 images per focal length for pinhole intrinsics and distortion; and, for each scene and focal length, stereo pairs of a 7×5 checkerboard from multiple viewpoints for extrinsics. Distortion coefficients are held fixed while extrinsics are computed.

For evaluation, they ran four shallow-DoF rendering models and three defocus deblurring models zero-shot on four scenes at five focal lengths. Because these DoF models expose unintuitive knobs (blur radius K, blur strength, normalized focus plane disparity, defocus scale), the authors ran multi-parameter grid searches per focal length, running inference on every image of a scene and collecting PSNR, LPIPS and SSIM, so each model is compared at its best achievable configuration. Outputs and the ground-truth blurry target are aligned to the sharp input before metrics are computed to avoid pixel shift. They also collected human rankings from 10 reviewers using mean opinion rank, and ran an extra experiment feeding stereo depth from FoundationStereo into the DoF models.

Why This Matters

• Research impact: The dataset targets the gap between synthetic, low-resolution training data and real high-resolution camera optics. Because it varies focal length and aperture systematically on the same scenes, it lets researchers separate blur caused by aperture width from blur caused by subject distance — something the authors argue prior datasets such as RealBokeh cannot do. It also shows that current models, even when tuned per focal length, degrade as focal length grows, and that their tunable parameters do not respond proportionally to the input image's optics — an argument for architectures that take focal parameters as explicit inputs. The accompanying calibration sets (which most prior datasets lack) allow recalibration as algorithms evolve or as lens drift occurs.

• Real-world applications (as named in the paper):

  • Augmented and virtual reality, where shallow DoF rendering helps users focus on sharp virtual objects placed among real ones.
  • Smartphones, where automatic DoF effects render portrait-style blur without professional knowledge.
  • Autonomous driving and robotics, where defocus deblurring supplies sharp regions for safe navigation.
  • Surveillance/security and forensic analysis, where blurred imagery is processed to reconstruct clearer regions.

• Industry relevance: The dataset targets the camera and imaging pipeline stack — DSLR/mirrorless manufacturers, smartphone computational photography teams, and robotics/AR companies deploying stereo or multi-camera systems. The paper's quantitative demonstrations of failure (cat-eye vignetting, non-uniform intensity bokeh, incorrect focal plane placement, tiling artefacts, order-of-magnitude time differences between methods — Dr.Bokeh at 120.4 s per image versus NRKNet at 0.35 s) give engineering teams a concrete basis for choosing or redesigning models. The paper also frames MODEST toward calibration-sensitive tasks such as novel-view synthesis, Gaussian Splatting, NeRF reconstruction and physics-based image formation.

Future Directions

1. Expand scenes while keeping the protocol. The authors state that future releases will add more indoor and outdoor scenes under the same acquisition protocol; the current release contains only 10 scenes, which limits the breadth of scene diversity relative to large low-resolution datasets.

2. Design optics-aware architectures. The finding that blur parameters of BokehMe, Dr.Bokeh and BokehDiff do not change proportionally with input focal length, and that Bokehlicious's F-parameter varies nonlinearly with focal length, motivates models that accept focal length and aperture as inputs for intuitive, proportional tuning.

3. Revisit calibration as algorithms advance. Because MODEST ships raw intrinsic and extrinsic calibration images (global intrinsics set, per-scene per-focal-length extrinsics), the authors explicitly intend for researchers to recompute parameters with newer calibration methods, adapt to lens drift, and develop calibration-aware computational photography — including geometrically accurate stereo depth supervision, physically grounded defocus/bokeh rendering from camera intrinsics, and focal-length-dependent geometric consistency studies.

4. Close the synthetic-to-real generalization gap. The paper positions itself as a bridge between synthetic low-resolution training data and inference on real camera optics; an open question is whether fine-tuning or evaluating on MODEST can make shallow DoF rendering and defocus deblurring hold up as focal length and aperture change, and whether the observed focus-region ambiguity and tiling artefacts can be eliminated.

Target Audience

• Computer vision and computational photography researchers working on shallow DoF rendering, bokeh synthesis, and defocus deblurring who need a real, high-resolution, optically varied benchmark.
• Camera calibration and stereo/multi-view researchers, who benefit from the global intrinsics set, per-scene per-focal-length extrinsics sets, and the sub-0.09px RMS reprojection errors reported for the 20MP sensors.
• Practitioners building AR/VR, smartphone camera, and robot-vision pipelines who need stereo depth, realistic bokeh, or restored sharpness in real optics.
• Model designers and benchmark maintainers interested in parameter-sensitivity analysis and in whether tunable knobs should be tied to the physical focal configuration of the input image.

Note: The paper reports 20,000 images and 10 scenes in its abstract and contributions, while one row of its dataset comparison table lists 9 scenes for MODEST; the analysis above follows the abstract/contribution figures. The dataset and tools are released under CC BY-NC-ND 4.0 for non-commercial, academic use only.

Authors’ abstract

Training and evaluation of state-of-the-art computer vision algorithms for reliable shallow depth of field (DoF) rendering and defocus deblurring remain constrained by a persistent lack of large-scale, full-frame, high fidelity, real-image datasets. Optical effects of shallow DoF and defocus blur depend intimately on camera optical configuration set with focal length and aperture; requiring rigorous evaluation of the models when these parameters systematically change. Further, modern applications such as AR, VR, smartphones, industrial robots, etc. deploy stereo or multi-camera systems. We present MODEST - the first ultra-high-resolution(5472x3648px, 20MP), multi-optics depth of field stereo DSLR dataset that methodically varies focal length and aperture for a series of complex, real-world scenes, capturing the optical realism and complexity of professional camera systems. With 20,000 images across 50 distinct optical configurations, focal length in 28-70mm, aperture in f/22-f/2.8 for multiple stereo viewpoints for 10 scenes; this ultra-high-resolution, full-range optics coverage enables controlled analysis of geometric and optical effects for shallow DoF rendering and defocus deblurring. Each scene is curated to have challenging visual elements: reflective surfaces, transparent glass walls, fine-grained details, point lights, and multi-scale depth illusions. In addition, we provide intrinsics and extrinsics calibration images to support ever-evolving calibration methods. We evaluate several SOTA DoF and defocus deblurring methods and demonstrate failure cases and limitations. Our comprehensive tuning analysis demonstrates how MODEST evaluates sensitivity of the SOTA DoF models for the actual focal configuration of input images. This work attempts to bridge the realism gap between synthetic, low-resolution training data and inference generalization on high-resolution real-camera optics.