InFlux: A Benchmark for Self-Calibration of Dynamic Intrinsics of Video Cameras

AI summary

Overview

Research area: Computer vision — camera calibration, 3D reconstruction/geometry, and benchmark dataset construction.

Technical level: Advanced. The paper assumes familiarity with camera intrinsics, focal length, distortion models, structure-from-motion pipelines, and calibration toolboxes, though the core idea is describable plainly.

One-sentence scope: The paper introduces InFlux, a real-world benchmark of 386 high-resolution videos (143K+ frames) with per-frame ground truth camera intrinsics, built by mapping per-frame lens metadata to intrinsics through calibration-derived lookup tables, and shows that six existing intrinsics-prediction baselines struggle on it.

What This Paper Is About

Most 3D algorithms assume camera intrinsics — the parameters describing how 3D points project onto the 2D image plane — stay fixed for an entire video, but real footage from zoom and autofocus lenses violates that assumption. The problem is that no benchmark previously offered per-frame ground truth intrinsics for real-world videos with changing intrinsics, so the constant-intrinsics assumption could not be properly tested or corrected. The authors build that benchmark and use it to measure how badly current methods fail.

Key Contributions

1. The first benchmark with per-frame ground truth camera intrinsics for real-world videos with dynamic intrinsics. Prior benchmarks either hold intrinsics fixed or report lens focal length rather than actual camera intrinsics.

2. A dataset of 386 high-resolution videos covering 143K+ annotated frames, spanning 126 indoor scenes and 260 outdoor locations, with diverse camera motions and intrinsic changes.

3. An extended version of the Kalibr calibration toolbox modified to improve accuracy and robustness on the authors' data, validated against synthetic board and drone experiments rendered in Blender.

4. An evaluation of six baseline intrinsics-prediction methods on InFlux, showing that all of them struggle to predict per-frame intrinsics accurately.

Main Findings

• Constant-intrinsics assumption is broken in the wild: The authors note that DSLR cameras with zoom lenses change intrinsics when zoom or focus is adjusted, and even smartphone cameras do so due to autofocus, yet many 3D algorithms still assume fixed intrinsics.

• Per-frame ground truth is achievable without stop-motion filming: Using an ARRI Alexa Mini camera with a Canon CINE-SERVO 17-120 mm PL Mount lens ("canon17") and a Fujinon Premista 80-250 mm lens ("premista80"), both of which record /i Technology lens metadata containing per-frame lens focal length (LFL) and focus distance (FD), the authors retrieve intrinsics from precomputed per-lens lookup tables (LUTs).

• Baselines fail on dynamic intrinsics: In Table 1, GeoCalib reports 56.5% fx error and 56.5% fy error with 52.9% of frame-point pairs under the 300-pixel EPE threshold (the text states 54.1% for the same metric). WildCamera reports 45.6% fx error and 47.2% of pairs under threshold; UniDepthV2 reports 50.6% fx error and 46.1%; DroidCalib reports 68.1% fx error and 28.0%; Perspective Fields reports 64.6% fx error and 17.8%; COLMAP reports 1270% fx error and 7.85%.

• GeoCalib is best but still inaccurate: The paper states that GeoCalib achieves the best performance among the baselines, yet only 54.1% of its frame-point pairs achieve EPE under 300 pixels while image dimensions are 3424 × 2202.

• Two baselines fail to produce output for many frames: COLMAP fails to produce intrinsics predictions for 92% of input frames even when run with PER_IMAGE camera mode; DroidCalib fails to predict intrinsics for 15% of input frames, often on videos with little motion because it relies on large amounts of optical flow for keyframe selection.

• Modified Kalibr outperforms the original on synthetic data: In synthetic canon17 experiments, the modified Kalibr achieves consistently low error, whereas the original exhibits occasional large error spikes and fails to converge on some experiments.

• Scene and motion diversity: The dataset includes subjects ranging from static structures such as buildings and furniture to dynamic elements such as people, vehicles, and animals, and camera motion from static shots to pans, tilts, fast sweeps, tracking and dolly movements, object scans, bumpy handheld footage, and wild random swings. Intrinsic changes include fixed settings, controlled monotonic changes, periodic variations, and wild non-monotonic fluctuations affecting one or both parameters.

Methodology in Plain English

The central obstacle is that calibrating a camera for every single frame would be computationally expensive, time consuming, and would force stop-motion filming that destroys natural motion. The authors sidestep this by shifting the work from per-frame calibration to a one-time per-lens setup.

First, they use hardware that records per-frame lens metadata: LFL and FD, the two values that characterize a lens's optical state. Then they build a lookup table for each lens by running calibration experiments across many LFL-FD combinations and interpolating between the results. Querying the table with a frame's metadata yields that frame's ground truth intrinsics.

For the calibration experiments, the size of the camera's field-of-view spatial footprint (FSF) varies dramatically across LFL-FD settings, so the calibration target must scale with it. For small and medium FSF settings, they use AprilGrid boards — an 8 × 11 array of AprilTags — in four sizes: 100 × 75 mm, 200 × 150 mm, 400 × 300 mm, and 800 × 600 mm, with the smallest board's tags at 6 mm and 1.8 mm spacing, scaling proportionally. Each board yields 352 detectable tag corners. Boards are held in five orientations (one parallel to the sensor, four tilted at ±45 degrees) and moved jerkily through the FSF so keyframes are sharp and intermediate frames blurry; adaptive non-maximum suppression (ANMS) with a robustness parameter of 1 and an elbow-method threshold selects keyframes.

For large FSF settings, where boards become impractical or the FSF clips into the ground, they use a drone calibration target: a Holybro X500 V2 quadcopter carrying a 3-watt red LED underneath for conspicuous 2D localization and a Septentrio Mosaic X5 RTK chip on top for centimeter-level 3D positioning. Flight paths consist of 24 hover points, and data is only captured when the drone hovers still, triggered over TCP/IP by an onboard Raspberry Pi that flashes the LED for 0.25 seconds.

LFL sampling uses exponentially increasing step size (giving canon17 values of 17, 18, 20, 24, 32, 48, 80, and 120 mm; premista80 values of 80, 81, 83, 87, 95, 111, 143, 207, and 250 mm). FD is sampled roughly uniformly in inverse depth (canon17: 0.85, 1.69, 1.93, 2.25, 2.7, 3.38, 4.5, 6.75, and 13.5 m; premista80: 1.5, 2.5, 2.81, 3.21, 3.75, 4.5, 5.62, 7.5, 11.25, and 22.5 m).

Kalibr, chosen for its recognized accuracy, is extended in three ways: the vanishing-point-based camera focal length initialization is replaced with a thin-lens approximation using known LFL and FD; a "fixed point" scheme periodically resets the principal point to the image center during Levenberg–Marquardt distortion initialization; and multiple rollouts are run with the median aggregate result selected, reverting to pre-final-optimization intrinsics if the principal point drifts too far. Distortion follows the Brown–Conrady model.

Interpolation over the LUT uses trapezoidal bilinear interpolation in quadrilateral grid regions formed by board experiments, and Delaunay triangulation with barycentric interpolation in irregular regions involving drone experiments. Leave-one-out cross-validation is used to check interpolation accuracy.

Why This Matters

Impact on research: The constant-intrinsics assumption underpins a large family of structure-from-motion, SLAM, and 3D reconstruction methods. By supplying per-frame ground truth for genuinely dynamic intrinsics, InFlux makes it possible to measure and eventually close that gap. It also gives the community a scoring target for what "good" looks like on this problem, which previously could not be quantified.

Real-world applications:

• Robotics and depth estimation, where accurate intrinsics are needed to convert pixels to 3D points correctly.
• AR/VR and CGI, where seamless virtual-real integration depends on matching the real camera's projection.
• Cinematography and documentary production, where smooth zoom plus dolly or panning shots are common and change intrinsics continuously.
• Smartphone and consumer camera pipelines, where autofocus silently changes intrinsics frame to frame.

Industry relevance: Camera and lens manufacturers, visual-effects studios, autonomous-vehicle and drone perception teams, and developers of photogrammetry or 3D-reconstruction software all depend on intrinsics accuracy. The method's reliance on recorded lens metadata also points at a practical lever: hardware that logs its optical state makes accurate self-calibration far easier.

Future Directions

1. Closing the accuracy gap: With six baselines still far from accurate — the best reaching only a minority of frame-point pairs under a 300-pixel EPE threshold — there is clear room for methods designed specifically for smoothly varying intrinsics rather than per-frame independent prediction.

2. Training data rather than only evaluation: The authors note that existing synthetic dynamic-intrinsics datasets are limited (some use identical intrinsics for all images, some provide no ground truth intrinsics, some lack scene diversity), leaving open how to train models for this task.

3. Scaling the metadata-to-intrinsics approach: The LUT strategy works for "virtually any video we can physically record," but it depends on lenses that report LFL and FD — extending it to hardware that does not report metadata is an unaddressed question.

4. Improving calibration tooling further: The modified Kalibr addresses convergence, principal point drift, and variance, but the underlying calibration still requires physical calibration experiments per lens, so reducing that one-time cost remains an open problem.

Target Audience

Researchers and engineers working on structure-from-motion, SLAM, 3D reconstruction, camera calibration, and AR/VR, as well as anyone building benchmarks or evaluation protocols for geometry from video. It is also relevant to practitioners in visual effects, robotics perception, and computational photography who need reliable per-frame intrinsics from real footage. Readers without a background in camera geometry will need to work through the preliminaries section, which defines camera focal length, lens focal length, lens-to-object distance, focus distance, and zoom lenses.

Authors’ abstract

Accurately tracking camera intrinsics is crucial for achieving 3D understanding from 2D video. However, most 3D algorithms assume that camera intrinsics stay constant throughout a video, which is often not true for many real-world in-the-wild videos. A major obstacle in this field is a lack of dynamic camera intrinsics benchmarks--existing benchmarks typically offer limited diversity in scene content and intrinsics variation, and none provide per-frame intrinsic changes for consecutive video frames. In this paper, we present Intrinsics in Flux (InFlux), a real-world benchmark that provides per-frame ground truth intrinsics annotations for videos with dynamic intrinsics. Compared to prior benchmarks, InFlux captures a wider range of intrinsic variations and scene diversity, featuring 143K+ annotated frames from 386 high-resolution indoor and outdoor videos with dynamic camera intrinsics. To ensure accurate per-frame intrinsics, we build a comprehensive lookup table of calibration experiments and extend the Kalibr toolbox to improve its accuracy and robustness. Using our benchmark, we evaluate existing baseline methods for predicting camera intrinsics and find that most struggle to achieve accurate predictions on videos with dynamic intrinsics. For the dataset, code, videos, and submission, please visit https://influx.cs.princeton.edu/.