Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation

AI summary

Overview

Research area: Robotics and robot learning — specifically visual perception for imitation learning in robotic manipulation, focused on wrist-mounted fisheye cameras.

Technical level: Intermediate. The paper is empirical rather than mathematically dense, but readers get the most out of it with some familiarity with imitation learning (behavioral cloning, diffusion policies) and basic camera concepts such as field of view (FoV) and lens intrinsics.

Scope: The authors run a controlled simulation-and-real-world study isolating how the wide FoV and radial distortion of wrist-mounted fisheye lenses affect policy spatial localization, scene generalization, and cross-hardware generalization. Published 2026-03-02 as arXiv:2603.02139v1 [cs.RO] by Han Xue, Nan Min, Xiaotong Liu, Wendi Chen, Yuan Fang, Jun Lv, Cewu Lu, and Chuan Wen.

What This Paper Is About

Fisheye cameras are increasingly used in robot manipulation projects (the paper names UMI, RDT2, GEN-0, and π0.5) because their field of view often exceeds 180°, giving policy networks a much wider view of the scene than standard pinhole cameras. That adoption, however, has moved faster than any systematic understanding of what fisheye lenses actually do to a learned policy's performance. This paper asks whether the wide FoV genuinely improves spatial reasoning, whether it helps or hurts generalization to new backgrounds, and whether a policy trained on one fisheye lens survives being deployed on a different lens with different intrinsics.

Key Contributions

1. The first systematic empirical study of wrist-mounted fisheye cameras for imitation learning. The authors frame the analysis around two defining lens properties — a wide FoV and severe radial distortion — and derive three research questions: spatial localization, scene generalization, and hardware generalization.

2. A working fisheye camera simulation pipeline inside MuJoCo. To the authors' knowledge, no existing robotics simulation benchmark provides native fisheye support, so they implement a two-stage projection pipeline: six virtual cameras arranged along cardinal directions produce a cubemap, which is stitched into an equirectangular panorama and then reprojected into a fisheye view using a specific projection model. This enables large-scale controlled experiments on the Robomimic and MimicGen benchmarks.

3. Random Scale Augmentation (RSA), a data augmentation method for cross-camera transfer. Instead of a fixed-scale random crop (for example 0.95), RSA samples a random scale factor from a wide uniform distribution, reported as U(0.7, 1.3), per training image, then center-crops and resizes. When the sampled scale is greater than 1.0 the operation becomes a zoom-out: the source image is resized down and the surrounding canvas is padded with black.

4. Concrete, actionable guidance for large-scale fisheye data collection, covering which environments to collect in, how much scene diversity to gather, and what augmentation to apply during training.

Main Findings

• The wide FoV improves spatial localization, but only in visually rich scenes. In the real world, the performance gain from moving to feature-rich backgrounds averaged +0.39 for the fisheye camera versus +0.18 for the pinhole camera. The authors attribute this to the stronger CLIP encoder and complex real-world textures that the fisheye's FoV can exploit, and note the advantage diminishes in environments lacking distinct visual features such as solid-colored backgrounds.

• Simulation results show fisheye outperforming pinhole across the board. In the six-task simulation comparison, single-camera average success rates were 0.57 (fisheye, feature-poor) and 0.66 (fisheye, feature-rich), versus 0.31 and 0.34 for the single pinhole camera. For double-camera setups the averages were 0.72 and 0.75 (fisheye) versus 0.38 and 0.45 (pinhole).

• Large gains are concentrated in the hardest precision tasks. Single fisheye cameras improved Threading from 0.30 to 0.56 (+0.26) and Assembly from 0.24 to 0.48 (+0.24) when moving from feature-poor to feature-rich backgrounds — the two largest individual gains reported in Table 1. Not every task improved: single fisheye Coffee dropped from 0.80 to 0.76 (−0.04).

• Probing the encoder confirms the localization explanation. The authors fine-tuned the trained visual encoder with a lightweight MLP head to regress the robot's end-effector pose, using lower error as a proxy for better spatial awareness. Encoders trained with fisheye cameras consistently produced lower proprioception error; in Pick Cup, translation error fell from 12.309 cm (pinhole, feature-poor) to 2.362 cm (fisheye, feature-rich), and rotation error fell from 15.345° to 3.394°. The same ordering held in Fold Towel (4.204 cm to 2.908 cm) and Hang Chinese Knot (14.168 cm to 5.143 cm).

• Scene generalization scales steeply with diversity for fisheye policies. Holding the total data volume fixed and varying the number of unique training scenes — N = {1, 8, 16, 32} in simulation and N = {1, 2, 4, 6, 8} in the real world — fisheye policies showed a steeper zero-shot performance curve on unseen scenes than pinhole policies. In the real-world Pick Cup task, the fisheye policy's zero-shot success rate on unseen environments exceeded 95% with just eight diverse training scenes.

• The simulation scaling curve is less steep than the real-world one. The authors attribute this to two differences: the visual encoders (a non-pre-trained ResNet-18 in simulation versus a pre-trained CLIP model in the real world) and the comparatively lower visual complexity of simulated background imagery.

• Cross-camera transfer fails because policies overfit to absolute object scale. When trained on one camera configuration ("Seen Param") and deployed zero-shot on five unseen configurations with varying distortion and FoV, the baseline policy — trained with normal augmentations — exhibited severe performance drops on lenses such as "Param 3" and "Param 4." The authors trace this to policies perceiving objects as closer or farther than they are when absolute pixel scale changes.

• RSA mitigates the drop across all unseen configurations. Policies trained with RSA maintained higher success rates across every unseen lens configuration, averaged over the six simulation tasks, supporting the hypothesis that learning relative scale (for example, object scale relative to the gripper) is the key to cross-camera robustness.

• Fisheye-trained policies overfit more easily in simple scenes. The paper frames this as the flip side of the generalization result: the same capacity that makes fisheye policies excel with sufficient scene diversity also makes them prone to overfitting when that diversity is absent.

Methodology in Plain English

The authors built a controlled comparison rather than a single system demo. To study the wide FoV, they compare a fisheye camera against a standard pinhole camera under otherwise identical conditions, always wrist-mounted with no third-view camera, since wrist mounting is what recent large-scale data collection projects use and the short observation distance amplifies the differences between lens types.

For simulation, they added fisheye rendering to MuJoCo. Six virtual cameras placed along the front, back, left, right, up, and down directions produce six images assembled into a cubemap. The cubemap is stitched into an equirectangular panorama, and that panorama is reprojected through a lens projection model to produce the final fisheye view, allowing different lens characteristics to be simulated. The pipeline draws on omnidirectional camera functionality inspired by OmniCV-Lib.

The learned policy is a Diffusion Policy with a U-Net noise-prediction network and a DDIM scheduler. A deliberate design choice is that the policy is state-free: proprioceptive inputs such as end-effector pose and joint states are omitted entirely, so the model must localize using vision alone. This isolates the visual encoder's capability and, by extension, the fisheye's FoV advantage. Visual encoders differ by domain: ResNet-18 without pre-training in simulation (chosen for comparability with existing benchmarks) and CLIP ViT features in the real world (chosen for robustness to novel textures and lighting).

Evaluation uses two protocols. Simulation follows the standard Robomimic and MimicGen pipelines, reports Success Rate, selects the best-performing training checkpoint, and runs 50 rollouts. Real-world evaluation uses a normalized multi-stage score because binary success is too sparse for complex physical manipulation: each task is decomposed into typically 2–3 stages, a point is awarded per completed stage, and the reported Normalized Score is total points earned divided by total stages, averaged over N = 20 trials per setup. Before every rollout, the robot pose and all relevant object poses are reset to predefined initializations.

The simulation benchmarks are Robomimic and MimicGen. Six tasks are used: Square, Tool Hang, Coffee, Threading, Assembly, and Mug Cleanup, spanning high-precision manipulation (Tool Hang, Threading), spatial generalization (Square, Assembly), and long-horizon execution (Coffee, Mug Cleanup). Trajectory counts are 200 each for Square and Tool Hang (RoboMimic, Proficient-Human data) and 500 each for Coffee (MimicGen, broadened reset distribution), Threading, Assembly, and Mug Cleanup (MimicGen, default reset distribution). Simulation camera configurations are one or two wrist-mounted pinhole cameras at 90° FoV, or one or two wrist-mounted fisheye cameras at 235° FoV.

Real-world experiments use one Flexiv Rizon 4 robot arm with a DH AG-160-95 gripper, with data collected by teleoperation using Meta Quest 3. Camera configurations are a single wrist-mounted pinhole camera at 60° FoV or a single wrist-mounted fisheye camera at 180° FoV. Three tasks are used: Pick Cup (spatial generalization), Fold Towel (deformable object manipulation), and Hang Chinese Knot (high-precision rotational manipulation).

Training hyperparameters are held constant across comparisons. In simulation: 128×128 image resolution, batch size 16, 2000 training epochs, EMA decay 0.75, learning rate 1×10⁻⁴ for both U-Net and encoder, weight decay 1×10⁻⁶, AdamW optimizer, cosine decay schedule. In the real world: 224×224 image resolution, batch size 64, 500 epochs, EMA decay 0.9999, U-Net learning rate 1×10⁻⁴, encoder learning rate 1×10⁻⁵. Both domains use spatial softmax pooling, a conditional U-Net1D denoising network, relative action space, action horizon 8, observation horizon 2, and prediction horizon 16.

For the encoder probing experiment, an MLP head with two fully connected layers of 256 hidden units and ReLU activations is attached to the CLIP ViT encoder, and the whole network is fine-tuned with Mean Squared Error loss to regress the end-effector's 3D position and quaternion orientation. Rather than splitting existing data, the authors collected a dedicated test set of 30 additional trajectories per task setup, gathered under the same environmental settings as training data but held out strictly.

Why This Matters

Impact on research. The paper shifts fisheye cameras from an assumed-good default to an empirically characterized design choice. Rather than reporting that fisheye cameras work, it identifies the specific conditions under which the wide FoV pays off (feature-rich environments, high scene diversity) and the specific failure mode of the distortion (overfitting to absolute object scale). It also fills a concrete benchmarking gap: existing fisheye datasets lack robotic manipulation tasks, and popular robotics benchmarks omit fisheye streams. The released MuJoCo fisheye pipeline gives other groups a way to run comparable experiments.

Real-world applications:

• Large-scale robot data collection. Teams deciding what cameras to mount on teleoperation rigs and how to spend collection budget get direct guidance: prioritize visually complex, feature-rich environments and maximize background diversity.
• Fleet deployment and hardware upgrades. As robot hardware is upgraded over time, policies must stay backward-compatible with legacy data; RSA offers a simple training-time fix for surviving lens changes.
• Fine-tuning pretrained generalist policies. As fisheye-based pretraining datasets become more common, downstream users will inevitably deploy on custom setups with different intrinsics; the cross-camera results show this is partially addressable rather than hopeless.
• Simulation-based robot learning. The fisheye rendering pipeline extends standard benchmarks that previously had no fisheye support.

Industry relevance. The paper is aimed at anyone shipping vision-based manipulation policies, particularly organizations building generalist or foundation policies from large in-the-wild datasets. The guidance is data-centric and cheap to apply — choosing environments differently and adding one augmentation — which makes it practical for production data collection pipelines rather than only for research prototypes.

Future Directions

• Extending RSA beyond scale. The authors state that the cross-camera challenge is not insurmountable and that data-centric approaches are promising, but RSA addresses only the scale component. Whether other contributors to cross-lens transfer failure can be similarly addressed with augmentation is left open.

• Closing the simulation-to-real generalization gap. The scaling curve for scene diversity was less steep in simulation than in the real world, which the authors attribute to encoder differences (non-pre-trained ResNet-18 versus pre-trained CLIP) and lower simulated background complexity. Testing whether better simulated backgrounds or stronger encoders close that gap is a natural follow-up.

• Cross-camera transfer verified more broadly in the real world. The RQ3 experiments were conducted extensively in simulation because evaluating numerous hardware configurations physically is costly, with real-world verification deferred to the supplementary material. Expanding real-world lens diversity remains open.

• Applying RSA to real-world fine-tuning of pretrained fisheye policies at scale. The paper argues these findings lay a foundation for large-scale fisheye data collection, but the actual training of a generalist policy across many lens types is not demonstrated here.

Target Audience

• Robot learning researchers studying imitation learning, diffusion policies, or vision-language-action models who need to justify camera hardware choices.
• Practitioners building real-world manipulation systems who are choosing between fisheye and pinhole wrist cameras and need to know the operational trade-offs.
• Data collection and annotation teams designing teleoperation setups and deciding how much environmental diversity to capture.
• Simulation and benchmark developers who want to add fisheye rendering to existing manipulation benchmarks; the two-stage cubemap-to-equirectangular-to-fisheye pipeline is directly reusable.
• Readers with intermediate background in robotics or computer vision who want an empirical, numbers-driven answer to "should we use a fisheye camera?" rather than a purely architectural proposal.

Authors’ abstract

The adoption of fisheye cameras in robotic manipulation, driven by their exceptionally wide Field of View (FoV), is rapidly outpacing a systematic understanding of their downstream effects on policy learning. This paper presents the first comprehensive empirical study to bridge this gap, rigorously analyzing the properties of wrist-mounted fisheye cameras for imitation learning. Through extensive experiments in both simulation and the real world, we investigate three critical research questions: spatial localization, scene generalization, and hardware generalization. Our investigation reveals that: (1) The wide FoV significantly enhances spatial localization, but this benefit is critically contingent on the visual complexity of the environment. (2) Fisheye-trained policies, while prone to overfitting in simple scenes, unlock superior scene generalization when trained with sufficient environmental diversity. (3) While naive cross-camera transfer leads to failures, we identify the root cause as scale overfitting and demonstrate that hardware generalization performance can be improved with a simple Random Scale Augmentation (RSA) strategy. Collectively, our findings provide concrete, actionable guidance for the large-scale collection and effective use of fisheye datasets in robotic learning. More results and videos are available on https://robo-fisheye.github.io/