HRoPE: Homography Relative Positional Encoding

AI summary

Overview

Research area: Computer vision — geometric positional encoding for multi-view transformers, evaluated primarily on feed-forward novel view synthesis.

Technical level: Advanced. The method builds on projective geometry (plane-induced homographies, PRoPE-style projection matrices, Sherman–Morrison inversion) and assumes familiarity with transformer attention and multi-view novel view synthesis.

Scope: The paper proposes HRoPE, a positional encoding that attaches a slanted 3D tangent plane to every patch token and modulates cross-view attention with the relative plane-induced homography between token pairs, then benchmarks it against view-, projection-, and ray-based encodings on RealEstate10K, CO3Dv2, Objaverse, and DL3DV.

What This Paper Is About

Transformers that compare image patches across multiple viewpoints are permutation-equivariant, so camera geometry has to be injected explicitly. Existing geometric positional encodings do this in a sensor-centric way: they describe the camera frustum a token is seen through (PRoPE) or the 3D point where a ray through the token terminates (RayRoPE, URoPE), but they never model the orientation of the physical surface the patch depicts.

HRoPE's goal is to make positional encoding surface-aware. Each patch is treated as a small slanted plane, following the slanted support-window assumption of classical multi-view stereo, and pairs of tokens are related through the homography that such a plane induces between their two views. This is meant to give attention a built-in prior for rectifying perspective distortion before comparing features.

Key Contributions

1. A stated principle of surface-aware positional encoding: the authors argue that local 3D surface geometry, not the sensor, is the natural anchor for comparing patch neighborhoods across views, and that surface orientation is the missing degree of freedom in prior encodings.

2. HRoPE itself: a per-patch relative encoding in which each token gets its own homography-structured transform, derived from a plane normal, inverse plane distance, rotation, and translation, with a closed-form inverse obtained via the Sherman–Morrison identity so that attention logits depend only on the relative factor between token pairs.

3. A geometry-free instantiation: the 13 per-patch scalars are regressed from patch features by a lightweight head trained jointly with the view-synthesis model under the view-synthesis objective alone, optionally using DINOv2 or VGGT features, plus a calibrated variant that composes PRoPE's projection matrix with a predicted per-patch tangent frame.

4. An empirical study across four datasets and several ablations: comparisons against LVSM (no geometric PE), GTA, PRoPE, RayRoPE, and URoPE, plus controlled ablations on transformation parameterization, feature source, calibrated vs. predicted geometry, oracle geometry, and out-of-distribution view settings.

Main Findings

• Best variant beats evaluated baselines on three datasets: with VGGT-predicted parameters, HRoPE reaches 20.025 dB PSNR / 0.647 SSIM / 0.462 LPIPS on CO3Dv2, 27.410 / 0.848 / 0.091 on RealEstate10K, and 31.622 / 0.937 / 0.057 on Objaverse, versus PRoPE's 19.495 / 0.626 / 0.583, 26.142 / 0.826 / 0.111, and 30.588 / 0.929 / 0.078 respectively.

• Gains over PRoPE are quantified: 0.53 dB, 1.27 dB, and 1.03 dB PSNR on CO3Dv2, RealEstate10K, and Objaverse, with higher SSIM and lower LPIPS in all three cases.

• Feature source matters monotonically: on RealEstate10K, HRoPE with a from-scratch learnable predictor reaches 26.419 PSNR, DINOv2 features 26.701, and VGGT features 27.410, so the gain does not depend on a single backbone but stronger geometric features add more.

• Not the top score everywhere: URoPE reports 26.836 PSNR / 0.838 SSIM / 0.099 LPIPS on RealEstate10K and 30.890 / 0.930 / 0.073 on Objaverse, exceeding the learnable and DINOv2 HRoPE variants on RealEstate10K; URoPE results on CO3Dv2 are not reported in the table.

• Scaling holds on DL3DV with a 53M LVSM: HRoPE gives 19.284 / 0.544 / 0.447 at 2 views, 20.373 / 0.624 / 0.270 at 4 views, and 20.940 / 0.635 / 0.257 at 8 views, versus PRoPE's 18.574 / 0.503 / 0.459, 19.957 / 0.602 / 0.289, and 20.653 / 0.621 / 0.271. PSNR and SSIM gains are largest with two views.

• The homography structure is the right inductive bias: among parameterizations compared after 80k training steps (depth term, SO(3), calibrated P·F, SE(3), HRoPE, unconstrained 4×4), HRoPE achieves the highest PSNR with both frozen VGGT features and a from-scratch predictor, and it outperforms the unconstrained 4×4 matrix despite having fewer degrees of freedom.

• Additive encoding is not equivalent: a 768-dimensional additive encoding built from the same VGGT features performs worse than the homography formulation, so injecting this geometry multiplicatively into attention matters.

• Local geometry alone helps: composing PRoPE's ground-truth projection with a predicted per-patch tangent frame (P·F) raises RealEstate10K PSNR from 26.142 to 27.115 and SSIM from 0.826 to 0.840, lowering LPIPS from 0.111 to 0.104; the fully predicted HRoPE then reaches 27.410 / 0.848 / 0.091.

• Oracle geometry gives a further nudge: on Aria Synthetic Environments, ground-truth planes and cameras reach 28.751 PSNR / 0.820 SSIM / 0.192 LPIPS versus 28.555 / 0.809 / 0.198 for VGGT-predicted parameters, with PRoPE at 27.829 / 0.802 / 0.264.

• Robustness to distribution shift: HRoPE retains its advantage when the target focal length is scaled from 1× to 5×, when target views are extrapolated 50% beyond the reference interval, and when input spacing shifts from the training range [25, 100] to [75, 150]; the calibrated P·F variant degrades more under large zoom.

• Transfer beyond novel view synthesis: the encoding transfers unchanged to a multi-view diffusion model and to the visual tokens of a spatial-reasoning MLLM, improving both tasks (reported in Appendix A.6).

Methodology in Plain English

The starting point is the classical homography relation: if a patch's surface lies on a plane with unit normal n at signed distance d from the camera, then points on that plane project into a view through a 3×3 matrix built from the camera intrinsics, rotation, and translation, with a rank-one correction involving n/d. Two views of the same plane are then related by the product of one such matrix and the inverse of the other.

HRoPE takes that construction and makes it per token. Each patch (i, u) gets its own transform built from a predicted rotation, translation, unit normal, and inverse plane distance κ — 13 scalars in total, with the rotation coming from a 6D continuous representation passed through Gram–Schmidt. The query of a patch is multiplied by the transform transposed, the key by its inverse, applied block-diagonally over 3×3 chunks of the head dimension, so the attention logit ends up depending on the relative factor between the two tokens. The remaining channels get ordinary 2D RoPE for relative image position.

Because every key needs an inverse, the authors avoid matrix inversion by decomposing the intrinsic-free part as a rotation plus a rank-one update and applying the Sherman–Morrison identity, giving both the forward and inverse transforms analytically in one pass. The denominator of that inverse equals the determinant and vanishes exactly when the camera center lies on the predicted plane, so the translation, κ, and the denominator are clamped for numerical stability.

The 13 parameters are produced by a lightweight regression head over patch features and trained only through the view-synthesis loss — no ground-truth depth, normals, or planes are required. A calibrated alternative keeps PRoPE's projection matrix and predicts only a local frame per patch, yielding a 4×4 composition that reduces exactly to PRoPE when all patches share one frame. Experiments use the LVSM backbone at roughly 25M parameters on CO3Dv2, RealEstate10K, and Objaverse with two reference views, and a roughly 53M variant on DL3DV with 2, 4, and 8 reference views following the RayZer protocol, reporting PSNR, SSIM, and LPIPS.

Why This Matters

Impact on research. The paper reframes what a geometric positional encoding should carry. Prior work progressively increased fidelity from image coordinates to full projection matrices to per-token ray termination, but all of it stayed sensor-centric. HRoPE shows that adding a first-order surface term — orientation, not just position — produces consistent gains, and that the benefit comes from the structure of the transform rather than from extra capacity or better features, since an unconstrained 4×4 matrix and a same-feature additive encoding both do worse.

Real-world applications (drawn from the paper's stated domains):

• Novel view synthesis and 3D content creation from sparse handheld imagery.
• Multi-view stereo depth estimation, where slanted support windows are the classical analogue.
• Camera and structure recovery from image collections.
• Camera-controlled video and world modeling, plus spatial reasoning in multimodal LLMs, both of which the authors test as transfer targets.

Industry relevance. The method needs no ground-truth geometry and trains under the view-synthesis objective alone, so it can be dropped into existing multi-view transformer pipelines as a positional-encoding swap rather than a new architecture. The observation that gains are largest with only two reference views is relevant to settings where input coverage is sparse, and the out-of-distribution results on zoom and baseline changes matter for deployed systems that encounter camera settings different from training.

Future Directions

• Enforcing cross-view consistency. The authors state as a limitation that plane parameters are predicted per patch and per view, so there is no formal guarantee that the same physical surface is encoded identically across views, unlike methods that enforce such uniqueness by construction. A hard guarantee is left as future work.

• Closing the gap to oracle geometry. Ground-truth planes and cameras still improve PSNR from 28.555 to 28.751 on the synthetic benchmark, so better surface estimation — through stronger 3D-aware backbones or additional supervision — is a clear route to further gains.

• Extending the transfer results. The improvements on a multi-view diffusion model and on a spatial-reasoning MLLM's visual tokens are reported in the appendix; characterizing how far this encoding scales across model families and tasks is open.

• Handling non-planar and mixed-geometry neighborhoods. The formulation assumes a local planar patch, and the authors note that for patches on different planes the relative factor is not guaranteed to represent a physical image warp, which invites finer surface models.

Target Audience

Researchers and engineers working on multi-view transformers, feed-forward novel view synthesis, and 3D-aware attention mechanisms — particularly those already familiar with PRoPE, GTA/CaPE, RayRoPE, or URoPE and interested in geometric inductive biases. It also suits practitioners in multi-view stereo and camera geometry who want to see classical slanted-support-window reasoning carried into learned attention. Readers without a background in projective geometry will find the method section demanding, though the motivation and empirical comparison are accessible.

Authors’ abstract

Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry. Such sensor-centric formulations, however, describe only where a token is observed from, leaving the rich surface geometry of the scene region depicted by the token. In this paper, we advocate a shift toward surface-aware positional encoding that explicitly grounds patch tokens in the geometry of physical surfaces. We introduce HRoPE, which represents the local geometry of each patch as a slanted 3D plane, following the slanted support-window assumption of classical multi-view stereo. We inject token-level, plane-induced homographies into attention as relative transformations. This equips the transformer with a geometric prior for rectifying perspective distortions and aligning patch features across viewpoints. Our encoding requires no ground-truth geometry: the plane parameters can be predicted end-to-end under a view-synthesis objective alone, or derived from foundation models. Extensive experiments on RealEstate10K, CO3Dv2, and DL3DV show that our best variant of HRoPE consistently outperforms the evaluated geometric positional encodings for novel view synthesis. Extensive ablations and analysis show the effectiveness of the proposed encoding. Please refer to our project page at: https://gaohchen.github.io/HRoPE