PGGT-Loc: Primitive-Grounded Geometry Transformer for Feed-Forward Camera Localization

AI summary

Overview

Research area: Computer Vision / 3D Vision — specifically 6-DoF camera localization against structured, non-photorealistic maps (dense planar surface maps and minimal 3D layout models).

Technical level: Advanced. The paper assumes familiarity with vision transformers, DUSt3R-style multi-view architectures, plane-based map representations, PnP/RANSAC registration, and render-and-compare localization.

Scope (one sentence): The paper proposes a new localization paradigm — "primitive-grounded localization" — and instantiates it as PGGT, a cross-modal transformer that takes a single query image plus a set of map primitives and predicts camera pose feed-forward, evaluated on ScanNet, 12Scenes, and Structured3D.

What This Paper Is About

Most camera localization systems need dedicated maps built from imagery (triangulated point clouds, NeRF/3DGS, meshes), which are expensive to construct, store, and maintain. Meanwhile, structured maps made of compact, human-interpretable primitives — planes, walls, floors, doors — are often already available, but the "modality gap" between such primitives and pixels has kept them from being used directly for pose estimation. The paper's goal is to close that gap: given one query image and a set of planar map primitives from a previously unseen scene, predict the 6-DoF camera pose in a single forward pass, without matching, rendering, or per-scene training.

Key Contributions

1. A new concept: primitive-grounded localization. The task is reframed as grounding query observations in map primitives, rather than solving pose from explicit 2D–3D correspondences or matching the query against exhaustive renderings of the map.

2. PGGT, a cross-modal transformer network. A two-branch architecture (inspired by DUSt3R) ingests the query image and map primitives as direct inputs and predicts multiple localization targets — patch–primitive assignments, primitive parameters, per-pixel camera rays, and poses — enabling feed-forward 6-DoF pose inference.

3. Three pose-inference pathways from one network. The predicted targets support a Param.-based route (point-to-plane registration), a Ray-based route (pose from the predicted ray map), and a Token-based route (pose decoded directly from a prepended camera token), letting the same base model be used in different ways.

4. One shared architecture for two map types. The same base model handles dense surface maps and minimal layout models, the latter via two adaptations (semantics injection and room-isolation injection), surpassing prior work on both.

Main Findings

• State-of-the-art on dense surface maps (ScanNet). The Ray-based variant reaches mean/median rotation errors of 7.5°/2.7° and translation errors of 0.25 m/0.14 m, with pose recall of 69.0 / 92.7 / 95.4 % at the (0.2 m, 10°), (0.5 m, 15°) and (1.0 m, 30°) thresholds. The strongest prior pipeline, PlanaReLoc +Refine, reports 17.2°/3.8°, 0.60 m/0.20 m, and 48.5 / 73.1 / 81.8 %.

• 28.6% average relative gain in pose recall. The authors report this average relative improvement across the three pose recall metrics over PlanaReLoc +Refine on ScanNet.

• Fast inference. The Ray-based model runs at approximately 0.107 s per query on a single NVIDIA RTX 3090 GPU, compared with approximately 0.544 s for PlanaReLoc +Refine and approximately 14.19 s for FreeReg.

• Holds up out-of-domain (12Scenes). With 1023 queries, the Ray variant achieves median errors of 3.6° / 0.15 m, pose recall of 68.6 / 90.1 / 93.8 %, and primitive-matching Precision 77.7 %, Recall 74.3 %, F1 76.0 %, AP 92.4 % — versus 4.8° / 0.28 m, 34.9 / 66.7 / 79.9 %, and 63.9 / 54.2 / 58.6 / 87.8 % for PlanaReLoc.

• Strong on minimal layout models (S3D-official). Ray-based PGGT reports 11.2 cm / 0.42° median 2D errors and pose recall of 40.9 / 91.3 / 93.6 / 93.5 %, compared with SPVLoc's 12.9 cm / 1.26° and 33.5 / 81.6 / 84.5 / 84.2 %. The paper notes it is the only method that is both feed-forward (no exhaustive map rendering) and free of FoV priors.

• Also leads on S3D-resampled. At 90° FoV: 0.38° yaw / 8.4 cm 2D translation error, recall 56.0 / 94.5 / 95.6 / 95.5 % versus SPVLoc's 0.83° / 9.2 cm and 50.1 / 89.7 / 92.1 / 91.9 %. At 120° FoV: 0.49° / 11.9 cm and 40.9 / 91.7 / 96.4 / 96.3 % versus 0.87° / 11.7 cm and 39.5 / 88.0 / 92.1 / 91.9 %. Note that SPVLoc is better on 3D translation error at 120° FoV (11.7 cm vs. 14.7 cm) and on 3D rotation at 90° FoV (0.83° vs. 0.81° is actually in PGGT's favor; at 120° FoV SPVLoc's 0.87° beats PGGT's 0.95°).

• Auxiliary targets help. Removing the assignment loss drops ScanNet pose recall at (0.2 m, 10°) from 62.0 % to 54.7 %; removing the primitive-parameter loss gives 54.9 %; removing the pose-decoding loss gives 59.7 % (vs. 62.0 % full). Matching F1 falls from 81.0 % to 80.0 % without the parameter loss.

• Layout adaptations are synergistic. On S3D-resampled-90°, base Ray performance is 0.69° yaw / 15.2 cm with recall@(1 m, 30°) of 90.5 %. Semantics alone hurts (0.88° / 17.3 cm, 91.6 %), isolation alone helps (0.53° / 11.2 cm, 93.8 %), and the two combined give the best result (0.46° / 9.3 cm, 95.5 %).

• Explicit plane parameters are optional. On ScanNet, removing the primitive positional encoding gives 3.2° / 0.17 m; removing both the positional encoding and the parameter loss gives 3.4° / 0.19 m, versus 3.0° / 0.16 m for the full model. Coarse recall stays largely unchanged, which the authors read as evidence that grounding need not depend on explicit plane parameters — a hint toward non-planar primitives.

• Multi-room scaling is only partially reported. The paper states an evaluation following the Integrated Rooms protocol (12Scenes rooms arranged on a 2D grid with 5 m cells, progressively merged), but the content provided is truncated and does not report the resulting numbers.

Methodology in Plain English

Set up the two inputs. A query image is cut into patches and encoded with a frozen DINOv2 ViT-Large into 1200 tokens. Each map plane is turned into one token: 1024 points sampled on the plane go through a PointNet encoder to capture shape and position, and the plane's own equation (normal and offset) is passed through a small MLP to act like a positional code.

Let the two modalities talk. The tokens run through two parallel stacks of 12 transformer decoder layers, one stack per modality, with cross-attention in every layer letting image patches and plane tokens exchange information. A DPT layer turns patch tokens back into pixel-aligned dense feature maps.

Train on four prediction targets at once. (1) A patch→plane assignment matrix built from descriptor similarity between the two branches, scaled by per-token confidence scores and normalized with a softmax over primitives; patches above a confidence threshold τ are grounded to their best-matching primitive. (2) Plane parameters, predicted as the 3-vector n/d, anchored in the query camera's coordinate frame so the reference stays consistent across samples. (3) A per-pixel ray map of 6D vectors (direction plus origin in the map frame); averaging ray origins yields translation, and decomposition of a ray-direction homography yields rotation and intrinsics. (4) Poses read off two prepended tokens — a camera token and a map token, the latter supervised toward the identity pose — decoded through a shared MLP.

Pick a pose route at inference. The Param. route solves a point-to-plane registration using the assignments, predicted parameters, and map primitives. The Ray route derives pose from the ray map. The Token route reads pose straight off the camera token. All three come from the same trained network.

Train and scale data. The multi-task loss sums a negative log-likelihood term on assignment, an L1 plus cosine-similarity term on parameters, and L1 terms on rays and pose. Training uses roughly 400K query–map pairs from ScanNet, ScanNet++, and Matterport3D (over four thousand indoor scenes/regions), with strict scene separation from evaluation. Optimization uses AdamW with a cosine schedule at peak learning rate 2×10⁻⁴, 125K iterations, bfloat16, batch size 128, and about 2 days on 8× H20 GPUs.

Adapt for layout maps. Layout primitives carry one of five labels (wall, floor, door, window, ceiling), so five learnable semantic embeddings are fused into the primitive tokens and an auxiliary segmentation loss supervises the query branch. Because layout primitives are organized per room, attention masks in the map branch's self-attention prevent tokens from attending across rooms, matching the convention that doorways are treated as opaque. This model is trained from scratch on 3250 Structured3D layout models with queries sampled from 21,835 panoramic images at 45°–135° FoV, using 256 points per primitive.

Why This Matters

Impact on research. The paper argues that high-level map primitives — already available in many pipelines and far cheaper than photorealistic reconstructions — are an underexploited resource for localization. By showing a single feed-forward transformer can beat both a plane-matching pipeline (PlanaReLoc) and a render-and-compare panorama-retrieval pipeline (SPVLoc), it reframes the problem from "match or render" to "ground," and supplies a shared base architecture across two very different map types.

Real-world applications (as motivated in the paper):

• Augmented reality, where devices need fast, low-cost pose against structured indoor maps rather than photorealistic ones.
• Robot navigation, where floorplans and layout models are more readily maintained than dense visual reconstructions.
• Multi-agent collaboration, where agents must share a common spatial reference frame.
• Indoor environments with CAD/floorplan-style assets, i.e., the building models and 2D floorplans the paper surveys in related work, which avoid texture-heavy map construction and maintenance.

Industry relevance. Maps that are cheap to store and maintain, plus inference around 0.107 s per query on a single RTX 3090, make the approach attractive for deployment where re-scanning a building is impractical. The method also requires no camera-intrinsics prior on the layout task (marked † in the results table), which reduces calibration burden on commodity devices.

Future Directions

• Non-planar primitives. The ablation showing that removing both plane-specific components barely changes coarse recall suggests the framework could extend beyond planes; the authors explicitly motivate this, but no non-planar results are reported.
• Scaling to larger, multi-room maps. The paper begins an Integrated Rooms evaluation (12Scenes rooms on a 5 m grid, progressively merged), but the provided content is truncated and does not report those outcomes — a natural open question is how the approach degrades as maps grow and rooms accumulate.
• Making layout adaptations converge better. Semantics injection alone reduced accuracy (0.88° / 17.3 cm vs. 0.69° / 15.2 cm base), which the authors attribute to slower convergence from semantic fusion; better fusion or training schedules could unlock more of the semantic signal.
• Closing the remaining 3D-error gap on wide-FoV layout queries. On S3D-resampled at 120° FoV, PGGT trails SPVLoc on 3D rotation (0.95° vs. 0.87°) and 3D translation (14.7 cm vs. 11.7 cm), even while leading on 2D metrics and recall — an accuracy gap worth investigating.

Target Audience

Researchers and engineers working on visual localization, 3D vision, SLAM/AR/robotics perception, and cross-modal representation learning — particularly those interested in localization against non-photorealistic maps (CAD, floorplans, layout models, planar surface reconstructions). Readers should be comfortable with transformer architectures and camera geometry; the paper is not an introductory treatment. Practitioners weighing map-maintenance costs against localization accuracy will also find the benchmark comparisons and inference-time numbers directly useful.

Authors’ abstract

Camera localization methods are mostly multi-stage pipelines relying on dedicated and complex maps that are costly to construct, store, and maintain over time. Alternatively, well-structured maps that organize scenes with compact yet expressive primitives are often readily available, providing rich structural and semantic cues that humans can interpret. However, their potential for direct camera localization remains largely unexplored due to the significant modality gap between map primitives and visual observations. This motivates a new concept for localization, which estimates camera poses by grounding query observations to map primitives. In this paper, we instantiate the concept in the context of planar primitives with PGGT, the Primitive-Grounded Geometry Transformer network. Given a single query image and a set of planar map primitives, the network is trained to predict multiple localization targets for feed-forward 6-DoF camera localization that generalizes to unseen environments. With this straightforward and scalable architecture, our method pushes the state-of-the-art localization performance on both dense surface maps and minimal layout models. The code and models are available at https://github.com/3dv-casia/PGGT-Loc