MvFFN: Multi-view Floor plan Feed-forward Network for Unposed Wide-Baseline Panorama Layout Reconstruction

AI summary

Overview

Research area: Computer vision for indoor 3D scene understanding — specifically structured floor plan reconstruction and camera pose estimation from sparse, unposed 360° panoramas.

Technical level: Advanced. The paper assumes familiarity with multi-view geometry, transformer architectures, feed-forward 3D reconstruction models (VGGT, DUSt3R), and panoptic/instance segmentation.

Scope: MvFFN proposes a feed-forward multi-view transformer plus a deterministic post-processing pipeline that reconstructs globally consistent multi-room floor plans and coarse camera poses from a handful of unposed wide-baseline panoramas, together with a new re-annotated benchmark (ZInD-CrossView).

What This Paper Is About

Existing floor plan reconstruction methods generally assume that camera poses are already known, that panoramas were captured with small baselines, or that a human will adjust the result afterward. This paper targets the harder, largely unaddressed setting of sparse, unposed, wide-baseline panoramas, where each view sees only part of the layout and geometry-only fusion produces duplicated walls, missing walls, and wrong junctions. The authors argue that predicting a complete multi-room floor plan as one global output is too brittle to learn, and instead recast the problem as predicting local, cross-view wall instances and within-room wall connectivity, which are then assembled into a global floor plan.

Key Contributions

1. MOIS (Multi-view Ordered Wall Instance Segmentation): A new task formulation that replaces direct global wall-sequence prediction with an anchor-conditioned local representation. Each image column acts as an anchor and predicts the wall instance it hits plus its K − 1 clockwise physically adjacent walls within the same room (the paper uses K = 3).

2. MvFFN architecture: A feed-forward multi-view transformer with two branches — a geometry encoder (VGGT-inspired, adapted to panoramic column tokens via an EfficientNet-v2 backbone and a Height Compression Module) that predicts 3-DoF poses, dense per-column floor boundaries, floor point maps, and wall-line parameters; and a floor-layout semantic branch that predicts the dense MOIS probability field. Training is two-stage (geometry first, then frozen-geometry semantic training).

3. Layout Chaining (LC): A deterministic post-processing pipeline that consolidates dense, redundant MOIS predictions into per-view wall instances, merges them into global wall identities via a cross-view same-wall graph, derives directed within-room adjacency into room-level wall sequences, and fits metric wall endpoints. The predicted pixel-to-wall correspondences also enable fully automatic downstream refinement (evaluated with planar bundle adjustment and BADGR).

4. ZInD-CrossView (ZInD-XV) dataset: A re-annotation of ZInD with globally unique wall instances and cross-view segmentation labels, containing 2,706 floors and 66,834 panoramas, plus a smaller auxiliary benchmark HM3D-XV (372 floors, 17,943 panoramas) for cross-dataset transfer to furnished, mesh-rendered scenes.

Main Findings

• Layout accuracy far exceeds modular baselines. On ZInD-XV 5-view, MvFFN reaches 88.5 wall precision, 87.7 recall, 88.1 F1 at 13.9 cm matched-wall distance, and 79.2 / 68.2 / 73.3 junction precision/recall/F1 at 13.8 cm. The strongest modular baseline (CovisPose + LGT-Net + LGT post-processing) reaches 75.6 / 71.4 / 73.4 wall F1 at 21.3 cm and 60.0 / 54.8 / 57.3 junction F1 at 21.3 cm.

• Gains hold at 8 views and are not just better poses. At 8 views MvFFN reports 89.4 / 87.5 / 88.4 wall F1 at 19.0 cm and 81.0 / 69.8 / 75.0 junction F1 at 19.1 cm, versus 76.3 / 72.7 / 74.5 and 61.7 / 56.7 / 59.1 for the LGT-PP baseline. MvFFN also outperforms the LGT-PP baseline given ground-truth poses (78.8 / 70.5 / 74.4 wall F1), indicating the improvement comes from the cross-view wall-instance formulation rather than pose accuracy alone.

• Learned top-down vectorization backends collapse in this setting. CovisPose + RoomFormer yields only 31.9 / 44.2 / 37.1 wall F1 at 84.9 cm on 5-view, and CovisPose + PolyRoom 33.6 / 29.3 / 31.3 at 86.6 cm, with junction distances of 137.8 cm and 129.7 cm respectively.

• Pose estimation is competitive but not the main target. Against CovisPose + GST on 5 views, MvFFN has slightly higher mean position error (12.5 cm vs 12.6 cm), far lower standard deviation (10.8 vs 23.3), but a higher median (10.2 vs 9.3 cm); at 8 views, 17.0 vs 17.3 cm mean, 15.6 vs 37.3 std, 13.8 vs 11.8 median. VGGT adapted to panoramas performs substantially worse (30.7 cm mean / 13.3 cm median at 5 views; 53.4 cm mean at 8 views), which is why it is excluded from layout evaluation.

• Planar bundle adjustment gives modest but broad improvement. It improves 68.0% of 5-view samples (mean distance 13.9 → 13.7 cm; on the improved subset 14.8 → 12.3 cm mean and 7.5 → 5.9 cm median) and 66.0% of 8-view samples (19.0 → 18.8 cm overall; 20.6 → 17.8 cm on the improved subset). BADGR improves only 45.9% (5-view) and 48.5% (8-view) and worsens aggregate error (13.9 → 17.7 cm; 19.0 → 21.0 cm), which the authors attribute to its assumption of complete initial layouts and clean correspondences.

• Architecture ablations confirm both design choices matter. Removing geometry-to-semantic fusion destroys recall (wall F1 36.5, recall 22.7, junction F1 24.6). A shared encoder degrades pose (5-view mean 35.8 cm, median 26.4 cm) and layout (wall F1 77.2 at 31.2 cm). Single-stage training is unstable (pose mean 46.8 cm, median 30.1 cm; wall F1 79.0 at 33.9 cm). The full two-stage model gives 88.1 wall F1 at 13.9 cm and 12.5 cm mean pose error.

• Inductive biases address data scarcity. Injecting rotation priors from the 2-view CovisPose model (globally aligned via a minimum spanning tree, following BADGR) into the angular encoder, and masking the first four global attention layers to a ±40-column window around precomputed 2-view correspondences, stabilizes 3-view-to-5-view training where naive training is reported as unstable.

Methodology in Plain English

The system takes a handful of panorama images of one floor — captured far apart, with no known camera positions — and produces a top-down floor plan plus rough camera placements.

Rather than asking a network to output an entire multi-room plan at once, the authors break the problem into many small, repeated questions. Every vertical column of pixels in every panorama is treated as an "anchor." For that anchor, the network answers: which wall does this ray hit, and what are the next two walls clockwise around the same room? Answering the same question from hundreds of columns produces heavily overlapping evidence, which is far easier to supervise than a single global plan and more tolerant of noisy labels.

The architecture has two parts. A geometry branch — modeled on VGGT, but operating on panoramic column tokens — compresses each panorama into feature columns, adds angular and camera-position encodings (including indoor vanishing-angle information to keep layouts axis-aligned), and runs a 12-layer alternating-attention transformer across views. It predicts poses, per-column floor boundaries, and floor point maps with uncertainty estimates. A second semantic branch, conditioned on the frozen geometry features, uses anchor-conditioned wall-slot queries and a four-layer cross-attention decoder to score, for every anchor and every column in every view, the probability of belonging to each of the three ordered wall slots.

Training is deliberately split in two stages because the geometry and semantic objectives compete: stage one trains geometry for 80k iterations, stage two freezes it and trains the semantic branch from scratch with a factorized loss (wall-presence plus ordered-slot classification).

Finally, Layout Chaining turns the dense probability field into a plan: group same-view current-wall masks into per-view wall instances, merge those across views into global wall identities, use the neighboring slots to infer which walls are adjacent around a room, extract room wall sequences as longest directed paths in a filtered room graph, and fit metric wall lines and corners from the point maps. Since the network already knows which pixels belong to which wall, standard bundle adjustment can then refine poses and geometry with no human input.

Why This Matters

Impact on research: The paper reframes multi-room layout estimation from a single global sequence-prediction problem into a dense local instance-prediction problem, and shows empirically that this reframing beats modular geometry-first pipelines even when those pipelines are handed ground-truth poses. The released ZInD-CrossView annotations — globally unique wall identities plus cross-view segmentation — fill a genuine data gap, since the authors note that datasets pairing structured multi-room layouts with camera-registered multi-view panoramas are scarce.

Real-world applications:

• Indoor digital twins and real estate capture: producing floor plans from a quick walkthrough with a consumer 360° camera, without survey equipment or manual CAD cleanup.
• Augmented reality and immersive walkthrough generation: pose-and-layout initialization for AR experiences where camera positions are unknown at capture time.
• Autonomous indoor navigation and robotics: coarse metric maps plus 3-DoF pose graphs for agents entering unfamiliar buildings.
• Scalable annotation pipelines: automatic pre-labeling of wall instances and room connectivity for large indoor datasets, reducing manual annotation effort.

Industry relevance: The pipeline is fully automatic end to end — no human-in-the-loop adjustment, no known poses, no small-baseline capture constraint — which is the practical requirement for deployment at scale in real-estate scanning, insurance, facilities management, and AR content creation. The demonstration that predicted correspondences support automatic bundle adjustment means the output can be refined without extra sensors.

Future Directions

• Improving global refinement. Planar bundle adjustment helps a majority of scenes but aggregate gains are modest (13.9 → 13.7 cm mean at 5 views), and BADGR degrades overall results. Better refinement formulations that tolerate incomplete layouts and imperfect correspondences are an open problem.
• Handling occlusion more explicitly. The paper acknowledges that layouts may remain open under occlusion and that small structural details are obscured by limited observations; extending the method to close rooms reliably under heavy occlusion is unresolved.
• Mitigating multi-view data scarcity. The reliance on 2-view CovisPose priors for rotation and correspondence is a workaround for limited N-view training data. Whether these priors can be removed as more multi-view data becomes available — or whether the priors limit ultimate accuracy — is untested here.
• Scale and generalization. Cross-dataset transfer is only partially explored (HM3D-XV results are relegated to supplementary material), leaving open how the approach behaves on non-residential or sparsely annotated scene types. The paper also does not report inference latency or compute cost, which matters for deployment.

Target Audience

Researchers and graduate students in computer vision working on indoor scene reconstruction, multi-view geometry, panoramic perception, or feed-forward 3D foundation models. It is also relevant to practitioners building AR, digital-twin, or robot-mapping products who need floor plans from unposed consumer captures, and to dataset curators interested in cross-view instance-level annotation of indoor panoramas. Readers should already be comfortable with transformer attention mechanisms and 3D vision terminology; the conceptual insight about local versus global prediction, however, is accessible to a broader technical audience.

Authors’ abstract

Reconstructing accurate camera poses and floor-plan layouts from sparse, unposed, wide-baseline RGB panoramas remains a challenging open problem. Direct multi-room floor-plan prediction is especially difficult in this setting: pose, geometry, semantics, and global layout structure are tightly coupled, yet a complete floor-plan is hard to learn as a single global output. Our key insight is to keep the model's predictions strictly local. Multi-view Ordered Wall Instance Segmentation (MOIS) is designed around this concept. Each panorama column predicts only the wall it observes plus the next two adjacent walls in clockwise order. The global wall partition, room layouts, and shared coordinates emerge from deterministic aggregation of overlapping local predictions. Building on MOIS, we propose Multi-view floor-plan Feed-Forward Network (MvFFN), which jointly predicts coarse camera poses, dense global geometry, and per-wall semantic outputs. A downstream Layout Chaining (LC) post processing pipeline then assembles these predictions into multi-room floor-plans with shared global coordinates, consistent wall identities, and connected room structure. The dense MOIS predictions recover cross-view wall identity and local within-room connectivity more accurately than composed modular baselines, and the assembled layouts substantially outperform prior methods on wall and junction level accuracy. We also introduce the ZInD-CrossView dataset, which augments ZInD with globally unique wall instances and cross-view segmentation labels for this task.