HouseLayout3D: A Benchmark and Training-Free Baseline for 3D Layout Estimation in the Wild

AI summary

Overview

Research area: Computer vision — 3D scene understanding, specifically vectorized (polygonal) layout estimation for indoor buildings.

Technical level: Intermediate. The paper is readable without deep prior knowledge, but familiarity with 3D reconstruction, semantic segmentation, and F1-style metrics helps when engaging with the method and evaluation details.

Scope: This paper introduces HouseLayout3D, a real-world benchmark of hand-annotated CAD layouts for multi-floor houses derived from Matterport3D, together with MultiFloor3D, a training-free pipeline that outperforms existing learned layout estimators on both the new benchmark and prior datasets.

What This Paper Is About

Current 3D layout estimation models are trained almost exclusively on synthetic data containing single rooms or single-floor apartments, so they cannot natively handle large buildings with multiple floors. The standard workaround — splitting a building into floors or rooms, predicting each part separately, and merging the results — destroys global spatial context, which matters for structures like staircases that physically connect levels. This paper provides both a benchmark of real multi-floor buildings and a training-free baseline that already beats existing state-of-the-art models on this harder setting.

Key Contributions

1. HouseLayout3D benchmark. The first real-world dataset for 3D layout estimation in large-scale, multi-floor buildings, with manual CAD annotations of walls, floors, ceilings, staircases, windows, and doors (including door opening directions). Each structural element is annotated as a 3D polygon, and per-vertex room IDs and object instances are inherited from Matterport3D.

2. MultiFloor3D baseline. A training-free method that combines modern 3D reconstruction (Gaussian Splatting + Poisson meshing), 2D semantic segmentation, superpoint clustering, polygon fitting, and 2D floorplan-based room extrusion to produce building-scale layouts.

3. Empirical exposure of existing method limitations. Extensive experiments showing that RoomFormer and SceneScript degrade sharply on multi-room, multi-floor buildings when evaluated per floor and per room.

4. New evaluation machinery. A generalized entity distance based on the Hausdorff distance between polygon surfaces, extending SceneScript's corner-based F1 metric to non-rectangular entities such as walls, floors, and ceilings with varying vertex counts.

Main Findings

• MultiFloor3D outperforms learned baselines despite requiring no training. On HouseLayout3D it reaches an F1@0.5 of 0.40 for structures versus 0.28 for SceneScript (per floor) and 0.24 for RoomFormer (per floor), and 0.38 average F1 versus 0.26 and 0.22 respectively.

• It is the only method that predicts stairs. MultiFloor3D achieves 0.42 F1@0.5 for staircases, a class the baselines cannot represent at all, which is directly tied to the multi-floor capability the paper is arguing for.

• Doors and windows are handled meaningfully. The method reports 0.55 F1@0.5 for doors and 0.43 for windows, whereas RoomFormer scores 0.07 and 0.07 and SceneScript 0.16 and 0.11 on doors and windows (per floor).

• Depth consistency also improves on a dataset without layout labels. On ScanNet++, MultiFloor3D reaches 67.8 (Δ5) and 84.7 (Δ10), compared to 55.1/68.5 for SceneScript and 36.8/48.9 for RoomFormer, though at roughly twice the vertex count.

• Both ablated stages matter substantially. Removing prototype fitting drops average F1 from 0.381 to 0.214; removing room segmentation drops it to 0.359; the raw skeleton plus mesh simplification only reaches 0.223.

• Baselines struggle most on large, open, multi-room areas. RoomFormer and SceneScript also cannot represent non-rectangular geometry such as sloped ceilings, and they perform better when given privileged per-room segmentation than per-floor.

• Cost is the clear trade-off. MultiFloor3D takes one to two hours per scene on an RTX 4090, versus one to two minutes for the feed-forward baselines.

Methodology in Plain English

The pipeline proceeds in four stages.

Reconstruct the scene. From unposed RGB images, the authors follow DN-Splatter: COLMAP provides camera poses, Metric3D provides monocular depth, a 3D Gaussian Splatting model is trained, and a Poisson surface mesh is extracted. The result is a dense triangle mesh with depth maps.

Extract a layout skeleton. The authors run OneFormer on the input images and map its classes into four semantic groups: structural components (walls, ceilings, floors, large furniture) which are kept; geometrically unreliable surfaces (windows, mirrors) which are discarded because depth estimation fails there; small objects, which are removed but used later; and stairs, handled separately. Labels are transferred to the mesh by back-projecting 5,000 sampled pixels per image onto the nearest vertices, then smoothed by majority voting within superpoint clusters. The structural subset becomes the skeleton.

Fit a layout prototype. The skeleton has holes where furniture occluded surfaces or where windows were removed. The authors initialize planar polygons from the skeleton superpoints, enforce vertex coplanarity, and optimize vertex positions and plane equations with three losses: a proximity term pulling skeleton vertices onto polygon surfaces, an emptiness term that penalizes polygons crossing rays known to pass through free space (derived from camera poses and depth), and a connectivity term pulling polygon vertices toward other polygons to close gaps. A simplicity term shrinks unshared edges. Periodically, nearby vertices are merged, polygons are simplified with Ramer-Douglas-Peucker, and near-coplanar adjacent polygons are combined. Holes are closed by projecting object meshes down onto floors and by extending wall and ceiling edges to the floor when the observed free space permits it.

Turn the prototype into a scene graph and then back into 3D. Floor-classified polygons identify building levels. Each level's 2D floorplan is formed by merging floor and ceiling polygons, then segmented into rooms using Hov-SG's algorithm, with openings narrower than 1.5 m labeled as doors and wider ones as generic openings. Connected components of the stair mesh become graph edges linking levels. Each room's 2D floorplan is then extruded to its ceiling: the floorplan is triangulated with a constrained Delaunay triangulation, each triangle is assigned to a ceiling polygon or plane via upward ray casting, and the result becomes a closed room shell with floor, ceiling, and wall geometry. Windows are detected by back-projecting window-classified pixels onto layout walls and clustering them with DBSCAN, fitting axis-aligned rectangles to clusters of at least 10 points.

Why This Matters

Impact on research. The paper reframes layout estimation from a per-room or per-floor problem into a building-scale one, and shows that a non-learned, geometry-driven method already surpasses large-scale learned models in that setting. This is a strong signal that current training data — not just architecture — is the bottleneck, and it provides both the data and the evaluation protocol needed to attack the problem.

Real-world applications:

• Indoor navigation and wayfinding. The authors demonstrate feeding the resulting scene graph (rooms as nodes, doors and stairs as edges) as JSON to an LLM, which returns turn-by-turn directions to a requested location.

• Robotics and embodied agents. Full-building layouts give mobile robots and assistive devices the global structural context needed for cross-floor planning, which per-room predictions cannot supply.

• Architectural modeling and digital twins. The CAD-style polygon annotations are directly useful for floorplan reconstruction, building information modeling, and as-built documentation from scans or photos.

• Generative scene synthesis and synthetic data. Large structural annotations provide a foundation for generating coherent multi-floor environments, which can in turn produce training data for other 3D perception models.

Industry relevance. Companies working on indoor mapping, AR/VR scene understanding, real-estate digitization, smart-home platforms, and warehouse or facility robotics all depend on recovering clean structural geometry from noisy scans. A benchmark with real multi-floor buildings and an open baseline lowers the barrier to measuring progress in that commercial setting.

Future Directions

• Learned models for full-building reasoning. The clearest open problem is training a feed-forward network that reasons across entire buildings rather than per floor, combining the speed of SceneScript or RoomFormer with the structural coverage of MultiFloor3D.

• Multi-floor training data at scale. Since the bottleneck appears to be data rather than architecture, generating or annotating large numbers of multi-floor layouts — possibly using HouseLayout3D as a seed for generative augmentation — is a natural next step.

• Closing the runtime gap. MultiFloor3D takes one to two hours per scene versus one to two minutes for learned baselines. Distilling its optimization-based fitting into an amortized model, or accelerating the reconstruction stage, would make deployment practical.

• Robustness to outdoor geometry and windows. The method occasionally fails to remove outdoor elements seen through large windows, introducing artifacts. Better handling of transparent surfaces and boundaries between interior and exterior is needed.

Target Audience

Researchers and engineers working on 3D scene understanding, indoor reconstruction, and layout estimation will get the most value, particularly those interested in scaling from single rooms to whole buildings. It is also relevant to practitioners building indoor navigation, digital twin, or robotics systems who need to understand the current state of structural extraction from scans. Graduate students entering the field will find the dataset statistics, the comparison table of existing benchmarks, and the training-free baseline to be a useful entry point, though the supplementary material is needed for full implementation details of the optimization and scene-graph construction steps.

Authors’ abstract

Current 3D layout estimation models are primarily trained on synthetic datasets containing simple single room or single floor environments. As a consequence, they cannot natively handle large multi floor buildings and require scenes to be split into individual floors before processing, which removes global spatial context that is essential for reasoning about structures such as staircases that connect multiple levels. In this work, we introduce HouseLayout3D, a real world benchmark designed to support progress toward full building scale layout estimation, including multiple floors and architecturally intricate spaces. We also present MultiFloor3D, a simple training free baseline that leverages recent scene understanding methods and already outperforms existing 3D layout estimation models on both our benchmark and prior datasets, highlighting the need for further research in this direction. Data and code are available at: https://houselayout3d.github.io.