Raster2Seq: Polygon Sequence Generation for Floorplan Reconstruction

AI summary

Overview

Research area: Computer vision and computer graphics — specifically raster-to-vector floorplan reconstruction (also called raster-to-vector conversion), with ties to scene understanding, CAD workflows, and 3D scene generation.

Technical level: Intermediate. The paper assumes familiarity with transformer decoders, causal/masked attention, deformable attention, anchor-based prediction from object detection, and cross-entropy/L1 training objectives. The core idea, however, is describable without that background.

Scope (one sentence): The paper introduces Raster2Seq, a sequence-to-sequence framework that converts a rasterized floorplan image into a vectorized representation of labeled polygons (rooms, windows, doors), and evaluates it on Structured3D-B, CubiCasa5K, Raster2Graph, and WAFFLE.

What This Paper Is About

Floorplans are usually drawn as vector graphics in CAD software but are distributed as rasterized images, which destroys the underlying geometry and semantics. Raster2Seq reconstructs that lost information by treating reconstruction as a sequence-generation problem: given an image, the model autoregressively emits an ordered sequence of labeled polygon corners, one token at a time, until an end-of-sequence token is produced. The goal is to recover both the structure (accurate polygon outlines) and the semantics (which corners belong to a kitchen, bedroom, door, window, and so on) of complex real-world floorplans containing many rooms and irregular polygon shapes.

Key Contributions

1. A labeled polygon sequence representation for floorplans. Instead of intermediate elements like edges, corners, or room segments, the method represents each floorplan entity directly as a polygon whose corners are tokens of the form (x, y, semantic probability vector). Multiple polygons are concatenated with a <SEP> token and wrapped in <BOS> and <EOS> tokens, with a token-type probability vector over three categories (<CORNER>, <SEP>, <EOS>).

2. An anchor-based autoregressive decoder. The decoder combines masked (causal) attention, deformable attention, and a feed-forward layer. Learnable anchors act as spatial coordinates in image space; the model predicts residuals relative to these anchors rather than regressing raw continuous coordinates, and deformable attention samples a sparse set of relevant image locations around the normalized anchor points. A "FeatFusion" concatenation merges image features with coordinate-token embeddings into a fused vector of shape [L_I + L, D].

3. Token-level semantic supervision. A separate semantic cross-entropy loss supervises every corner token directly, and room-level labels are obtained at inference by majority voting across the token predictions in a polygon sequence. This contrasts with approaches that average corner embeddings over uniform-length sequences (which include padding corners) or that require per-corner neighbor-room annotations.

4. State-of-the-art results and demonstrated generalization. The method reports the best results across geometric (Room, Corner, Angle) and semantic (Room Semantic, Window & Door) F1 metrics on Structured3D-B, CubiCasa5K, and Raster2Graph, and shows strong zero-shot and cross-dataset generalization, including on WAFFLE. An ablation on Structured3D-B isolates the contribution of FeatFusion, learnable anchors, and left-to-right polygon ordering.

Main Findings

• Best overall results on structured and semantic metrics. On Structured3D-B, Raster2Seq reports Room / Corner / Angle F1 of 99.6 / 98.3 / 92.7, compared with RoomFormer at 95.1 / 91.7 / 83.2, PolyRoom at 98.9 / 96.0 / 91.9, FRI-Net at 96.5 / 85.4 / 83.3, and HEAT at 94.7 / 84.5 / 79.6. Room Semantic is 76.9 versus RoomFormer's 74.2, and Window & Door is 98.5 versus RoomFormer's 94.1.

• Largest advantages on the more complex dataset. On CubiCasa5K, Raster2Seq reports 88.7 / 59.4 / 37.4 for Room / Corner / Angle, versus RoomFormer at 83.5 / 55.5 / 34.1, HEAT at 78.2 / 53.7 / 32.3, FRI-Net at 77.1 / 50.8 / 38.0, and PolyRoom at 54.1 / 37.1 / 23.0. On this dataset Room Semantic is 63.8 (RoomFormer: 63.0) and Window & Door is 77.8 (RoomFormer: 78.5).

• Strong results on Raster2Graph's own benchmark. Raster2Seq reports 97.0 / 80.3 / 66.6 for Room / Corner / Angle and 85.1 Room Semantic, compared with Raster2Graph at 95.0 / 78.3 / 67.3 and 83.4, RoomFormer at 91.9 / 74.5 / 51.1 and 79.5, FRI-Net at 91.5 / 72.3 / 52.8, HEAT at 95.9 / 79.7 / 50.9, and PolyRoom at 56.9 / 42.4 / 23.8.

• Baselines vary sharply across datasets. PolyRoom and FRI-Net perform very well on the simpler Structured3D-B dataset but drop substantially on CubiCasa5K and Raster2Graph, where polygon lengths are more diverse and shapes are irregular. The authors attribute this to PolyRoom's reliance on segmentation proposals and FRI-Net's dependence on line assembly to form rooms. Raster2Seq is described as strong and stable across all three datasets.

• Fixed query budgets limit competing methods. RoomFormer operates with a fixed number of room queries (the paper gives an example of 2800). Exceeding that capacity causes out-of-memory errors and increased computation due to quadratic attention costs. Raster2Seq instead generates ordered, non-redundant output sequentially and is not limited by a fixed number of predictions.

• Performance gap widens with floorplan complexity. The paper reports Room F1 for RoomFormer, FRI-Net, and Raster2Seq across varying numbers of polygons and corners on Structured3D-B and CubiCasa5K. The models perform similarly on simpler cases, but the baselines show a notable drop in complex scenes with over 15 polygons or 150 corners.

• Generalization to unseen real-world data. Cross-evaluation heatmaps over training/test dataset combinations (using RoomF1 for CubiCasa5K and Raster2Graph, and IoU for WAFFLE) show Raster2Seq outperforming FRI-Net and RoomFormer across settings, with significant gaps on the WAFFLE test set. WAFFLE contains 20K real-world floorplan images scraped from the Internet, of which only approximately 100 are annotated, so evaluation there is zero-shot.

• Every architectural component helps. The ablation on Structured3D-B (all models trained for 1,350 epochs) shows Room / Corner / Angle F1 of 94.1 / 91.1 / 82.0 for the base model, 96.3 / 93.7 / 82.6 with FeatFusion, 97.4 / 95.3 / 86.0 adding learnable anchors, and 99.6 / 98.3 / 92.7 adding left-to-right ordering. The authors state these ablation numbers do not reflect the best results the model can achieve.

• Qualitative advantages. RoomFormer is reported to produce "short-cut" triangular polygons, while Raster2Graph often fails to recover complete floorplan structures. Raster2Seq is described as reconstructing structure more accurately.

• A stated limitation on doors and windows. The model occasionally fails to localize windows and doors accurately, producing artifacts such as cross-over windows placed inside rooms.

Methodology in Plain English

The input is an RGB floorplan image I of size H × W × 3, encoded by a feature extractor into an image feature vector f_img of shape L_I × D. The feature extractor is a ResNet-50 backbone followed by a transformer encoder, initialized with ImageNet pretrained weights and finetuned end-to-end.

The output is a single flat sequence. Each polygon corner becomes a token carrying a spatial position and a semantic probability vector over C categories. Rooms, windows, and doors are all treated as polygons; windows and doors are simply two extra semantic categories on top of room types. Polygons are separated by a <SEP> token, and the whole sequence is bracketed by <BOS> and <EOS>. Because the model regresses continuous values rather than using a discrete tokenizer, each token also carries a probability vector over three token types: <CORNER>, <SEP>, or <EOS>.

The decoder is the core of the system. Continuous 2D coordinates are quantized into a discrete 1D embedding space using a learnable codebook C of size H_b × W_b × D (quantization bins × embedding dimension). Three inputs feed the decoder: image features, the coordinate token sequence (f_poly of shape L × D), and learnable anchors v_anc of shape L × 2.

Inside the decoder, a masked attention layer applies a causal mask so each token can only attend to preceding tokens, encouraging left-to-right generation. The query vector combines coordinate-token embeddings with positional embeddings from the anchors; keys and values come from the fused image-plus-token feature vector (FeatFusion), which the authors say is crucial for precise coordinate regression, since image features act as a prefix every token can attend to. The output then goes to a deformable attention module, which normalizes anchors to [0, 1] with a sigmoid, predicts offsets with a linear layer, adds those offsets to the anchor points, and samples only a small set of points around each reference rather than the whole feature map. Three lightweight heads sit on top of the last autoregressive block: a token head for token type, a semantic head for labels, and a coordinate head that produces residual offsets combined with the learnable anchors to yield continuous coordinates.

Training uses three losses: an L1 coordinate loss over non-padded tokens (masked), a cross-entropy token-type loss over the three token classes, and a cross-entropy semantic loss over the C predefined room classes. The total loss is a weighted sum with coefficients λ_coord, λ_token, and λ_sem. To induce a geometric inductive bias, polygons are ordered during training by top-left coordinates using top-to-bottom, left-to-right scanning priority. At inference the model predicts tokens until <EOS>, and room labels are decided by majority voting — taking the highest-probability class per token and then the most frequent class across the sequence. Within each room, corners are predicted in counterclockwise order.

Data and evaluation setup. Four datasets are used. Structured3D contains 3,000/250/250 training/validation/test samples with 16 room types and is converted into binary raster images (Structured3D-B) using ground-truth annotations. CubiCasa5K has 4,199/399/399 samples with 11 classes and is preprocessed into separate floorplan instances, increasing the described dataset size from 5,000 to 6,281 samples (5,267 train / 503 val / 511 test); the paper states 10 semantic room classes are selected (Outdoor, Wall, Kitchen, Living Room, Bed Room, Bath, Entry, Railing, Storage, Garage, Undefined) along with Window and Door. Raster2Graph has 9,803/500/499 samples with 12 classes. WAFFLE is evaluated zero-shot only. All experiments use a fixed 256 × 256 resolution.

Metrics. Predictions are matched to ground truth by room IoU (valid match above 0.5), and metrics are computed at room, corner, and angle levels as Precision, Recall, and F1. A corner counts as correctly recovered if its L2 distance is within 10 pixels and the angle difference is under 5 degrees. Room type predictions are additionally used for semantic matching. WAFFLE is evaluated with IoU for the segmentation task.

Baselines. HEAT, RoomFormer, and FRI-Net — originally designed for point-cloud density maps — are finetuned for raster input. Raster2Graph is compared only on its own dataset because it requires per-corner neighboring room class annotations. PolyRoom is also included in the comparison table.

Why This Matters

Research impact. The paper reframes raster-to-vector floorplan conversion from detector-style, fixed-budget prediction into an autoregressive sequence-to-sequence problem, which naturally handles variable-length polygons, dense layouts, and non-redundant ordered outputs without corner sampling, image augmentation, or post-processing to extract semantics. It also reports that methods built around fixed query counts degrade when complexity exceeds capacity, motivating sequence generation as an alternative paradigm for structured geometric output.

Real-world applications named or implied by the paper:

• Automated editing of floorplans in CAD-style workflows, which the authors list alongside floorplan understanding and generation as tasks limited by rasterization.
• Floorplan-guided 3D reconstruction and controllable 3D scene generation — the paper demonstrates using its vectorized floorplan as guidance for generating controllable 3D scenes, and Figure 1 shows 2D boundaries extended vertically into 3D visualizations.
• Modeling historical buildings defined by hand-drawn floorplans, motivated by the Café Helms example and the real-world Internet images from WAFFLE.
• A VLM-based vectorization refinement step that builds on the polygon sequence representation to further improve reconstruction accuracy, suggesting a route to integrating higher-level reasoning modules.

Industry relevance. The DOI and venue (SIGGRAPH Conference Papers '26, published in ACM Transactions on Graphics) place this work at the intersection of graphics and computer vision, where vector outputs are directly usable by design, architecture, and construction tooling. Accurate vectorization plus semantic labels is described as a prerequisite for computational tasks over floorplans, and the authors argue that recovering accurate vectorized representations will become more important as generative models grow more powerful.

Future Directions

• Better handling of doors and windows. The paper's stated limitation is inaccurate localization of these less prevalent semantic structures, producing artifacts such as cross-over windows. The authors suggest tailored architectural changes or modeling these elements separately from room entities.

• Robustness to out-of-distribution data. The authors propose incorporating mechanisms such as appearance-based augmentations to improve results on in-the-wild internet floorplans.

• Open-vocabulary semantics. Combining the system with open-vocabulary predictions could allow reconstructing the richer semantics reflected in diverse real-world floorplans.

• Conditioned decoding. The authors suggest explicitly incorporating semantic conditions during inference, for example through a lightweight condition adapter, enabling controls such as using input room semantic labels to steer decoding toward desired room coordinates.

Beyond these, the appendix points to further analyses the paper opens up, including training on density map inputs rather than rasterized floorplans, robustness to noisy density map inputs, runtime comparisons, and ablations over sequence length, quantization resolution, and the coordinate loss coefficient.

Target Audience

Researchers and graduate students working on floorplan understanding, raster-to-vector conversion, structured prediction with transformers, and generative models for architectural or 3D scenes. It is also relevant to practitioners in CAD, architecture, construction, and real-estate tooling who need vectorized, semantically labeled floorplans as inputs to downstream editing, analysis, or 3D generation pipelines. Readers should be comfortable with attention-based models and standard detection-style evaluation metrics; the paper is not an introductory treatment of those topics.

Authors’ abstract

Reconstructing a structured vector-graphics representation from a rasterized floorplan image is typically an important prerequisite for computational tasks involving floorplans such as automated understanding or CAD workflows. However, existing techniques struggle in faithfully generating the structure and semantics conveyed by complex floorplans that depict large indoor spaces with many rooms and a varying numbers of polygon corners. To this end, we propose Raster2Seq, framing floorplan reconstruction as a sequence-to-sequence task in which floorplan elements--such as rooms, windows, and doors--are represented as labeled polygon sequences that jointly encode geometry and semantics. Our approach introduces an autoregressive decoder that learns to predict the next corner conditioned on image features and previously generated corners using guidance from learnable anchors. These anchors represent spatial coordinates in image space, hence allowing for effectively directing the attention mechanism to focus on informative image regions. By embracing the autoregressive mechanism, our method offers flexibility in the output format, enabling for efficiently handling complex floorplans with numerous rooms and diverse polygon structures. Our method achieves state-of-the-art performance on standard benchmarks such as Structure3D, CubiCasa5K, and Raster2Graph, while also demonstrating strong generalization to more challenging datasets like WAFFLE, which contain diverse room structures and complex geometric variations.