From Geometry to Function: A Multi-Task LiDAR Benchmark Dataset for Semantic and Functional Parsing of Urban Facades

AI summary

Overview

Research area: Computer vision and 3D scene understanding — specifically LiDAR point cloud parsing of building facades, with ties to urban digital twins, BIM (Building Information Modelling), and architectural knowledge representation.

Technical level: Intermediate. The paper is readable with a general machine-learning background, but the benchmark results assume familiarity with point cloud segmentation architectures, instance segmentation metrics (mAP, AP50, AP25), and standard evaluation conventions.

Scope: The paper introduces HFX3D, a multi-task LiDAR facade benchmark of roughly 492 million points from 26 Halifax buildings that adds per-instance functional attributes on top of semantic and instance labels, and benchmarks seven semantic segmentation backbones, learned and clustering-based instance methods, cross-dataset transfer to ZAHA, and functional attribute prediction.

What This Paper Is About

Existing 3D facade benchmarks stop at geometry and semantic labels: they can tell you a region is a window, but not whether that window is operable, whether a balcony provides access, or whether a column is load-bearing. The authors argue that this functional layer is what actually drives structural analysis, energy modelling, and BIM reconstruction, and that no existing LiDAR facade dataset supplies it. HFX3D is built to close that gap by annotating every facade element at three tiers — semantic class, instance identity, and a 15-attribute functional vector — and by testing how well current models handle each tier.

Key Contributions

1. A three-tier facade benchmark. To the authors' knowledge, HFX3D is the first and largest RGB-coloured LiDAR facade benchmark to provide semantic, instance, and per-instance functional annotations: approximately 492 million points from 26 buildings across five Halifax neighbourhoods, averaging 18.9 million points per building, captured across four seasons, day and night, and varied weather.

2. A semi-automatic annotation pipeline. A scalable procedure that pairs geometry-guided instance generation (DBSCAN clustering tuned per building, then manual refinement) with IFC- and Getty AAT-grounded functional labelling, refined by geometric and contextual evidence and validated by human annotators using scanner imagery and street-level imagery.

3. A unified multi-task benchmark. Full-coverage evaluation of seven semantic backbones (PointNet, PointNet++, DGCNN, KPConv, RandLA-Net, Point Transformer v2, and Point Transformer v3) and seven instance methods spanning three learned approaches (PointGroup, ISBNet, SoftGroup) and four classical clustering baselines (DBSCAN, HDBSCAN, MeanShift, KMeans), reported in separate tables because the clustering baselines receive ground-truth semantics.

4. A cross-city generalization study against the ZAHA facade benchmark, remapping both datasets to a shared 11-class taxonomy and training with xyz-only input for a feature-neutral comparison, plus an evidence ablation for functional attribute labelling.

Main Findings

• Semantic segmentation is far from solved, and class frequency dominates. KPConv leads all seven backbones at 47.26% mIoU, 86.64% OA, 67.06% mP, 53.54% mR, and 57.15% mF1. The next best is PointNet++ at 29.19% mIoU, followed by Point Transformer v3 (28.18), PointNet (26.30), Point Transformer v2 (20.65), RandLA-Net (16.94), and DGCNN (15.37).

• Architecture ordering differs from indoor benchmarks. PointNet and PointNet++ outperform Point Transformer v2 on HFX3D, and DGCNN ranks last — a different ordering than is commonly observed on S3DIS. The authors suggest class imbalance may partly explain why architectural sophistication does not translate into higher overall performance.

• Rare classes collapse. Arch accounts for only 0.3% of points and receives negligible IoU across all models. KPConv's per-class IoU ranges from 93.94 on vegetation and 91.59 on terrain down to 0.00 on roof and 0.01 on arch. Window is the hardest of the frequent classes for every backbone, and residual annotation disagreements in the second review round were most often wall/window confusion.

• Learned instance segmentation is weak and pipeline-sensitive. SoftGroup leads with 0.2032 mAP, 0.2923 AP50, and 0.4044 AP25, ahead of ISBNet (0.0422 mAP, 0.0883 AP50) and PointGroup (0.0145 mAP, 0.0427 AP50). The authors trace much of this to evaluation mechanics: scoring subsampled predictions against full-resolution ground truth caps mask IoU, giving ground-truth masks only 0.31 AP50 on one validation building versus 0.91 when masks are propagated to every point; they therefore propagate masks and group PointGroup and SoftGroup at 7.5 cm.

• Failures are class-specific. PointGroup over-fragments — the 52 trees of one test building receive 358 confident vegetation predictions. ISBNet scores well only on balconies (0.54 AP50). Column and stairs, with only 14 and 5 test instances, supply about 70% of SoftGroup's mAP. SoftGroup's window AP falls from 0.56 at IoU 0.25 to 0.22 at IoU 0.50.

• Density clustering given oracle semantics is still limited. HDBSCAN reaches 0.2533 mAP, 0.3430 AP50, and 0.4521 AP25, ahead of DBSCAN (0.2725 AP50), MeanShift (0.0871 AP50), and KMeans (0.0615 AP50). HDBSCAN recovers well-separated elements such as balconies (0.62 AP50) and columns (0.66 AP50) but not windows (0.05 AP50), because density clustering merges neighbouring panes.

• Cross-dataset transfer is asymmetric. KPConv leads every cross-dataset setting, but training on HFX3D and testing on ZAHA loses 29.5 percentage points, while the reverse direction loses only one. Every model tested on ZAHA does worse when trained on HFX3D instead. Tested on HFX3D, transfer is architecture-dependent: Point Transformer v2 actually improves when trained on ZAHA, and Point Transformer v3 is essentially unaffected. The authors report this as an observation rather than evidence of superior robustness, attributing it to HFX3D's broader variability, label harmonization, and annotation-convention differences.

• Mixed training has no consistent winner. Combining both datasets helps DGCNN and Point Transformer v3 on HFX3D but hurts KPConv, PointNet, and RandLA-Net, with a smaller effect either way on ZAHA.

• Functional labelling is the hardest task. On the Ambiguous subset, the semantic prior alone reaches 0.4808 mF1; geometry alone reaches 0.7039 and context alone 0.6746, but neither resolves the ambiguity fully, and fusing all three sources reaches 0.6606 with an exact set match of 0.5434. On the full test set, S+G+C gives the best mF1 at 0.7691, while the semantic prior alone gives the best exact set match at 0.6905. A learned MLP over 21 per-instance descriptors (mean over 10 seeds) reaches 0.7304 mF1 and 0.5590 exact set match on All, and 0.6354 / 0.4141 on Ambiguous.

• Ambiguity is defined quantitatively. An attribute is ambiguous for a class when its ground-truth rate falls between τ = 0.05 and 1 − τ; the training split contains 16 ambiguous pairs across seven semantic classes, and the All and Ambiguous test subsets contain 1,131 and 714 evaluated instances.

Methodology in Plain English

Data capture. An operator walked each facade at ground level with a GeoSLAM ZEB Horizon handheld mobile LiDAR scanner and a co-mounted ZEB Vision camera that supplies per-point colour. The team surveyed 50 buildings across five Halifax neighbourhoods and selected 26 to span a range of facade complexity, height, material, and architectural era. Capture ran six to eight months across all four seasons, day and night, in snow, fog, and clear sun. Clouds were reconstructed and denoised in GeoSLAM Faro Connect, then cleaned and coloured in CloudCompare.

Semantic annotation. Four trained annotators labelled points in CloudCompare using the co-registered fisheye imagery as a visual reference, assigning one of 12 classes (wall, window, door, balcony, vegetation, stairs, terrain, roof, blinds, other, column, arch). Each building was assigned a primary annotator and then cross-reviewed twice by annotators who had not originally labelled it, with disagreements settled by majority vote. Review tracked IoU, Cohen's Kappa, and counts of regions added, deleted, or modified.

Instance generation. Each semantic class is clustered with DBSCAN, with the neighbourhood radius tuned per building between 0.09 and 0.7 so that individual elements separate. This produced 8,586 candidate instances, which annotators merged and split in CloudCompare to yield 6,418 final instances — a 25.3% reduction reflecting DBSCAN's tendency to over-segment.

Functional labelling. Each semantic class maps to a constrained candidate set of functional attributes drawn from IFC 4.3 and the Getty Art & Architecture Thesaurus, giving an initial semantic prior per class. Instance-dependent attributes are then refined using two kinds of evidence: geometric descriptors (dimensions, depth, orientation, elevation, ground relation) and contextual relations from a graph (proximity, overlap, attachment, structural support). The prior, geometric evidence, and contextual evidence are fused in the odds domain with no class- or attribute-specific weights. Because that fusion assumes conditional independence only approximately, the result is treated as a confidence score rather than a calibrated posterior, and presence is suggested when the fused value is at least 0.5. Annotators then verify, correct, and supplement the suggestions in 3D against scanner imagery and Google Street View.

Label audit. Across two review rounds on ten buildings spanning all splits, the rounds agree on 93.2% of applicable attribute decisions; on two buildings reviewed separately by two second-round annotators, agreement reaches 99.0% with a Cohen's κ of 0.93. The authors state that attributes such as load-bearing are assessed visually from exterior geometry and imagery, not verified against structural drawings.

Benchmarking protocol. All models train on 18 buildings (343.7M points), validate on 4 (68.4M), and test on 4 (79.7M), with splits at the building level so no building spans two splits. Evaluation enforces 100% point coverage before computing any metric: six of seven models use a shared possibility-guided KNN-crop procedure with crop sizes from 4,096 to 120k points, while KPConv uses its reference implementation's potential-guided spherical sampling. No test-time augmentation is used. Inputs are 7-dimensional: mean-centred XYZ at metric scale, RGB normalised to [0, 1], and intensity normalised by training-split p1/p99 percentiles.

Why This Matters

Research impact. The paper reframes facade parsing as a three-tier problem rather than a labelling problem. It gives the community a benchmark where a model must not only name an element but group it and infer what it does — and it shows that the gap between semantic and functional performance is large, with instance separation of repetitive structures and functional reasoning both substantially harder than semantic segmentation. The cross-dataset asymmetry finding also gives researchers a concrete, measurable phenomenon in facade domain shift.

Real-world applications:

• Urban digital twins: function-aware models of windows, balconies, and columns let a city-scale model represent what elements do, not just where they are.

• Heritage modelling: the 26-building, multi-season capture supports documentation and condition monitoring of architecturally varied facades.

• Sustainable city and energy analysis: attributes such as thermal envelope, solar shading, natural lighting, and ventilation feed directly into energy modelling.

• BIM reconstruction and structural analysis: load-bearing, access, and fall-protection labels are the properties BIM tools need to turn a scan into an element-level model.

Industry relevance. Surveying, AEC (architecture, engineering, construction), and digital-twin vendors already produce dense mobile LiDAR scans at centimetre resolution; what they lack is a scalable route from raw geometry to element-level function. The semi-automatic pipeline described here — ontology-driven priors, geometric and contextual refinement, then targeted human review — is a template for producing that annotation at dataset scale, and the audit numbers (93.2% and 99.0% agreement, κ of 0.93) give a sense of what quality control costs.

Future Directions

• Extending functional annotations to interior scans, which would let load-bearing, access, and circulation attributes be reasoned about across whole buildings rather than exterior envelopes only.

• Adding complementary sensing modalities such as thermal infrared, which could supply direct physical evidence for attributes like thermal envelope membership that currently must be inferred from exterior geometry.

• Architectures that jointly optimise semantic, instance, and functional reasoning through shared functional-geometric representations, instead of the current staged pipeline.

• Imbalance- and repetition-aware methods. The authors argue that improving rare facade elements may require class-balanced re-weighting or focal loss in addition to architectural advances, and that the persistent difficulty with dense repeated windows and blinds motivates methods that explicitly model facade-scale geometry, spatial regularity, and repetition.

Target Audience

Researchers working on 3D point cloud segmentation, instance segmentation, and scene understanding who need a facade-specific benchmark with richer-than-semantic labels; practitioners in urban digital twins, BIM reconstruction, and AEC surveying who want to know what current models can and cannot deliver from mobile LiDAR; and architectural-informatics researchers interested in grounding element properties in IFC and Getty AAT ontologies. The paper assumes comfort with standard point cloud architectures and segmentation metrics, so it suits readers with intermediate background rather than complete beginners.

Authors’ abstract

Understanding building facades is central to urban digital twins, heritage modelling, and sustainable city analysis, yet existing LiDAR benchmarks stop at geometry and semantic segmentation and offer little support for reasoning about the functional role of each architectural element. We introduce HFX3D, a large-scale urban facade benchmark of approximately 492 million LiDAR points from 26 buildings in Halifax, Canada, captured across four seasons, day and night, and varied…