A multi-centre, multi-device benchmark dataset for landmark-based comprehensive fetal biometry

AI summary

Overview

• Research area: Computer Vision / Medical Image Analysis (fetal ultrasound biometry, landmark detection, domain shift and generalisation)
• Technical level: Intermediate (a dataset-and-benchmark paper requiring some familiarity with deep learning evaluation metrics such as normalised mean error and domain shift concepts)
• Scope: The paper releases an open, multi-centre, multi-device dataset of 4,513 fetal ultrasound images from 1,904 subjects at three clinical sites using seven ultrasound devices, annotated with anatomical landmarks for head, abdomen, and femur biometry, and reports cross-dataset baseline results using the BiometryNet framework.

What This Paper Is About

Accurate fetal growth assessment depends on biometric measurements taken by manually placing anatomical landmarks on standard ultrasound planes, a process that is slow, operator-dependent, and sensitive to differences between scanners and clinical sites. There has been no publicly available dataset covering all primary fetal biometry planes with landmark annotations, which has made it impossible to fairly compare automated methods or to test how well they transfer between hospitals. This paper builds and releases that missing benchmark, together with standardised splits, evaluation code, and quantitative evidence of how much performance drops when a model is tested away from the site it was trained on.

Key Contributions

1. A new open multi-centre, multi-device dataset: 4,513 de-identified ultrasound images from 1,904 subjects acquired at three clinical sites on seven different ultrasound devices, combining the FP dataset, the HC18 head dataset, and an expanded UCL cohort (424 images from 51 pregnancies).
2. Comprehensive landmark annotations for all primary biometry measures: bi-parietal diameter (BPD) and occipito-frontal diameter (OFD) on the head plane, transverse abdominal diameter (TAD) and antero-posterior abdominal diameter (APAD) on the abdominal plane, and femur length (FL) on the femoral plane, provided as CSV files with pixel-level landmark coordinates and scale-recovery metadata.
3. Standardised benchmarking infrastructure: subject-disjoint train/test splits (images from the same subject appear in only one split), evaluation code, and training pipelines, plus a documented file organisation and annotation format.
4. Quantification of domain shift: baseline results with BiometryNet (a BiometryNet landmark regression framework using HRNet with a Dynamic Orientation Determination mechanism) showing that single-centre training substantially overestimates performance relative to multi-centre testing.

Main Findings

• Variability across sites is substantial: Analysis of structure position (centre-point distribution), size (area relative to image), and orientation (angle from horizontal) shows marked intra- and inter-dataset variability; structures are inconsistently centred, magnification levels are heterogeneous, and the femur in particular may appear at any angle depending on fetal pose. HC18 showed the tightest size range because all its images came from a single scanner under a controlled protocol, whereas FP and UCL include multiple devices and operators.
• Within-domain performance is good: Training and testing on the same dataset produced NME values typically below 0.1 for head and abdomen measurements (BPD, OFD, APAD, TAD) and below 0.05 for FL.
• Cross-domain performance degrades sharply: Averaged across all train–test pairs, head and abdomen NME roughly doubled relative to within-domain performance. Femur NME reached as high as 0.90 ± 0.54 in the FP→UCL setting.
• Multi-centre training improves generalisation: The multi-centre (m-c) model tested on UCL achieved NME = 0.02 ± 0.02 for BPD and 0.03 ± 0.11 for OFD, outperforming the UCL-trained model evaluated on its own test set (0.08 ± 0.18 for BPD and 0.05 ± 0.11 for OFD).
• Abdomen results: UCL- and m-c-trained models achieved the lowest within-domain errors (UCL→UCL: 0.08 ± 0.14 for both APAD and TAD; m-c→m-c: 0.07 ± 0.08 and 0.08 ± 0.08). Cross-domain transfer between FP and UCL degraded more modestly than the femur, with NME rising to 0.16–0.31.
• Femur errors reflect instance ambiguity: The m-c-trained model achieved the lowest femur error on the m-c test set (0.03 ± 0.09), but cross-site generalisation for FL remained limited. Inspection showed many high-NME failures came from the model selecting the wrong femoral instance when more than one long bone was visible, rather than misplacing endpoints on the correct bone.
• Preprocessing matters: HC18 was originally preprocessed with ellipse-centric parameters, leaving 84.5% of landmarks outside the 64×64 heatmap space; recomputing with image-centric parameters reduced out-of-bounds landmarks to 0%. After this fix, HC18-trained models showed competitive within-domain performance and moderate cross-domain degradation (HC18→FP = 0.06 ± 0.07; HC18→UCL = 0.15 ± 0.16) instead of catastrophic failure.
• Clinically meaningful agreement: On the m-c test set, the m-c model's median head errors (approximately 0.2 mm for BPD and OFD) correspond to clinically acceptable gestational age variation in late pregnancy.

Methodology in Plain English

The authors assembled three existing image collections rather than collecting everything from scratch. The FP dataset (Barcelona, 3,090 images from 1,047 subjects) already had manual landmark annotations made by an obstetrician using the VIA annotation tool; the UCL dataset (424 images from 51 pregnancies, collected under IRAS ID 230125) was annotated the same way by a clinical research fellow under a senior consultant's supervision. HC18 (999 head images from 806 subjects, Netherlands) only had head circumference segmentation masks, so the authors derived BPD and OFD landmarks by fitting a least-squares ellipse to the mask and taking its major and minor axes.

To make the three sources comparable, they standardised everything: landmarks were stored as floating-point pixel coordinates with a consistent ordering convention, images were converted to grayscale, cropped to remove text overlays and scale bars, and normalised to the [0,1] intensity range. For the HRNet-W18 model input, each image was cropped and scaled to 256×256 using image-centric parameters (crop centre at the image centre, scale factor = max(w,h)/(1.7 × 256)). They also handled a subtle problem: different datasets ordered the two landmarks of a measurement in opposite directions (for example, FP placed the first femur point on the right, UCL on the left). They used BiometryNet's Dynamic Orientation Determination mechanism, extended to save the learned orientation vector in checkpoints and apply it at inference, so predictions were automatically corrected to match the training convention instead of manually swapping points. Pixel-to-millimetre conversion was recovered by detecting ruler markers with template matching, using a separate median inter-tick spacing approach for Aloka devices where not all ruler markers are displayed.

For evaluation, they used normalised mean error (NME), which divides the endpoint localisation error by twice the ground-truth inter-landmark distance and takes the minimum over standard and swapped endpoint correspondences, making it invariant to endpoint ordering. They trained and tested every combination of the four dataset configurations (FP, HC18, UCL, m-c) and reported mean and standard deviation NME, plus normalised Bland–Altman plots and absolute error boxplots in millimetres.

Why This Matters

Impact on research: The paper directly addresses a reproducibility gap. Prior work such as AutoFB demonstrated automated biometry on limited single-site data, and no previous public dataset covered all anatomical planes needed for fetal weight estimation. By releasing standardised splits, annotation code, and baseline results, this work gives the community a common measuring stick for comparing methods and for studying domain adaptation under realistic clinical variability.

Real-world applications:

• Fetal growth screening: Automated biometry that maintains accuracy across hospitals could support gestational age estimation and growth monitoring in routine prenatal care.
• Reducing operator burden and variability: Landmark annotation takes on average 20 seconds per plane, versus 70 seconds for manual structure segmentation delineation, and inter-operator variability in fetal biometry currently ranges between ±4.9% and ±11.1% (intra-operator ±3% to ±6.6%).
• Cross-site deployment of AI tools: The quantified domain shift gives developers a realistic picture of how much a model trained at one hospital will degrade at another, informing whether retraining or adaptation is needed.
• Training and quality assurance: The annotation conventions, quality-control steps (plane verification, anatomical landmark checks, measurement validation against expected ranges, automated outlier detection), and agreement plots offer a template for standardising measurements in clinical training programmes.

Industry relevance: Ultrasound device manufacturers (the paper notes General Electric, Philips, and Hitachi machines as sources of variability) and medical AI companies building prenatal tools now have a public benchmark to validate cross-device robustness rather than single-scanner performance. Healthcare providers and regulators gain a reference point for evaluating claims that an automated biometry product generalises beyond its training site.

Future Directions

• Extend to temporal and video data: The authors propose using temporal sequences and video to leverage anatomical motion consistency for more robust biometry.
• Address plane selection variability: The current benchmark evaluates landmark placement only within pre-selected, clinically appropriate planes; a fully automated pipeline would need plane verification or robustness to near-boundary and non-standard planes, and the variability introduced by plane selection was not quantified in this work.
• Include pathological cases and anomalous presentations: The authors call for extending the dataset to ensure robustness across the full range of clinical scenarios.
• Add more anatomical planes: They suggest incorporating additional planes such as the fetal abdomen and thorax to enable comprehensive fetal assessment.
• Solve femur instance ambiguity: Robust femur biometry may require anatomy-specific strategies such as region-of-interest priors or multiple-instance learning, given that high cross-domain FL errors often stem from selecting the wrong long bone rather than misplacing endpoints.

Target Audience

Researchers in medical image analysis and computer vision working on landmark detection, domain adaptation, and multi-centre generalisation; fetal medicine clinicians and sonographers interested in where automation stands relative to manual measurement; and industrial or regulatory teams who need a public benchmark to evaluate the cross-site robustness of prenatal AI tools. The paper is most useful to readers who already understand basic deep learning evaluation, since much of its value lies in the benchmark infrastructure and the cross-dataset NME comparisons rather than in a new model architecture.

Authors’ abstract

Accurate fetal growth assessment from ultrasound (US) relies on precise biometry measured by manually identifying anatomical landmarks in standard planes. Manual landmarking is time-consuming, operator-dependent, and sensitive to variability across scanners and sites, limiting the reproducibility of automated approaches. There is a need for multi-source annotated datasets to develop artificial intelligence-assisted fetal growth assessment methods. To address this bottleneck, we present an open, multi-centre, multi-device benchmark dataset of fetal US images with expert anatomical landmark annotations for clinically used fetal biometric measurements. These measurements include head bi-parietal and occipito-frontal diameters, abdominal transverse and antero-posterior diameters, and femoral length. The dataset comprises 4,513 de-identified US images from 1,904 subjects acquired at three clinical sites using seven different US devices. We provide standardised, subject-disjoint train/test splits, evaluation code, and baseline results to enable fair and reproducible comparison of methods. Using an automatic biometry model, we quantify domain shift and demonstrate that training and evaluation confined to a single centre substantially overestimate performance relative to multi-centre testing. To the best of our knowledge, this is the first publicly available multi-centre, multi-device, landmark-annotated dataset that covers all primary fetal biometry measures, providing a robust benchmark for domain adaptation and multi-centre generalisation in fetal biometry and enabling more reliable AI-assisted fetal growth assessment across centres. All data, annotations, training code, and evaluation pipelines are made publicly available.