Patient Pose Assessment Using a CT-Based Framework for Synthetic Data Generation

AI summary

Overview

Research area: Medical computer vision / deep learning for radiography, specifically synthetic data generation from CT scans for patient pose assessment using Time-of-Flight depth images.

Technical level: Intermediate. Familiarity with convolutional neural networks, transfer learning, and medical imaging concepts is helpful, but the framework is explained step by step and remains accessible to readers outside radiology.

Scope: The paper presents a CT-based framework that synthetically generates depth image and radiograph pairs of upper ankle joints, and evaluates whether pretraining on this synthetic data improves pose-quality assessment on real data.

What This Paper Is About

The diagnostic quality of a radiograph depends most heavily on how the patient is positioned, but positioning is not standardized and depends on the patient and the radiographer's experience. The authors want to give radiographers automatic feedback on the patient's pose before the radiograph is taken, using depth images from Time-of-Flight cameras, but collecting paired depth images and radiographs is blocked by ethical, data protection, and regulatory hurdles. Their solution is to generate the required training data synthetically from Computed Tomography (CT) scans that have already been acquired.

Key Contributions

1. A clinical-realistic synthetic data generation framework that produces paired synthetic depth images and synthetic radiographs from existing CT scans, including realistic scene composition on a recorded X-ray room table, multiple pose rotations, and patient-shape augmentations.
2. A synthetic dataset of upper ankle joints: 3077 synthetic radiographs and 30,770 synthetic depth images generated from 10 CTs of different patients, described as by far the largest dataset linking depth images to diagnostic quality.
3. A newly acquired real-world clinical dataset of 18 living subjects (10 male, 8 female, aged 21 to 33) covering 108 poses, mirroring clinical practice, released alongside the synthetic dataset.
4. Evidence that synthetic pretraining transfers to real data, improving pose assessment of real upper ankle joints by up to 11 percentage points (pp).

Main Findings

• Synthetic data alone supports accurate pose assessment. Training a randomly initialized model solely on the synthetic dataset achieved a Mean Absolute Error (MAE) of 0.23 and Spearman correlation of 0.85 without augmentation, and an MAE of 0.22 and correlation of 0.86 when all augmentations were combined. Diagnostic accuracy was 85.45% without augmentation and 86.94% with all augmentations combined.
• Augmentation generally improves results. Across the synthetic-only experiments, training with augmented data improved results for all metrics. Size augmentation alone produced the strongest correlation (0.87) and the best accuracy (87.6%), while combining size and bump augmentation (all) produced the best specificity (80.21%) and the best diagnostic accuracy (86.94%).
• Synthetic pretraining beats both training from scratch and ImageNet pretraining. On the anatomical preparations dataset, models pretrained on the synthetic dataset and then finetuned reached up to 92.8% diagnostic accuracy (camera view specific pretraining, size augmentation) and up to 90.45% accuracy (unified camera view, size augmentation), compared with 89.08% diagnostic accuracy and 79.25% accuracy for models trained from scratch, and 84.2% diagnostic accuracy and 77.37% accuracy for ImageNet-pretrained models.
• View-specific synthetic pretraining gives the highest specificity and diagnostic accuracy. Camera view specific pretraining with size augmentation reached 95.34% specificity and 92.8% diagnostic accuracy; with bump augmentation it reached 93.87% diagnostic accuracy and 94.88% specificity.
• The reported transfer gain reaches 11 percentage points. The abstract states that pretraining on the synthetic dataset of 3077 image pairs improves pose assessment of real upper ankle joints by up to 11 percentage points.
• Pretraining also helps on a realistic clinical dataset. The authors report that pretraining on the synthetic dataset can improve performance on the newly acquired dataset of 18 living subjects. The specific numbers for this experiment (Experiment 2b) and for the linear probing experiment (Experiment 2c) are not included in the content provided.
• Synthetic radiographs were judged usable by experts. The synthetic radiograph is not as detailed as a real radiograph, but expert radiologists validated that the visual quality is suitable for assessing the diagnostic quality for a given pose.

Methodology in Plain English

The authors built a semi-automatic pipeline that turns an existing CT scan into realistic-looking training data:

• Extract the anatomy. The CT scan is converted into a point cloud using the Marching Cubes Algorithm with a threshold of -500 Hounsfield units to remove air around the patient. Clustering (DBSCAN) and cropping remove the imaging table and any points not belonging to the surface of the target anatomy.
• Define rotation and support plane. For the upper ankle joint, the rotation axis is the longitudinal axis, found by computing the centers of two areas of the point cloud (proximal to the ankle and around the calf). A plane is placed directly beneath the target anatomy using three manually selected points.
• Augment for body variety. Normal vectors of the point cloud allow points to be shifted outward or inward, simulating different patient shapes and sizes. Bumps simulating swellings or edema are added using a Gaussian distribution, with 2 to 3 bumps placed locally in two additional point clouds, resulting in four augmented point clouds in total.
• Compose a realistic scene. The X-ray room and its imaging table are recorded in advance as a point cloud. The rotated target anatomy is matched onto the table plane and then optimized so it sits realistically on the table, using an iterative algorithm that aligns the proximal and distal points closest to the table plane and translates the anatomy onto it.
• Render depth images. A 2D projection of the anatomy and room point clouds, accounting for intrinsic and extrinsic camera parameters and distortion coefficients, produces the synthetic depth image for any rotation, augmentation, and camera view.
• Render radiographs. The target anatomy is cropped out of the CT and the voxels are converted into material voxels (air, soft tissue, bone, or titanium, based on the HU value) and mass-density voxels. Instead of moving the anatomy, the position of the X-ray device is changed. A Monte Carlo simulation with the MCGPU tool and PENELOPE 2006 material files simulates 2·10^10 X-ray beam paths, and the raw image is converted with a non-linear value mapping to look like a real radiograph.
• Label and train. Every radiograph was assessed by 4 radiologists on a quality scale of 1 to 3 in increments of 0.5, where 1 is ideal and 3 is inadequate; labels in [1, 2.5) count as diagnostic and labels of 2.5 or above as non-diagnostic. Training treats the task as a regression, using two EfficientNet-B0 models (one per camera view) that output a continuous value between 1 and 3. Images are resized to 336×336 pixels, the Adam optimizer is used with an initial learning rate of 10^-3 decreased 4 times by a factor of 10 in the last 10,000 steps, batch size is 16, and models were trained for 200,000 steps. All experiments were repeated 10 times with different seeds.
• Compare pretraining strategies. The authors compare random initialization (from scratch), ImageNet pretraining, and synthetic pretraining in two variants: a unified camera view approach (both models initialized with the same weights from both views) and a camera view specific approach (each model initialized from its own view). Across all experiments they conducted 21 experimental configurations, amounting to 980 downstream training runs and 1100 trained models including the synthetic pretraining runs.

Why This Matters

The work addresses a bottleneck that is common across medical imaging: the best training data is often the hardest to collect ethically. Rather than radiographing subjects without an indication or in non-diagnostic poses, the framework reuses CT scans that already exist, with no additional radiation exposure, and can generate multiple poses — including non-diagnostic ones — from a single scan. This makes the approach substantially more scalable than prospective data acquisition and offers a template for other anatomy-specific pose assessment tasks.

Real-world applications:

• Real-time feedback to radiographers on patient pose before the radiograph is taken, reducing retakes.
• Reduced radiation exposure and lower costs for hospitals by avoiding repeat examinations.
• Pose assessment in settings where the anatomy is difficult to standardize, such as extremity radiography, since the pipeline is not inherently limited to the upper ankle joint.
• Retrospective reuse of existing CT archives to build large custom synthetic datasets for specific X-ray rooms and camera configurations.

Industry relevance: The pipeline targets practical deployment constraints. The authors deliberately chose EfficientNet-B0 because it should be able to run on a CPU to keep cost and complexity low, while being fast enough to process frames from a live Time-of-Flight camera feed in real time. The framework explicitly incorporates recorded room geometry and camera parameters, so it can in principle be adapted to different X-ray rooms, X-ray devices, clinical environments, and camera configurations.

Future Directions

• Full automation versus human oversight. The authors believe fully automated data generation is possible in principle but consider human oversight essential to ensure data quality; resolving this trade-off is left open.
• Extending to other anatomies. The preprocessing steps, in particular the rotation axis and the support plane, are partly anatomy-specific, so transferring the framework to other joints or limbs would require corresponding setup adaptations.
• Automatic labeling of synthetic radiographs. The authors note that labels could be generated automatically, for example using the framework presented in Mairhöfer et al. (2021), while still considering expert manual assessment essential.
• Closing the gap to fully labeled clinical data. The real-world clinical dataset uses surrogate labels of 1 and 3 rather than radiographic ground truth because subjects had no indication for radiography; whether the textbook pose always corresponds to a true diagnostic quality label of 1 remains unverified.

Target Audience

This paper is most useful for medical imaging and computer vision researchers working on synthetic data generation, transfer learning, and pose or quality assessment; for radiographers and clinical physicists interested in standardized patient positioning and dose reduction; and for industry engineers building camera-based assistance systems for X-ray devices. Readers primarily interested in clinical validation should note that the real-world dataset is weakly labeled and that the specific results for the clinical and linear probing experiments are not available in the provided content.

Authors’ abstract

An adequate diagnostic quality of radiographs is essential for reliable diagnoses and treatment planning. The patient's pose during radiography is one of the most important factors determining the diagnostic quality. Since patient positioning is difficult and not standardized, an automated AI-based approach using depth images to automatically assess the patient's pose before the radiograph has been taken would be helpful. Due to regulatory hurdles, however, it is difficult in practice to acquire the required depth images and corresponding radiographs. In this paper, we present a framework that can generate such training data synthetically from Computed Tomography scans. We further show that by pretraining on our generated synthetic dataset consisting of 3077 image pairs of upper ankle joints, the pose assessment of real upper ankle joints can be improved by up to 11 percentage points.