Bridging the Markerless Tracking Validation Gap: Intraoperative X-ray Workflow Integration

AI summary

Overview

Research area: Computer vision for surgical guidance — specifically markerless RGB-D tracking of spinal anatomy and the validation infrastructure needed to benchmark it.

Technical level: Intermediate. The paper assumes familiarity with 2D–3D registration, point-cloud registration (ICP), camera calibration and rigid transform chains, but the core argument is accessible to anyone working on surgical data.

Scope (one sentence): The paper presents a clinically deployable pipeline that derives per-vertebra 6-DoF ground-truth pose from routine intraoperative C-arm fluoroscopy and co-registers it to an RGB-D camera, then uses it to evaluate two markerless spine registration methods on real surgical data.

What This Paper Is About

Markerless tracking of the spine from RGB-D cameras could replace the invasive bone-anchored fiducials used in surgical navigation, but nobody has been able to prove how accurate these markerless methods actually are in a real operating room, because no clinical dataset contains a trustworthy reference for the subsurface position of each vertebra. Existing clinical ground truth is approximated from exposed bone surfaces, which says nothing about the pose of the vertebra underneath. The authors build a way to recover that missing reference from X-ray images the surgical team already takes during routine level checks — no extra radiation, no extra hardware in the wound — and use it to score markerless registration methods for the first time against an independent subsurface reference in a clinical setting.

Key Contributions

1. A multimodal capture and labelling pipeline that co-registers intraoperative RGB-D recordings with ground-truth vertebral poses derived from routine intraoperative fluoroscopy — requiring no additional radiation, no instrumentation inside the surgical field, and no deviation from standard surgical workflow.

2. The first quantitative evaluation of markerless spine registration methods on clinical data against an independent subsurface reference, reporting per-vertebra translational and rotational error across three surgical sequences (11 vertebrae, 1,500 frames).

3. A validated, reusable evaluation framework, with ex vivo end-to-end accuracy of 2.06 ± 0.29 mm and 3.35 ± 0.93°, intended as reusable infrastructure for benchmarking markerless registration methods.

4. Open release commitment: upon publication, CAD files for the intraoperative marker array and calibration phantom, plus evaluation code and annotated data.

Main Findings

• Ex vivo end-to-end accuracy: across four ovine thoracic vertebrae, the pipeline achieved a combined TRE of 2.06 ± 0.29 mm and rotational error of 3.35 ± 0.93° (per-vertebra TREs of 1.96, 2.32, 1.62 and 2.32 mm; rotations of 4.06°, 3.40°, 3.94° and 1.98°).

• Comparison to clinical thresholds: these values approach the ≤2 mm and ≤5° thresholds commonly cited for spinal navigation systems, and the authors argue this precision is sufficient to act as independent ground truth for markerless methods whose errors are substantially larger, particularly in rotation.

• Component-level validation against an optical tracker: both modalities agreed with the Polaris Vicra reference at sub-millimetre translation and sub-degree rotation. Worst-case reported values were 1.19 mm / 0.07° for AP X-ray tracking, 0.96 mm / 0.50° for oblique X-ray tracking, and 0.70 mm / 1.85° for RGB-D tracking with the sterile probe cover in place. The largest RGB-D translation error (0.10 mm) occurred for motion along the sensor optical axis, which produced the largest RGB-D rotational error (1.85°).

• Rigid full-spine registration beat per-vertebra ICP on rotation: 4.52° ± 1.00° versus 8.36° ± 3.69°, with substantially lower variance across all metrics, while mean translation errors were comparable with overlapping distributions (4.78 ± 1.10 mm versus 4.52 ± 3.00 mm overall).

• Neither markerless method yet meets navigation thresholds: both achieve mean translational errors below 5 mm, but neither meets ≤2 mm and ≤5°, which the authors read as evidence of both the value of independent pose evaluation and the headroom remaining.

• The rotational gap is real, not noise: the 3.84° difference in rotation between the two markerless methods exceeds the reference's own uncertainty (3.35 ± 0.93°), while translational differences are not statistically different.

• Surface geometry is the bottleneck: the posterior vertebral surface exposed during surgery is geometrically ambiguous, leaving ICP unable to robustly resolve pose, particularly rotation — a limitation the authors expect will not be fixed by better initialization alone.

Methodology in Plain English

The trick is to piggyback on something surgeons already do. During lumbar-thoracic fixation cases, the C-arm is routinely used to confirm which vertebra is being operated on, typically with an AP view plus oblique views. The authors designed a flat marker array — five radiopaque ArUco tags, each 30 × 30 mm, set into a radio-translucent PETG body with 3D-printed PLA XRS insets, arranged non-coplanarly so no tags overlap at AP or oblique angles — that is visible in both the X-ray and the RGB-D camera. It sits on a boom arm next to the exposure site, under the C-arm detector and inside the Kinect's field of view, under a sterile ultrasound probe cover.

Each C-arm source is modelled as a pinhole camera. Intrinsics were calibrated before surgery using a custom multi-plane phantom with nine radio-opaque ArUco markers at known depths, solved by bundle adjustment, from AP and oblique angles within ±30°. During surgery, the marker array is detected automatically in each fluoroscopic shot and its pose relative to the detector recovered by PnP; the first shot is treated as the reference frame, and relative transforms between shots are chained through the shared marker coordinate system. Those projection matrices feed intensity-based multi-view 2D–3D registration in ImFusion, which optimizes six rigid-body parameters per segmented vertebra by maximizing mutual information between digitally reconstructed radiographs and the acquired fluoroscopic images. Preoperative CT bone was segmented with the network from Meng et al.

The same array is detected in the Kinect frames, giving the two transforms needed to close the loop: vertebra-to-reference-detector and marker-to-RGB-D. Composing them yields the vertebra-to-camera transform, so any markerless estimate produced in camera coordinates can be compared directly against an X-ray-derived reference.

Validation happened in two stages. First, an infrared tracking array was rigidly bolted to the marker array, which was moved between four distinct points in front of each sensor, isolating the error of each coordinate-recovery stage against the optical tracker. Second, the full chain was tested on four excised ovine vertebrae, each fitted with two plastic ArUco markers fixed by plastic screws and kept as invisible as possible on X-ray; because those markers can be localized inside the CT volume, the vertebrae can also be tracked directly from the Kinect, bypassing the transform chain entirely. Two landmark points — the spinous process tip and a point on the superior laminar margin — defined the target registration error. Imaging used a TAU 2020 C-arm with two shots at 30° angular disparity, close to the 25.6° mean disparity seen in the clinical cases.

For the clinical demonstration, two markerless strategies were run on three sequences of 500 frames each. The first follows Daly et al.: a preoperative spine mesh is deformed by a kinematic chain and rigidly ICP-registered to segmented intraoperative points, with semi-automated optimization and manual refinement. The second is fully automatic, inspired by prior work: a two-stage PointNet++ heatmap network trained on manually annotated spinous process tips localizes tips (recall 0.95, F1 0.89, localization error 3.98 ± 1.99 mm), then a PointNet segmentation network isolates points within 40 mm of each detection (IoU 0.71, F1 0.83), and individual CT vertebra meshes are registered to those patches. Both were scored with a 3 mm correspondence distance.

Why This Matters

Impact on research: the field has been measuring markerless spine tracking against surrogate surface metrics, which cannot detect subsurface pose error. This paper supplies an independent, workflow-compatible reference and shows it discriminates between methods — the rigid approach's rotational advantage is larger than the reference's own uncertainty. It converts "our method looks plausible on clinical data" into a measurable claim, and the committed code, CAD and annotations give other groups a common yardstick.

Real-world applications:

• Surgical navigation and robotic guidance for lumbar-thoracic fixation, where tracking currently depends on bone-anchored fiducials that add operative time and complication risk.
• Augmented-reality overlay of preoperative CT onto the live surgical scene, as shown in the paper's intraoperative setup figure.
• Benchmarking and regulatory-style validation of emerging markerless trackers against a quantifiable reference rather than a surrogate.
• Broader hospital settings: the pipeline relies on C-arm fluoroscopy, which the authors note is broadly accessible, rather than intraoperative CT.

Industry relevance: the approach is deliberately built on equipment already in the room — a standard C-arm, an Azure Kinect, and commercial registration software — and adds no radiation dose. That makes it adoptable as validation infrastructure by companies developing camera-based navigation, and it sets an evidentiary bar (≤2 mm, ≤5°) that current markerless products can be measured against.

Future Directions

• Scale the clinical evidence. The paper reports initial results from a large-scale ongoing collection study: three sequences and ovine ex vivo validation. More labelled cases are needed to broaden the evaluation.
• Learn pose instead of aligning surfaces. Because the exposed posterior surface is geometrically ambiguous, the authors propose a learned mapping from intraoperative point clouds to vertebral pose that exploits appearance, anatomical context and priors beyond ICP's reach.
• Remove the manual bottleneck. The winning rigid method requires manual deformation tuning, which the authors identify as a barrier to clinical deployment and an explicit avenue for future work.
• Tighten the reference itself. End-to-end accuracy is limited by the 2D–3D registration stage, which depends on image quality, angular separation and the number of fluoroscopic views — so improving that stage directly lowers the floor on measurable markerless error.

Target Audience

Researchers and engineers working on surgical computer vision, RGB-D tissue tracking and 2D–3D registration; clinical scientists and surgeon-investigators running observational studies in spine surgery; and anyone who needs a defensible ground truth for evaluating image-guided navigation, including groups building or assessing commercial spinal navigation systems. Readers without a background in medical imaging will find the framing clear but the methodology dense.

Authors’ abstract

Markerless intraoperative tracking of spinal anatomy via RGB-D sensors is a promising alternative to invasive bone-anchored fiducials, yet no existing clinical dataset provides independent subsurface ground truth for validating these methods. We address this gap with a clinically viable pipeline that derives per-vertebra 6-DoF pose from routine intraoperative C-arm fluoroscopy, co-registered to the RGB-D camera frame through a multimodal fiducial array detectable in both imaging modalities. Critically, the pipeline requires no additional radiation, instrumentation, or deviation from standard surgical workflows, relying solely on the fluoroscopic images already acquired during routine level checks. A calibrated transform chain links the vertebral pose recovered via multi-view 2D-3D CT-to-fluoroscopy registration to the RGB-D coordinate frame, enabling direct comparison of any markerless registration estimate against a subsurface reference. Ex vivo validation against optically tracked ovine vertebrae confirms end-to-end accuracy of 2.06 +- 0.29 mm and 3.35 +- 0.93 degrees, sufficient to serve as independent ground truth for methods with substantially larger errors. We demonstrate the pipeline's utility by evaluating two markerless registration strategies on clinical data spanning 11 vertebrae across 1,500 frames from three surgical sequences: the first such evaluation against an independent subsurface reference in a clinical setting. The proposed framework provides reusable infrastructure for benchmarking emerging markerless tracking methods on real surgical data.