X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction

AI summary

Overview

Research area: Computer vision and computer graphics — 3D reconstruction and novel-view synthesis, specifically 3D Gaussian Splatting extended to jointly model surface appearance (RGB) and internal volumetric structure (X-ray/CT).

Technical level: Advanced. The paper assumes familiarity with Gaussian splatting, volumetric rendering, the Beer–Lambert law, multi-view stereo, and point-cloud registration (PCA/ICP).

Scope: X-Splat proposes and evaluates a single unified Gaussian representation that reconstructs an object's exterior appearance and interior density from paired RGB photographs and X-ray projections, with geometry-based cross-modal alignment and joint optimization (published at SIGGRAPH Asia 2026 Conference Papers).

What This Paper Is About

Real objects have both an outside and an inside, but 3D reconstruction pipelines capture only one: RGB methods (NeRF, 3DGS) recover surface appearance while treating the interior as unobserved, and X-ray methods (NAF, X-Gaussian, R²-Gaussian) recover internal density but no surface appearance. The few existing attempts to do both — most notably InsideOut — train two separate Gaussian sets and stitch them together afterwards, which doubles the parameter count and lets floaters and optimization artifacts corrupt the alignment. X-Splat's goal is a single set of Gaussian primitives, sharing one geometry, that renders both modalities well and is aligned by construction.

Key Contributions

1. A unified Gaussian representation. Every primitive carries shared geometry {μ, R, S} and an X-ray density ρ, while RGB appearance attributes {α, c} are attached only to a surface subset S of primitives — motivated by the physics that RGB signal comes from surface reflection. This removes the redundancy of two independent geometries.
2. A geometry-based alignment pipeline. Instead of registering two already-optimized Gaussian sets, the method registers clean geometric proxies: a multi-view stereo point cloud (from MVSFormer) against a mesh extracted from a coarse X-ray reconstruction via Marching Cubes, using PCA initialization followed by point-to-point ICP with scaling.
3. A joint optimization scheme with X-ray-derived geometric supervision. Both modalities update the shared geometry through their own objectives, and an edge-aware surface-depth consistency loss uses CT-derived depth as a stop-gradient edge prior to guide RGB surface reconstruction.
4. A new evaluation dataset and protocol. The authors introduce the Flower & Insect dataset — two flower scenes (Tulip, Eustoma) from existing scans plus three newly captured insect specimens (E02, E15, E32) — extending evaluation beyond the four InsideOut scenes to natural objects.

Main Findings

• RGB quality improves without hurting X-ray quality. On InsideOut, X-Splat averages 32.59 PSNR / 0.9765 SSIM / 0.0293 LPIPS versus 31.20 / 0.9726 / 0.0323 for 3DGS with MVS initialization. On Flower & Insect it averages 36.44 / 0.9871 / 0.0262 versus 33.72 / 0.9811 / 0.0300. The paper characterizes this as roughly 1 dB PSNR on InsideOut and over 2 dB on Flower & Insect.
• X-ray reconstruction stays at parity with the specialist. X-Splat averages 49.81 2D-PSNR, 0.9963 2D-SSIM and 38.16 3D-PSNR on InsideOut versus R²-Gaussian's 49.76 / 0.9963 / 38.25; on Flower & Insect it is 48.92 / 0.9971 / 35.27 versus 48.98 / 0.9972 / 35.31. Differences are described as minor fluctuations, with individual metrics going slightly either way.
• Better initialization alone does not close the gap. A 3DGS baseline initialized from the FDK volume — which already encodes the full internal structure — does not beat the MVS-initialized baseline (31.20 PSNR average on InsideOut for both), which the authors read as evidence that persistent geometric supervision, not a better starting point, is what matters.
• The reproduced InsideOut baseline trails badly on RGB. InsideOut† averages 21.22 PSNR / 0.9122 SSIM / 0.0344-class LPIPS of 0.1107 on InsideOut and 19.10 / 0.8923 / 0.1368 on Flower & Insect. Its X-ray numbers coincide exactly with R²-Gaussian because its X-ray stage follows that method.
• Geometry-based alignment beats aligning optimized Gaussians. Chamfer Distance on InsideOut: X-Splat (PCA + ICP) 0.0122, PCA only 0.0250, InsideOut† hierarchical alignment 0.0978, ICP only 0.1453. ICP alone is the worst of the four, showing that a good initialization matters as much as the refinement step.
• FDK remains far behind Gaussian methods for X-ray. Classical FDK averages 33.86 2D-PSNR / 0.9040 2D-SSIM / 28.57 3D-PSNR on InsideOut and 34.42 / 0.8937 / 27.66 on Flower & Insect.
• Qualitative gains. The authors report sharper surface texture and fewer floater artifacts than RGB baselines (e.g., Matryoshka, Tulip), and CT slices and X-ray projections closely matching R²-Gaussian.

Methodology in Plain English

The pipeline has four moving parts:

1. One cloud of points, two kinds of properties. The object is represented by N Gaussian primitives. Every primitive stores the same position, rotation and scale, plus a density value used only by the X-ray renderer. Because an RGB camera only ever "sees" the outer shell, the opacity and color attributes are given only to primitives lying within a distance threshold (τ_d = 0.02) of the object surface; interior primitives keep their density but have no appearance. The authors note that only the geometry is shared — the optical attributes remain modality-specific.

2. Learn the inside first, then register the outside to it. Stage 1 is an X-ray-only fit of 3,000 iterations starting from an FDK reconstruction, using the X-ray photometric loss plus a 3D total-variation term (λ_tv = 0.05), reusing the R²-Gaussian training schedule. The resulting coarse volume is voxelized, its outer shell isolated by binarization and morphological closing, and a mesh extracted with Marching Cubes. Separately, MVSFormer runs on the RGB views to produce a surface point cloud. PCA aligns the two clouds' centroids, principal axes and variances to give a starting similarity transform; ICP with scaling refines it; the resulting transform is applied to the RGB camera poses to bring both modalities into one coordinate frame.

3. Initialize appearance on the surface, then optimize jointly. Primitives within τ_d of the CT mesh form the surface set. Their initial opacity follows a Gaussian decay from the surface with reference opacity α_surf = 0.5, and their color is taken from the nearest MVS point. Stage 2 runs 12,000 iterations with λ_rgb = λ_xray = 1.0, λ_tv = 0.05 and λ_geo = 0.05, and keeps the X-ray loss active rather than freezing the X-ray side — otherwise RGB gradients could drift the shared geometry and degrade X-ray fidelity. The CT mesh is re-extracted every 500 iterations. The depth-consistency loss does not force RGB depth to match CT depth numerically; it only penalizes RGB depth gradients where the CT depth has no edge, evaluated on 180 virtual cameras sampled with a Fibonacci sphere.

4. Density control adapted to two supervisors. Densification keeps separate position-gradient statistics per modality, so under-fitting in either one spawns new primitives; the RGB criterion only applies to surface primitives, while the X-ray criterion is unrestricted. Pruning requires both opacity and density to be low, so a primitive invisible to RGB but radiographically meaningful survives. Primitives whose opacity falls below threshold are demoted out of the surface set rather than deleted, so they keep contributing to X-ray rendering.

Evaluation covers nine scenes. X-ray projections are re-synthesized from ground-truth CT with TIGRE (50 training and 120 testing projections over [0°, 360°), 512×512) so all baselines share identical acquisition geometry; RGB images are downsampled to width 800 with every fourth view held out. Training takes roughly 14–21 minutes per InsideOut scene on a single NVIDIA RTX A6000. Because the official InsideOut code is not public, the authors reimplemented it as InsideOut†, omitting its final X-ray refinement stage since that stage's opacity-based rendering is incompatible with their density-based renderer.

Why This Matters

For research. The paper reframes cross-modal 3D reconstruction as a representation-design problem rather than an alignment-after-the-fact problem, and gives evidence that a shared geometry can absorb supervision from two physically different forward operators. It also supplies a second dataset and a Chamfer-Distance alignment metric, which is a useful evaluation axis that per-modality rendering scores do not capture.

Real-world applications (as identified in the paper and supported by the reconstructed results):

• Digital cultural heritage — displaying an artifact's exterior and its internal construction in one aligned model.
• Industrial inspection — checking surface defects and internal voids against a common geometry.
• Physically-grounded visualization — rendering an object consistently as surface and as volume from any viewpoint.
• Biological and natural-specimen study — the Flower & Insect scenes show the approach applying to organic objects, not just manufactured ones.

Industry relevance. The compactness argument matters for deployment: carrying geometry once instead of twice reduces parameter count and storage, which the paper measures in number of Gaussians (thousands) and megabytes. The method is implemented on the publicly available R²-Gaussian codebase, and training in tens of minutes on a single high-end GPU makes it practical for small-batch scanning workflows rather than only offline pipelines.

Future Directions

• Replace the staged, hand-tuned schedule. The paper's own design hinges on a strong asymmetry (X-ray anchors geometry, RGB refines appearance); whether a learned or adaptive weighting could remove the two-stage split, the fixed iteration counts, and the several scalar weights is an open question.
• Remove the dependence on a good MVS surface. The alignment and appearance initialization both rely on MVSFormer output, which will degrade on textureless, transparent, or specular objects — a natural stress test given that the method's physics argument assumes opaque surfaces.
• Move beyond opaque objects. The surface-restricted appearance design is explicitly justified only for the opaque objects considered; translucent or thin structures where "the surface" is ill-defined are not addressed in this work.
• Extend to real X-ray acquisition rather than re-synthesized projections. The paper's X-ray protocol re-synthesizes cone-beam projections from ground-truth CT; validating on genuinely measured, noisy, and possibly beam-hardened acquisitions would test robustness under the conditions the motivating applications actually face.

Target Audience

Researchers and graduate students in computer graphics and computer vision working on Gaussian splatting, inverse rendering, and tomographic reconstruction; practitioners who need aligned external and internal models of real objects for heritage, inspection, or biological imaging; and readers interested in how shared representations can be designed around the physics of two different sensing modalities rather than patched together after independent optimization.

Authors’ abstract

Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.