InsideOut: Integrated RGB-Radiative Gaussian Splatting for Comprehensive 3D Object Representation

AI summary

Overview

Research area: Computer vision / 3D reconstruction, specifically multi-modal fusion of RGB and X-ray imaging within 3D Gaussian splatting (3DGS).

Technical level: Advanced. The paper assumes familiarity with 3DGS, radiative/differentiable X-ray rasterization, SfM point clouds, ICP registration, and K-means clustering.

Scope: The paper introduces InsideOut, a five-stage pipeline that fits RGB Gaussian splats and radiative (X-ray) Gaussian splats into one geometrically and visually aligned 3D object representation, and releases a new paired RGB–X-ray dataset for evaluation.

What This Paper Is About

RGB images record only what is visible on an object's surface, while X-ray images reveal only the internal density structure, and the two modalities capture information in fundamentally different ways, so combining them into a single 3D model is difficult. Existing approaches either keep the two modalities in separate pipelines or only align them in 2D, and no paired RGB–X-ray 3D dataset was previously available. InsideOut's goal is a unified 3D representation that simultaneously preserves high-fidelity external surface appearance and internal subsurface structure, using 3DGS as the shared representation medium.

Key Contributions

1. A framework that integrates RGB images and X-ray images into 3DGS, producing a single unified 3D model that carries both external appearance and internal structure.
2. A hierarchical fitting procedure (global, local with K-means clustering at multiple resolutions, and fine alignment using ICP) that bridges the geometric domain gap — position, scale, and rotation — between the SfM-based RGB model and the scanner-coordinate X-ray model.
3. A new X-ray reference loss that uses cross-sectional X-ray images as pseudo-ground-truth guidance, combining L1, D-SSIM, and a zero-one opacity term to sharpen layer boundaries and reduce structural mismatch between the RGB and X-ray 3D models.
4. A new, publicly released dataset of paired RGB images and X-ray projections covering five objects across medical imaging, cultural heritage preservation, and manufacturing quality control.

Main Findings

• Internal detail quality (perceptual): InsideOut achieved the lowest PIQE score across all five objects — 25.28 (Matryoshka), 30.35 (Pharaoh), 25.35 (Terracotta), 20.01 (Skull), and 20.74 (Toy Gun) — against comparison methods including 3DGS, 3DGS-MCMC, DoF-Gaussian, MaskGaussian, Analytic-Splatting, RS-NeRF, NAF, and X-Gaussian, indicating superior perceptual quality of internal structure.

• PSNR and SSIM against reference X-ray images: InsideOut improved PSNR by 1.67 dB over X-Gaussian and 3.33 dB over 3DGS on average, and improved SSIM by 0.018 over X-Gaussian and 0.042 over 3DGS. Per-object InsideOut PSNR values were 18.86, 15.14, 19.56, 20.33, and 20.87; SSIM values were 0.61, 0.64, 0.55, 0.65, and 0.66. The paper notes no ground-truth images exist for simultaneously evaluating both internal and external structure, so a no-reference metric (PIQE) is used for cross-sections.

• Noise removal and layer sharpening: Qualitatively, 3DGS produced internal noise and scattered splats, while InsideOut concentrated splats along the object surface and made internal layers visible that X-Gaussian left blurred.

• Geometric alignment (ablation, chamfer distance): Baseline chamfer distances were 38.56, 27.93, 21.29, 722.45, and 23.02 across the five objects. Adding hierarchical alignment reduced them to 8.41, 12.52, 11.44, 9.53, and 4.59; adding internal rendering on top reduced them further to 6.45, 4.10, 2.42, 7.95, and 3.42 — the lowest values. Surface points for this metric were extracted using Hidden Point Removal so that only visible surface points contributed.

• Mesh reconstruction: Compared against R²-Gaussian (radiative volume reconstruction with Marching Cubes) and GS2Mesh (RGB-based mesh reconstruction), InsideOut captured finer geometric details than the radiative mesh while preserving overall shape, and reconstructed both external surface and internal structure.

• Spherical occupancy: At 42 bins InsideOut had the highest occupancy (0.443 mean, 0.452 median), followed by GS2Mesh (0.437 mean, 0.429 median) and R²-Gaussian (0.431 mean, 0.429 median). At 242 bins GS2Mesh led with 0.476 mean and 0.475 median, InsideOut recorded 0.471 for both, and R²-Gaussian was lowest at 0.468 mean and 0.466 median. The reported interpretation is that InsideOut captures more diverse normal distributions than R²-Gaussian and internal layer normals that GS2Mesh cannot capture.

• Stated limitation: Because a voxelization method is used for the radiative Gaussian splats, surface texture quality tends to be lower than GS2Mesh.

Methodology in Plain English

The approach is a five-stage pipeline that starts by training each modality separately and then progressively fuses them.

1. Separate coarse training. A standard 3DGS model is trained for 20k iterations to learn the RGB surface, and X-Gaussian is trained for 5k iterations to learn a radiative (X-ray) splat model. X-ray splats have no color; each carries a radiative feature vector instead, and are initialized with an angle cuboid uniform initialization (ACUI) that samples points inside a cuboid enclosing the scanned object using the scanner's intrinsic and extrinsic parameters.

2. Hierarchical fitting. The two point clouds live in different coordinate systems: SfM output is relative, X-ray output is in absolute physical units. So the method approximates each full splat set as one big Gaussian, matches their centroids, normalizes scale, and uses PCA-derived rotation, then solves with ICP (global fitting). It then repeats this at progressively finer granularity using K-means clustering at ⌊N/10⌋, then ⌊N/2⌋, then all N Gaussians, aligning each cluster by its mean and covariance (local fitting), and finally aligning individual Gaussians (fine fitting).

3. Internal rendering with cross-sectional guidance. Orthographic slices of the X-ray volume at 15 mm intervals along the axial and coronal planes serve as pseudo-ground-truth. For RGB splats, a Canny edge detector extracts surface contours from these slices; when several edge layers appear, the outermost is selected by choosing the contour whose bounding box has the largest area. This keeps RGB splats on the surface and empties the interior. Only the geometry parameters (position, covariance, opacity) of RGB splats are updated, since the pseudo-GT is grayscale edges with no color.

4. X-ray reference loss. The total loss is (1 − λs)·L1 + λs·D-SSIM + λz·L_zero-one, with λs = 0.2 and λz = 0.005. The zero-one term pushes opacities toward 0 or 1, which forms thin planar structures at layer locations, sharpens layer boundaries, and removes internal noise.

5. Visual alignment. Radiative splats receive color by k-nearest-neighbor matching to the nearest RGB Gaussian splat. Then detail cloning runs: the largest eigenvalue of each RGB splat's covariance is computed, the bottom 95% by covariance magnitude are kept and the top 5% discarded, and the selected fine splats are added to the radiative set to enrich surface detail without disturbing internal structure.

Training uses the Adam optimizer with an initial learning rate of 5e-4 on an NVIDIA A6000 GPU; the internal rendering stage runs for a further 20k iterations.

Dataset: Five objects — Matryoshka, Pharaoh, Terracotta, Skull, and Toy Gun — spanning medical imaging, cultural heritage preservation, and manufacturing quality control. RGB images per object: 555, 644, 210, 90, and 416, captured with smartphones and DSLR cameras in natural lighting with over 60% overlap between consecutive views, at resolutions from 3024×1848 to 5712×4284. X-ray data was captured with a cone-beam CT scanner over 360° rotation with 2048 detector elements, voltage from 185 to 330 kV, current from 0.49 to 1.50 mA, and 2016–2048 Z-slices. X-ray projections per object were 360, 360, 2160, 360, and 360.

Why This Matters

Impact on research. The paper argues that full 3D reconstruction from combined RGB and X-ray data was previously unresolved, with existing work limited to 2D image alignment or to separate processing of each modality even for the same object. InsideOut extends 3DGS beyond surface-only rendering and beyond X-ray-only volume reconstruction, and the released paired dataset addresses a stated lack of RGB image sets with corresponding X-ray images.

Real-world applications (as described in the paper):

• Medical diagnostics, where external appearance and internal anatomy are both relevant.
• Cultural heritage restoration and preservation, where objects must be examined non-destructively.
• Manufacturing quality control and non-destructive testing.
• Real-world asset diagnosis and interactive/physical simulation, supported by cross-sectional analysis and quantitative assessment of internal structures.

Industry relevance. The unified model supports visualization and simulation from a single 3D asset, which the authors position as useful for inspection workflows where disassembly or destructive testing is not possible. The underlying technique is built on readily available components — PyTorch with CUDA, standard 3DGS, X-Gaussian, and Marching Cubes — and the dataset is released for others to use.

Future Directions

• Improving surface texture quality. The paper explicitly notes that surface texture from the voxelization-based radiative splats is lower quality than GS2Mesh, leaving room for a better meshing or surfacing approach.
• Broadening the dataset. The current paired dataset covers five objects; scaling to more objects, materials, and application domains would test whether the hierarchical fitting and X-ray reference loss generalize.
• Establishing ground-truth evaluation. Because no ground truth exists for simultaneously evaluating internal and external structure, evaluation currently relies on no-reference PIQE for cross-sections plus PSNR/SSIM against X-ray references — a limitation that better benchmarks could address.
• Reducing capture and training cost. X-ray projection counts vary widely across objects (2160 for Terracotta versus 360 for others) and the pipeline trains in separate stages, raising questions about how far acquisition and optimization can be streamlined.

Target Audience

Researchers and graduate students working on 3D Gaussian splatting, multi-modal 3D reconstruction, differentiable rendering, or neural radiance and radiative volume methods. It is also relevant to practitioners in medical imaging, cultural heritage digitization, and industrial non-destructive testing who need a combined external-and-internal 3D representation of a physical object. Readers without prior exposure to Gaussian splatting or X-ray volumetric reconstruction will find the framework's five-stage pipeline conceptually accessible, but the fitting, loss, and rasterization details require an advanced background.

Authors’ abstract

We introduce InsideOut, an extension of 3D Gaussian splatting (3DGS) that bridges the gap between high-fidelity RGB surface details and subsurface X-ray structures. The fusion of RGB and X-ray imaging is invaluable in fields such as medical diagnostics, cultural heritage restoration, and manufacturing. We collect new paired RGB and X-ray data, perform hierarchical fitting to align RGB and X-ray radiative Gaussian splats, and propose an X-ray reference loss to ensure consistent internal structures. InsideOut effectively addresses the challenges posed by disparate data representations between the two modalities and limited paired datasets. This approach significantly extends the applicability of 3DGS, enhancing visualization, simulation, and non-destructive testing capabilities across various domains.