SmartSplat: Feature-Smart Gaussians for Scalable Compression of Ultra-High-Resolution Images

AI summary

Overview

Research area: Computer vision — image compression using 2D Gaussian Splatting for ultra-high-resolution (UHR) imagery.

Technical level: Advanced. The paper assumes familiarity with Gaussian Splatting, differentiable rasterization, covariance parameterization, and image compression rate–distortion trade-offs.

Scope: The paper introduces SmartSplat, a feature-aware 2D Gaussian image representation framework with a new sampling and color-initialization pipeline, evaluated on the DIV8K benchmark and a newly constructed DIV16K dataset (arXiv:2512.20377v1, 23 Dec 2025).

What This Paper Is About

Ultra-high-resolution images (8K, 16K and beyond) are now common thanks to generative AI, but they are large — the paper reports DIV8K images averaging 5736 × 6120 at 53.56 MB and DIV16K images averaging 12684 × 15898 at 235.52 MB. Traditional formats such as JPEG cap out around 50× compression, and existing neural or Gaussian-based image representations either need many Gaussian primitives or fail entirely under aggressive compression. SmartSplat's goal is to reconstruct high-fidelity images from a strictly limited budget of 2D Gaussians by placing, sizing, and coloring those Gaussians according to image content.

Key Contributions

1. A unified analysis linking compression ratio to Gaussian count. The authors formalize the maximum allowable number of Gaussians as N_g = 3HW / (7 · CR), showing that high compression ratios inherently constrain how many primitives are available and therefore make faithful reconstruction harder.

2. An adaptive, feature-guided Gaussian sampling strategy. Gradient-Color Guided Variational Sampling and Exclusion-based Uniform Sampling jointly initialize the positions and scales of Gaussians, targeting denser, smaller Gaussians in high-frequency or high-color-variance regions and sparser, larger ones elsewhere.

3. A scale-adaptive color initialization method. Each Gaussian's RGB value is set using a Gaussian-weighted median over a circular neighborhood whose radius equals the Gaussian's scale, replacing random or pixel-center initialization.

4. A new 16K benchmark (DIV16K). Built by applying 8× upsampling to DIV2K images using the Aiarty Image Enhancer tool, with evaluations spanning DIV8K and DIV16K.

Main Findings

• DIV8K gains over the runner-up: On DIV8K, SmartSplat improves over the runner-up method, 3DGS, by 1.53 dB in PSNR and 0.0201 in MS-SSIM at the same compression ratio. Relative to the 2D baseline GI (RS), it reports a 2.57 dB PSNR gain, and it maintains comparable quality at 500× compression where GI (RS) needs 200× to reach similar results.

• DIV8K full comparison at CR = 20–1000: SmartSplat reaches 33.26 / 0.9752 at CR = 20, 29.65 / 0.9482 at CR = 50, 27.49 / 0.9164 at CR = 100, 25.75 / 0.8745 at CR = 200, 23.82 / 0.8055 at CR = 500, and 22.66 / 0.7469 at CR = 1000 (PSNR / MS-SSIM). At CR = 200, 500, and 1000, ImageGS reports higher PSNR (26.80, 24.88, 23.50) but substantially lower MS-SSIM (0.7181, 0.6544, 0.6165). Both GI variants and LIG report "Fail" at CR = 500 and CR = 1000.

• DIV16K results and baselines failing: SmartSplat reaches 34.34 / 0.9267 at CR = 50, 33.00 / 0.9117 at CR = 100, 31.85 / 0.8897 at CR = 200, 29.40 / 0.8524 at CR = 500, 27.49 / 0.8226 at CR = 1000, 25.70 / 0.7966 at CR = 2000, and 24.72 / 0.7844 at CR = 3000 (PSNR / SSIM). 3DGS reports OOM at CR = 50, 100, and 200 on DIV16K, and ImageGS reports OOM at every listed DIV16K compression ratio. GI and LIG fail at CR ≥ 500, and only SmartSplat reports a result at CR = 3000.

• Average PSNR advantage on DIV16K: At lower compression ratios, SmartSplat achieves an average PSNR gain of about 5.64 dB over GI (RS).

• Faster convergence: On a 10848 × 16320 image at CR = 200, SmartSplat surpasses both 3DGS and GI quality within 1K iterations, outperforming their results at 10K iterations.

• Memory and time versus 3DGS: 3DGS requires approximately 2.56× the memory of SmartSplat and about 3.49× longer training time. In Table 2 (10K iterations), 3DGS uses 50.19 GB and 7841.80 s, versus SmartSplat at 19.59 GB and 2237.52 s.

• Efficiency versus GI: Against GI, SmartSplat shows a 10.66 dB PSNR gain within 1K iterations, with training time reduced to 25% of GI's. GI reports a higher FPS (62.33) than SmartSplat (32.35 at 10K, 33.12 at 1K).

• Ablation study (CR = 200, 10K iterations, 4416 × 6720 image): Full random initialization gives 22.34 dB / 0.8435 MS-SSIM; adding VS/US mean initialization gives 22.18 dB / 0.8270; adding VS/US scale initialization gives 23.12 dB / 0.8647; adding scale-adaptive color initialization (full SmartSplat) gives 24.38 dB / 0.8972, with training time of 456.12 s.

• Extreme compression behavior: The authors state SmartSplat rivals JPEG at 2000× and maintains visually pleasing results up to 5000×, while JPEG shows severe artifacts at its maximum compression ratio and GI struggles with scalability.

• Fast initialization: On 16K images, the initialization stage completes within 2 to 5 seconds using a CUDA-based query-to-reference KNN pipeline.

Methodology in Plain English

SmartSplat represents an image as a set of 2D Gaussians, each with a position, a 2D covariance (decomposed into rotation and scale), a color, and an opacity fixed at 1. The number of Gaussians is dictated by the target compression ratio. The research question is therefore how to place a small number of these blurry elliptical blobs so that they cover the image well.

The pipeline works in three initialization stages before any optimization:

1. Variational sampling. The image is split into tiles of size 1024. Inside each tile, the authors compute a per-pixel gradient magnitude and a local color variance, normalize both, and combine them into a sampling weight using a balance coefficient λ_m = 0.9. Pixels are then drawn by multinomial sampling according to that weight, so structurally complex and chromatically rich areas receive more Gaussians. Each sampled point gets a scale that shrinks exponentially with its sampling weight, anchored to a base scale derived from uniform circular coverage of the image area.

2. Exclusion-based uniform sampling. To cover flat, low-detail areas, additional points are placed subject to an exclusion radius — the maximum of the base scale and the median scale of the variational samples — so they do not stack on top of existing points. Their scales come from a K-nearest-neighbor distance estimate (K = 3) computed over all sampled points.

3. Scale-adaptive color initialization. Each Gaussian's color is the Gaussian-weighted median of pixels inside a circular neighborhood whose radius equals that Gaussian's scale, which the authors describe as more robust to noise and outliers than random or pixel-center color initialization.

The Gaussians are then rendered with differentiable rasterization and optimized with Adam for 50K steps using learning rates of 1e-4, 5e-3, 5e-2, and 1e-3, minimizing a composite loss of L1 distance and (1 − SSIM) with λ_l = 0.9. A variational sampling ratio λ_g = 0.7 splits the Gaussian budget between variational and uniform sampling.

Why This Matters

Research impact. The paper reframes UHR Gaussian image compression as a sampling and initialization problem rather than a purely architectural one, and provides a concrete formula tying compression ratio to Gaussian count. It also releases a 16K benchmark, which is rare in this literature.

Real-world applications.

• Storing and transmitting ultra-high-resolution photographs on end-user devices with limited storage.
• Serving 8K/16K content over constrained networks where per-image sizes of 53.56 MB (DIV8K) or 235.52 MB (DIV16K) are impractical.
• Archival of AI-generated or AI-upscaled UHR imagery, which the paper identifies as a growing source of such data.
• Progressive or on-demand decoding of very large images where a compact Gaussian representation replaces a raster file.

Industry relevance. Companies handling high-resolution media pipelines — cloud storage, streaming, medical or satellite imaging, and generative AI content platforms — face direct storage and bandwidth costs that scale with resolution. A representation that stays stable at compression ratios where established Gaussian baselines report "Fail" or "OOM" is directly relevant to those cost profiles.

Future Directions

• Advanced attribute compression. The conclusion explicitly states that this work focuses on Gaussian spatial distribution, and that future work will target better compression of Gaussian attributes.
• Closing the PSNR gap at extreme rates. On DIV8K at CR = 200, 500, and 1000, ImageGS reports higher PSNR than SmartSplat even though its MS-SSIM is much lower; understanding and addressing that perceptual-versus-pixel trade-off is an open question.
• Decoding speed. GI achieves 62.33 FPS versus SmartSplat's 32.35 (10K iterations), so real-time rendering claims rest on further optimization.
• Scaling beyond the tested settings. Whether the approach holds at resolutions above 16K, at compression ratios beyond the reported 5000× visual claim, and for datasets other than DIV8K and DIV16K is not evaluated here.

Target Audience

Researchers and engineers working on image compression, learned image representations, or Gaussian Splatting, particularly those dealing with 8K-and-above content. It is also relevant to practitioners building storage, delivery, or rendering pipelines for ultra-high-resolution media who need to know where existing Gaussian-based methods break down and what a viable alternative looks like.

Authors’ abstract

Recent advances in generative AI have accelerated the production of ultra-high-resolution visual content, posing significant challenges for efficient compression and real-time decoding on end-user devices. Inspired by 3D Gaussian Splatting, recent 2D Gaussian image models improve representation efficiency, yet existing methods struggle to balance compression ratio and reconstruction fidelity in ultra-high-resolution scenarios. To address this issue, we propose SmartSplat, a highly adaptive and feature-aware GS-based image compression framework that supports arbitrary image resolutions and compression ratios. SmartSplat leverages image-aware features such as gradients and color variances, introducing a Gradient-Color Guided Variational Sampling strategy together with an Exclusion-based Uniform Sampling scheme to improve the non-overlapping coverage of Gaussian primitives in pixel space. In addition, we propose a Scale-Adaptive Gaussian Color Sampling method to enhance color initialization across scales. Through joint optimization of spatial layout, scale, and color initialization, SmartSplat efficiently captures both local structures and global textures using a limited number of Gaussians, achieving high reconstruction quality under strong compression. Extensive experiments on DIV8K and a newly constructed 16K dataset demonstrate that SmartSplat consistently outperforms state-of-the-art methods at comparable compression ratios and exceeds their compression limits, showing strong scalability and practical applicability. The code is publicly available at https://github.com/lif314/SmartSplat.