GaussianImage++: Boosted Image Representation and Compression with 2D Gaussian Splatting

AI summary

Overview

Research area: Computer Vision — image representation and learned image compression using 2D Gaussian Splatting (GS).

Technical level: Advanced (assumes familiarity with neural image compression, implicit neural representations, and Gaussian Splatting).

Scope: A single paper introducing GaussianImage++, a 2D Gaussian-Splatting-based image codec that improves on GaussianImage via distortion-driven densification, content-aware Gaussian filters, and quantization-aware training.

What This Paper Is About

Implicit neural representations (INRs) can fit images with high fidelity, but they train slowly and consume large amounts of memory because they rely on MLPs. Gaussian Splatting (GS) offers a faster, more compact alternative by storing images as a set of explicit 2D Gaussian primitives, but the existing 2D GS method GaussianImage requires far too many Gaussians to reach high visual quality. GaussianImage++ targets this exact gap: it shows that with smarter densification, filtering, and quantization, a small, fixed budget of 2D Gaussian primitives can deliver both better image fitting and better rate-distortion (RD) performance than GaussianImage and INR-based codecs like COIN, while keeping decoding real-time.

Key Contributions

1. Distortion-driven densification (D³): A progressive, image-quality-oriented scheme that grows new 2D Gaussians at the top-k highest-distortion pixels and prunes invalid Gaussians, capping the total at a user-controlled maximum M. This replaces the failed adoption of 3D-GS-style Adaptive Density Control (ADC) in 2D GS.

2. Content-aware Gaussian filters (CAF): Per-Gaussian adaptive low-pass filters with variance s_i that change over training — strong filtering early (to prevent holes from undersampling) that weakens as Gaussians densify (to allow fine detail). The filtered covariance Σ + sI is stored, so no extra storage is required.

3. Attribute-separated learnable quantization with QAT: LSQ+ quantizers with different bit depths per attribute (12-bit for position μ, 10-bit for covariance Σ, 6-bit for color c), combined with quantization-aware training to compress the Gaussian primitives into a compact bitstream.

4. General-purpose components: Both D³ and CAF are demonstrated to be drop-in enhancements for other 2D GS variants (GaussianImage with Cholesky or RS factorization, LIG), independent of the covariance parameterization.

Main Findings

• Representation gains scale up with Gaussian budget. On Kodak, GaussianImage++ reaches 31.83 dB PSNR with 5k Gaussians and 35.41 dB with 10k Gaussians, versus GaussianImage's 29.85 / 32.48 dB. On DIV2K it reaches 28.14 / 33.75 dB versus 26.54 / 31.45 dB.

• Ablation isolates the value of D³ and CAF. D³ alone adds roughly 2 dB PSNR over GaussianImage (Cholesky) at equal Gaussian counts; combining D³ and CAF yields up to 3 dB over GaussianImage and 4 dB over LIG.

• Compression beats COIN and GaussianImage across most RD points. GaussianImage++ consistently outperforms INR-based COIN in PSNR and MS-SSIM, surpasses GaussianImage at most bitrate points, and outperforms JPEG at low bitrates (0.1–0.7 bpp on Kodak, 0.1–0.5 bpp on DIV2K). It still lags behind state-of-the-art VAE-based codecs at high bitrates.

• Decoding speed is the standout advantage. GaussianImage++ maintains real-time decoding (e.g., ~1839/1666 FPS on Kodak at low/high bpp), dramatically faster than JPEG2000, Ballé17/18, and COIN, and faster than GaussianImage itself. Encoding is slower than traditional codecs due to quantization-aware training.

• Quantization design matters a lot. Replacing LSQ+ with FP16 or Residual Vector Quantization (RVQ) for either position or color degrades BD-Rate by 17–147% and BD-PSNR by up to ~2.5 dB. The chosen 12/10/6-bit configuration balances bitrate and quality better than alternatives.

• Densification adds only modest training cost. Roughly 16 seconds of extra encoding time yields ~1.9 dB PSNR gain, with no penalty to rendering speed.

• Low variance across random seeds. Standard deviations of 3.21×10⁻⁶ bpp, 1.97×10⁻³ PSNR, and 9.03×10⁻⁷ MS-SSIM show the method is stable.

• Mirage-style ADC has memory risks. The paper notes that applying 3D GS's ADC directly to 2D scenes leads to uncontrolled Gaussian growth and out-of-memory errors during training.

Methodology in Plain English

The authors start by throwing out the MLP-based approach entirely. Instead, they represent an image as a collection of soft, colored, 2D elliptical blobs (Gaussians) that are summed together to paint the picture. The challenge is deciding where to put each blob.

They begin with only half the final budget of Gaussians, placed randomly. Every 5,000 training steps, they look at where the current rendering differs most from the real image, and drop new Gaussians at those worst pixels, copying the real pixel color at those locations. This repeats until the budget is exhausted. Meanwhile, every 100 steps, they remove any Gaussians whose covariance matrix has become mathematically invalid.

A second trick addresses the "holes" problem when few Gaussians are present. If each blob is too small, gaps appear between them. Their solution is to temporarily inflate every Gaussian with an extra "spread" value (a low-pass filter), then shrink that spread as more Gaussians are added. This is analogous to blurring the image early on, then sharpening as detail arrives. Crucially, the effective covariance (Σ + sI) is what gets stored, so this filtering is free at decompression time.

For compression, they first overfit the Gaussians to the image as above, then run a warm-up phase (6,000 iterations), then switch to quantization-aware training: the position, covariance, and color of each Gaussian are each sent through a separate learnable quantizer (LSQ+), with precision tuned per attribute (12/10/6 bits), so the network learns to survive the quantization error. The final quantized attributes form the compressed bitstream.

Why This Matters

Impact on research: This paper argues that 2D Gaussian Splatting is a viable third paradigm for image compression, sitting alongside VAEs/hyperpriors and INRs. It shows GS-based codecs can combine competitive low-rate RD with far lower memory and much faster decoding. The densification and filtering techniques are shown to generalize to other 2D GS methods, so they can be adopted widely.

Real-world applications:

• Real-time image streaming and display on edge devices — decoding overhead is measured in fractions of a millisecond per image, and memory footprint is under 1 GB.
• Mobile and embedded content delivery, where RAM is limited but instant loading is required.
• Interactive editing pipelines — because Gaussians are explicit primitives with semantic attributes (position, color, shape), they are easier to manipulate than a monolithic neural network.
• Training-data storage and retrieval in ML pipelines that need fast random-access image decoding.

Industry relevance: The approach competes directly with JPEG on bitrate at low quality settings while decoding orders of magnitude faster than learned codecs, making it attractive for CDN infrastructure, cloud photo services, and any system that is decode-bound rather than encode-bound. The explicit, per-image Gaussian representation is also a natural fit for GPU-accelerated rendering pipelines already common in graphics.

Future Directions

• Closing the high-bitrate RD gap: GaussianImage++ still trails state-of-the-art VAE-based codecs at high bitrates. The authors suggest more advanced attribute encoding schemes and entropy models (e.g., context modeling on Gaussian attributes) as the way forward.

• Faster encoding: Encoding time remains far from real-time because of the quantization-aware training loop. Distillation, better initialization, or amortized encoder networks could help.

• Better color attribute compression: The paper notes that 2D GS lacks the spherical harmonic coefficients of 3D GS, which are naturally suited to vector quantization. Designing a compact color representation for 2D GS remains open.

• Extending to higher dimensions: The framework is 2D by design. Whether D³ and CAF can be extended to 3D scenes, video, or light fields — where temporal coherence and view-dependence come into play — is an open question.

Target Audience

Researchers and engineers working on learned image compression, implicit neural representations, Gaussian Splatting, or real-time rendering systems. Practitioners evaluating codec trade-offs between rate, distortion, memory, and decode latency on edge or GPU hardware will find the quantitative comparisons especially useful. Readers new to neural compression will find the paper's framing accessible but should have prior familiarity with VAE-based codecs (Ballé et al.) and Gaussian Splatting to follow the technical details.

Authors’ abstract

Implicit neural representations (INRs) have achieved remarkable success in image representation and compression, but they require substantial training time and memory. Meanwhile, recent 2D Gaussian Splatting (GS) methods (\textit{e.g.}, GaussianImage) offer promising alternatives through efficient primitive-based rendering. However, these methods require excessive Gaussian primitives to maintain high visual fidelity. To exploit the potential of GS-based approaches, we present GaussianImage++, which utilizes limited Gaussian primitives to achieve impressive representation and compression performance. Firstly, we introduce a distortion-driven densification mechanism. It progressively allocates Gaussian primitives according to signal intensity. Secondly, we employ context-aware Gaussian filters for each primitive, which assist in the densification to optimize Gaussian primitives based on varying image content. Thirdly, we integrate attribute-separated learnable scalar quantizers and quantization-aware training, enabling efficient compression of primitive attributes. Experimental results demonstrate the effectiveness of our method. In particular, GaussianImage++ outperforms GaussianImage and INRs-based COIN in representation and compression performance while maintaining real-time decoding and low memory usage.