A$^2$TG: Adaptive Anisotropic Textured Gaussians for Efficient 3D Scene Representation

AI summary

Overview

• Research area: Computer vision and real-time 3D scene representation, specifically textured Gaussian Splatting built on 2D Gaussian Splatting (2DGS).
• Technical level: Advanced. The method rests on differentiable rasterization, spherical-harmonic color, per-pixel alpha compositing, and positional-gradient derivations, though the central idea (allocating texture size where it is needed) is conceptually simple.
• Scope: The paper introduces A²TG, a representation that gives each 2D Gaussian its own anisotropic (rectangular) texture whose resolution and aspect ratio are chosen by gradient-guided rules, and evaluates it against fixed-texture Gaussian Splatting baselines on three standard datasets.

What This Paper Is About

Textured Gaussian Splatting methods attach a learnable texture patch to every Gaussian to enrich appearance, but they give every primitive a fixed-size square texture even though Gaussians differ enormously in size, shape, opacity, and visibility. This wastes memory on tiny, transparent, or occluded Gaussians that gain little from texture detail, while also failing to match the elongated (anisotropic) shape of many splats. The paper's goal is to keep textured Gaussians' visual fidelity while cutting the texture memory cost, by deciding per Gaussian how large and what shape its texture should be.

Key Contributions

1. Adaptive anisotropic textured Gaussians (A²TG): a generalization of Textured Gaussians in which each primitive carries its own texture with individually chosen width and height rather than a uniform square patch.
2. Gradient-based adaptive texture control: a strategy that jointly determines texture resolution and aspect ratio, using gradient-driven Gaussian selection followed by adaptive texture upscaling, so allocation is non-uniform and detail-aware.
3. Empirical demonstration of memory efficiency: experiments across Mip-NeRF 360, Tanks and Temples, and Deep Blending showing improved memory efficiency with comparable visual fidelity relative to fixed-texture textured Gaussian methods.
4. A two-stage training recipe that couples MCMC densification into 2DGS for the first 30,000 iterations, then fixes the Gaussian count and trains textures with adaptive control for another 30,000 iterations.

Main Findings

• Fixed memory budget results (Table 1): A²TG reached 28.51 PSNR / 0.838 SSIM / 0.174 LPIPS on Mip-NeRF 360 with 700k Gaussians and 199.68MB; 23.56 / 0.828 / 0.153 on Tanks and Temples with 690k Gaussians and 199.51MB; and 29.86 / 0.900 / 0.187 on Deep Blending with 700k Gaussians and 189.42MB.
• Comparison at that budget: Under the same constrained setting, Textured Gaussians* used a fixed 4×4 texture and obtained 28.37 / 0.832 / 0.188 (410k Gaussians, 200.00MB), 23.41 / 0.824 / 0.164 (410k, 200.00MB), and 29.51 / 0.897 / 0.198 (410k, 200.00MB). The paper reports that A²TG achieves higher overall visual quality with less memory, particularly on PSNR, and that other texture-based methods suffer from excessive texture parameters under a fixed memory constraint.
• Fixed Gaussian count results (Table 2): At 1M Gaussians, A²TG used 291MB (+25%) on Mip-NeRF 360 versus 488MB (+110%) for Textured Gaussians*, 292MB (+26%) on Tanks and Temples versus 488MB (+110%), and 277MB (+19%) on Deep Blending versus 488MB (+110%). Textured Gaussians* gained roughly 0.4 dB in PSNR while taking up to more than four times the texture memory.
• Trade-off summary at fixed counts: The paper states that BBSplat and Textured Gaussians* typically have slightly better visual quality than A²TG but require substantially more memory because of inefficient texture parameter use.
• Sparse allocation: On the Garden scene from Mip-NeRF 360, 62.4% of Gaussians retained 1×1 textures under gradient-driven selection, indicating resolution is allocated sparingly. Non-square textures concentrated along sharp edges such as gaps between table planks.
• Texture resolution ranges: Final textures in Textured Gaussians* remained 4×4, whereas A²TG textures varied within {1, 2, 4} × {1, 2, 4}, including both square and anisotropic shapes.
• Texture decomposition: Removing textures (RGB textures set to zero, alpha textures set to one) caused a clear loss of high-frequency detail such as leaf boundaries, grass structure, and fine variations in carpets and fabrics. Removing the SH base color produced overly dark renderings with missing global shading. The authors conclude that SH color provides a smooth, lighting-aware low-frequency foundation while textures encode local high-frequency detail.
• Ablation results (Table 3, averaged across the three datasets): Without texture upscaling, PSNR was lowest at every Gaussian count (27.03 at 1M, 26.76 at 500k, 25.72 at 100k, 24.87 at 50k) with the lowest memory. Without anisotropic upscaling, quality was close to the full method but memory was higher (298.22MB vs 286.54MB at 1M; 152.10MB vs 145.92MB at 500k).
• Rendering speed: Textures are packed into a single GPU texture-atlas array, and the custom CUDA rasterization kernel can require up to 16 global-memory load operations per texture lookup. Despite this overhead, the system maintains real-time rendering speeds above 30 FPS.

Methodology in Plain English

The method starts from 2D Gaussian Splatting, where each scene element is a flat, oriented 2D Gaussian with a position, 2D scale, rotation, opacity, and spherical-harmonic color. The authors add two small texture images to each Gaussian: an RGB texture for spatially varying color and an alpha texture for spatially varying opacity. Colors are read from the texture by converting a pixel's intersection point into the Gaussian's local uv-coordinates, rescaling that range by the texture's width and height, and using bilinear interpolation. The final pixel color combines the spherical-harmonic base color with the texture color, and the alpha combines the Gaussian's opacity, its Gaussian kernel value, and the alpha texture value.

Training happens in two stages. First, the scene is trained for 30,000 iterations using 2DGS with an MCMC densification scheme, which lets the authors control the total number of Gaussians; during this stage every texture is fixed at 1×1 with RGB and alpha values of zero, meaning no textures are used. Second, the Gaussian count is frozen, texture alpha is set to 1, and training continues for another 30,000 iterations with the adaptive texture control.

The adaptive control has two parts. Gradient-driven Gaussian selection computes the positional gradient of the rendering loss (an L1 plus SSIM loss) with respect to each Gaussian's position, accumulating the absolute gradient magnitude over covered pixels, following AbsGS. If this accumulated magnitude exceeds a threshold k_G, the Gaussian is flagged as containing high-frequency content and selected as a candidate for upscaling. This filters out Gaussians that cover few pixels, have low opacity, are occluded by foreground splats, or lie in homogeneous regions such as uniformly colored walls.

Adaptive texture upscaling then uses the ratio of the two semi-axes of the Gaussian to decide how to grow the texture. If s_x/s_y exceeds k_A while s_y is below k_S, only the texture width is doubled. If s_y/s_x exceeds k_A while s_x is below k_S, only the height is doubled. Otherwise both dimensions are doubled. New texture values are initialized from the nearest pixel of the previous texture and then refined by further optimization. Upscaling is applied every 500 iterations so gradients can accumulate. The reported thresholds are k_A = 4.0, k_S = 0.01, and k_G = 0.00002, with upscaling applied at iterations 500 and 1000.

Why This Matters

The work targets a concrete inefficiency in a popular representation: attaching uniform square textures to every Gaussian, including ones that contribute almost nothing visible. By making texture allocation follow the geometry and error signal of each primitive, it offers a route to textured Gaussian scenes that fit in less memory without giving up detail. The authors note this direction is orthogonal and complementary to existing 3DGS compression work such as vector quantization, splat-count reduction, and context models.

Real-world applications:

• Immersive content creation and view synthesis, the application areas the authors cite for Gaussian Splatting, where richer per-primitive appearance comes at lower storage cost.
• Memory-constrained deployment, such as mobile, embedded, or VR/AR headsets, where the paper's gains are largest at high Gaussian counts (1M and 500k) that are most costly to store.
• Reconstruction and mesh extraction pipelines built on 2D Gaussian Splatting, since A²TG keeps the 2DGS uv-parameterization and rasterization framework intact.
• Scene streaming and storage workflows, where reducing trainable-parameter memory for the same visual fidelity lowers transfer and storage demands.

Industry relevance: the method is a drop-in augmentation of an existing 2DGS pipeline, uses standard CUDA rasterization, and still renders above 30 FPS, which matters for any product that needs real-time rendering alongside a fixed memory footprint.

Future Directions

• Downscaling as well as upscaling: the authors note they currently only upscale texture resolution and suggest reducing resolution when appropriate to improve texture efficiency further.
• Texture compression and primitive control: compressing textures and controlling the shape and size of Gaussians are proposed as additional memory-efficiency levers.
• More flexible primitives: combining adaptive textures with primitives beyond 2D/3D Gaussians, specifically Deformable Radial Kernel Splatting (Huang et al., 2025), which have sharper and more flexible boundary shapes.
• Dynamic scenes: extending A²TG to dynamic Gaussian Splatting models such as 4DGS.

Target Audience

Researchers and graduate students working on Gaussian Splatting, neural rendering, and differentiable rasterization, who will care about the gradient-driven allocation rule and the memory-versus-quality trade-offs. Graphics and XR engineers deploying real-time 3D scene representations on memory-limited hardware will benefit from the practical results and the serialized-texture-atlas implementation detail. Readers who only want the conceptual idea (adaptive anisotropic textures instead of fixed squares) can follow the abstract, introduction, and qualitative figures without the derivation-heavy sections.

A note on the source: the provided content is truncated during Appendix A.2, cutting off mid-sentence after the statement that the rasterizer was not explicitly optimized for maximum performance; the remaining details of that section are not reported here.

Authors’ abstract

Gaussian Splatting has emerged as a powerful representation for high-quality, real-time 3D scene rendering. While recent works extend Gaussians with learnable textures to enrich visual appearance, existing approaches allocate a fixed square texture per primitive, leading to inefficient memory usage and limited adaptability to scene variability. In this paper, we introduce adaptive anisotropic textured Gaussians (A$^2$TG), a novel representation that generalizes textured Gaussians by equipping each primitive with an anisotropic texture. Our method employs a gradient-guided adaptive rule to jointly determine texture resolution and aspect ratio, enabling non-uniform, detail-aware allocation that aligns with the anisotropic nature of Gaussian splats. This design significantly improves texture efficiency, reducing memory consumption while enhancing image quality. Experiments on multiple benchmark datasets demonstrate that A TG consistently outperforms fixed-texture Gaussian Splatting methods, achieving comparable rendering fidelity with substantially lower memory requirements. Project page: http://github.com/Rickyeeeeee/A2TG.