StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video

AI summary

Overview

• Research area: Computer vision / computer graphics, specifically free-viewpoint video (FVV), dynamic scene reconstruction with 3D Gaussian Splatting (3DGS), and efficient model compression for network streaming.
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, deformation fields, NeRF-based dynamic scene reconstruction, spherical harmonics, video codecs (H.264 / HEVC / libx265), and adaptive bitrate transmission.
• Scope: A streaming representation for dynamic multi-view video that decouples a dynamic 3DGS scene into canonical Gaussians (stored as images) plus temporal features (stored as a video), reporting PSNR, SSIM, LPIPS, frame size, keyframe size, decoding time, rendering time, FPS, and training time on the N3DV and MeetRoom datasets.

What This Paper Is About

Real-time free-viewpoint video requires a 4D scene representation that is small enough to transmit over a network and fast enough to render interactively. Recent 3DGS-based FVV methods render quickly but can require up to 10 MB of storage per frame, which makes real-time streaming impractical. The authors propose StreamSTGS, a representation that stores canonical 3D Gaussians as 2D images and temporal motion features as a video, so that standard video codecs and variable bitrate transmission can be used without any extra training.

Key Contributions

1. A streamable spatial-temporal Gaussian grid representation. StreamSTGS decouples a dynamic scene into canonical 3D Gaussians, a set of temporal features, and a deformation field, and applies a sliding window over the temporal Gaussian grids to capture the motion relationships between adjacent frames.
2. Spatial and temporal smoothness losses. These regularize the spatial and temporal Gaussian grids so they can be compressed into images and a video respectively, substantially reducing model size and enabling adaptive bitrate transmission based on network conditions without retraining.
3. A transformer-guided auxiliary training strategy. A transformer learns global motion and distills it into StreamSTGS via a self-distillation loss, while the shared deformation field lets the transformer be removed at inference time so rendering speed is not affected.
4. Two optimization techniques for limited Gaussian budgets: dynamic-aware density (L1 loss over the whole image but SSIM loss restricted to dynamic pixels) and a Gaussian relocate operation that repositions Gaussians instead of pruning them when the ~150k Gaussian budget is reached.

Main Findings

• Reconstruction quality on N3DV: StreamSTGS reaches 32.30 PSNR, 0.943 SSIM, and 0.147 LPIPS, compared with 31.52 / 0.941 / 0.143 for 4DGC, 31.32 / 0.939 / 0.147 for HiCoM, 30.73 / 0.935 / 0.147 for 3DGStream, 30.07 / 0.900 / 0.299 for TeTriRF, and 27.45 / 0.871 / 0.213 for VideoGS.
• Reconstruction quality on MeetRoom: StreamSTGS reaches 27.41 PSNR, 0.92 SSIM, and 0.21 LPIPS, versus 27.11 / 0.91 / 0.23 for 4DGC, 26.69 / 0.90 / 0.23 for HiCoM, and 26.41 / 0.90 / 0.24 for 3DGStream.
• Storage: The average frame size of StreamSTGS is only 170 KB, which the paper states is 4X smaller than GS-based methods. On N3DV the reported per-frame storage is 173.6 KB with a keyframe size of 3.86 MB; on MeetRoom it is 142 KB with a 2.8 MB keyframe.
• Speed: StreamSTGS reports 100 FPS on N3DV with 8 ms decoding and 10 ms rendering, and 126 FPS on MeetRoom with 6 ms decoding and 7.93 ms rendering. Training takes 67 s on N3DV and 29 s on MeetRoom.
• Abstract-level summary: The abstract reports StreamSTGS increases PSNR by an average of 1 dB while reducing the average frame size to just 170 KB.
• Sliding window ablation (N3DV): Removing the sliding window (W = 1) drops PSNR to 32.01, SSIM to 0.941, and increases storage to 298.06 KB. W = 3 gives 32.30 PSNR / 0.944 SSIM / 173.59 KB, and W = 5 gives 32.26 / 0.944 / 176.05 KB, so increasing the window to 5 gives no significant benefit.
• GOP length ablation: GOP-30 gives 32.32 PSNR / 0.944 SSIM / 228.7 KB / 133 s training; GOP-60 (used) gives 32.30 / 0.943 / 173.6 KB / 67 s; GOP-100 gives 32.06 / 0.942 / 161.9 KB / 40 s.
• Component ablation (N3DV): Removing auxiliary training reduces PSNR to 31.99 and training to 29 s; removing dynamic-aware density reduces PSNR to 32.07 while storage drops to 114.15 KB; removing temporal regularization raises storage to 319.48 KB with PSNR 32.23; removing Gaussian relocate reduces PSNR to 32.11.
• Compression QP tradeoff: With libx265, QP 20 is chosen as the optimum (32.30 PSNR / 173.59 KB on N3DV; 27.41 PSNR / 142.53 KB on MeetRoom). Even at QP 28 or 32 the method still outperforms benchmark methods in reconstruction quality while significantly reducing storage size, per the paper.
• Qualitative result: In regions of complex human interaction, StreamSTGS reconstructs hands and objects accurately where comparison methods yield blurred results, which the authors attribute to dynamic-aware density concentrating Gaussians in dynamic areas combined with the transformer module. The paper also states StreamSTGS reconstructs high-light effects despite using a simplified view-independent color model.

Methodology in Plain English

Instead of storing a separate model for every frame, StreamSTGS splits a long video into groups of pictures (GOPs), and each GOP is reconstructed independently. Within a GOP, the scene is described by a set of canonical 3D Gaussians that live in a reference space, a set of temporal features, and a small deformation field implemented as MLPs. The deformation field takes a sliding window of adjacent temporal features plus a position-encoded timestamp and predicts how each canonical Gaussian should move, change size and orientation, change opacity, and change color at that moment.

The key trick for streaming is formatting. The canonical Gaussian attributes are sorted into 2D grids using the PLAS algorithm and written out as attribute images, while the temporal features become a feature video. Because images and video are already well-supported by standard codecs, the scene can be compressed and streamed at different bitrates for different network conditions with no retraining. To make compression effective, the authors add a spatial smoothness loss on the attribute images and a temporal consistency loss on neighboring feature frames, using a Huber loss so that static Gaussians are held consistent (reducing video size) while dynamic Gaussians are smoothed in a way that tolerates outliers.

Because a short sliding window only sees local motion, the authors add a transformer in a second "auxiliary pass" during training that looks at all timestamps as a sequence and learns global motion. The deformation field is shared between the normal pass and the auxiliary pass, and a self-distillation loss pulls the representation's features toward the transformer's output. At inference the transformer is simply deleted, so FPS is not reduced. Two more techniques address quality under a Gaussian budget capped at approximately 150k: dynamic-aware density weights the SSIM loss toward dynamic pixels (identified by per-camera standard deviation across 30 frames, or optionally SAM), and a relocate operation moves unneeded Gaussians to better positions rather than pruning them.

Why This Matters

• Research impact: The paper argues that decoupling spatial and temporal components, and formatting them so that existing image and video codecs apply directly, is a route around the storage burden that has kept 3DGS-based FVV from being streamable in real time. It also introduces a training-time transformer distillation pattern that improves motion learning without paying a runtime cost.
• Real-world applications:
  • Immersive VR experiences where a viewer chooses their own viewpoint, which the paper lists as the primary application domain of FVV.
  • Education and remote training, where learners move around a captured 3D scene.
  • Industrial and entertainment content, both explicitly cited in the paper as areas with significant potential.
  • Network-constrained delivery of volumetric video, because the image/video encoding supports variable bitrate transmission according to network conditions without extra training.
• Industry relevance: Because the representation is compressed with conventional codecs (lossless JPEG XL for attribute images, libx265 for the temporal feature video) and supports variable bitrate, it maps onto existing video delivery pipelines and adaptive-streaming infrastructure rather than requiring bespoke transport. Reported 100 FPS rendering and 8 ms decoding also matter for latency-sensitive interactive products.

Future Directions

• Static/dynamic separation: The conclusion notes that every 3D Gaussian currently needs temporal features even though some Gaussians stay static within a GOP. Assigning temporal features only to a dynamic subset is proposed as a way to further reduce storage and improve FPS.
• Dynamic masking alternatives: The dynamic mask is computed from per-camera standard deviation over 30 frames against a threshold; the paper suggests SAM or other models could produce masks instead, leaving room to compare masking quality and cost.
• Compression and bitrate policy: QP, sliding window length, and GOP length were each ablated, but the paper treats QP 20, W = 3, and GOP 60 as choices made for a good tradeoff rather than as optimized settings — adaptive selection of these values per scene is left open.
• Method scope: The approach is evaluated on two real-world multi-view datasets (N3DV with six scenes at 18 to 21 cameras, MeetRoom with three scenes at 12 cameras). Behavior on other capture setups, resolutions, or frame rates is not reported.

Target Audience

Researchers and engineers working on volumetric video, free-viewpoint video, telepresence, and real-time neural rendering; practitioners implementing 3D Gaussian Splatting pipelines who need to compress, stream, or transmit dynamic scenes; and readers with a background in 3DGS, deformation fields, and video codecs who want to understand how codec-friendly layouts and training-time transformer distillation can be combined to make dynamic 3DGS practical for real-time delivery.

Authors’ abstract

Streaming free-viewpoint video~(FVV) in real-time still faces significant challenges, particularly in training, rendering, and transmission efficiency. Harnessing superior performance of 3D Gaussian Splatting~(3DGS), recent 3DGS-based FVV methods have achieved notable breakthroughs in both training and rendering. However, the storage requirements of these methods can reach up to $10$MB per frame, making stream FVV in real-time impossible. To address this problem, we propose a novel FVV representation, dubbed StreamSTGS, designed for real-time streaming. StreamSTGS represents a dynamic scene using canonical 3D Gaussians, temporal features, and a deformation field. For high compression efficiency, we encode canonical Gaussian attributes as 2D images and temporal features as a video. This design not only enables real-time streaming, but also inherently supports adaptive bitrate control based on network condition without any extra training. Moreover, we propose a sliding window scheme to aggregate adjacent temporal features to learn local motions, and then introduce a transformer-guided auxiliary training module to learn global motions. On diverse FVV benchmarks, StreamSTGS demonstrates competitive performance on all metrics compared to state-of-the-art methods. Notably, StreamSTGS increases the PSNR by an average of $1$dB while reducing the average frame size to just $170$KB. The code is publicly available on https://github.com/kkkzh/StreamSTGS.