SwitchSplat: Learning Discrete Object States from Casually Captured Scenes

AI summary

Overview

Research area: Computer vision / 3D scene reconstruction — specifically novel-view synthesis with 3D Gaussian Splatting (3DGS), applied to casually captured 360° video in which a person physically interacts with objects.

Technical level: Advanced. The paper assumes familiarity with Gaussian Splatting, volume rendering, differentiable optimization, and dynamic (4D) scene representations.

Scope: The paper introduces SwitchSplat, a state-aware 3DGS representation that decomposes a casually captured scene into a shared static base plus per-object "slots" holding two binary states, so that users can render arbitrary unseen combinations of object configurations from any viewpoint.

What This Paper Is About

Current 3D reconstruction pipelines assume the world is static, or — in the case of 4D Gaussian Splatting — condition the scene on time rather than on what state an object is in. That means if someone opens a door, a hood, or a trunk while filming, the model either averages the two configurations into ghosting artifacts or discards the minority state entirely, and it can never render a combination of states that was never filmed.

SwitchSplat's goal is to factor discrete object states out of time: learn which Gaussians belong to which object's "on" and "off" configurations directly from an unannotated casual video, then let a user at test time freely switch each object on or off and render the resulting scene from any novel viewpoint.

Key Contributions

1. Problem identification: The authors identify and formalize a gap in the 3DGS literature — existing representations cannot model scenes with discrete, user-triggered state changes (objects that open, switch, or move), and they show that naïvely applying dynamic methods such as 4DGS fails to address it because time-conditioned models conflate when a state was observed with what state the scene is in.

2. The base + slot decomposition: SwitchSplat is presented as the first 3DGS representation to address this problem, using a shared static base for the invariant scene plus one slot per interactable object, where each slot independently models that object's two binary states (on/off). Each Gaussian carries a learnable state-logit vector over a base channel plus an off and an on channel per slot.

3. A differentiable slot table: Slot activation over time is parameterized as a smooth window function (normalized product of two sigmoids with learnable endpoints and temperature), turning state activation into a per-frame Bernoulli probability. Because the whole pipeline is differentiable, Gaussian-to-slot assignment and slot timing are learned from video without manual annotation.

4. A new casual-capture dataset and evaluation: A 12-scene dataset captured with a consumer Insta360 X4 dual-fisheye 360° camera, including a "golden set" with controlled novel-camera and novel-state-combination test sequences, on which SwitchSplat outperforms static and dynamic baselines.

Main Findings

• Best average PSNR on the main benchmark: Across ten scenes, SwitchSplat (3DGUT) reaches an average PSNR of 32.65 dB, versus 32.41 for SwitchSplat (3DGS), 30.55 for 4DGS, 30.30 for DeGauss, 28.99 for FullCircle (3DGUT), and 28.88 for FullCircle (3DGS).

• Consistent per-scene improvement: SwitchSplat (3DGUT) leads on every reported scene, e.g. 26.97 on Balcony (vs. 26.36 for 4DGS and 25.77 for FullCircle/3DGUT), 32.55 on Kitchen 1 (vs. 30.83 for 4DGS), and 36.04 on the concurrent-indoor scene M Kitchen 1 (vs. 34.10 for DeGauss and 30.95 for 4DGS).

• Generalization to novel cameras: On the golden-set novel-camera split, SwitchSplat (3DGUT) achieves PSNR 34.26 / SSIM 0.975 / LPIPS 0.154 on Kitchen 2 and 32.55 / 0.973 / 0.185 on Oslo, beating DeGauss (32.38 / 0.965 / 0.174 and 29.40 / 0.960 / 0.207) and 4DGS (32.05 / 0.967 / 0.168 and 29.93 / 0.966 / 0.194).

• Rendering states never observed in training: On novel state-combination frames, SwitchSplat (3DGUT) reaches 32.58 PSNR on Kitchen 2 and 29.10 on Oslo, versus DeGauss (30.85 and 27.51) and 4DGS (29.63 and 26.53).

• Dynamic baselines were given an oracle advantage: For the golden-set comparisons, 4DGS and DeGauss were evaluated with a per-image oracle over time (selecting the highest-PSNR rendering from sampled timestamps), so their reported numbers are explicitly upper bounds — and SwitchSplat still outperforms them.

• Cleaner geometry than dynamic baselines: Qualitatively, 4DGS and DeGauss produce pervasive floating artifacts around the car hood, when the refrigerator is in its on state, and during cabinet/wardrobe interactions in the Flat scenes, while SwitchSplat keeps static and state-specific structure disentangled.

• Ablation (partially reported): Ablations are run on the "Kitchen 2" scene with the golden test set, reporting PSNR averaged over the standard 8th-frame test set, novel camera frames, and novel state-combination frames, each averaged over five independent runs. Removing interaction slots (which also removes slot optimization and its regularizers) produces what the authors describe as a substantial performance drop; the truncated text does not include the Table 3 numbers themselves, so the exact ablation values are not reported here.

Methodology in Plain English

The system starts from a casual 360° video in which a person walks around and physically changes objects — opening a car door, lifting a hood, opening a trunk.

Splitting the scene. Everything that never changes (walls, floor, the car body) goes into a shared "static base." Every object that gets interacted with gets its own slot, and each slot stores two separate sets of 3D Gaussians: one for the "off" configuration and one for the "on" configuration. Two separate sets are needed because a closed door and an open door occupy genuinely different regions of space, which a single shared set of primitives cannot represent. Sharing one base across all configurations avoids duplicating the whole scene for every slot.

Assigning Gaussians to parts. Each Gaussian carries a small learnable score vector that says how strongly it belongs to the base, or to the on/off state of each slot. A softmax turns those scores into probabilities. Because assignment is soft and differentiable, the model figures out on its own which primitives belong to which object state, with no masks or human labels.

Knowing when each state is active. The key assumption is that an interaction is compactly supported in time: a slot is "on" only during a bounded window of the video. Each slot gets a smooth window function — a normalized product of two sigmoids with learnable start and end times and a learnable temperature controlling how sharp the transition is. This yields a per-frame probability that the slot is on, and thus a Bernoulli distribution over the slot's state at each time.

Rendering. At training time, the per-frame slot probabilities are combined with each Gaussian's state probabilities to produce a time-dependent effective opacity, and standard volume rendering is applied. At test time, the user instead supplies a hard on/off vector for all slots, producing a state-conditional opacity and letting any combination be rendered from any viewpoint.

Bootstrapping. Since a casual user won't annotate when interactions happen, the number of slots and their approximate time windows are initialized by prompting a vision-language model (the paper specifies GPT-5.6) with a sequence of frames to extract start and end timestamps per interaction. Slots are initialized with a high temperature so gradients flow over a wide temporal range early in optimization.

Training details. The implementation builds on the 3DGUT architecture, initializes Gaussians from COLMAP point clouds with uniform base/on/off membership, and follows 3DGUT's optimization schedule and densification/pruning protocols, with the modification that cloned or split Gaussians inherit their parent's state logits. The loss combines an L1 photometric term, an SSIM term, a sparsity loss (negative log-likelihood on the base probability, pushing regions to stay static unless justified), and a compactness loss (opacity-weighted variance of Gaussians around each slot's center of mass, encouraging spatially cohesive clusters).

Data and evaluation protocol. Capture used an Insta360 X4; raw dual-fisheye data is preprocessed following FullCircle to mask the camera operator, calibrated in COLMAP, and undistorted to pinhole images. The dataset has 12 scenes: 10 standard scenes (8 indoor, 2 outdoor) with 3–9 object interactions each, plus 2 golden-set scenes with controlled novel-camera and novel-state test sequences. Every 8th frame is held out, but test frames coinciding with active state transitions are excluded to reduce metric noise, while those transition frames remain in the training set.

Why This Matters

For research: The paper reframes an assumption that has been quietly baked into almost all 3DGS pipelines — that scene content can be indexed by time (dynamic) or treated as fixed (static). It argues for a third axis, discrete object state, and shows that 4D methods, even when given a per-image oracle over time, cannot compose configurations that were never co-observed. The base + slot design also draws an explicit analogy to PCA-based morphable models, where the static base is the mean and each slot's on/off representations are the eigenvectors.

Real-world applications:

• Digital twins and real-estate tours: capture a home once while opening closets, cabinets, and appliances, then let a viewer toggle each one instead of re-scanning the property per configuration.
• Product and furniture visualization: film a configurable object (a car with its hood, trunk, and doors) once, then render any combination of open/closed states for e-commerce or configurators.
• Interactive training and simulation content: generate scene variants for robotics or VR walkthroughs without physically re-staging each configuration.
• Consumer mobile scanning: the pipeline tolerates non-expert capture with a handheld 360° camera and automatically masks the operator, which lowers the barrier for casual users of apps in the vein of Scaniverse or Hyperscape.

Industry relevance: The authors situate this within commercial 3D capture ecosystems (Matterport digital twins, Niantic Scaniverse, Meta Horizon Hyperscape), where the inability to represent a scene in more than one configuration forces redundant re-capture. Since the method builds on 3DGUT and consumes standard 360° video, it is architecturally close to deployable pipelines.

Future Directions

• Beyond binary states: each slot holds exactly two states (on/off); objects with more than two meaningful configurations — multi-position recliners, multi-leaf doors, multi-setting appliances — are not modeled.
• Removing the VLM dependency: slot count and timing are bootstrapped by prompting GPT-5.6 with frames, so the pipeline inherits an external model's judgment; learning slot discovery fully from photometry alone is an open question.
• Handling non-rigid and fuzzy interactions: the authors contrast their approach with state-aware methods that assume rigid motion of segmented entities, but SwitchSplat itself discards the transient motion between states rather than modeling it — how well it handles deformation-heavy interactions is untested.
• Full ablation reporting and scaling: the truncated text cuts off before the ablation numbers (Table 3) and the remaining ablation variants are not visible; broader study of how the method scales with many simultaneous interactions across larger environments is a natural next step.

Target Audience

Researchers and practitioners in 3D vision and graphics working on Gaussian Splatting, novel-view synthesis, and neural scene representations — particularly those interested in controllable or editable reconstructions. It is also relevant to engineers building consumer 3D capture products (digital twins, real-estate tours, mobile scanning) who need scene representations that survive human interaction during capture. Readers without a background in differentiable rendering and 3DGS will find the methodology section demanding.

Authors’ abstract

3D Gaussian Splatting has made high-fidelity 3D reconstruction fast and accessible, yet existing representations assume a static world. This assumption breaks down in casual captures, where a user naturally interacts with the environment -- opening doors, toggling switches, or moving objects -- introducing discrete state changes that no current framework can model. Time-conditioned dynamic methods such as 4DGS conflate when a state was observed with what state the scene is in, limiting them to replaying seen sequences rather than composing arbitrary object configurations. We present SwitchSplat, the first 3DGS representation to address this gap via a base+slot decomposition: a shared static base captures the invariant scene, while each slot independently holds the two binary states (on/off) of one interactable object. Because the assignment of Gaussians to slots is fully differentiable, the model discovers the geometry of each state directly from video, without manual annotation. At inference, the user can freely combine slot states and render the resulting scene from any viewpoint. We validate SwitchSplat on a new casual-capture dataset and show that it outperforms static and dynamic baselines in novel-view synthesis quality, while uniquely supporting rendering of state combinations never seen during capture. Project page: https://switchsplat.github.io