SpaceFlow: Locally Controllable 3D Generation

AI summary

Overview

Research area: Computer vision and 3D generative modeling, specifically spatially controllable text- and image-conditioned 3D asset generation.

Technical level: Advanced. The paper assumes familiarity with rectified flow models, structured latents (SLAT), sparse voxel grids, cross-attention, and training-free test-time guidance.

Scope: The paper introduces a training-free inference-time framework built on the TRELLIS 3D generator that allows users to control, per region, both how strictly the generated geometry follows an input shape and which text or image appearance cue applies to which part.

Venue/status: arXiv:2610.12399v1 [cs.CV], dated 08 Oct 2026, licensed CC BY 4.0. Affiliations listed are ETH Zürich, Stanford University, and Microsoft.

What This Paper Is About

Existing 3D generators can turn a text prompt or image into a textured asset, but they apply condition controls uniformly across the whole object. If you want one part of an object to strictly match an input shape while another part is left free for the model to invent, or you want a red cushion only on the cushion and not the whole sofa, current methods cannot do it. SpaceFlow addresses this by letting users assign a geometric control strength and an appearance cue to each individual part of a decomposed input, so instructions apply only where they are meant to.

Key Contributions

1. Local geometric control. The paper introduces a method to vary the strength of geometric conditioning spatially across a generated asset, so that some parts strictly preserve user-specified geometry while others are left to the generative prior.

2. Local appearance conditioning. The paper introduces routing of separate text or image appearance cues to specific semantic parts of the generated structure, using PartField descriptors and K-means clustering to build a per-voxel routing volume that stays consistent even when low-control geometry drifts from its original boundary.

3. A unified framework. Both control mechanisms are combined within a single training-free pipeline built on TRELLIS, exercised jointly on multi-part assets in the qualitative results.

4. An evaluation suite. The paper reports an 83-conditioning evaluation dataset, a vision-language-model judge protocol (using GPT 5.6 Sol), regional geometry and feature-space metrics, and a human study of 337 trials across 29 evaluators.

Main Findings

• Localized geometric control beats uniform control on fidelity metrics. Against SpaceControl at τ = 3, SpaceFlow wins 78.3% on prompt and geometric fidelity (95% CI [68.3, 85.8]) and 75.9% overall (95% CI [65.7, 83.8]), but only 43.4% on realism (95% CI [33.2, 54.1]).

• Against strong uniform control, realism flips in SpaceFlow's favor. Against SpaceControl at τ = 10, SpaceFlow wins 66.3% on prompt and geometric fidelity (95% CI [55.6, 75.5]), 85.5% on realism (95% CI [76.4, 91.5]), and 68.7% overall (95% CI [58.1, 77.6]).

• Regional metrics show genuine spatial selectivity. SpaceFlow achieves a Chamfer Distance delta (CD_low − CD_high) of 6.05 and a feature deviation delta of 0.15, while both SpaceControl baselines sit near zero (−0.14 and 0.01 for τ = 3; 0.11 and 0.00 for τ = 10). Its CD_high is 3.65, versus 15.99 for τ = 3 and 0.94 for τ = 10.

• Semantic alignment is not sacrificed. CLIP text-image similarity is comparable across methods: 0.233 for SpaceFlow, 0.232 for SpaceControl at τ = 3, and 0.226 at τ = 10.

• Human evaluators prefer SpaceFlow overall. Across 337 trials and 29 evaluators, SpaceFlow wins 48% of comparisons against the τ = 3 baseline (13% ties, 39% losses) and 71% against the τ = 10 baseline (10% ties, 19% losses).

• Local appearance routing improves prompt faithfulness. In pairwise judging, SpaceFlow beats TRELLIS with a 63.9% prompt-faithfulness win rate (95% CI [53.1, 73.4]) and 59.0% overall (95% CI [48.3, 69.0]), and beats GuideFlow3D with 58.5% prompt faithfulness (95% CI [47.7, 68.6]) and 57.3% overall (95% CI [46.5, 67.5]).

• Both baselines win on texture detail, however. SpaceFlow's texture-detail win rates are 31.3% against TRELLIS (95% CI [22.4, 41.9]) and 37.5% against GuideFlow3D (95% CI [27.7, 48.5]), which the authors attribute to localized routing prioritizing part-level prompt adherence at some cost to unconstrained texture quality.

• Absolute ratings favor SpaceFlow overall. On a 1–10 scale over 83 cases, SpaceFlow scores 7.07 prompt faithfulness, 6.43 color/material accuracy, 6.49 texture detail, and 6.78 overall, compared with 6.13/5.79/6.87/6.42 for TRELLIS and 6.40/5.96/6.76/6.46 for GuideFlow3D.

• Image-conditioned routing works standalone. Scored without baselines (which cannot process per-part image conditions), SpaceFlow reaches 6.53 style fidelity, 7.07 local routing accuracy, 6.33 texture detail, and 6.73 overall.

Methodology in Plain English

SpaceFlow builds on TRELLIS, a pretrained 3D generator that produces assets in two stages: first a coarse sparse voxel structure, then the local geometry and appearance features attached to those voxels.

The user supplies the scene layout as a set of editable superquadric primitives (compact, easily edited shapes), each tagged with a control strength of either τ_low = 3 or τ_high = 10 and optionally an appearance cue.

For geometry, the method splits the primitives into high-control and low-control groups. It encodes both the full set and the high-control subset into TRELLIS's latent space. The full set is noised to the τ_low timestep and denoising begins, which fixes the coarse layout of the flexible parts. At each denoising step, the clean high-control latent is blended back into the latent state inside the high-control regions, using a spatially masked weighted average. A low-control mask is built by fitting a bounding box around the low-control superquadrics with 7% diagonal padding, voxelizing at 64³, downsampling to the 16³ latent resolution with 4×4×4 average pooling, and binarizing at > 0.

A RePaint-inspired resampling loop runs K times over the interval [τ_low, τ_high]: noise is added back to the earlier timestep, the model denoises forward one solver step, and the masked blending is re-applied. This smooths the transition between constrained and free regions.

For appearance, the generated structure is decoded to a mesh and a PartField triplane feature field is extracted. Features at active voxels are clustered with K-means into K_PF semantic clusters, and each cluster is assigned as a whole to its nearest input primitive. This produces a routing volume that stays part-consistent even when low-control geometry has moved away from its original scaffold.

Each cue is encoded separately (frozen CLIP for text, DINOv2 for images). Inside cross-attention, each latent voxel attends only to the embedding its routing volume selects, so cues stay in their region. Because self-attention can still smear features across parts, a soft bias of log β (β > 1) is added to attention logits for query-key pairs sharing the same condition. Finally, a test-time supervised contrastive loss over the PartField clusters pulls features within a part together and pushes different parts apart, and the lowest-loss latent along the guidance trajectory is selected for decoding. No network weights are modified at any point.

Why This Matters

Impact on research. The paper attacks a limitation the authors identify in prior controllable generation work: conditioning is applied globally, so a single control strength must compromise across the entire asset. SpaceFlow demonstrates that spatially varying both geometric control strength and appearance routing is possible in a training-free setting, and provides regional metrics (Chamfer Distance delta, PartField feature deviation delta) designed to measure spatial selectivity rather than whole-object quality alone. It also provides an alternative to post-hoc texturing, positioning appearance control inside the generative flow process.

Real-world applications:

• Game and immersive media asset creation, where designers iterate on individual parts (a character's frame, a vehicle's fins) rather than regenerating whole objects.
• Digital content creation and 3D printing workflows, where certain regions must match specified dimensions exactly while others can be synthesized freely.
• Virtual try-on and product configuration, where a base shape stays fixed and materials or colors are swapped per component.
• Scene layout prototyping from proxy shapes such as superquadrics or decomposed meshes, with per-region style references for mood or material direction.

Industry relevance. The method requires no retraining and no modification of pretrained weights, which lowers the cost of adoption for teams already using TRELLIS-family generators. The paper reports funding from the Rafael del Pino Foundation, an ETH AI Center doctoral fellowship, SNSF Advanced Grant 216260, and the Stanford Doerr School of Sustainability.

Future Directions

• Reducing the texture-detail trade-off. SpaceFlow loses to both TRELLIS and GuideFlow3D on texture detail in pairwise judging, and the authors attribute this to routing prioritizing prompt adherence. Closing that gap without losing part-level fidelity is an open problem.
• Scaling the evaluation. The findings rest on an 83-conditioning dataset, a 337-trial user study with 29 evaluators, and a single VLM judge (GPT 5.6 Sol). Broader asset categories and additional judges would test generalization.
• Robustness of part-to-primitive assignment. The routing relies on clustering PartField descriptors and assigning whole clusters to primitives; the paper notes that direct per-voxel geometric assignment is unreliable when low-control regions are reshaped. How this degrades on highly deformed or ambiguous geometry is not settled.
• Extending beyond the two-level control scheme. Control strength is currently binary (τ_low = 3 or τ_high = 10). Whether finer gradations of control, or control over other attributes such as topology or articulation, are achievable within the same framework is untested.

Target Audience

Researchers and practitioners in 3D generative modeling, especially those working on controllable generation, training-free guidance, or diffusion/flow-based 3D synthesis. It is also relevant to technical artists and graphics engineers evaluating whether generative 3D tools can fit into iterative, part-specific production pipelines, and to readers interested in VLM-as-a-judge evaluation protocols for 3D assets.

Authors’ abstract

Current 3D generation methods lack explicit local control: geometric adherence is often defined by a global control strength, and appearance cannot be specified locally. We present SpaceFlow, a training-free pipeline for locally controllable 3D generation from text descriptions and a collection of geometric primitives. Each primitive serves as a proxy for an object part and is assigned a local control level, enabling users to specify whether regions should strictly follow the input shape or allow generative completion. During structure generation, we enforce these spatial constraints within the generative flow process. For appearance synthesis, the generated structure is segmented and matched to the primitives. Each generated part is conditioned only on its assigned text or image cue, thereby limiting cross-part leakage. Regional geometry metrics demonstrate that SpaceFlow preserves the specified geometry in high-control regions and enables plausible shape variation in low-control areas. A user study further indicates that the resulting balance between geometric fidelity and generative freedom remains competitive in overall quality. When evaluating appearance on fixed geometry, text-conditioned routing achieves state-of-the-art prompt faithfulness and color/material accuracy. Qualitative results additionally show localized routing of image cues. The project page is available at SpaceFlow3D.github.io.