SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation

AI summary

Overview

Research area: controllable text-to-image generation, specifically 3D-aware (9-DoF) pose control of multiple objects. Technical level: Advanced. Scope: A single paper proposing a dataset (ObjectPose9D), a pose encoding (CNOCS map), a two-stage training scheme with reinforcement learning, and an inference technique (Disentangled Object Sampling) for manipulating the 3D location, size, and orientation of multiple objects in generated images.

What This Paper Is About

Most controllable image generation methods work in 2D and rely on dense control maps such as depth images, while the few 3D-aware approaches either lack orientation information, depend on users supplying reference images or precise object shapes, or are tied to one-step generative models. This paper aims to let a user specify, for each object in a scene, its 3D location, size, and orientation from the camera view (the 9 degrees of freedom), and have a standard text-to-image model generate an image that respects all of those constraints simultaneously.

Key Contributions

1. ObjectPose9D dataset: A new dataset aggregating images from diverse sources with 9D pose annotations. It is built from selected subsets of OmniNOCS (common indoor and street-scene objects) plus MS-COCO images annotated for 9D poses, with human annotators filtering low-quality samples and refining inaccurate annotations, and captions generated by a Multimodal Large Language Model.
2. CNOCS map: A new pose representation — Cuboid Normalized Object Coordinate System — that encodes 9D pose from the camera view using a coarse cuboid abstraction rather than a category-specific CAD shape, retaining geometric cues while remaining category-agnostic.
3. Two-stage training with reinforcement learning: Stage one learns basic pose controllability on ObjectPose9D; stage two fine-tunes with a reward-based objective on rebalanced data to improve performance on low-frequency poses.
4. Disentangled Object Sampling (DOS): An inference technique that combines multiple noisy latents at each denoising step using region masks so each object matches its own pose condition, mitigating concept confusion and insufficient generation in multi-object scenes. Personalized user weights can also be loaded for customized pose control of reference subjects.

Main Findings

• Single-object front-facing performance: On ObjectPose-Single-Front, SceneDesigner reaches Acc_ls of 50.20, mIoU of 57.21, Abs.Err of 13.23 degrees, and Acc@22.5° of 89.47. LOOSECONTROL scores 23.89, 27.12, 87.26, and 23.08; C3DW scores 2.02, 19.61, 50.01, and 60.32.
• Back-facing objects: On ObjectPose-Single-Back, SceneDesigner reports Acc_ls 52.56, mIoU 60.66, Abs.Err 17.47, and Acc@22.5° 83.33, versus LOOSECONTROL at 24.36, 30.49, 132.26, and 7.05. C3DW is reported to be unable to generate back-facing objects and to only handle the front 180 degrees of azimuths.
• Multi-object performance: On ObjectPose-Multi, SceneDesigner reports Acc_ls 47.16, mIoU 52.16, Abs.Err 23.14, and Acc@22.5° 80.79, against LOOSECONTROL at 14.85, 22.58, 147.42, and 4.80. C3DW only handles single-object generation and is therefore not compared in this setting.
• Image quality and text alignment: FID is 24.91 for SceneDesigner, 37.89 for LOOSECONTROL, and 67.39 for C3DW. CLIP scores are 0.345, 0.293, and 0.267 respectively.
• Pose conditioning ablation: Replacing the CNOCS map degrades results. C-CNOCS map gives Acc_ls 40.45, mIoU 49.86, Abs.Err 37.86, Acc@22.5° 73.70; direct pose embedding gives 32.51, 40.73, 49.65, 47.15; the prompt-only baseline gives 12.90, 14.32, 88.43, 25.31. The full method on ObjectPose-Single gives 51.12, 58.55, 14.87, 87.10.
• Dataset ablation: Training without MS-COCO data drops ObjectPose-Single performance to Acc_ls 41.69, mIoU 50.07, Abs.Err 74.89, Acc@22.5° 24.32, with poor generalization to unseen classes such as rabbit.
• RL fine-tuning ablation: Removing the second stage gives Acc_ls 43.18, mIoU 50.32, Abs.Err 43.85, Acc@22.5° 52.36; the paper states this checkpoint fails to generate back-facing objects.
• DOS ablation: Removing Disentangled Object Sampling on ObjectPose-Multi drops results to Acc_ls 36.68, mIoU 45.91, Abs.Err 42.92, Acc@22.5° 59.39, versus 47.16, 52.16, 23.14, and 80.79 with it.
• User study: With 20 volunteers rating normalized scores, SceneDesigner leads on image quality (0.96 vs. 0.64 for LOOSECONTROL and 0.47 for C3DW), location fidelity (0.98 vs. 0.88 vs. 0.05), size fidelity (0.96 vs. 0.82 vs. 0.02), orientation fidelity (0.91 vs. 0.39 vs. 0.62), and text-to-image alignment (0.94 vs. 0.51 vs. 0.63).
• Evaluation setup: Benchmarks ObjectPose-Single (split into Front with 247 samples and Back with 156 samples) and ObjectPose-Multi with 229 cases, derived by estimating 9D poses from the COCO validation split. Metrics use Grounding DINO for detection and Orient Anything for orientation; FID reference images are randomly sampled from LAION.
• Which CNOCS variant was chosen: Three variants are defined — C-CNOCS (constant function, e.g. Euler angles), I-CNOCS (identity function on normalized coordinates), and S-CNOCS (spherical harmonics) — and based on empirical results the paper uses I-CNOCS.
• Not reported / not compared: The paper states that the codes of ORIGEN and of Parihar et al. were not open-sourced at the time of the experiment, so they are not discussed in the comparisons. Zero-1-to-3 depends on a user reference image and shows poor generalization to real-world images, so it is likewise not used as a direct T2I comparison.

Methodology in Plain English

The researchers start from the observation that a pose condition must be delivered to the generator in a form the network can actually use. Instead of projecting location, size, and orientation separately into embeddings fed through attention, or using plain 3D bounding boxes that cannot express which way an object faces, they render a control image from the camera's viewpoint. For every pixel, they find where the view ray hits the object's 3D bounding box, translate that point into the object's own coordinate frame, and normalize it using the box side lengths. That produces an RGB-style map carrying position, scale, and orientation together, without needing an exact 3D mesh of each object category.

Training data is assembled by taking OmniNOCS subsets as a base and then annotating MS-COCO images: candidate objects are filtered by mask area and by whether their orientation is unambiguous (ambiguous categories such as "bottle" are excluded), orientation is estimated with Orient Anything and low-confidence predictions are filtered out, scene geometry comes from a 3D reconstruction method, and bounding boxes are fitted to the object points. Human annotators then check and refine everything, and an MLLM writes captions.

The generation model is a ControlNet-style branched network added to a pre-trained Stable Diffusion 3.5 flow-matching backbone, trained on 512x512 images with batch size 48 on 6 NVIDIA A800 80G GPUs, AdamW at a learning rate of 5e-6, for 45K iterations in the first stage. Because real data over-represents certain orientations, a second 5K-iteration reinforcement-learning stage maximizes a reward combining IoU between the detected box and the projected 3D box (location/size) with a KL divergence between the target orientation distribution and the one estimated by Orient Anything. To keep the memory cost feasible, they use randomized truncated backpropagation and gradient checkpointing in the manner of AlignProp, feeding the reward function a coarse image from an intermediate denoising step rather than the clean output.

At inference they sample the whole scene and then, for each object, additionally sample a version conditioned only on that object plus its own map; these are blended back into the scene latent through the object's mask from the CNOCS map. Conditions are injected only in the first 15 of 20 denoising steps.

Why This Matters

This work pushes controllable image generation from 2D layout toward full 3D spatial control over multiple objects at once, which existing methods had not achieved in a way compatible with widely used multi-step diffusion/flow frameworks. It also shows that a compact, category-agnostic geometric encoding can outperform both embedding-based pose injection and the coordinate maps used in prior 3D-aware control work.

Real-world applications suggested by the paper:

• Virtual and augmented reality content creation, where spatial control of objects is essential.
• Product design, where designers need to place and orient items in a scene.
• Scenarios such as arranging multiple pieces of furniture in a room, each with distinct sizes and orientations.
• Image generation where a subject faces away from the camera, e.g. a pet dog turned away and gazing at the landscape ahead.
• Customized pose control of user-provided reference subjects through personalized weights.

Industry relevance: the method targets designers and general users who need 3D-aware layout control without professional equipment or resources, and its compatibility with standard multi-step image generation pipelines (rather than one-step models) makes it more practical to adopt than prior orientation-control approaches. The paper also flags misuse risk for deceptive content in areas such as political manipulation and social media, and proposes ethical guidelines, usage restrictions, and traceability measures as mitigations.

Future Directions

• Controlling the precise shape of objects, which SceneDesigner cannot currently do; it only manipulates pose via cuboid abstractions.
• Improving multi-object performance, which the paper says is constrained by the inherent capability of the base model, since more semantic concepts in the prompt exacerbate insufficient generation and attribute leakage.
• Reducing the extra computation introduced by Disentangled Object Sampling while retaining its disentangling benefit.
• Enhancing alignment with input conditions in multi-object generation while maintaining computational efficiency, which the authors explicitly name as future work.

Target Audience

Researchers and practitioners working on controllable image generation, text-to-image diffusion and flow-matching models, 3D-aware content creation, and dataset/annotation construction for pose-conditioned generation. It is also relevant to applied engineers in AR/VR, design tooling, and generative content pipelines who need multi-object spatial control, and to readers interested in applying reinforcement learning from preference-style objectives to generative models. The paper is written at an advanced level and assumes familiarity with ControlNet-style architectures, NOCS representations, and diffusion or flow-matching sampling.

Authors’ abstract

Controllable image generation has attracted increasing attention in recent years, enabling users to manipulate visual content such as identity and style. However, achieving simultaneous control over the 9D poses (location, size, and orientation) of multiple objects remains an open challenge. Despite recent progress, existing methods often suffer from limited controllability and degraded quality, falling short of comprehensive multi-object 9D pose control. To address these limitations, we propose SceneDesigner, a method for accurate and flexible multi-object 9-DoF pose manipulation. SceneDesigner incorporates a branched network to the pre-trained base model and leverages a new representation, CNOCS map, which encodes 9D pose information from the camera view. This representation exhibits strong geometric interpretation properties, leading to more efficient and stable training. To support training, we construct a new dataset, ObjectPose9D, which aggregates images from diverse sources along with 9D pose annotations. To further address data imbalance issues, particularly performance degradation on low-frequency poses, we introduce a two-stage training strategy with reinforcement learning, where the second stage fine-tunes the model using a reward-based objective on rebalanced data. At inference time, we propose Disentangled Object Sampling, a technique that mitigates insufficient object generation and concept confusion in complex multi-object scenes. Moreover, by integrating user-specific personalization weights, SceneDesigner enables customized pose control for reference subjects. Extensive qualitative and quantitative experiments demonstrate that SceneDesigner significantly outperforms existing approaches in both controllability and quality. Code is publicly available at https://github.com/FudanCVL/SceneDesigner.