Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation

AI summary

Overview

Research area: Computer vision and generative AI, specifically diffusion-based image and video editing with geometric (camera pose) control.

Technical level: Advanced. The paper assumes familiarity with diffusion models, flow matching, ControlNet-style conditioning, VAE latent spaces, and camera extrinsics/axis-angle representations.

Scope: A single paper describing Ctrl&Shift, an end-to-end diffusion framework that relocates and reorients objects in images and videos under explicit relative camera pose control, without explicit 3D reconstruction at inference, plus the new GeoEditBench benchmark.

What This Paper Is About

Moving or rotating an object in a photo or video while keeping the scene realistic is hard: geometry-based methods give precise control but need explicit 3D reconstruction and generalize poorly, while diffusion-based editors generalize well but cannot follow precise geometric instructions. Ctrl&Shift tries to break that trade-off by decomposing object manipulation into object removal and reference-guided inpainting driven by a relative camera pose descriptor, all inside one 2D diffusion process.

Key Contributions

1. Conceptual innovation: An end-to-end framework for geometry-consistent object manipulation that injects relative camera pose control directly into the diffusion process, without requiring explicit 3D representations at inference.
2. Architectural design: A multi-task, multi-stage training strategy that disentangles background, object identity, spatial masks, and geometric transformations by assigning each signal a distinct role across three tasks (main manipulation, object removal, reference inpainting with camera control).
3. Data construction pipeline: A scalable method producing real-world paired supervision with camera control, using 3D object reconstruction, differentiable camera pose estimation, rendering of novel views, and a learned object-pasting (harmonization) network.
4. Systematic benchmarking: Evaluation across multiple datasets plus the introduction of GeoEditBench, a new benchmark of 346 curated image pairs for geometry-aware editing, with reported state-of-the-art results in fidelity, controllability, and viewpoint consistency.

Main Findings

• Zero-shot object translation (ObjectMover-A): Ctrl&Shift reports PSNR 28.69, DINO 88.07, CLIP 93.58 and DreamSim 0.075, compared with ObjectMover at 25.27 / 85.07 / 93.16 / 0.142 and MagicFixup at 23.82 / 78.49 / 91.06 / 0.198. The paper describes this as outperforming existing approaches by a large margin.
• GeoEditBench results: Ctrl&Shift reports PSNR 28.71, DINO-Score 85.23, CLIP-Score 90.44, DreamSim 0.112, Pose MAPE 17.70%, and Obj IoU 0.83. The strongest listed baseline, Nano-Banana, reports 26.38 / 78.05 / 85.63 / 0.145 / 24.36% / 0.78.
• Pose control vs. spatial-signal methods: Methods supplied with spatial signals (Drag-Anything, VACE, Nano-Banana) achieve relatively high DINO, CLIP and DreamSim scores and improved object IoU, but exhibit higher pose MAPE. Methods relying only on language and coordinates (3DiT, Flux-Kontext) struggle with spatial accuracy.
• Stage I matters for geometry: Removing Stage 1 increases Pose MAPE to 32.50% and reduces Obj IoU to 0.68.
• Stage II matters for photorealism: Removing Stage 2 drops PSNR to 24.83 and raises DreamSim to 0.195.
• Auxiliary tasks both contribute: Dropping Aux 1 reduces CLIP-Score to 86.32 and DINO-Score to 80.47 with Pose MAPE rising to 23.80%; dropping Aux 2 reduces Obj IoU to 0.65 and raises Pose MAPE to 28.60% while lowering CLIP-Score to 83.75.
• Remaining difficulty: The authors state that precise object manipulation in real-world images remains a complex challenge, and that video manipulation exhibits lower visual fidelity than image editing.

Methodology in Plain English

The model takes a source video or image, a reference image of the object cropped from the first frame, a mask marking the object to be removed, an estimated target mask showing roughly where the object should go, and an 8-dimensional relative camera pose descriptor. It outputs edited frames with the object moved and rotated to match the new viewpoint.

Representing the camera change. The object is treated as sitting at the world origin, with a look-at camera described by yaw, pitch, distance, and two normalized-device-coordinate shifts. For the source and target views the pipeline computes the relative rotation and translation, then encodes the relative rotation as an axis-angle vector. These components are concatenated into the 8-dimensional descriptor f, Fourier-position-encoded and passed through lightweight MLPs into 8 tokens of dimension 4096. Relative rather than absolute pose is used so that control behaves like an intuitive drag relative to the input view.

Architecture. Ctrl&Shift uses a ControlNet-style DiT with hidden size 1536, built on the Wan-1.3B backbone with eight control blocks in the conditioning branch. Source frames and the reference image are encoded by a VAE; masks are not VAE-encoded (which would confuse their binary semantics with appearance) but instead mapped to the latent grid using a space-to-depth/pixel-unshuffle operation matching the VAE stride. Camera conditioning enters via cross-attention; visual conditioning is injected into DiT blocks through zero-initialized convolutions.

Estimating the target mask. Because the true target mask is unknown at inference, it is approximated from the source mask by tightening the source mask to its bounding box, scaling by the source-to-target distance ratio, shifting on the image plane, and truncating anything outside the frame.

Training decomposition. Three tasks share one conditioning interface. The main task performs full object manipulation. Auxiliary task 1 performs object removal by using a pure white reference image, an all-zero target mask, and a pose descriptor with NDC shifts pushed beyond [-1, 1] so nothing is placed in frame, with the clean background as supervision. Auxiliary task 2 performs reference inpainting with camera control by setting the source mask to zeros and feeding background frames, with the target frames as supervision. Tasks are weighted 8:1:1 (main:aux1:aux2). Training uses flow matching: a linear interpolation path between the target latent and noise, with a velocity network trained to predict the noise-minus-latent velocity.

Two-stage training. Stage I pretrains on roughly 2M synthetic image pairs generated by rendering 3D object meshes under randomized camera poses against uniform white backgrounds, for 50k steps with AdamW at learning rate 5×10⁻⁵ and a One-Cycle scheduler, jointly optimizing both branches. Stage II freezes the main branch and fine-tunes only the conditioning branch for 5k steps on a curated set of 100K high-quality image and video pairs, targeting background preservation and photorealism.

Data pipeline. Starting from approximately 400k Pexels videos (first frame used as source) plus roughly 100k object-centric images synthesized with HunyuanT2I, the authors filter out non-manipulable categories and broken instances. Masks come from Grounded-SAM-2; textured watertight meshes from Hunyuan3D-2, yielding about 100k meshes after filtering. Source pose is recovered by optimizing the rendering silhouette's IoU against the source mask with differentiable rendering, keeping only instances with IoU ≥ 0.90. Target views are moderate perturbations of the source pose, rendered as RGB. The original object is removed with MiniMax-Remover to get a background plate, and an object-pasting network trained in a self-reconstruction regime blends the rendered object in with color matching, relighting and boundary harmonization. For video, mesh reconstruction, pose estimation and rendering are all done on the first frame.

Inference setup. Mixed precision with bfloat16, DeepSpeed ZeRO stage 2, trained on 32 A100 GPUs at 640×960 for images and 61×640×960 for videos. Camera pose conditioning is dropped with probability 0.1 during training; inference uses classifier-free guidance scale 1.5 with the UniPC sampler for 40 steps.

Why This Matters

Impact on research. The paper argues for a shift away from lifting content into 3D: instead of reconstructing geometry at inference, it conditions a 2D diffusion process on relative camera pose. It also contributes GeoEditBench, a 346-pair benchmark aimed at measuring geometry-aware editing rather than only perceptual quality, and a reusable real-world paired-data pipeline with pose annotations.

Real-world applications:

• Film post-production, where a prop's placement or a camera angle in a frame must be adjusted while remaining photorealistic.
• Augmented reality previews, such as changing the camera angle of a product in an AR view.
• Creative visual editing tools that let users relocate or reorient objects in photos and short videos.
• Product and content creation workflows needing controllable placement under viewpoint changes.

Industry relevance. The framework is built on an open Wan-1.3B backbone with a ControlNet-style branch, is trained on widely used open components (Hunyuan3D-2, Grounded-SAM-2, MiniMax-Remover), and avoids explicit 3D modeling at inference, which matters for deployment cost. The authors span academic institutions and industry labs, and the stated goals align with tooling needs for AR, film, and generative content platforms.

Future Directions

• Usability of the pose descriptor: Mapping 2D mouse drags, rotational gestures, or 3D gizmo input analytically onto the 8D descriptor so users do not have to specify pose parameters by hand.
• Physical realism beyond geometry: Explicitly modeling lighting, variable shadows and specular reflections, since the current data-driven approach can fail when an object moves to locations with drastically different illumination.
• Generalization boundaries of the data pipeline: Rigid-geometry assumptions exclude non-rigid objects (cloth, hair) and topological changes (smoke, fluids); transparent or highly reflective materials (glass, mirrors) get reconstructed with baked-in background textures; and the remove-and-inpaint strategy cannot handle complex depth relationships such as moving an object behind another scene element.
• Video manipulation quality: Leveraging 4D representations to handle non-rigid deformations and complex spatiotemporal dynamics, which the current rigid reconstruction pipeline does not capture.

Target Audience

Researchers and practitioners in generative vision working on controllable image and video editing, diffusion model conditioning, and geometry-aware synthesis. It is also relevant to graphics and VFX engineers evaluating whether learned 2D diffusion can replace explicit 3D pipelines, and to benchmark developers interested in geometry-aware editing evaluation. Readers need a working understanding of diffusion models, flow matching, and camera pose representations to follow the method section in detail.

Authors’ abstract

Object-level manipulation, relocating or reorienting objects in images or videos while preserving scene realism, is central to film post-production, AR, and creative editing. Yet existing methods struggle to jointly achieve three core goals: background preservation, geometric consistency under viewpoint shifts, and user-controllable transformations. Geometry-based approaches offer precise control but require explicit 3D reconstruction and generalize poorly; diffusion-based methods generalize better but lack fine-grained geometric control. We present Ctrl&Shift, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations. Our key insight is to decompose manipulation into two stages, object removal and reference-guided inpainting under explicit camera pose control, and encode both within a unified diffusion process. To enable precise, disentangled control, we design a multi-task, multi-stage training strategy that separates background, identity, and pose signals across tasks. To improve generalization, we introduce a scalable real-world dataset construction pipeline that generates paired image and video samples with estimated relative camera poses. Extensive experiments demonstrate that Ctrl&Shift achieves state-of-the-art results in fidelity, viewpoint consistency, and controllability. To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation, without relying on any explicit 3D modeling.