Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models

AI summary

Overview

Research area: Computer Vision, specifically text-driven 3D and 4D scene editing powered by vision-language models and large language model (LLM) agents.

Technical level: Advanced. The paper assumes familiarity with neural scene representations (NeRF, 3D/4D Gaussian-splatting), hash-based neural encodings, vision-language models, and tool-calling LLM agents.

Scope: The paper introduces Hash-Atlas, a network that maps 3D and monocular 4D scene views onto 2D foreground/background atlases, and CE3D++, a dialogue framework in which an LLM interprets free-form user text and schedules visual tools to edit those atlases.

What This Paper Is About

Existing text-driven 3D editing frameworks tie 2D editing models tightly to a 3D reconstruction pipeline (iterative, supervised, or denoised designs), which makes swapping in new visual models difficult and forces users into fixed input patterns. The authors separate the two stages: a Hash-Atlas network converts scene views into editable 2D atlases, and an LLM-centered dialogue system called CE3D++ reads arbitrary user text, decides which visual tools to call, and has them operate on the atlases. The same framework is extended to monocular 4D scenes by constraining moving objects during atlas mapping.

Key Contributions

1. Hash-Atlas network. A hash-structured network that translates 3D and 4D scene editing into manipulation of 2D atlases, decoupling the editing and reconstruction processes so that conventional intricate pipelines are avoided. It maps each view pixel to two UV coordinates plus a foreground weight, and reconstructs edited views without retraining.

2. CE3D++ dialogue framework. An LLM-based system with a mechanism for parsing user text and formulating responses, an Executor implementing atlas-editing logic, and a Backend/Frontend managing multiple visual models and scene files. It supports free-form text rather than fixed input templates.

3. Trajectory-tuning dataset and 4D extension. The authors build a trajectory-tuning dataset (1,000 dialogue samples) that lets lightweight LLMs reliably orchestrate more than 30 different vision tools, and add a pivot motion loss so the Hash-Atlas model also handles dynamic 4D scenes.

4. Extensive experiments. Comparisons against 3D and 4D editing baselines, demonstrations of richer editing functionality, and analysis of how reliably different LLMs perform the editing tasks.

Main Findings

• Atlas quality beats the LNA video-atlas baseline. Across six dataset groups (LLFF, TanksAndTemples, IBRNet-collect, CE3D-collect, Dycheck, DynamicNeRF), Hash-Atlas improves PSNR by 1.1 to 5.1 dB, accelerates training by 14.2 to 18.6x, and increases inference FPS by 7.6 to 9.0x. For example, on LLFF PSNR rises from 23.04 to 24.66, training time falls from 10.75 h to 0.60 h, and FPS rises from 0.66 to 5.93; on DynamicNeRF PSNR goes from 24.32 to 28.26 and training from 11.52 h to 0.69 h.

• Atlas fidelity metrics improve consistently. LPIPS drops sharply, e.g., 0.40 to 0.17 on LLFF, 0.53 to 0.12 on CE3D-collect, and 0.58 to 0.14 on Dycheck; SSIM rises, e.g., 0.63 to 0.84 on CE3D-collect.

• Better CLIP alignment than editing baselines. CE3D++ scores 0.304, 0.352 and 0.321 CLIP Similarity on LLFF, CE3D-collect and NeRF-Art respectively, versus 0.188/0.212/0.201 for EditSplat, 0.224/0.240/0.229 for RoMaP, and 0.193/0.233/0.218 for DN2N. On 4D it reaches 0.313 and 0.320 (DyCheck, DynamicNeRF) versus 0.223/0.251 for Instruct-4DGS and 0.236/0.267 for Dynamic-eDiTor.

• Higher CLIP Directional Score. CE3D++ records 0.192, 0.248 and 0.211 on the three 3D datasets, compared with 0.121/0.153/0.136 for EditSplat, 0.145/0.181/0.159 for RoMaP and 0.129/0.174/0.157 for DN2N; on 4D it reaches 0.193 and 0.214.

• Faster editing and competitive memory. 3D editing takes CE3D++ 5.6, 8.8 and 8.4 minutes (DN2N: 26.5, 36.3, 31.5; RoMaP: 21.5, 24.9, 24.2; EditSplat: 16.4, 18.7, 17.7). 4D editing takes 15.9 and 8.2 minutes versus CTRL-D at 66.8/35.4, Dynamic-eDiTor at 51.2/25.8, and Instruct-4DGS at 43.5/22.6. VRAM peaks are 12.6/12.7/11.9 GB for 3D and 17.4/14.2 GB for 4D.

• Ablations confirm each component matters. Removing the initial loss, the inpainting loss, or the Executor lowers CLIP Similarity and CDS across datasets; the "w/o SSN" variant fails to produce edited results. On DynamicNeRF, removing the motion loss drops CLIP from 0.320 to 0.295.

• Trajectory tuning rescues small LLMs. Mainstream LLMs handle the tasks generally, while LLaMA-8B and Qwen-14B show noticeable gaps before fine-tuning; after fine-tuning with the trajectory data they improve significantly. For Qwen-14B, failure rates fall from 17% to 7% for Conditional Editing, from 21% to 10% for Hard complexity, and from 7% to 3% for Enhancement.

• The tuned models generalize to unseen tools. The test set contains eight visual tools absent from the training data, and the fine-tuned model preserves scheduling accuracy on them.

• Pivot-frame selection is the most robust option. In 4D ablations, pivot frame gives PSNR 28.21 versus 28.02 (first frame), 27.93 (median frame) and 27.86 (last frame). Higher atlas resolution helps up to a point (25.17 at 250x250, 27.59 at 500x500, 28.21 at 1000x1000, 28.35 at 1500x1500), and longer video or more moving objects hurt (33.64 at 10 frames down to 27.46 at 120 frames; 28.85 with one moving object down to 26.46 with three).

• Demonstrated dialogue capability. Fig. 8 shows a 12-round dialogue, and CE3D++ supports object removal or replacement, text- and image-driven style transfer, depth prediction, scene regeneration, VQA about a scene, and basic text dialogue.

Methodology in Plain English

The pipeline starts by turning a scene into a picture you can edit with ordinary image tools. A network called F_m takes a pixel at position (x, y) in view t and predicts two UV coordinates plus a transparency value α in [0, 1]. A second network, F_h, uses a multi-resolution hash encoding to predict RGB values at those UV coordinates. One UV coordinate is squeezed into [0, 0.5] for the foreground atlas and the other into [0.5, 1] for the background atlas, so the same network weights serve both. Any original pixel is then rebuilt as α times the foreground color plus (1 − α) times the background color, which means edited atlases can be mapped back to scene views without retraining.

Training proceeds in stages. Early on, only the first view is used with a positional loss that keeps coordinates close to where they started, plus a loss on α that uses a VQA model to identify the foreground and a segmentation model to produce a mask, with a sparsity term to keep foreground and background cleanly separated. Afterwards, the model is supervised on reconstruction, and because background atlas content gets dropped, the authors run ProPainter to inpaint masked backgrounds and add an inpainting reconstruction loss, plus rigid and flow constraints borrowed from atlas work on video.

For 4D, movement wrecks the atlas. The authors pick the view where the moving object covers the most pixels as the "Pivot-View," sample key points on the object, and use CoTracker3 to track those points across nearby frames. A motion loss forces all tracked positions to map to the same atlas coordinates, keeping the dynamic object stable.

Editing itself uses a merge-split strategy: the LLM and a VQA model identify the region to change; if it is background-only, the background atlas is edited directly, otherwise the foreground atlas is overlaid on the background, edited as one image, and then separated again using the original foreground mask and the new object mask.

The dialogue layer hides file complexity from the LLM by representing scenes as a meaningless string such as "xxx.scn," which prevents the model from inventing scene names; a Frontend assembles replies and a Backend allocates real scene files. Each visual tool is annotated with its name, when to use it, required parameters, and input examples. Because pre-injecting knowledge for 30+ tools degrades smaller models, the authors fine-tune on 1,000 trajectory samples covering tool attributes, scheduling reasoning, and decision-making.

Why This Matters

The work turns 3D and 4D scene editing from a bespoke engineering problem into a workflow that reuses the fast-moving ecosystem of 2D image models, and puts a natural-language interface on top of it. Instead of designing a new pipeline for each editing effect, the system swaps tools in and out through an LLM that also handles multi-round conversation.

Real-world applications:

• Interactive 3D content creation and design tools where artists type instructions rather than tune pipelines.
• AR/VR and gaming workflows that need object replacement, relighting, or stylization inside reconstructed scenes.
• Film and video post-production for monocular dynamic footage, where 4D editing covers moving subjects.
• Downstream simulation or visualization pipelines that need scene edits (clarity enhancement, defocus, recoloring, depth and normal map extraction) computed quickly.

Industry relevance: The measured editing times (as low as 5.6 minutes for 3D and 8.2 minutes for 4D) and VRAM peaks (11.9–17.4 GB) matter for deploying such tools on practical hardware, and the demonstration that a fine-tuned 8B or 14B model can schedule 30+ tools suggests the approach is not locked to expensive proprietary LLM APIs.

Future Directions

• Robustness in 360-degree scenes. The authors report that atlas distortions in 360-degree scenes can reduce compatibility with 2D visual models and cause editing failures.
• Better tracking under difficult motion. Rapid non-rigid deformation or severe occlusion can break the tracking model's trajectories and produce artifacts in the generated atlas.
• Scaling tool coverage and dialogue data. The trajectory dataset contains only 1,000 dialogue samples; whether more data or more tools continues to improve scheduling accuracy is left open.
• Extending beyond the evaluated representations. Atlas quality is measured with a fixed set of reconstruction backends (TensoRF, 3D Gaussian-splatting, 4D Gaussian-splatting); applicability to other scene representations is not established here.

Target Audience

Researchers and graduate students working on 3D/4D scene editing, neural rendering, and LLM-based visual agents will get the most from this paper, particularly those interested in decoupling 2D editing from 3D reconstruction. Practitioners building interactive content-creation tools or multimodal agent systems will also find the tool-scheduling and trajectory-tuning design directly relevant. Readers without background in neural scene representations or LLM tool-calling will find the system-level architecture understandable but will need supplementary reading for the atlas formulation and loss design.

Authors’ abstract

Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.