Clarify Before Executing: A Self-Evolving Agent for Resolving Intent Asymmetry in 3D Tool Orchestration

AI summary

Overview

Research area: Computer Vision / 3D content generation, LLM-based multi-agent systems, interactive clarification, and self-evolving agent training.

Technical level: Advanced. The paper assumes familiarity with LLM agent frameworks, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), LoRA, and 3D generation/reconstruction toolchains.

Scope: The paper introduces Clare, a four-role multi-agent framework that detects and resolves underspecified user requests before invoking 3D tools, plus the 3D-Clarify benchmark (620 scenarios) and a simulated-user self-evolution training loop using a Multi-turn Reward.

What This Paper Is About

3D generation, reconstruction, and editing tools require precise inputs—explicit file paths, exact output formats, chosen models, and numerical parameters—but ordinary users typically give vague or partly wrong instructions, such as "reconstruct this object" without supplying an image. Existing 3D agents assume a single-turn, fully specified request and either execute blindly, fail on hidden parameter contradictions, or never ask for what is missing.

The paper's goal is to make a 3D agent proactively clarify before executing: detect ambiguity, missing information, and mistaken details, ask targeted questions, then translate the resolved intent into executable code. It also trains that clarification policy automatically through simulated multi-turn interaction rather than hand-written rules.

Key Contributions

1. Clare, a multi-agent clarification framework for 3D tool orchestration. It decouples the pipeline into four specialized roles—Proactive Approver (gates execution by detecting ambiguity, missing information, and mistaken details), Contextual Summarizer (distills multi-turn dialogue into a normalized task manifest), 3D-Aware Programmer (maps user constraints to 3D tools and synthesizes executable Python code), and Feedback Provider (asks clarification questions or reports execution status). It covers five domains: text-to-3D generation, single-view reconstruction, multi-view reconstruction, point cloud editing, and post-processing.

2. The 3D-Clarify benchmark. 620 interaction scenarios with systematically injected underspecification: 500 single-step atomic tasks spanning five 3D domains and 120 multi-step tasks chained from 15 workflow templates, each perturbed with Ambiguity, Missing Information, or Mistaken Details to simulate a "lazy user." The benchmark is designed to be tool-agnostic, so adapting it to new 3D tools requires only establishing a new model mapping.

3. A self-evolution training paradigm. The agent samples candidate responses, rolls out full interaction trajectories against a stateful simulated user initialized with a hidden goal, scores them with a Multi-turn Reward, and updates policy via SFT or DPO—learning when to clarify versus when to execute without manual annotation.

4. Reported state-of-the-art results on 3D-Clarify. 60.40% and 43.34% success rates on single-step and multi-step tasks respectively, described by the authors as more than doubling existing baselines.

Main Findings

• Proactive clarification is essential. On single-step tasks, single-turn paradigms (Zero-shot, Few-shot, CoT) reach only 0–6% success rate on ambiguity and missing-info tasks despite moderate completion rates (31–35% and 17–18% CR). Dynamic frameworks (ReAct, Reflexion) show similar failures at 1.6–8.4% SR. The best task-interface-matched adapter reaches only 24.40% average SR on single-step tasks and 9.44% on multi-step tasks, while Clare-base reaches 48.60% and 27.22% respectively.

• Self-evolution via Multi-turn Reward substantially improves clarification. On ambiguity single-step tasks, Clare-SE-DPO achieves 60.00% SR (CR 70.52%) versus Clare-base's 45.80% SR—a 14.20% absolute improvement. On missing-info tasks, Clare-SE-SFT reaches 52.00% SR (CR 69.47%) and Clare-SE-DPO reaches 59.40% SR (CR 75.32%). Overall, Clare-SE-DPO averages 60.40% SR (CR 73.51%) across single-step underspecifications, versus the strongest baseline at 24.40% SR.

• Multi-step tasks amplify the advantage of structured clarification. Baselines collapse to 0% SR on ambiguity and missing-info multi-step tasks, while Clare-SE-DPO maintains 34.17–49.17% SR (CR 53.41–68.74%)—a 34–49% absolute gap. Clare-SE-DPO averages 43.34% SR (CR 64.97%) on multi-step tasks, against 4.44–7.78% SR for dynamic baselines and 11.39–13.89% SR for single-turn paradigms.

• SFT beats DPO on mistaken-detail tasks. Clare-SE-SFT attains 66.80% SR (CR 85.43%) on single-step mistaken tasks versus Clare-SE-DPO's 61.80%. The authors attribute this to the conflict between preference alignment and error correction: SFT clones firm-correction trajectories, while DPO biases the model toward compliant, agreeable responses when it must contradict the user.

• Baselines show inflated completion but collapsed success on mistaken details. Single-step Zero-shot reaches 77.55% CR but only 34.80% SR: the underlying LLM configures most standard parameters correctly, but blind execution of deliberately injected fatal errors breaks the run.

• Clarification is cheap relative to its payoff. ReAct and Reflexion use only 1.34–1.51 turns on single-step tasks yet reach merely 7.8% and 16.1% SR. Clare-base uses 3.65 turns to reach 48.6% SR on single-step tasks and 2.69 turns for 27.2% SR on multi-step tasks. MR optimization adds only 0.07 and 0.76 turns for Clare-SE-DPO while raising SR to 60.4% and 43.3%.

• The Proactive Approver matters most in ablation. Removing it drops single-step SR from 48.60% to 27.67% (−20.93% absolute) and CR from 66.26% to 52.04% (−14.22%); multi-step SR falls from 27.22% to 13.06% (−14.16%). Removing the Summarizer costs less: single-step 65.25% CR / 48.20% SR and multi-step 45.97% CR / 20.28% SR.

• Qualitative results. Baselines hallucinate a generic frog for an ambiguous text-to-3D request, crash on missing image paths, or pass invalid parameters; Clare clarifies to produce the intended blue poison-dart frog, reconstructed fruit mesh, and segmented room point cloud.

Methodology in Plain English

The researchers first separate the job of a 3D agent into four roles so that ambiguity is caught before any expensive 3D model runs. The Proactive Approver makes a binary Execute-or-Clarify decision on each incoming instruction. If it chooses Clarify, the Feedback Provider turns that rejection into a natural, targeted question. Once the instruction is actionable, the Contextual Summarizer aggregates the dialogue history into a structured task manifest, resolving references like "remove it" and extracting paths and quality preferences. The 3D-Aware Programmer then consults a model knowledge base to map preferences such as "high quality" or "fast speed" onto a specific 3D tool and its parameters, and emits executable Python code.

To teach the agent when to clarify rather than hard-code rules, the authors build a simulation loop. A stateful simulated user is initialized with a hidden target goal plus a vague opening query, and only reveals critical parameters when the agent explicitly identifies the information gap. The agent samples several candidate responses, rolls each one out into a full interaction trajectory, and scores it with a Multi-turn Reward: a verifier's [0,1] score for task completion minus a penalty λ times the number of turns. The verifier checks that the predicted task type matches the ground truth and then averages per-parameter similarity scores (exact match for numbers, LLM-judged similarity for text). Top-ranked trajectories become SFT data, and pairwise chosen/rejected rankings become DPO data, applied via LoRA fine-tuning to the Proactive Approver.

Evaluation uses Qwen3-8B for both the agent components and the user simulator, a knowledge base covering Worldgen, Trellis, Triplaneturbo, Hunyuan-3D, VGGT, Fast3R, Concerto, and Open3D, a maximum of 8 turns, and a penalty coefficient λ of 0.1. Fine-tuning used LLaMA-Factory on two NVIDIA RTX 3090 GPUs for 2 epochs with a learning rate of 5×10⁻⁶, total batch size 4, and a de-duplicated subset of 3,000 multi-turn trajectories filtered by Multi-turn Reward. Performance is measured by Goal Completion Rate (CR) and Task Success Rate (SR), checking module name, text-to-3D prompt and parameters, input file path, and output file extension.

Why This Matters

Impact on research. The paper reframes intent asymmetry in 3D pipelines not as noise or an execution error but as a dialogue opportunity, and shows the clarify-versus-execute decision is learnable from simulation rather than rule-based. It contributes a controlled benchmark (620 scenarios across three underspecification types) and a training recipe that requires no manual annotation, offering a template for other fragile, parameter-strict toolchains.

Real-world applications:

• Game and entertainment asset creation, where non-expert designers describe characters or props in vague terms and need them turned into usable 3D models.
• Virtual reality and architectural visualization, where clients request "reconstruct this room" or "high quality" without specifying files, formats, or fidelity settings.
• Industrial design, where an underspecified edit request (for example, deleting points from a scanned part) could otherwise waste compute or corrupt an asset.
• Point cloud post-processing and format conversion pipelines, where mismatched input modalities or wrong output extensions are common and currently cause cryptic runtime failures.

Industry relevance. The framework sits directly on top of commercial and open 3D foundation models (Trellis, Hunyuan-3D, VGGT, Fast3R, Open3D and others) and its tool-agnostic benchmark design means it can be re-pointed at upgraded models by establishing a new mapping. That makes it a practical interaction layer for 3D content platforms rather than a replacement for the underlying generators.

Future Directions

• Scaling the backbone. The authors note that absolute performance still indicates room for growth, explicitly mentioning larger backbones, since all components here use Qwen3-8B.

• Resolving the SFT–DPO trade-off on error correction. DPO improved ambiguity and missing-info outcomes but degraded on mistaken details relative to SFT. Finding a training objective that preserves both proactive correction and compliant helpfulness is an open problem the paper raises directly.

• Extending self-evolution beyond the Approver. The reported LoRA fine-tuning targets the Proactive Approver; whether the Summarizer, Programmer, and Feedback Provider also benefit from trajectory-based optimization is not reported.

• Broadening tool coverage and workflow depth. The benchmark's tool-agnostic design anticipates future tools, and the multi-step setting (120 tasks from 15 workflow templates) leaves headroom, since even the best variant reaches 43.34% multi-step SR. The paper content, as provided, does not report a dedicated limitations section, nor metrics such as wall-clock cost or user-satisfaction studies.

Target Audience

Researchers and engineers working on LLM agents, tool orchestration, and human-in-the-loop interfaces for 3D content creation will benefit most, along with practitioners building interactive pipelines on top of 3D foundation models. It is also relevant to anyone studying clarification dialog, reward design for multi-turn agents, or preference optimization in domain-specific settings. Readers should already be comfortable with agent architectures and fine-tuning methods to follow the training and evaluation details.

Authors’ abstract

A fundamental intent asymmetry plagues modern 3D asset creation: while state-of-the-art 3D toolchains demand precise, executable parameters, ordinary users typically provide vague, underspecified instructions. Current 3D agents treat this ambiguity as noise, defaulting to blind execution under a single-turn assumption. To address this limitation, we introduce CLARE, a clarification-aware and evolutionary 3D agent that treats intent asymmetry not as an execution error, but as an opportunity for strategic dialogue. By decoupling the generation pipeline into four specialized cognitive roles, CLARE intercepts and resolves underspecified instructions before invoking computationally expensive 3D tools to seamlessly execute tasks across five diverse domains: text-to-3D generation, single-view reconstruction, multi-view reconstruction, point cloud editing, and post-processing. Crucially, rather than relying on rigid manual rules, CLARE self-evolves its clarification policy via simulated multi-turn interactions. By optimizing a Multi-turn Reward, the agent internalizes the delicate balance between interaction efficiency and task completion. To rigorously test this, we construct 3D-Clarify, a comprehensive benchmark comprising 620 interaction scenarios with systematically injected ambiguity, missing information, and mistaken details. CLARE achieves state-of-the-art performance, with 60.40% and 43.34% success rates on single-step and multi-step tasks, respectively, more than doubling existing baselines. Both quantitative and qualitative results demonstrate that proactive clarification is the missing key to robust 3D execution. Code is available at https://github.com/xyzhu1225/CLARE.