BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

AI summary

Overview

Research area: Computer vision and multimodal LLM agents, specifically the evaluation of agents that build 3D content inside the Blender software environment.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), agent harnesses, tool-calling protocols such as MCP, and 3D rendering pipelines, though the core argument is stated plainly.

Scope (one sentence): The paper introduces BlenderWorld, a 223-task benchmark with a dual-judge evaluation protocol for measuring how well frontier visual agents reconstruct, modify, and design editable 3D Blender projects from text, image, video, and structured specification inputs.

What This Paper Is About

Multimodal models are increasingly used as agents that operate software tools to produce 3D models and scenes, but existing 3D benchmarks cover only narrow slices of this work, such as single-object modeling, camera edits, or predefined geometric targets. The authors argue that real requests are open-ended: they arrive as photographs, illustrations, videos, or written design briefs, they may have no single correct answer, and they demand both visual plausibility and correct underlying scene structure.

BlenderWorld is built to measure that broader setting. Each task hands an agent a production brief plus optional references and specification files, and the agent must return an editable .blend project with rendered media, which is then scored on requirement satisfaction and visual quality.

Key Contributions

1. A 223-task benchmark. BlenderWorld contains 223 production briefs spanning 9 main task categories and 22 subcategories, and serving 16 application scenarios. Inputs include text, images, and videos, with 59 tasks additionally carrying structured specification files in Markdown, YAML, or JSON.

2. A unified, evidence-routed evaluation protocol. The authors combine Automatic Agent Render (AaR), which selects eight task-conditioned views and renders them at 1024×1024 with 64 samples in Cycles, with 2,651 task-specific binary rubrics and eight holistic visual-quality dimensions scored from 1 to 10.

3. A dual-judge design. A VLM-as-Judge handles the 1,107 criteria tied to rendered views, while an Agent-as-Judge programmatically inspects the submitted project for the 1,544 criteria that rendered images cannot reliably verify, such as exact dimensions, topology, and internal structure.

4. A baseline over thirteen frontier models plus failure analysis. All agents run the same tasks under the same protocol, establishing comparable baselines and exposing recurring failure modes such as view-dependent geometry and violated contact constraints.

Main Findings

• Highest overall score: GPT-6-Astra at 79.9. It leads on text-only tasks (78.6) and multimodal tasks (80.7), and also ranks first on both component metrics, with a Rubric mean of 87.3 and a Dimension mean of 72.6.

• Runners-up are well behind. Opus-5 reaches 73.0 overall (74.8 Text, 72.1 MM), and GPT-5.6-Sol reaches 67.7 (71.7 Text, 65.5 MM). DeepSeek-V4.1-Flash follows at 65.7, then Qwen3.8-Max (60.6), GLM-5.3-Flash (59.7), Grok-4.6 (57.0), Opus-4.8 (57.0), Seed2.1pro-0915 (51.3), Qwen3.8-27B (47.7), Kimi-K3 (46.5), Gemini-3.8-Flash (34.2), and MiniMax-M3 (31.8).

• No model fully satisfies the rubrics. GPT-6-Astra passes every rubric criterion on only 42.2% of tasks; Opus-5 reaches 25.6%, and no other agent exceeds 15.7%.

• Spending more tool calls does not buy quality. GPT-5.6-Sol averages 39.9K output tokens and 44.0 tool calls per task, while DeepSeek-V4.1-Flash averages 209K tokens and 168 calls, yet their overall scores are close (67.7 vs. 65.7). Opus-5 uses roughly twice the Blender executions (45.5 vs. 23.3) and image views (27.1 vs. 15.0) of GPT-6-Astra per task.

• More looking does not mean better fixing. Seed2.1pro-0915 and DeepSeek-V4.1-Flash make the most visual queries (33.9 and 31.3 calls per task) but score moderately or poorly, while GPT-6-Astra achieves the top multimodal score of 80.7 with 15.0 image inspections.

• Category strengths are uneven. GPT-6-Astra leads eight of nine categories among the five agents compared, with its best scores on Procedural & Game Assets (86.8) and Products & Manufacturing (83.9). Its largest lead over Opus-5 appears in Characters & Animals, where its own score (78.7) falls below its overall average. Opus-5 beats GPT-6-Astra on Rendering & Simulation, and Qwen3.8-Max beats DeepSeek-V4.1-Flash on Nature & Landscapes despite ranking lower overall.

• Trade-offs appear inside a single model's profile. For GPT-6-Astra, Art & Animation shows a high Rubric score (89.6) but the lowest Dimension score (64), while Rendering & Simulation shows the lowest Rubric score (70.6) alongside a higher Dimension score (71.6).

• Four recurring failure modes. The qualitative audit identifies missing dynamic interactions, viewpoint inconsistency where subjects vanish from alternate angles, fine-grained geometric distortions such as malformed animal anatomy, and contact constraint violations such as a figure floating above a lawn.

• Rubric design detail. There are 6–22 criteria per task (mean 11.9); Function and Geometry together account for 41.7% of criteria; 83.3% of criteria carry major importance (weight 3); 41.8% are routed to rendered views and 58.2% to scene inspection; 500 criteria (18.9%) come from explicit requirements in specification files.

• Not reported. The paper content provided does not report per-task latency, monetary cost, or repeated-run variance for the main results table; repeated runs are listed only as planned future work.

Methodology in Plain English

The team first built a diverse set of realistic 3D briefs. They varied the production domain, scene complexity, assumed user expertise, and request length, and they curated reference images covering interiors, architecture, products, mechanical structures, and natural environments, keeping images with clearly identifiable structure and detail. Each reference was assigned one of three roles: recreation (preserve main shapes and spatial relationships), modification (make requested changes while retaining specified structures), or reference-based design (draw on form, style, or layout to create something new).

Each brief was then grounded in a practical scenario and written as a natural-language request stating the production goal and key requirements, sometimes with Markdown, YAML, or JSON attachments holding dimensions, material lists, camera settings, or asset budgets. Briefs passed two validation tracks: structural checks for metadata compatibility, file accessibility, parseability, and explicit requests for both renders and an editable project; and a semantic review by a model distinct from the task generator, checking image–text alignment, constraint consistency, and feasibility. Failed briefs went back for revision.

Requirements were converted into binary pass/fail criteria, each tagged with a category, an importance weight (3 for major, 2 for secondary, 1 for detail), and an evidence type (render or scene_summary). Drafts were refined through pilot evaluations using GPT-5.6-Sol, then cross-checked by at least two independent reviewer models (drawn from GPT-5.6-Sol, Gemini 3.1 Pro Preview, and Claude Opus 4.8) using concrete counterexamples, with an editor resolving issues before the schema was frozen.

At test time each agent works in an isolated Linux container with one NVIDIA GPU and Blender LTS, using shell, file, image-viewing, and Blender MCP tools. Rubrics are kept invisible during generation and artifacts are evaluated in a separate container, which the authors describe as a guard against rubric hacking. Eight GPT-6-Astra and GPT-5.6-Sol runs use Codex; the other eleven systems use Claude Code, each tested once per task at its highest available reasoning intensity.

For grading, the Automatic Agent Render stage opens the submitted .blend files and renders eight complementary views without repairing any defects, permitting up to two re-render attempts and failing the evaluation if new views cannot be produced. Criteria are then split by evidence type: the Agent-as-Judge inspects scene structure through Blender tools, and the VLM-as-Judge reviews the rendered views, the agent's own exported media, and sampled video frames, with original exported media taking precedence when evidence conflicts. A separate blind pass generates prompt-free descriptions used only as auxiliary grounding for the visual judge. The same VLM also rates eight dimensions on a 1–10 scale.

Scores combine a weighted rubric satisfaction percentage and a dimension percentage, fused with λ = 0.5 so that requirement satisfaction and aesthetic quality count equally on a 0–100 scale, then averaged over all 223 tasks. Failed evaluations score zero and stay in the denominator, and submissions lacking a .blend file automatically score zero.

Why This Matters

Impact on research. The paper reframes 3D agent evaluation away from a single rendered image or a narrow geometric metric. By separating what can be seen from what must be inspected inside the project file, it makes a specific claim testable: that visually plausible outputs can hide structurally broken scenes. The finding that extra tool calls and extra image views do not correlate with better scores is a direct challenge to the intuition that more iterations equal better agents.

Real-world applications. The benchmark's 16 application scenarios are drawn from the paper's task set, including:

• Real-estate marketing and renovation proposals, where clients supply photographs and expect editable, revisable scenes.
• Product launches and e-commerce visuals, where appearance, materials, and camera framing drive the deliverable.
• Classroom teaching and scientific illustration, where accuracy of internal structure matters more than surface polish.
• Safety training, construction reporting, and storyboarding/previsualization, where spatial relationships and working mechanisms must be correct.

Industry relevance. The benchmark targets the actual bottleneck of 3D production tooling: not whether a model can produce a pretty render, but whether it can deliver a project a professional can reopen, modify, and reuse. The failure catalogue—view-dependent shortcuts, floating objects, distorted anatomy—maps onto concrete acceptance criteria that studios and visualization teams already apply. The observation that a leading model can satisfy every criterion on only 42.2% of tasks sets a realistic expectation for near-term deployment in production pipelines.

Future Directions

• Interactive and parametric judging. The authors plan to give the agent judge Computer-Use Agent tools so it can manipulate rigs, animations, and simulations and observe resulting states, paired with deterministic scripts for dimensions, tolerances, and geometric constraints. They name voxel intersection-over-union as one possible supplement, using consistent alignment and scale conventions.

• Harder, more professional tasks. Planned additions include localized editing, mesh fault diagnosis and repair, topology optimization, retopology, engineering drawings with dimensional tolerances, and revision tasks that test whether an agent preserves existing content and keeps a project editable.

• Expert-aligned and graded rubrics. Working with professional 3D artists to calibrate rubrics toward production acceptance standards, and exploring graded assessment of individual parts for organic subjects where binary checks discriminate poorly.

• Broader ecosystem and platform coverage. Extending to other DCC applications such as Maya and 3ds Max, integrating external asset repositories such as the Mixamo motion library, evaluating more models across thinking-effort settings and harnesses with reported token usage, latency, tool calls, and cost, and moving beyond the current Linux evaluation to Windows and macOS with GUI-based Blender operation.

Target Audience

Researchers and engineers building multimodal agents that manipulate software tools will get the most from this paper, particularly those working on long-horizon, long-trajectory tasks where visual feedback must be converted into corrective action. It is also useful for benchmark designers interested in rubric construction, dual-judge protocols, and separating visible evidence from programmatic scene inspection. Practitioners in 3D production, architectural visualization, game asset creation, and technical illustration will find the failure taxonomy and application scenarios directly relevant to setting expectations for agent-assisted workflows. Readers seeking a gentle introduction to 3D generation may find the Blender-specific tooling and rubric vocabulary demanding.

Authors’ abstract

Recent advances in multimodal large language models (MLLMs) have enabled them to act as autonomous agents, creating and editing 3D models via code or direct software manipulation. In practical workflows, these models must infer object shapes, materials, spatial layouts, and lighting from diverse references while strictly following instructions to reconstruct, modify, or design 3D content. However, existing benchmarks provide limited coverage of such open-ended requests across real-world scenarios. To bridge this gap, we introduce BlenderWorld, the first comprehensive benchmark dedicated to evaluating frontier models on realistic 3D creation tasks. BlenderWorld comprises 223 tasks spanning architecture, products, natural environments, and scientific visualization, driven by text, image, video, and structured specification inputs. To rigorously assess these complex deliverables, we establish a unified evaluation protocol: we render the submitted Blender projects from multiple viewpoints and evaluate them against 2,651 task-specific binary rubrics alongside holistic visual quality scores. Benchmarking thirteen frontier models reveals that GPT-6-Astra achieves the highest overall fused score of 79.9, followed by Opus-5 (73.0) and GPT-5.6-Sol (67.7). Ultimately, our findings reveal a critical bottleneck: while current models can parse broad instructions, they still profoundly struggle with fine-grained 3D spatial coherence and accurate structural reproduction in Blender, leaving substantial headroom for future 3D-native agentics.