AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

AI summary

Overview

Research area: Computer vision and multimodal agentic systems, specifically long-horizon agents that transform multimodal source documents into human-facing visual artifacts (posters, slides, videos, webpages). The paper sits at the intersection of agent scaffolding ("harness") design, self-improving agent loops, and benchmark construction for design tasks.

Technical level: Advanced. The paper assumes familiarity with LLM/MLLM agent scaffolds, execution trajectories, VLM-as-judge evaluation, and train/development acceptance gating.

Scope: The paper proposes AutoDesign, a meta-harness optimizer that recursively rewrites a design agent's harness using rollout feedback, and instantiates it for academic paper-to-poster generation, evaluated with a new 100-paper benchmark called PosterBench.

What This Paper Is About

Existing design systems for turning multimodal sources (such as academic papers) into structured artifacts (such as posters) treat human feedback as a transient signal that fixes one output at a time, rather than as reusable design knowledge that persistently improves the producing system. The goal of this work is to make the surrounding system, or "harness," the thing that gets optimized: a meta-harness observes rollouts across many tasks and rewrites the harness itself, so that multimodal evidence, structural constraints, and human preferences accumulate into design priors. The authors instantiate this idea as DesignHarness for paper-to-poster generation and build PosterBench to measure whether the resulting posters are faithful, dense, readable, and visually usable.

Key Contributions

1. AutoDesign, a meta-harness optimization framework that converts a static design harness into a recursively improving system for human-aligned multimodal design. Over 7 days of evolving traces, it invokes 224 subagents, records at least 123 recursive iterations, and accumulates 54 harness updates that convert human-designed reference artifacts, rollout traces, rendering diagnostics, and evaluator feedback into persistent design priors.

2. DesignHarness, an executable academic paper-to-poster system evolved by AutoDesign. It grounds a tool-using Designer in paper context and combines editable generation with rendering, rule-based validation, and visual critic feedback for localized revision, producing source-grounded posters that remain directly usable and editable.

3. PosterBench, a comprehensive evaluation protocol for paper-to-poster evaluation. It assesses scientific communication quality and executable-artifact reliability through a seven-dimensional rubric spanning faithfulness, coverage, density, visual evidence, layout, readability, and aesthetics. The paper reports that Claude Code (Claude 4.8) achieves 70.01 on PosterBench, and that attaching DesignHarness can improve it by +8.31, surpassing the best commercial design agent Claude Design by 7.45 points.

4. A fully autonomous long-horizon demonstration: in a poster generation run, DesignHarness executes 253 tool calls and 11 editing turns within 40 minutes for less than $3, with negligible human intervention. A demo is available as a research-preview platform for interactive use and localized revision of editable posters.

Main Findings

• Main Track leaderboard: On the 100-paper PosterBench Main Track, AutoDesign attains the highest PosterBench Score at 78.32, exceeding Claude Design (70.87) by 7.45 points and OpenDesign (69.45) by 8.87 points, all under a matched Claude Code and Claude 4.8 configuration.

• Harness generalizes across model–agent pairs: Across seven controlled code agent–model configurations on PosterBench-mini, attaching DesignHarness raises the average PosterBench Score from 54.99 to 67.39 (+12.40 points), with per-configuration gains ranging from 5.01 to 19.56 points. Figure 1(b) reports that the optimized harness improves all Coding Agents by 5.0 to 19.6 points, reaching a best overall score of 81.5.

• Largest and smallest reported gains: The largest gain is 19.56 points for DeepSeek V4 Pro with Claude Code. The smallest reported in Table 4 is +5.01 for Claude Code with Claude 4.8 (69.55 to 74.56); Codex with GPT-5.5 improves from 75.87 to 81.46 (+5.59); Claude Code with Kimi K2.7 rises from 57.20 to 70.12 (+12.92); Seed 2.1 Pro with Claude Code rises from 54.01 to 71.83 (+17.82).

• PosterBench-mini standings: AutoDesign with DesignHarness reaches 81.46 with Codex/GPT 5.5 and 74.56 with Claude Code/Claude 4.8, compared with native Codex at 75.87 and standalone Claude Code at 69.55.

• Controlled track isolation: In the Design Harness Track (Claude Code and Claude 4.8 fixed), AutoDesign scores 74.56 versus OpenDesign at 70.36 and Claude Design at 66.83. In the Coding Harness Track (AutoDesign and GLM 5.2 fixed), Kimi Code leads at 82.31, followed by ZCode at 69.53, OpenCode at 67.87, and Claude Code at 64.33. In the Model Track (AutoDesign and Claude Code fixed), Claude 4.8 leads at 74.56, followed by Seed 2.1 Pro at 71.83 and Kimi K2.7 at 70.12.

• Low-cost access: With DesignHarness, LongCat-2.0 reaches 55.13 at approximately $0.27 per poster. The reported cost reflects LongCat-2.0's pricing policy at evaluation time, where cached context incurs no charge on a cache hit.

• Human agreement with the automatic protocol: Across 933 valid system-blind pairwise judgments, AutoDesign receives the highest Bradley–Terry preference estimate at 64.0% (95% interval: 55.2–77.8%). For pairs separated by at least 20 PosterBench points, participants prefer the PosterBench-preferred poster in 74.4% of cases.

• Baseline comparison of handcrafted workflows: On the Main Track, human-crafted workflows score lower than the agentic systems reported, with PosterGen at 56.71, Any2Poster at 49.09, and Paper2Poster at 44.61. Dimension-level results show a trade-off: Paper2Poster obtains the highest Density score in Table 1 (8.36) while scoring 44.61 overall.

• Human-in-the-loop matters at plateaus: Figure 1(a) tracks one representative paper across meta-harness iterations and reports that autonomous optimization improves the initial harness before reaching a plateau, after which human guidance redirects the search and yields a further gain.

Methodology in Plain English

The authors reframe design generation as an optimization problem over the system surrounding a fixed model, not the model weights. They split a "design harness" into five components: Context and Memory (sources, prompts, skills, assets, persistent state); Tools and Specifications (tools and editable artifact specs for layout, typography, provenance); Execution Runtime (workspace for authoring, rendering, validating, exporting); Orchestration (task routing, attempt budgets, loop control, candidate selection, fallback, finalization); and Evaluation and Feedback (rule-based validation, model-based critique, localized revision feedback).

AutoDesign then runs two nested loops. The inner loop is the harness itself: a Designer module produces or revises an artifact, and a Critic module scores it and returns feedback, repeated until the artifact passes blocking checks or an attempt budget is exhausted. The outer loop improves the harness: it rolls the current harness out across a training task set, evaluates the artifacts with an evaluator built from human-annotated reference artifacts along the seven quality dimensions, then has a coding agent acting as planner and code editor propose one bounded update. Each iteration is restricted to exactly one of the five harness components, which keeps credit assignment interpretable. A candidate update is accepted only if it improves training performance and does not degrade an independent development set; development results are never shown to the optimizer, so they act as an overfitting guard. An optimization record stores the harness checkpoint, trajectories, scores, chosen component, plan, code changes, and the accept/reject decision, giving the next iteration persistent context and rollback ability. The loop keeps a single active harness and does not perform tree search over variants.

For paper-to-poster, the resulting DesignHarness ingests the source into a provenance-aware context (metadata, section outline, key passages, figures and tables with source locations), organizes it into a content brief and a medium-specific artifact plan, and generates an editable HTML artifact. A rule-based validator applies deterministic blocking checks (unsafe or missing assets, broken provenance links, severe overflow or overlap, typographic and layout violations) and returns localized diagnostics plus non-blocking checks on coverage, density, and numerical consistency. Failing candidates are rendered to a preview (PNG, PPTX, or MP4) and inspected by a critic VLM for design-context compliance, layout, readability, and aesthetics. The harness permits at most 12 refinement attempts; passing candidates go straight to finalization, and if the budget is exhausted, fallback mechanisms select a deliverable candidate while retaining safety and integrity constraints.

PosterBench itself is a frozen external evaluator, separate from the optimization-time evaluator. It contains a 100-paper Main Track and PosterBench-mini, a shared 10-paper subset, spanning five disciplines: AI/ML, biomedicine and health, climate and earth environment, economics and policy, and physics and astronomy. It returns scores in [0,10] across the seven dimensions and aggregates them with the fixed weights (10, 10, 15, 10, 20, 25, 10), then applies the strictest active record-level ceiling bounding severe layout damage, insufficient presentation viability, confirmed visible failures, and protected render-integrity violations; inactive ceilings are 100, and a standard P0 gate caps a score at 40. Because of these ceilings, the Overall score is the mean of capped poster scores and cannot generally be recovered by reweighting the displayed dimension means.

Why This Matters

Impact on research: The paper argues that the unresolved question in multimodal design is how to convert multimodal evidence, structural constraints, feedback, and human preferences into persistent capabilities of the production system. By making the harness rather than the artifact the optimization target, and by providing a seven-dimensional frozen protocol with a system-blind human validation, the work offers a reproducible template for evaluating self-improving agent scaffolds. The paper also reports pipeline statistics (224 subagents, at least 123 recursive iterations, 54 harness updates over 7 days) that make the optimization process auditable.

Real-world applications:

• Automatic generation of ready-to-use conference posters from submitted papers, which the authors report reaching average conference-poster quality in human evaluation.
• General human-facing artifact production, since DesignHarness supports academic posters, presentation slides, videos, and web pages within the same harness abstraction.
• Low-cost design assistance for resource-constrained users: LongCat-2.0 with DesignHarness reaches 55.13 at approximately $0.27 per poster, and the autonomous run completes within 40 minutes for less than $3.
• Editable, source-traceable outputs, where provenance links allow statements and visual materials to be traced back to the source paper and checked during revision, useful for scientific and technical communication contexts.

Industry relevance: The comparison includes closed-source commercial systems (Claude Design, Claude Code, Codex, Doubao, GLM, Kimi, DeepSeek, Seed), and the paper reports that a learned open harness outperforms a commercial design agent by 7.45 points under a matched configuration. The controlled tracks show that harness choice and model choice contribute separately and that the harness transfers across seven model–agent pairs, which is directly relevant to teams deciding where to invest: scaffold optimization rather than model replacement. Affiliations span Meituan, MBZUAI, Huazhong University of Science and Technology, Peking University, Tsinghua University, The Chinese University of Hong Kong, and Shanghai Jiao Tong University.

Future Directions

• Beyond paper-to-poster: The harness abstraction is defined generally over multimodal sources and target media, and the paper states DesignHarness supports posters, slides, videos, and web pages, but the reported evidence is concentrated on the poster task. Extending and measuring the other media remains open.

• Reducing evaluator bias: The paper notes that the meta-harness receives no external signal to identify or correct bias in its optimization-time evaluator, so evaluator revision requires explicit human input. Automating detection and correction of evaluator bias is an unresolved problem.

• Escaping optimization plateaus: Figure 1(a) shows autonomous optimization reaching a plateau that requires human guidance to overcome. The paper leaves open how to detect such plateaus automatically or to build guidance into the loop, and it explicitly notes the outer loop maintains a single active harness and does not perform tree search over variants.

• Better matching human judgment: The human study reports 64.0% Bradley–Terry preference for AutoDesign and 74.4% agreement for pairs separated by at least 20 PosterBench points, indicating that the automatic protocol and human preferences diverge in some cases; closing that gap is a stated concern of the protocol design.

Target Audience

This paper is most useful to researchers and engineers building agentic systems for multimodal content generation, particularly those working on self-improving scaffolds, harness or scaffold optimization, and agent memory. It is also relevant to benchmark designers interested in combining rule-based algorithmic checks with rubric-guided VLM judges and validating that protocol against system-blind human preference. Practitioners evaluating design agents for production use will benefit from the controlled-track analysis comparing harnesses, coding agents, and models under matched conditions, as well as the reported cost and latency figures. Readers without background in agent scaffolds or VLM-as-judge evaluation will find the framing advanced and should first familiarize themselves with the inner-loop/outer-loop separation and the model-versus-scaffold distinction the paper builds on.

Authors’ abstract

Transforming multimodal sources into condensed and structured media outputs can be fundamentally conceptualized as a long-horizon agentic process centered on a model-harness system. While an ideal harness system should align with human design priors and accumulate reusable experience through empirical exploration to drive recursive self-improvement, existing paradigms remain static and fall short of this capability. In this paper, we present AutoDesign, a framework that aligns with human design priors, where a meta-harness optimizer guides a code agent to recursively improve harness based on rollout feedback. To instantiate and evaluate this framework, we focus on the academic paper-to-poster generation task and introduce PosterBench, comprising a 100-paper Main Track spanning five disciplines and PosterBench-mini, a shared 10-paper subset for controlled evaluation. On the PosterBench Main Track, AutoDesign achieves the highest score of 78.32, surpassing the closed-source commercial system Claude Design by 7.45 points. Across seven controlled code-agent-model configurations, integrating the learned DesignHarness consistently improves performance, increasing the average PosterBench Score from 54.99 to 67.39 (+12.4%). In a fully autonomous long-horizon loop, it executes 253 tool calls and 11 editing turns within 40 minutes for under $3, reaching average conference-poster quality in human evaluation. A system-blind human study further demonstrates that AutoDesign achieves the highest human preference among evaluated systems.