PerfGuard: A Performance-Aware Agent for Visual Content Generation

AI summary

Overview

• Research area: LLM-powered multi-agent systems for visual content generation (text-to-image generation and instruction-based image editing).
• Technical level: Advanced — assumes familiarity with agent architectures (Analyst/Planner/Worker/Evaluator), diffusion models, and preference optimization methods such as DPO/SPO.
• Scope: The paper proposes PerfGuard, a framework that replaces vague textual tool descriptions with measured, multi-dimensional performance profiles, and feeds those profiles into both tool selection and task planning.

What This Paper Is About

Most LLM agent frameworks for image generation assume that whenever they call a tool, the call succeeds — and they describe tools with loose natural-language blurbs like "can generate images matching the input text." That assumption breaks down in AIGC, where tools like FLUX, SD3, or Step1X_Edit have sharply different strengths and weaknesses across attributes such as color, spatial relations, or style transfer. PerfGuard's goal is to quantify each tool's actual performance boundaries, use those quantified profiles to pick the right tool for each sub-task, and correct the profiles over time based on how tools actually perform in real executions.

Key Contributions

1. Performance-Aware Selection Modeling (PASM). A multi-dimensional scoring system that replaces generic tool descriptions with numerical performance boundaries. Generation tools are scored on seven dimensions drawn from T2I-CompBench (color, shape, texture, 2D spatial, 3D spatial, non-spatial semantics, numeracy); editing tools are scored on seven dimensions from ImgEdit-Bench (addition, removal, replacement, attribute alteration, motion change, style transfer, background change). The Worker converts each sub-task into a preference-weight vector and multiplies it against a normalized tool-by-dimension score matrix to rank candidate tools.

2. Adaptive Preference Updating (APU). A feedback loop that corrects benchmark-derived scores using real execution outcomes. It explores by taking the top-m ranked tools plus n randomly sampled ones, compares the theoretical ranking against the actual ranking determined by a multimodal evaluator, and nudges the performance matrix in the direction of the discrepancy.

3. Capability-Aligned Planning Optimization (CAPO). An extension of Step-aware Preference Optimization (SPO) to the Planner's autoregressive decision process. At each step the Planner samples k candidate sub-tasks, the Self-Evaluator scores each resulting output across global and local semantics, and the highest- and lowest-scoring candidates form a DPO-style winning/losing pair used to update the Planner.

4. Experimental validation across three benchmarks plus efficiency analysis. Comparisons on T2I-CompBench, OneIG-Bench, and Complex-Edit, together with ablations on each module, hyperparameter sweeps on the update step size, and scaling tests with tool libraries of up to 200 tools.

Main Findings

• Basic generation: PerfGuard leads across all T2I-CompBench dimensions. It reaches 0.8753 on color, 0.7366 on shape, 0.8148 on texture, 0.6120 on spatial, 0.3754 on non-spatial, and 0.5007 on the complex metric, beating GenArtist (0.8482 / 0.6948 / 0.7709 / 0.5437 / 0.3346 / 0.4499) and T2I-Copilot on every column. The gain is largest on spatial relations, where PerfGuard scores 0.6120 versus 0.2863 for FLUX and 0.5437 for GenArtist.

• Advanced generation: reasoning is where performance-awareness pays off. On OneIG-Bench, PerfGuard scores 0.834 on alignment, 0.684 on text, 0.350 on reasoning, and 0.395 on style. The authors note that alignment and text margins over T2I-Copilot (0.821 / 0.679) are small because the underlying toolset caps generation quality — the real edge is in reasoning (0.350 vs 0.318), which they attribute to smarter planning rather than better tools.

• Complex editing: best overall on Level-3 of Complex-Edit. PerfGuard achieves 8.95 instruction following, 9.02 perceptual quality, 8.56 identity preservation, and 8.84 overall, versus 8.13 overall for OmniGen and 8.10 for Step1X_Edit. AnySD scores highest on identity preservation (9.08) only because it barely edits Level-3 samples, which drags its instruction-following score down to 4.13.

• Textual tool descriptions are close to useless for selection. On the T2I-CompBench complex subset, selecting tools from text descriptions alone yields a 77.8% error rate with QWen3-14B and still 72.2% with GPT-4o. Adding a retrieval-based experience module only reaches 68.1%. A benchmark-initialized performance matrix cuts it to 30.5%, and adding APU reaches 14.2% — roughly a five-fold reduction over the strongest text-based baseline.

• APU step size needs tuning. With η = 0.1 the error rate falls too slowly; with η = 0.15 it converges fast but oscillates badly. η = 0.13 reaches the minimum 14.2% error at step 800.

• Ablations confirm each module contributes. With no modules, color/spatial/complex scores are 0.8239 / 0.5600 / 0.4327. CAPO alone gives 0.8466 / 0.5756 / 0.4493. PASM alone gives 0.8521 / 0.5919 / 0.4412. PASM + APU gives 0.8596 / 0.6005 / 0.4738. The full stack reaches 0.8753 / 0.6120 / 0.5007.

• Efficiency improves rather than degrades with scale. PerfGuard uses less time than GenArtist and T2I-Copilot for planning, tool selection, and image evaluation. In simulated libraries of 10 to 200 tools, token consumption for text-based selection grows catastrophically with tool count, while PerfGuard stays nearly flat — its cost scales with the number of performance dimensions, not the number of tools.

• Planner training changes behavior qualitatively. With CAPO, the Planner learns that operation ordering matters — for example, editing a background first with Step1X_Edit can introduce inaccuracies that damage later edits to other entities like a table.

Methodology in Plain English

PerfGuard keeps the standard four-role agent structure (Analyst, Planner, Worker, Self-Evaluator) but changes what information flows between roles.

The Analyst reads the user's image or text instruction and produces a task summary, target semantics, and evaluation goals. The Planner breaks the task into ordered sub-tasks, using both the task summary and a table of tool performance profiles. The Worker takes each sub-task and picks a tool. Instead of reading prose descriptions, it decides which performance dimensions matter most for this particular sub-task — that becomes a weight vector — and multiplies that vector against a normalized matrix of tool scores. The tool with the highest weighted score wins. The Self-Evaluator then scores the resulting image against the goals and feeds that back.

Two feedback loops refine this. The first is APU: whenever the Worker picks an tool, the system also tries a few alternatives and compares the actual quality rankings against what the score matrix predicted. If a tool outperformed its predicted rank, its scores are nudged up in proportion to how much the current task cared about each dimension; if it underperformed, they are nudged down. The second is CAPO: rather than only updating the tool table, the Planner itself is fine-tuned. At every step it generates five candidate sub-tasks, watches which one produces the best evaluated output and which the worst, and updates its policy with a DPO-style loss to favor the winner. A memory of previously successful sub-task sequences, retrieved by CLIP similarity, seeds 40% of those candidates to avoid wasting exploration.

Why This Matters

The paper challenges a quiet but pervasive assumption in agent research: that tool calls succeed and that tool capability can be conveyed in text. If that assumption fails, entire planning pipelines — not just individual tool calls — become unreliable, because a Planner that does not know a tool's limits cannot order operations sensibly. PerfGuard shows that measuring tools and feeding those measurements back into both selection and planning produces large, consistent gains on standard benchmarks, and that the approach scales gracefully to tool libraries far larger than any current agent could manage with text descriptions.

Real-world applications:

• Creative and marketing pipelines. A campaign tool that must produce images meeting specific attribute requirements (brand color accuracy, correct object counts, spatial layout) can route each requirement to the model that demonstrably handles it best rather than defaulting to one general-purpose generator.
• Photo editing and retouching assistants. Multi-round editing where each step depends on the previous one — the CAPO ordering result directly addresses the failure mode where editing the background first corrupts later subject edits.
• Agent platforms with large tool marketplaces. As third-party image models proliferate, an orchestrator needs to onboard new tools without hand-written descriptions and keep rankings current as models are updated or deprecated. APU's initialization of unseen tools from similar tools' average scores addresses exactly this.
• Robotics and design tools that chain heterogeneous models. Any domain where a planner dispatches to specialized models with different competence profiles — 3D asset generation, layout synthesis, video editing — inherits the same performance-boundary problem.

Industry relevance: the token-consumption result is the most directly commercial finding. Text-based tool selection becomes economically unusable as tool libraries grow, while dimension-based matching stays roughly constant in cost. That makes the approach viable for platforms where the tool catalog changes frequently and per-call LLM cost matters.

Future Directions

• Dynamic tool integration. The conclusion explicitly names this: adding new tools to the library at runtime is currently handled by averaging similar tools' scores, which is a placeholder rather than a principled solution. How should a newly registered tool be probed and profiled efficiently?
• Multimodal task expansion. The framework is built around images. Extending the performance-boundary concept to video, audio, or 3D generation raises the question of whether the same dimension-based scoring applies, or whether these modalities need their own benchmark-derived taxonomies.
• Reducing dependence on static benchmarks. The initial performance matrix comes directly from T2I-CompBench and ImgEdit-Bench scores. How much does PerfGuard degrade when a deployed tool differs from its benchmark configuration, or when the benchmark's dimensions do not match a user's real priorities?
• Scaling the planner's training signal. CAPO's learning signal comes from a Self-Evaluator that measures semantic alignment. Whether this signal can catch subtler failure modes — aesthetic degradation, stylistic drift across a multi-step edit — or whether it saturates once semantic goals are met remains open.

Target Audience

Researchers and engineers working on LLM agent frameworks, tool orchestration, and AIGC systems will get the most out of this paper, particularly those who have hit the wall of unreliable tool selection in multi-model pipelines. It is also relevant to practitioners building production image generation or editing products who need to choose among many competing models. Readers without background in DPO-style preference optimization or multi-agent LLM architectures will find Sections 4.2 and 4.3 dense, though the core idea of scoring tools on measurable dimensions is accessible on its own.

Authors’ abstract

The advancement of Large Language Model (LLM)-powered agents has enabled automated task processing through reasoning and tool invocation capabilities. However, existing frameworks often operate under the idealized assumption that tool executions are invariably successful, relying solely on textual descriptions that fail to distinguish precise performance boundaries and cannot adapt to iterative tool updates. This gap introduces uncertainty in planning and execution, particularly in domains like visual content generation (AIGC), where nuanced tool performance significantly impacts outcomes. To address this, we propose PerfGuard, a performance-aware agent framework for visual content generation that systematically models tool performance boundaries and integrates them into task planning and scheduling. Our framework introduces three core mechanisms: (1) Performance-Aware Selection Modeling (PASM), which replaces generic tool descriptions with a multi-dimensional scoring system based on fine-grained performance evaluations; (2) Adaptive Preference Update (APU), which dynamically optimizes tool selection by comparing theoretical rankings with actual execution rankings; and (3) Capability-Aligned Planning Optimization (CAPO), which guides the planner to generate subtasks aligned with performance-aware strategies. Experimental comparisons against state-of-the-art methods demonstrate PerfGuard's advantages in tool selection accuracy, execution reliability, and alignment with user intent, validating its robustness and practical utility for complex AIGC tasks. The project code is available at https://github.com/FelixChan9527/PerfGuard.