Agentic Visual Generation: From Generative Models to Agentic Control

AI summary

Overview

• Research area: Computer vision, specifically visual generation (images, video, editing, 3D, world models, slides, and user interfaces), analyzed through the lens of LLM/VLM-driven agents and control.
• Technical level: Intermediate. The paper is a taxonomy, survey, and evaluation-design paper rather than a new model report; it uses a small amount of formal notation (state, action, policy, capability vector) that is explained in plain terms.
• Scope: The paper proposes and applies a single criterion — the latest point in a generation trajectory at which a controller can causally change a future generation decision — to organize agentic visual generation into the levels L0 Fixed Support through L4 Experience-Adaptive Control, and proposes a matched, level-conditioned evaluation design for it.

What This Paper Is About

Visual generation systems are increasingly wrapped in controllers (usually an LLM or VLM) that plan, pick tools, inspect intermediate outputs, and retry — but the field has no consistent rule for deciding when such a system is genuinely "agentic." The paper argues that planning depth, tool count, multi-role collaboration, and reinforcement learning are all poor criteria, because none of them determines which generation decisions the controller can actually make. Its goal is to replace those criteria with one causal test ("maximum causal reach") and to derive both a reproducible taxonomy and a matched evaluation protocol from it.

Key Contributions

1. A decision-based definition of agenticity. Agenticity is defined by the maximum temporal and causal reach of the decisions a controller can make, yielding a reproducible hierarchy: L0 Fixed Support (inclusion boundary), L1 Conditioning Control, L2 Execution Control, L3 Outcome-Adaptive Control, and L4 Experience-Adaptive Control. A category/subcategory taxonomy organizes systems by the object controlled and its technical realization.
2. A structured corpus. The authors release a corpus with level, task, mechanism, feedback, memory, resource, and provenance fields, whose temporal and cross-sectional statistics show a rapid shift toward within-trajectory feedback while persistent cross-task experience remains uncommon.
3. A design space of controlled variables and feedback paths. Representative systems are organized by the causal decisions that distinguish them rather than by enumerating paper titles.
4. Level-conditioned evaluation. Each comparison adds one class of controller decisions while holding the generator, tools, budget, and evaluator fixed, so the value of specification, execution, outcome feedback, and reusable experience can be isolated.

Main Findings

• One test replaces many informal labels. The assignment rule is applied from the highest level down: cross-task persistence gives L4; an outcome-to-action causal link gives L3; selection and invocation of a visual operation gives L2; specification construction for a predetermined executor gives L1; if none holds, the system is L0. Ambiguous cases are assigned the lower level until the paper demonstrates the missing causal link — so words like "multi-agent," "self-reflective," or "self-evolving" in a title are not evidence by themselves.
• A capability vector formalizes the hierarchy. Each system gets a vector c(S) = (c1, c2, c3, c4) with entries in {0,1} denoting demonstrated control over specification, execution, within-task outcome adaptation, and cross-task experience reuse; the primary level is L(S) = max{i : ci = 1}. The vector is kept as the "capability path," so L1+L2+L3 records all three while the system is assigned to L3.
• Task type does not determine controller level. The paper keeps a task axis (image, video, editing, slide/UI, 3D, world-grounded synthesis, interactive simulation) and a mechanism axis (intent grounding, planning, tool routing, retrieval, multi-agent collaboration, verification, memory, reinforcement learning) separate from the level, noting that an image system can have a longer adaptive trajectory than a video system, and a multi-shot video pipeline can remain open loop.
• L0 is a boundary, not a peer level. L0 records generators, editors, retrievers, evaluators, reward models, benchmarks, and fixed pipelines. ControlNet, T2I-Adapter, DreamBooth, IP-Adapter, InstructPix2Pix, MGIE, SmartEdit, CreatiLayout, CreatiDesign, MagicMotion, Seg2Any, and Re-Imagen are placed here: they execute a supplied condition along a predetermined path. Data-construction systems (ScaleEdit-12M, JAVEDIT, DataEvolver, AgentComp, Gen-n-Val) and fixed generator training (DPOK, AeSlides, FrontCoder, T2I-R1, ReasonGen-R1, OSPO, WorldCycle, Ask-Solve-Generate) are also L0 when the deployed inference path chooses no generation-level action, regardless of how agentic the offline process was.
• Benchmarks diagnose but do not act. T2I-CompBench and GenEval expose compositional and object-binding errors, VBench and EvalCrafter expose temporal and perceptual video defects, and AgenticVBench, DirectorBench, VideoArgus, DECKBench, 3DCodeBench, VisEditBench, and WeEdit extend evaluation to agentic video, slides, procedural 3D modeling, and visualization-code editing. These become part of an L3 loop only when another mechanism maps their judgment to a later revision, rerouting, or stopping action.
• The corpus is small at the ends and growing sharply at L3. After deduplication from seven task collections, only four reviewed records are assigned to L0, and L4 remains a small fraction of the literature. Half-year counts by first public release are: 1 (2022-H2), 5 (2023-H1), 3 (2023-H2), 10 (2024-H1), 11 (2024-H2), 37 (2025-H1), 45 (2025-H2), 149 (2026-H1), and 52 (2026-H2 to date, through August 24, 2026). Growth after 2025 is dominated by L3 Outcome-Adaptive Control.
• L3 is prevalent across all modalities, but means different things. Editing and user-interface tasks expose rendered states that can be inspected and revised, whereas video, 3D, and world tasks require state consistency across time or viewpoints. Image generation has the largest absolute count and also contains most L1 controllers, which are based on prompt construction, layout planning, and retrieval.
• Single controllers dominate every level. In the controller-organization cross-section, each system receives exactly one label: a single language or multimodal controller, a multi-role controller, or a unified multimodal policy. Multi-role organizations occur mainly from L1 through L3, while unified policies remain less common but appear at every controller level. Reinforcement learning is excluded from that comparison because it is an optimization procedure, not a controller organization, and is analyzed separately in the paper's training section.
• Repeated sampling can be mistaken for adaptivity. Because outcome-conditioned revision is the most common agentic behavior, the paper argues its evaluation must distinguish genuine outcome-driven repair from repeated sampling followed by selection, where apparent improvement may come from the selector rather than from adaptive revision.
• A generator-as-controller is not a new level. It is classified by the same maximum-causal-reach test; it would need to select tools or models, manage state, interpret outcomes, and alter its own trajectory rather than only update pixels or visual tokens.
• Not reported in the available content: the paper contains no benchmark scores, accuracy numbers, or quantitative comparisons of the surveyed systems. The provided text is truncated during Section V and Table IV, so the detailed L1–L4 subcategory analyses, the training analysis, and the full level-conditioned evaluation protocol are not available here.

Methodology in Plain English

The authors do not train a model or run a benchmark study. Instead, they define two roles — a controller that decides how generation proceeds, and a primary visual generator or editor that produces content — and formalize the interaction with a small loop: the controller picks action a_t from a policy π(a_t | g, s_t) given goal g and state s_t, and the state updates via s_{t+1} = F(s_t, a_t, o_{t+1}), where the observation can be a generated image, video segment, scene, critique, or retrieved reference. Agenticity is then measured by the latest point in that loop at which evidence can change a future generation decision: constructing a condition (L1), selecting an operation (L2), revising the current task (L3), or adapting future tasks (L4).

To make the classification reproducible, they apply the tests from highest to lowest level and default to the lower level when a causal link is not demonstrated, and they check the test against boundary cases in a table (for example: prompt rewrite plus fixed generator is L1; critique plus repair/regeneration is L3; current-task state only is at most L3; an RL-trained fixed generator is L0/L1 because level follows inference-time decisions, not the optimizer; candidate scoring without follow-up action is L0). They then deduplicate systems from seven task collections into a structured corpus annotated with first release date, primary level, capability path, modality, mechanism, controller type, visual executor, feedback boundary, evaluation type, cross-task persistence, and public resources, and report temporal and cross-sectional statistics derived from the same README records. The taxonomy has one explicit second-stage axis per level: support function for L0, generator-facing specification for L1, executable object for L2, decisive feedback source for L3, and persistent experience carrier for L4.

Why This Matters

Impact on research. The paper gives the field a single, reproducible criterion for agenticity instead of a patchwork of labels, which makes systems that differ causally comparable and makes claims in paper titles testable. It also reframes the L3-to-L4 transition as an open research challenge rather than a routine extension, and it warns that gains attributed to adaptive revision may actually come from a selector in a repeated-sampling pipeline. The level-conditioned evaluation design — matching generators, tools, budgets, and evaluators across levels — provides a template for measuring how much a broader controller decision scope is actually worth.

Real-world applications (drawn from the tasks and systems discussed in the paper):

• Creative image and video production, including multi-shot pipelines that handle script preparation, shot planning, character styling, and clip generation (as in ViMax), and image-to-video trajectory control (as in MagicMotion).
• Design and frontend work, where controllers generate and revise user interfaces and executable representations (Design2Code, FrontCoder, UI2Code^N) and slide decks with verifiable layout rewards (AeSlides, Slide Translation).
• 3D asset and scene construction with internal optimization under a fixed objective (DreamFusion, Magic3D, DreamGaussian) and procedural 3D modeling evaluation (3DCodeBench).
• World-grounded synthesis and interactive simulation, where generated content is not a terminal output but a persistent world state that later decisions depend on.

Industry relevance. The taxonomy bears directly on production tooling: node-graph and workflow systems such as ComfyUI-Copilot, tool-calling assistants such as Visual ChatGPT, and retrieval-augmented generators such as Re-Imagen all sit at specific levels, so vendors and teams can state precisely what their controller decides rather than what it contains. The paper's observation that within-episode feedback has grown fast while cross-task experience reuse is rare also points to where commercial differentiators — persistent memory, skill libraries, and capability profiles such as those in OctoT2I, GenEvolve, and COMFYCLAW — are still scarce.

Future Directions

• From L3 to L4. The corpus shows L4 Experience-Adaptive Control is a small fraction of the literature, so the paper frames persistent, reusable cross-task experience as an open challenge rather than a solved extension.
• Separating real repair from selection. Because outcome-conditioned revision is the most common agentic behavior, evaluation must distinguish outcome-driven repair from repeated sampling followed by selection.
• Making matched evaluation real. The proposed level-conditioned framework must be executed with generators, tools, budgets, and evaluators held fixed, so the value of specification, execution, outcome feedback, and reusable experience can each be isolated.
• The generator-as-controller regime. A future system in which the generator itself selects tools or models, manages state, interprets outcomes, and alters its own trajectory would be classified by the same maximum-causal-reach test — the paper leaves open how such systems should be built and evaluated.

Target Audience

Researchers, graduate students, and practitioners in visual generation and multimodal agents who need a consistent way to position, compare, or evaluate systems that combine LLM/VLM controllers with image, video, editing, 3D, world, slide, or UI generators. It is also useful to technical product and engineering teams who must describe what their generation pipeline actually decides, and to benchmark designers who need matched comparisons across controller capabilities rather than across model checkpoints.

Authors’ abstract

Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.