APT: Feedback-Driven Agentic Post-Training for Text-to-Image Generation

AI summary

Overview

Research area: Text-to-image (T2I) generation, specifically automated post-training via reinforcement learning and multi-agent (LLM/VLM) pipelines.

Technical level: Advanced. The paper assumes familiarity with diffusion and autoregressive image generators, RL fine-tuning (policy gradients, GRPO, DiffusionNFT, LoRA), and rubric-based VLM evaluation, though the high-level ideas are explained clearly.

Scope: The paper introduces APT, a fully automated, feedback-driven framework in which three agents build an evaluation set, diagnose a T2I model's failures, and generate model-specific RL training data in a repeating loop, and reports results across four diffusion models and one autoregressive model on five benchmarks.

What This Paper Is About

Post-training a text-to-image model currently requires humans to spot a model's weaknesses, hand-collect or synthesize training data for them, and repeatedly tune that data distribution. The paper argues this is the wrong framing: different base models fail in different ways, so an RL dataset that helps one model may be suboptimal for another, and the right training distribution should change as the model improves. APT's goal is to make this entire process automatic and model-adaptive, so that the model's own evaluation results decide what it trains on next.

Key Contributions

1. A three-agent, fully automated post-training framework. APT coordinates an Evaluation Set Generation Agent (builds prompt–rubric pairs from a user-specified scope, or from a general-purpose harness when no scope is given), a Diagnosis Agent (evaluates the current model at the instance and category level), and an RL Data Generation Agent (converts the diagnosis into a training prompt distribution that emphasizes underperforming capabilities while retaining moderate and mastered ones).

2. A recursive self-improvement (RSI) loop for T2I. Rather than using a fixed dataset, APT couples an outer loop of diagnostic-driven task adaptation with an inner loop of RL-based model optimization, so each updated model produces fresh evidence that reshapes the next round's training distribution.

3. Empirical evidence for model-specific training distributions. A transfer study shows that RL data generated for one model and reused on another is consistently worse than data generated adaptively for the target model, supporting the claim that effective post-training distributions are model-specific.

4. Demonstrated generality across models, benchmarks, and architectures. The same agentic workflow is applied without model-specific prompt sets, reward designs, or hyperparameter searches to Lens, SD3.5-Medium, FLUX.1 [dev], LongCat-Image (diffusion), and Janus-Pro (autoregressive), with a single APT run per model evaluated on five benchmarks that are never used to construct training prompts.

Main Findings

• APT beats a handcrafted RL set on all four diffusion models (Table 1). Averages across GenEval (overall), OneIG-EN (average), GenEval2 (Soft-TIFA GM), DPG (overall), and CVTG-2K (word accuracy): Lens 71.26 with APT versus 68.61 handcrafted and 57.06 base; SD3.5-Medium 57.92 versus 55.94 and 48.34; FLUX.1 [dev] 57.81 versus 52.32 and 47.11; LongCat-Image 67.72 versus 64.05 and 55.92.

• One APT run improves most benchmarks without benchmark-specific tuning. For example, LongCat-Image moves from 0.426 to 0.821 on CVTG-2K word accuracy and from 0.727 to 0.832 on GenEval, while FLUX.1 [dev] moves from 0.260 to 0.513 on CVTG-2K word accuracy and from 22.23 to 31.01 on GenEval2 Soft-TIFA GM.

• Model-specific data wins over transferred data (Table 2). In the average-over-three-benchmarks transfer matrix, the diagonal (APT data generated for the same target model) is best in every row: Lens 61.40 versus 60.03/60.88/58.93 off-diagonal; SD3.5 47.43 versus 45.87/46.15/44.78; FLUX 50.67 versus 47.19/48.53/48.97; LongCat 57.07 versus 56.04/55.19/56.13. This transfer study post-trains each model for 200 steps for efficiency.

• The APT-generated evaluation set beats reusing a benchmark's own evaluation set for diagnosis (Table 3). On Lens, APT's evaluation set gives 71.26 average versus 70.85 with the OneIG-EN evaluation set and 66.45 with the GenEval evaluation set; on SD3.5-Medium, 57.92 versus 57.17 and 50.07. The paper attributes the gap to single-benchmark diagnosis biasing improvements toward one capability.

• Both the feedback loop and the sampling strategy matter (Table 4, Lens). APT reaches 71.26; removing the iterative feedback loop (still generating new prompts but not re-diagnosing) drops to 70.34; reducing the diagnostic criteria drops to 69.08; sampling only the hardest capabilities drops to 70.64.

• APT transfers to autoregressive generation (Table 5, Janus-Pro). Soft-TIFA GM rises from 15.25 (base) and 20.42 (handcrafted RL set) to 21.61 with APT; the GenEval "overall" column rises from 0.243 to 0.316 (handcrafted) to 0.318 (APT), and Perlin-Pro-scale metrics such as Count (0.538 to 0.606) and Color (0.891 to 0.910) improve.

• Results are robust to the LLM backbone of the RL Data Generation Agent (Table 6, Lens). Average scores range from 68.83 (Grok-4.6) to 72.27 (Qwen3.5-122B-A10B), with GPT-5.4 at 71.26, GPT-5.6-Sol 70.75, GPT-5.4-mini 70.76, GPT-4.1 70.71, Qwen3.5-9B 70.56, and Gemini-3.8-Flash 69.97 — all well above the 57.06 base.

• The evaluation set is large and automatically generated. The Evaluation Set Generation Agent produces 2,609 prompt–rubrics pairs, built from ten scenario categories (Human, Object, Animal, Plant, Scene, Food, Event, Fictional World, Text, and UI and Graphic Design), with up to 10 prompt-specific rubrics per prompt.

• Implementation settings. DiffusiionNFT is the default RL algorithm, with GRPO used for Janus-Pro; LoRA with rank 64 and alpha = 128 at a learning rate of 3e-4; each training round is 20 optimization steps; the Diagnosis Agent samples 120 prompt–rubric pairs per round; the RL Data Generation Agent produces 480 training prompts per round; FLUX.1 [dev] is post-trained for 200 steps and other models for 400 steps.

Methodology in Plain English

The pipeline runs in two phases.

Building the evaluation set (once). Starting from a broad set of image-generation categories, an LLM expands each into fine-grained sub-categories and items (the Human category, for instance, yields sub-categories like Race, Occupation, and Gender, with items such as Researcher and Doctor). Items are then combined with one to four randomly sampled descriptive dimensions — attributes, spatial relations, counts, interactions, colors — and passed to GPT-4.1 to write actual image prompts. Finally, GPT-4.1-mini writes up to 10 prompt-specific rubrics per prompt, specifying what must be checked (identity consistency, text correctness, style alignment, scene correctness, spatial relations, counting accuracy). The result is a prompt paired with its own grading checklist. If the user supplies an evaluation scope, the agent tailors examples to it; otherwise a general-purpose evaluation harness ensures broad coverage.

The training loop. At each round, the Diagnosis Agent samples 120 prompt–rubric pairs, generates images with the current model, and scores each rubric binary (0 or 1) using GPT-4.1-mini as the vision-language evaluator. An image's reward is the arithmetic mean of its rubric scores, and the round's objective is the expected reward under the current prompt distribution. The resulting evaluation report — prompts plus rubric-level pass/fail results — goes to the RL Data Generation Agent, implemented with GPT-5.4, which reads across many examples, identifies recurring failure patterns (for instance, counting or text rendering failing across unrelated semantic categories), and writes 480 new training prompts for the next RL update. Most target weak capabilities, but some cover moderate and already-strong ones to preserve diversity. The RL algorithm then updates the model on those prompts, and the updated model is re-diagnosed, so the training distribution shifts as the model changes. No benchmark is used to construct training prompts, and the same workflow, reward procedure, and configuration are reused across all models.

Why This Matters

Impact on research. The paper reframes post-training from "how do we optimize the model" to "what should the model be trained on," and shows that the answer is model-dependent and time-dependent. It also provides a concrete T2I instantiation of recursive self-improvement, plus an explicit negative result — transferring APT data between models is worse than generating it per model — that challenges the common practice of sharing a single curated RL dataset. Its comparison against a handcrafted RL set curated by human experts suggests human curation may not be necessary for this stage.

Real-world applications.

• Automated improvement pipelines for image-generation products, where a base or fine-tuned model can be sharpened without a human labeling team.
• Text rendering in generated images (signage, posters, packaging), where CVTG-2K word accuracy improved most dramatically in several models.
• Compositional and counting-heavy content such as product catalogs, instructional imagery, and e-commerce scenes, where GenEval and GenEval2 measure object co-occurrence, counts, colors, and spatial relations.
• Domain customization through user-specified evaluation scopes — the paper's UI and Graphic Design category points at design tooling, where a team could describe its target scenarios and let the agents build the evaluation and training data.

Industry relevance. The framework targets the most expensive and least automated part of T2I development: deciding what data to train on. Models like Lens, SD3.5-Medium, FLUX.1 [dev], LongCat-Image, and Janus-Pro all improve under the same workflow, and results hold across GPT, Qwen, Gemini, and Grok backbones, which lowers the barrier to adopting it in existing RL post-training stacks.

Future Directions

• Regenerating the evaluation set, not just the training data. The evaluation set is built once before post-training and held fixed; the paper does not report experiments on refreshing it, leaving open whether an evolving evaluation set would help further.
• Extending to other modalities and architectures. Only image generation is tested (four diffusion models plus one autoregressive model, Janus-Pro); the same agentic loop could be applied to video, editing, or other generators, but this is not reported.
• Cost and scaling of the agentic loop. Each round involves VLM scoring of 120 prompt–rubric pairs and LLM generation of 480 training prompts plus rubrics; sensitivity to these batch sizes and the resulting compute or API cost is not reported.
• Beyond visual-quality rubrics. The reward is a binary per-rubric score from a VLM evaluator; whether APT can handle objectives that are hard to express as rubrics, such as safety, fairness, or long-horizon user satisfaction, is not addressed.
• Understanding why off-diagonal transfer fails. The transfer study shows model-specific data wins, but the paper does not report an analysis of which specific capabilities transfer poorly between which models.

Target Audience

This paper is most useful to researchers and engineers working on post-training, reinforcement learning, and reward design for generative image models; to teams building agentic or LLM-driven data-generation pipelines; and to practitioners who need to improve a specific base model's weakest capabilities without hand-curating a dataset. Readers interested in recursive self-improvement and automated curriculum learning will also find the T2I instantiation relevant. A background in RL fine-tuning and diffusion or autoregressive image generation is helpful for the methodology details, though the conceptual framing is accessible to anyone familiar with text-to-image evaluation.

Authors’ abstract

Post-training is crucial for improving text-to-image models, yet existing pipelines typically rely on manually curated training data and fixed training distributions. We introduce APT (Agentic Post-Training), a fully automated framework that adaptively post-trains a base model according to its evolving capability profile. APT coordinates three agents to construct evaluation sets from user requirements, diagnose model-specific strengths and failure modes, and generate reinforcement learning data with an adaptive prompt distribution that emphasizes underperforming capabilities. By repeatedly evaluating, diagnosing, and updating the generator, APT lets the current model's behavior shape its next training tasks, forming a feedback loop inspired by recursive self-improvement. For general-purpose settings without user-specified evaluation scenarios, we further develop an evaluation harness that enables broad and diverse evaluation set generation. Experiments across multiple base models, benchmarks, and generation architectures demonstrate that APT is model-agnostic, benchmark-agnostic, architecture-agnostic, and fully automated.