Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

AI summary

Overview

Research area: Agentic AI / continual adaptation of frozen language-model agents, applied to professional graphic design (tool use, procedural skill memory, agent self-improvement).

Technical level: Advanced. The paper assumes familiarity with agentic tool-calling loops, LLM/VLM-as-judge evaluation, retrieval, and safe policy improvement concepts.

Scope: The paper describes a deployment-scale framework that evolves an external library of natural-language design "skills" around a frozen frontier model, growing the bank from 76 to 139 skills over five rounds on 1,406 briefs, and measures the resulting gains on general text-to-image and specialized graphic-design benchmarks across three backbones.

What This Paper Is About

Professional graphic design is a long-horizon agentic task: editable artifacts emerge from dozens of interdependent tool actions, but there is no reliable programmatic oracle for whether a design is good. The paper's goal is to let an agent improve from real user traffic without updating any model weights and without human reward labels, by treating an external library of natural-language design procedures (the "skill bank") as the object of learning. The system only changes SKILL.md files; the model, tools, renderer, evaluator, and evolution roles stay fixed.

Key Contributions

1. Skill evolution inside a professional graphic-design agent. A persistent library of reusable natural-language procedures becomes the learning target around a frozen foundation model, with mechanisms to both acquire and revise procedures from user traffic. The agent operates equivalents of Adobe Photoshop, Illustrator, and InDesign through more than 230 tools.
2. Conservative evolution under unverifiable feedback. A "matched replay gate" decides which proposed skill changes enter the deployed bank, designed to tolerate noisy VLM-judge scores and rollout variability.
3. Coupled acquisition and revision in deployment. Across five evolution rounds, multiple frozen backbones, and several benchmarks, the paper shows acquiring new skills (widening) and revising existing skills (deepening) are substantially more effective together than either alone.
4. A matched-replay protocol that holds upstream context fixed and judges candidate-vs-incumbent outputs pairwise under order randomization, so the only difference under test is the skill condition.

Main Findings

• The bank grows from 76 to 139 skills with no weight updates and no human labels. Five rounds over 1,406 briefs produced 1,869 automatically graded trajectories. Net growth was 63 skills above the 76 documentation-derived cold-start skills.
• Execution success on GenEval2 for Claude-Sonnet-4 rises from 72.7% to 99.3%, with a +11.99-point gain in generation quality on that benchmark. The same backbone's DPG-Bench success rate rises from 82.7% to 100%.
• Average generation quality improves across all three backbones: +3.83 (Claude-Opus-4.6), +7.70 (Claude-Sonnet-4), and +9.67 (Qwen3.6-27B), though individual benchmarks can regress (e.g., GenEval2 quality declines 61.24 to 57.92 for Opus, and DPG-Bench success declines 44.0% to 28.7% for Qwen).
• The evolved agent wins a majority of pairwise comparisons on specialized design benchmarks: 61.8% overall win rate for Claude-Sonnet-4 and 67.6% for Claude-Opus-4.6 against the no-skill agent, judged blind by GPT-5.4. Qwen3.6-27B reaches 62.8% overall but loses on GraphicBench (49.0%, -1.0%).
• Widening and deepening are superadditive. On 200 held-out briefs, widening alone reaches a 49.4% win rate over the no-skill agent and deepening alone 48.6%, while their combination reaches 58.5% (p = 0.025). Completeness rises from 68.62 (cold start) to 74.04, a +5.42 gain, of which +3.85 is attributed to the interaction.
• Cold start alone does not beat the baseline. The 76 documentation-derived skills score 68.62 versus 69.08 completeness for the no-skill agent, with a 46.4% win rate.
• Gains concentrate in completeness, not aesthetics. Aesthetics moves only 65.92 to 66.53, and critique scores remain largely unchanged.
• The gate rejects a large fraction of proposals. Across five rounds it rejected 100 of 231 rewrite proposals (committing 131) and 67 of 136 mint candidates (committing 69); deepening also discarded 6 superseded or merged skills.
• Minting and rewriting follow distinct schedules. Round 1 was almost entirely repair (39 of 59 rewrites and 4 of 10 mints committed); minting peaked in rounds 2–3 (22/46 and 26/40 committed), growing the bank from 77 to 124; round 5 committed 21 rewrites and only 4 mints.
• Progress is not monotonic. Round 4 fell below the no-skill baseline at completeness ≥ 0.3 (90% vs. 94%) while retaining a +7 pp gain at ≥ 0.9; the authors attribute this to round 4 holding the largest stock of never-revised v1 skills. Round 5 became the strongest round at every threshold.
• Widening covers genuinely new intents rather than paraphrasing the seed bank. Nearest-neighbour distance of minted skills to the cold-start bank (0.215 median) exceeds the seed bank's internal spacing (0.158 median; Mann–Whitney p < 10⁻⁸, Cliff's δ = 0.58).
• Latency overhead is modest. Evolve adds only 3.4%–6.2% mean latency across backbones, and sometimes reduces generation time (e.g., 113 to 82 s for Sonnet on DPG-Bench).
• Evolution adds no marginal token cost despite retrieval. Skill retrieval adds roughly 28% prompt tokens over Base, but Evolve uses fewer prompt/output tokens than cold start (436.6k/5167 vs. 445.2k/5298), which the authors attribute to more direct execution and fewer corrective retries.

Methodology in Plain English

The system keeps a frontier language model frozen and lets it drive real design software through more than 230 tools. Around it sits a bank of natural-language playbooks (SKILL.md files) that describe reusable design workflows, such as "double exposure: extract a subject, build masks, blend in another asset, refine." Before execution, retrieval injects a playbook and a reduced tool list into the model's context. Turning retrieval off recovers the original agent, which gives a clean baseline.

Four fixed roles run the loop: a Prompter poses design briefs drawn from user data and LLM-generated variants; the Solver is the agent itself, which turns a brief plus retrieved skills into a rendered image; a multimodal Grader scores each image and explains why each unmet requirement failed; and a Reflector turns those failures into targeted edits of SKILL.md files.

The bank changes along two axes. Widening extracts the subtasks each trajectory actually performed, flags subtasks that no retrieved skill covered, and pools them by canonical label; once a label accumulates k_min = 3 occurrences, an LLM distills those cases into a candidate skill. Deepening treats any trajectory scoring below τ = 0.6 as a failure against every skill it retrieved, selects each skill whose failure count reaches m = 2 (most-failing first), and gives the Reflector the failure rationales alongside successful uses of the same skill as a do-not-regress baseline, so it reasons over a concrete success-versus-failure divergence rather than failure text alone.

Both axes only propose changes. A replay gate decides what ships. It never uses absolute scores, because a VLM grader's score for the same image drifts between runs. Instead, it samples prompts that exercise the skill, generates several contexts per prompt with distinct retrieved assets and upstream state, freezes each context, and replays it fresh in the same batch under both arms — candidate versus incumbent for a rewrite, or candidate versus no-skill for a mint. Outputs are judged pairwise under order randomization. A prompt counts as won only if the candidate wins a majority of its contexts. A change ships only if no prompt was lost and at least one prompt was won.

Evaluation uses a separate set of 200 human-authored briefs, disjoint from the 1,406 evolution briefs, frozen and reused after each round. External evaluation samples 300 prompts from each of GenEval2, DPG-Bench, OneIG-EN, OneIG-ZH, OpenCOLE, GraphicBench, CreatiDesign, and BannerRequest400. General T2I benchmarks report Soft-TIFA geometric mean (GenEval2), Soft-TIFA arithmetic mean (DPG-Bench), and VQAScore judged by Qwen3-VL-8B-Instruct (OneIG); design benchmarks report pairwise Evolve-vs-Base win rates judged by GPT-5.4. Backbones are claude-opus-4.6 and claude-sonnet-4 via Amazon Bedrock, and Qwen3.6-27B via vLLM on 8× A100 GPUs with 65K-token context.

Why This Matters

Impact on research. The paper argues that procedural memory is a practical route to continual agent adaptation under noisy, unverifiable feedback — a setting where supervised learning needs costly demonstrations and outcome-based optimization must contend with both long-horizon credit assignment and imperfect proxy rewards. Separating proposal from admission, and gating changes by matched replay rather than absolute judge scores, offers a transferable pattern for updating deployed agents whose foundation models are externally hosted or impractical to update. It also documents a caution: evolution is not monotonic, and newly minted skills can misfire until they are revised.

Real-world applications include:

• Deployed creative software assistants that learn recurring design workflows from actual user traffic without retraining or shipping model updates.
• Marketing and brand asset production, such as banner creation, where measurable improvements appear on BannerRequest400 (71.3% win rate for Opus, 69.2% for Qwen).
• Multi-step image editing pipelines (double exposure, silhouette extraction, masking), where the paper's qualitative examples show the baseline often places assets but stops short of the required edit.
• Any domain with long-horizon tool use and no executable success oracle, where the replay-gate pattern could govern which proposed changes ship.

Industry relevance. Making the skill bank the learning target means the expensive artifact — a hosted frontier model — stays untouched, while improvements accumulate in a portable, human-readable store of procedures. Modest latency overhead (3.4%–6.2%) and no marginal token cost versus cold start make the approach plausible for production use, and the observed failure modes (regressions at lower completeness thresholds, model defaults overriding retrieved guidance) are the kind of behavior product teams would need to monitor.

Future Directions

• Preference-aware retrieval and stronger execution mechanisms. The authors note that a natural-language skill can describe a preferred procedure, but retrieval alone may not make the model follow it when the model has a strong default strategy; they call for preference-aware retrieval, deterministic primitives, structured plans, and more precise verification.
• Better handling of fine geometric operations, which the paper reports remain limited by both model perception and automated verification.
• Preserving fidelity in long procedures. Instructions lose fidelity as they accumulate over many execution steps, an open problem the paper explicitly leaves unaddressed.
• Extending the gate beyond the replay set. The replay gate is local to the evaluated cases, so rejecting observed regressions does not guarantee monotonic improvement over the full user-traffic distribution; a distribution-level guarantee remains open.
• Personalization. The paper mentions a personalization mechanism described in Appendix I that is excluded from the public skill pool and all main-paper experiments, leaving its behavior unexamined in the main results.

Target Audience

This paper is most useful to researchers and engineers building agentic systems that use external tools in open-ended domains, particularly those working on agent memory, procedural skill libraries, or deployment-time continual adaptation of frozen models. It is also relevant to practitioners in creative software and content production who need to improve agent behavior without fine-tuning hosted models or collecting human preference labels. Readers interested in LLM-as-judge reliability and conservative update gating will find the replay-gate design directly applicable. Beginners will find the system-level framing accessible, but the benchmark tables, metric definitions, and safe-policy-improvement reasoning assume an advanced background.

Authors’ abstract

Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.