OmniHarness: Harnessing Generalizable Visual Generation via Symbolic Policy Learning

AI summary

Overview

Research area: Machine learning, specifically multimodal large language models (MLLMs), agentic systems, and visual generation (text-to-image and image-to-image workflows).

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, agent architectures, ComfyUI-style node workflows, and benchmark conventions in generative vision.

Scope: The paper introduces OmniHarness, a framework that learns reusable, abstract "symbolic policies" for families of visual generation tasks — allowing a frozen MLLM to improve continually through self-directed practice, intermediate verification, and localized failure recovery rather than model fine-tuning.

What This Paper Is About

Current visual generation agents tend to memorize solutions to individual tasks, check their work only after finishing, and learn only when a downstream task forces them to. That means their experience rarely transfers: a system that succeeds on one editing job usually cannot explain why it succeeded or apply that principle to a nearby but different job. OmniHarness tackles this by extracting shared procedures from verified executions into abstract, reusable policies, verifying results at each intermediate step so errors are caught and repaired before they propagate, and generating its own practice tasks to build up capability before any real user request arrives.

Key Contributions

1. Symbolic policy learning. Verified execution traces are abstracted into symbolic policies that capture a task family's shared procedure and the conditions under which it applies, while stripping out instance-specific inputs (particular images, prompts, or parameter values). Equivalent policies are merged and tracked with usage and reliability statistics.

2. Feedback-guided execution. The harness instantiates, adapts, and composes stored policies into new workflows. A plan verifier checks step order and dependencies, intermediate outputs are verified against intended effects during execution, and a diagnoser locates the failing step and repairs only that component, preserving already-verified steps and preventing error propagation.

3. Self-directed inquiry. Before any downstream objective is specified, the system proposes its own practice tasks, selecting candidates that are both novel (underexplored context–capability pairs) and near its competence frontier (roughly 50% estimated success), following a "Goldilocks" learnability heuristic. Failure evidence and corrective strategies accumulate in a parallel failure library.

4. Empirical validation and reusable snapshots. Experiments span six benchmarks, three MLLM backbones, and three visual agent frameworks. The learned policy library can be exported as a frozen snapshot and plugged into external agents, and policies learned from image-only practice transfer to unseen video generation tasks.

Main Findings

• ComfyBench autonomous workflow construction: Both OmniHarness variants reach a 100.0% Pass rate across all subsets. GPT-4o + OmniHarness achieves 89.5% Total Resolve; Codex GPT-4o + OmniHarness reaches 92.5%, exceeding the strongest baseline (SymbOmni) by 6.5 points overall.

• Creative task gains are the headline result: On ComfyBench's Creative subset, OmniHarness reaches 95.0% Resolve versus 67.5% for the best competing system — a 27.5 percentage point improvement. Creative tasks test skill application beyond curriculum examples, which the authors argue reflects genuine policy generalization rather than task memorization.

• Text-to-image generation: OmniHarness achieves the highest GenEval overall score of 0.997 (1.00 in five of six categories, 0.98 on attribute binding) and the highest GenEval2 overall score of 89.78, leading on Attribute (94.0), Count (94.0), Position (76.9), and Verb (89.0). The Object score of 95.0 matches SymbOmni but trails Qwen-Image and Gemini 2.5 Flash Image.

• Knowledge-informed generation: On WISE, OmniHarness posts the highest overall WiScore of 0.86, exceeding both SymbOmni and GPT-Image-1 by 0.06, with leads in Time, Biology, Physics, and Chemistry and near-best Cultural and Space scores.

• Image editing: On Reason-Edit's Understanding Scenarios, OmniHarness leads all four metrics (23.89 dB PSNR, 0.86 SSIM, 0.05 LPIPS, 24.55 CLIP). In Reasoning Scenarios it posts the best SSIM (0.80) and CLIP (21.32) with LPIPS tied at two-decimal precision.

• Self-directed inquiry drives the largest ablation effect: Removing it drops Total Resolve from 92.5% to 87.0% and Creative Resolve from 95.0% to 72.5%. Removing online policy updates drops Total Resolve to 88.5%.

• Both task-selection scores matter and complement each other: Dropping capability novelty lowers Creative Resolve to 87.5%; dropping the competence frontier score lowers it to 85.0%; using both beats either alone.

• Execution-side components are load-bearing: On Complex tasks, removing planning drops Resolve from 83.3% to 55.0%, removing intermediate verification to 68.3% (Pass also falls from 100.0% to 75.0%), and removing localized recovery to 76.7%.

• Transfer and reuse hold up: Frozen policy snapshots improve existing visual agent systems through plug-and-play reuse, and policies acquired from image-only practice transfer to unseen video generation tasks.

Methodology in Plain English

The researchers treat visual generation as a library-building problem rather than a model-training problem.

They start with a knowledge graph of ComfyUI nodes — the standard building blocks for image-generation pipelines — plus a pool of source images. Rather than waiting for a user request, a "proposer" component invents practice tasks. Each candidate is scored on two axes: how little the system has explored that combination of context and capability, and how close the task sits to the edge of what the system can currently do. Tasks that are far too easy teach nothing; tasks that are far too hard produce noise. The product of these two scores picks the next practice task.

For each task, a planner assembles an ordered plan by instantiating, adapting, or composing existing symbolic policies, then a writer emits a Python-like program representing the workflow, which a reversible interpreter compiles into an executable graph. Execution is not all-or-nothing: the system verifies intermediate outputs, and when a step fails, a diagnoser identifies which step broke and pulls a corrective strategy from the failure library, repairing just that component while leaving verified steps intact.

After each task — practice or real — verified successes are distilled into new symbolic policies keyed by task family, with reliability tracked as the lower bound of a 95% Wilson confidence interval over usage and success counts. Failures are recorded with their root causes and remedies. A curator periodically merges redundant workflows and consolidates corrective strategies. Crucially, all of this happens while model parameters stay frozen; learning lives entirely in the policy and failure libraries, which can be exported as a snapshot for other agents to use.

Why This Matters

The work reframes agent improvement as knowledge acquisition rather than weight updates. If a system can build a portable, inspectable library of reusable procedures, then capability growth decouples from the cost and risk of retraining frontier models — and the resulting artifact is transferable across backbones and frameworks, as the experiments demonstrate.

Real-world applications:

• Creative production pipelines: Automating multi-step editing, style transfer, restoration, and content preservation in workflows that currently require a human expert to wire together dozens of nodes.
• E-commerce and advertising imagery: Generating product shots that must satisfy hard constraints on object counts, attributes, spatial relations, and brand-knowledge consistency — the exact compositional weaknesses GenEval2 and WISE probe.
• Scientific and educational illustration: Producing knowledge-grounded images across cultural, historical, biological, physical, and chemical domains, where factual alignment matters more than aesthetic polish.
• Agent infrastructure: Supplying frozen policy snapshots as a drop-in capability layer that improves third-party visual agent systems without modifying them.

Industry relevance: The plug-and-play snapshot mechanism is directly commercializable as a complement to existing model APIs — a way to upgrade an agent's reliability without retraining or switching vendors. The reliance on ComfyUI, an industry-standard tool, keeps the approach grounded in real production stacks rather than a research-only abstraction.

Future Directions

• Cross-modal generalization beyond the demonstrated case. The paper notes transfer from image-only inquiry to video generation. How far this extends — to 3D assets, audio, or joint multimodal outputs — is an open question, and the conditions under which policies are modality-agnostic versus modality-specific remain uncharacterized.

• Scaling the policy library. The framework includes a curator that merges redundancy, but the long-run behavior of a library growing over thousands of tasks — retrieval precision, composition depth, and whether reliability statistics stay meaningful — is not examined at that scale.

• Where the Object and Reasoning-scenario gaps come from. OmniHarness trails Qwen-Image and Gemini 2.5 Flash Image on GenEval2 Object score and trails baselines on Reason-Edit Reasoning-scenario PSNR. Diagnosing whether these reflect policy-library coverage limits or fundamental backbone constraints would sharpen the method's value proposition.

• Verifier quality as a ceiling. Every learned policy depends on intermediate verification being correct. Whether verifiers can be learned, self-improved, or made robust to adversarial or ambiguous generation tasks is largely unaddressed and likely bounds the whole approach.

Target Audience

Researchers and engineers working on multimodal LLM agents, agentic self-improvement, and automated workflow synthesis will get the most from this paper. It is also relevant to practitioners building production image-generation pipelines who need reliability and constraint satisfaction rather than one-shot generation, and to those studying memory and skill abstraction in LLM-based agents. Readers without a background in MLLM agent architectures, ComfyUI node graphs, or generative vision benchmarks will find the technical sections dense, though the high-level framing around symbolic policies and self-directed practice is accessible to anyone familiar with how modern AI agents are built.

Authors’ abstract

Unified multimodal large language models (MLLMs) and multi-agent systems have advanced visual generation. However, three limitations remain. (1) Existing methods often distill task-specific experience with limited generalizability. (2) Reflection is often deferred until task completion. (3) Knowledge is often acquired only in response to downstream task demands. To address these limitations, we introduce OmniHarness, a framework for generalizable visual generation via symbolic policy learning. OmniHarness abstracts verified executions into symbolic policies for visual generation task families, capturing shared procedures and applicability conditions while removing instance-specific inputs. The harness instantiates, adapts, and composes these policies for new tasks. Intermediate verification guides refinement and failure recovery during execution. Through self-directed inquiry, OmniHarness autonomously generates and executes practice tasks near its capability limits before downstream objectives are specified. Execution feedback continually refines the policies while model parameters remain fixed. Experiments across six benchmarks, three MLLM backbones, and three visual agent frameworks demonstrate strong performance and continual capability expansion. On ComfyBench's Creative tasks, OmniHarness achieves a 95.0% resolve rate, exceeding the strongest baseline by 27.5 percentage points. Frozen policy snapshots improve existing visual agent systems through plug-and-play reuse.