Heterogeneous Agentic Coevolution

AI summary

Overview

Research area: Autonomous self-improving AI systems, specifically the coevolution of domain-specific ("specialized") foundation models with LLM-based agents.

Technical level: Advanced. The paper combines a multi-stage training loop (supervised specialist updates, on-policy reinforced self-training for the agent, evolutionary harness search) with a learning-theoretic guarantee, and assumes familiarity with LLM post-training and evolutionary program search.

Scope (one sentence): The paper proposes HACE, a framework in which a converged specialized foundation model and an LLM agent repeatedly improve each other — the agent's verified contributions select what the specialist trains on, and the evolving specialist redefines what counts as a useful agent response — validated on recommendation, protein function prediction, and crystal stability prediction.

What This Paper Is About

Specialized foundation models (recommender architectures, protein language models, interatomic potentials) hold deep domain expertise but cannot self-improve: once supervised training converges, humans must manually invent the next design change. LLM agents can revise code and procedures autonomously, but existing self-evolving systems usually keep the LLM's parameters frozen and target models that are not domain specialists. HACE's goal is to let an LLM agent keep advancing a converged specialist even as that specialist changes underneath it, so that neither side is a fixed teacher and both keep learning.

Key Contributions

1. A heterogeneous coevolution framework (HACE). A single loop that joins supervised learning of a specialized foundation model with on-policy reinforced self-training of a heterogeneous LLM agent, where the two models have different architectures, data, and capabilities.

2. A reciprocal learning mechanism with verified-gain gating. An agent response counts only if it passes verification and its contribution exceeds the current specialist's quality on that instance (r_i(τ) = q_i^A(τ) − q_i^S > 0). Those verified gains then (a) train the agent and (b) determine which instances the specialist trains on with original task labels — note that an agent with lower average quality can still contribute on individual instances.

3. An evolutionary interaction harness. A searched, archived "harness" controls what task information and specialist context the agent sees, how its responses are interpreted, and which examples reach each learner's training budget. Candidate harnesses plus adapted specialists are accepted only if joint validation quality improves (ΔJ_t > 0), evaluated with two-fold cross-fitting.

4. A learning guarantee for evolving experience (Theorem 1). A source-risk bound, R_μ(S_T) ≤ U_T(v) + (L̂_T(S_T) + log(|F|/δ)) / ((1−e^{−1})v), separating whether the loop reaches useful training instances (exposure term U_T) from whether it learns and retains them (loss term), with a consistency condition (v_T → ∞, U_T(v_T) → 0, L̂_T(S_T)/v_T → 0).

Main Findings

• Recommendation: best in all 15 settings. HACE has the highest mean HR@10 for every dataset–architecture pair (5 datasets: ML-1M plus Amazon Beauty, CDs, Sports, Toys; 3 architectures: SASRec, HSTU, Wukong). Relative gains over the converged Base range from 2.9% to 18.5%, averaging 12.1%. Examples: SASRec on ML-1M 0.2557 → 0.2632; HSTU on Toys 0.0673 → 0.0788; Wukong on Beauty 0.0734 → 0.0856.

• Some evolution methods fall below the starting point. On ML-1M, several compared evolution methods score below the initial SASRec and HSTU models, while HACE improves all three architectures. On the four Amazon categories, HACE raises Wukong's HR@10 by 12.9% to 16.6%.

• Protein function prediction: large NDCG jumps. On CAFA, HACE lifts NDCG from 0.6285 to 0.6745 with a frozen ESM-2 backbone and from 0.6241 to 0.6739 with ProtT5-XL. Competing evolution methods improve over Base by at most 0.0040 and 0.0032 respectively, versus 0.0460 and 0.0498 for HACE. HACE also leads on F_max (0.2218 → 0.2438 for ESM-2; 0.2221 → 0.2317 for ProtT5-XL), AUPR, and Recall@10.

• Materials: smaller but consistent margin. On WBM with CHGNet, NDCG goes from 0.6388 to 0.6541, above the strongest competing mean by 0.0028, and F1 from 0.5159 to 0.5513. HACE has the best mean on all four additional metrics reported in Table 6.

• Component ablations: all pieces help, unevenly. Removing agent optimization causes the biggest drop on Toys (0.0706 → 0.0622). On CAFA, removing harness or specialist optimization costs 0.0438 and 0.0435 NDCG. Specialist optimization matters most on WBM (0.6541 → 0.6395, close to Base 0.6388). Removing agent-guided experience selection still lowers scores by 0.0044 (Toys), 0.0170 (CAFA), and 0.0034 (WBM), showing the agent helps through instance selection, not only through prediction.

• Specialist scaling gains are stable in absolute terms. With the agent fixed at Gemma-4 E4B, NDCG gains span 0.0409 to 0.0460 across ESM-2 sizes 35M, 150M, 650M, and 3B. The largest final score is the 3B model (0.6763); the largest gain is at 650M (+0.0460). The 35M model after HACE (0.6692) beats the initial 3B model (0.6339).

• Bigger agent is not automatically better. With the specialist fixed at ESM-2 650M over a shared 30-round configuration, Gemma-4 E4B gains 0.0460, while Gemma-4 E2B gains 0.0012 and Gemma-4 26B-A4B gains 0.0054 — gains do not increase monotonically with agent size.

• Improvement timing differs by domain. Recorded trajectories reach their best score with HACE in all three domains; recommendation scores rise mostly in early rounds, while protein and materials reach new maxima after longer intervals.

• Not reported: dataset sizes and split counts for ML-1M, the Amazon categories, CAFA, and WBM, as well as compute budget and wall-clock cost, are not given in the provided content.

Methodology in Plain English

The setup starts deliberately handicapped: each specialist is already trained to convergence, so ordinary continued training yields no further improvement. Everything after that must come from the interaction loop, which repeats in rounds.

1. Propose a new harness. A harness is a configuration of the working arrangement rather than a model: what specialist predictions the agent gets to see, how its answers are scored, and which examples get spent on training. A new candidate is created by modifying a parent from an archive of previously accepted harnesses.

2. Let the agent attempt the task blind. For each training input, the agent sees the input plus the current specialist's predictions, but not the ground-truth label. Its answer is scored against the label and compared to the specialist's score on that same instance. Only answers that are valid, checkable, and strictly better than the specialist survive verification.

3. Retrain the specialist only on what the harness selected. The specialist is updated with the ordinary supervised task loss on that chosen subset — agent exploration decides where to train, labels decide what is learned.

4. Judge the pair jointly, not separately. Because a harness change alters both the specialist's predictions and the specialist itself, the proposed harness plus adapted specialist are validated together. Candidate ranking blends specialist and agent scores via s_i(c) + λ·a_i(c), with the agent's rank contribution a_i(c) = 1/log2(u+1) when its candidate sits at position u, and λ fit by cross-fitting on validation folds. The pair is kept only if validation improvement is positive; otherwise everything is rolled back.

5. Train the agent on fresh, verified successes. With the retained specialist and harness, the agent re-answers, the largest positive-contribution answer per instance is kept, and the agent is fine-tuned with on-policy reinforced self-training (token-level negative log-likelihood over the selected prompt–response records). An empty selection leaves the agent unchanged.

6. Repeat. The updated agent changes what it explores next round, so the specialist's remaining weaknesses shift, which changes what the agent is rewarded for — the curriculum is continuously re-derived rather than fixed.

Why This Matters

The paper argues that model development itself can become an autonomous learning process for domain models, not just for general-purpose LLMs whose parameters stay fixed while only code and prompts evolve. It also reverses the usual teacher–student assumption in agent distillation: here an LLM agent that is weaker on average in the domain can still be a net source of improvement for a stronger specialist, with no fixed teacher role.

Real-world applications it points to:

• Recommender systems: squeezing further gains from already-converged large-scale ranking models (SASRec, HSTU, Wukong) without hand-designed feature engineering or architecture changes.
• Biology: improving protein function annotation on top of frozen protein language models such as ESM-2 and ProtT5-XL, where retraining the backbone is expensive.
• Materials discovery: better ranking of candidate crystal structures by stability using pretrained interatomic potentials like CHGNet, where each new candidate must be screened in simulation.
• General specialist pipelines: any setting with a converged domain model plus a measurable task metric, where an agent can supply exploratory proposals and verified gains act as the gate.

Industry relevance: the work comes from Meta AI with University of Virginia collaborators, and its headline claim — gains on top of models that have already plateaued, using a small open-weight agent (Gemma4-E4B by default) — is directly relevant to teams that own large production rankers or scientific surrogate models and want automated post-convergence improvement rather than a new pretraining run.

Future Directions

1. Explaining the non-monotonic agent scaling. E4B (+0.0460) far outperformed both the smaller E2B (+0.0012) and the larger 26B-A4B (+0.0054) on the same specialist. What property of an agent — verification reliability, exploration diversity, output format fidelity — actually drives contribution is left open, and the paper offers no mechanism for the pattern.

2. Broadening beyond three domains and one metric family. All evidence comes from recommendation, protein function prediction, and crystal stability ranking; the authors frame heterogeneous coevolution as "a universal approach," which remains untested outside these tasks.

3. Closing the gaps the ablations expose. Removing harness or specialist optimization costs 0.0438 and 0.0435 NDCG on CAFA, and specialist updates dominate on WBM — understanding when specialist updates, harness search, or agent learning is the binding constraint would inform where to spend compute.

4. Verification as the bottleneck. The whole loop depends on an automatically checkable, instance-level quality score and a strictly positive contribution test. Extending HACE to domains where quality is subjective, expensive, or only measurable in batch is an open engineering and theoretical question; the Theorem 1 bound likewise requires sufficient exposure across the source distribution, which may not hold for rare instances.

Target Audience

Researchers and engineers working on self-improving agents, agentic AutoML, and post-training of LLMs; practitioners who own large domain-specific models (ranking, scientific ML, bioinformatics) that have plateaued and want automated improvement; and learning-theory readers interested in guarantees under non-stationary, self-selected training distributions. Readers should be comfortable with reinforcement/self-training objectives and evolutionary search, but the conceptual sections and ablations are readable for a technically trained non-specialist.

Authors’ abstract

Recent advances in self-improving LLM agents have demonstrated the potential of AI systems that drive their own intelligence. However, many domain-specific applications rely on specialized foundation models that possess strong domain expertise but remain dependent on human-designed improvements. Extending recursive self-improvement to these models presents a key challenge: as a specialist improves, the strategies used to advance it must also adapt. To address this challenge, we propose Heterogeneous Agentic CoEvolution (HACE), a framework that jointly evolves specialized foundation models and LLM agents through reciprocal learning. HACE improves the specialist through supervised learning while training the agent with on-policy reinforced self-training, guided by verified performance gains on the current specialist. The agent's exploration, in turn, informs subsequent specialist training, enabling both models to adapt to each other's progress. To sustain this reciprocal learning, HACE further employs evolutionary search to optimize their interaction harness, adapting how the models exchange information and select training experiences as their capabilities evolve. We evaluate HACE across three challenging domains: recommendation systems, protein function prediction in biology, and crystal stability prediction in material discovery. We show that HACE outperforms the evaluated self-improving methods on the primary task metrics across all three domains. These results shed light on heterogeneous coevolution as a universal approach to improving specialized foundation models through reciprocal interaction with a learning agent.