ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents

AI summary

Overview

Research area: AI agents for scientific research — specifically agent harness optimization, reinforcement learning from task-specific rubrics, and recursive self-improvement.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO), agent architectures (ReAct loops, harnesses), and benchmark-based evaluation in biomedical domains.

Scope: ScienceBuddy is a released interactive scientific workspace where an agent's working procedures (the harness) and its underlying model weights improve in alternating, nested cycles driven by real researcher collaboration.

What This Paper Is About

Most scientific AI agents improve only within a single conversation: a researcher corrects an answer, but that correction never changes how the agent works on the next task. This paper asks how an agent can convert collaboration with researchers — their requests, feedback, and execution traces — into durable improvements in both its procedures and its capabilities.

The authors build ScienceBuddy, a workspace combining 224 scientific tools across 22 modules with a pluggable agent harness, and propose recursive-in-recursive self-improvement: alternating between evolving the harness (inner recursion, model frozen) and training the model under that harness (outer recursion, harness frozen).

Key Contributions

1. A released scientific research workspace. ScienceBuddy is delivered as an interactive product connecting researcher dialogue, executable analysis (Python, R, Bash), inspectable execution traces and artifacts, and a pluggable harness — with 224 tools spanning genomics, molecular and cancer biology, pharmacology, bioimaging, literature retrieval, and database queries.

2. Interaction-grounded tasks and supervision. A pipeline derives executable tasks and task-specific evaluation rubrics from full collaboration trajectories (not just task instructions), packaged as Harbor tasks that support both supervised fine-tuning via rejection sampling and RL via rubric rewards. Researcher replies inform criteria but are explicitly not treated as scientific ground truth.

3. Recursive-in-recursive self-improvement. The central paradigm: an inner recursion where a fixed auxiliary model diagnoses failures and proposes bounded harness edits, accepted only if they improve paired development evaluation; and an outer recursion where the model is trained with GRPO on rubric rewards under the selected harness. The two are coupled bidirectionally and both run in the background while the service stays online.

4. Case-study evidence for procedural and model learning. Experiments that isolate each mechanism — harness evolution with weights frozen, and model RL with the harness frozen — plus a three-cycle co-evolution run showing compounding gains.

Main Findings

• Harness adaptation works without touching model weights. Refining the harness on an adaptation set of LAB-Bench and Biomni-Eval1 tasks raised validation accuracy from 31.1% to 51.1% (a 20-point gain), with best observed adaptation-batch accuracy reaching 75.0%. The learned procedures included Python execution, resource and schema inspection, bounded record lookup, and domain-specific steps like gene-set membership checks and cytoband lookup.

• Model RL broadens the range of solvable problems under a fixed harness. After roughly two hours of RL, problem coverage (pass@4 — fraction of test problems solved at least once in four attempts) rose from 48.3% to 67.8%, a 19.5-point gain with the same harness and attempt budget.

• Co-evolution compounds across cycles. Over three cycles starting from Qwen3.5-4B, harness validation accuracy improved within each cycle (38.9%→44.4%, 34.4%→46.7%, 61.1%→70.0%) and mean training reward rose in each RL phase (33.3%→38.8%, 44.1%→60.5%, 57.8%→69.8%).

• End-to-end scientific task accuracy improved substantially. Single-attempt held-out test accuracy rose from 42.2% to 73.3%, with 33.3% of problems transitioning from incorrect to correct and only 2.2% regressing from correct to incorrect. Gains appeared across all four task families.

• Acceptance is strictly conservative. A candidate harness replaces its parent only when it satisfies edit-constraint checks and strictly outperforms the parent on identical development tasks, seeds, and budgets — with ties retaining the parent and previously passed tasks included to catch regressions.

• The optimizer itself does not improve. The auxiliary diagnosis model remains fixed, so improved task performance should not be misread as the improvement mechanism becoming stronger.

Methodology in Plain English

The system runs a ReAct-style loop: the agent reasons, calls a tool or executes code, observes the result, and repeats, with researcher replies feeding in as observations. The architecture deliberately separates the harness — instructions, reusable skills, and context settings — from the surrounding infrastructure, so procedural changes are explicit and testable.

Inner loop (harness evolution): With model weights frozen, a separate frozen auxiliary model reads recent trajectories and rubric evaluations, identifies which criteria were unmet, and proposes one bounded edit — adding, removing, or revising a single scoped skill, instruction, or context setting. A schema check enforces the edit budget. Parent and candidate are evaluated on the same development tasks and budgets; the candidate is adopted only if it wins.

Outer loop (model learning): The selected harness is used to calibrate task difficulty — augmenting tasks by varying inputs, conditions, or step dependencies so they stay challenging. Fresh on-policy rollouts are then scored by task-specific rubrics with weighted criteria, using executable checks where possible and a fixed judge for interpretive criteria. Those rubric scores become trajectory-level rewards for a GRPO update, with harness and rubrics frozen during training.

Coordination: All harness and environment versions are archived. After each outer update, the inherited harness is re-evaluated under the new model before deployment, and the updated pair returns to researchers, whose next interactions seed the following cycle. Everything happens asynchronously in the background while the live service continues.

Why This Matters

This work reframes the improvement path for scientific AI: instead of treating researcher corrections as one-off conversational fixes, it turns collaboration into a renewable supply of training tasks, evaluation rubrics, and diagnostic evidence. It also argues that procedural adaptation and weight-level learning are complementary rather than competing — each creates opportunities for the other, and neither alone reaches the same ceiling.

Real-world applications:

• Biomedical study design. A researcher scoping a JAK1 / immunotherapy / tumor-microenvironment study received an ordered plan distinguishing treatment-effect modification from prognosis, with named tools assigned to single-cell, signature-scoring, and cell-communication steps.

• Evidence-linked scientific presentations. An ARL4C study request produced panel-selection rationales and speaker notes that tied each claim to its supporting comparison (depletion vs. conditional knockout, blockade, kinetic and rescue assays).

• Multimodal target analysis. Uploaded pathway diagrams were interpreted into structured target-evidence tables, with retrieved evidence and gaps explicitly visible rather than silently assumed.

• Protocol troubleshooting and variant assessment. Benchmark families covering database judgments, protocol debugging, and gene/variant interpretation map onto routine wet-lab and clinical bioinformatics support.

Industry relevance: The paradigm targets any organization running AI agents in a high-stakes expert loop — pharma R&D, clinical genomics, research infrastructure providers. The separation of an editable harness from fixed infrastructure is a practical deployment pattern: improvements ship without retraining from scratch, and every harness and environment version stays auditable.

Future Directions

• Does the improvement mechanism itself need to improve? The reflector and auxiliary diagnosis model stay fixed. Making the meta-level editing procedure editable — as Hyperagents does — is a natural next step that the authors explicitly leave open.

• Can gains generalize beyond biomedicine? The tool catalog and data currently specialize in biomedicine even though document handling and execution interfaces are domain-agnostic. Testing the paradigm in chemistry, materials science, or physics is untested.

• How much of the gain comes from which edits? The paper acknowledges limits in attributing improvements to individual harness edits or particular feedback sources, and notes that rejected edits and score changes are retained in optimizer history but not fully analyzed.

• How robust is rubric-based supervision at the frontier? Rubrics derived from collaboration, with judges for interpretive criteria, risk reward hacking or drift on tasks where correctness is genuinely contested. Scaling to adversarial or ambiguous scientific claims is unresolved.

Target Audience

AI researchers working on agent architectures, harness or prompt optimization, and reinforcement learning from verifiable or rubric-based rewards. Also relevant to biomedical informatics engineers and research-software teams evaluating deployable scientific agents, and to AI-for-science strategists interested in continual learning from real expert interaction rather than static offline benchmarks. Readers need comfort with RL terminology and agent evaluation methodology; the workspace case studies are accessible to a broader scientific audience.

Authors’ abstract

We introduce and release ScienceBuddy, an interactive scientific research workspace that brings continually improving scientific agents into researchers' everyday workflows. ScienceBuddy supports researchers in carrying out scientific tasks while transforming their requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. At its core is recursive-in-recursive self-improvement, a paradigm that couples harness evolution with model reinforcement learning: the inner recursion improves the harness with the model fixed, while the outer recursion trains the model under the improved harness. Harness evolution shapes training experience, and model learning creates new opportunities for harness adaptation. We present case studies of researcher interaction, harness refinement, and model learning, with the benchmark cases spanning four scientific task families. By releasing ScienceBuddy as a research product, we make this paradigm available to the scientific community and take a step toward discovery intelligence: scientific AI that advances through sustained collaboration with researchers and evolves alongside the research it supports. Website: http://science-buddy.io