Trinity: Self-Evolving Vision-Language Models with a Self-Verifier

AI summary

Overview

Research area: Self-evolving (self-improving) vision-language models, multimodal reinforcement learning, self-play curriculum generation, and scientific reasoning agents — submitted to the Agentic AI for Biological Discovery (AgenticLS) workshop.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, reinforcement learning from self-generated data, and group-relative policy optimization, but the core idea (a model that grades itself) is explained in accessible terms.

Scope: The paper introduces Trinity, a single vision-language model (VLM) that plays three roles — Questioner, Solver, and Verifier — and trains on unlabeled images alone, using an exponential moving average (EMA) copy of the policy as a label-free self-verifier that screens questions, scores reasoning, and arbitrates between the model's own reference answers and a strong Solver consensus.

What This Paper Is About

Existing self-evolving VLMs generate their own training problems from unlabeled images and reward both the question-generating and answer-producing roles mainly by agreement among sampled answers. Agreement is a weak proxy for truth: it cannot detect whether a question is actually grounded in the image, whether the reference answer is correct, or whether a confident majority is systematically wrong. Trinity's goal is to insert an image-aware evaluator into that loop — without labels, external judges, tools, or a separate critic model — so that only verified, genuinely difficult questions become training supervision.

Key Contributions

1. A three-role, single-model self-evolving framework. One VLM acts as Questioner, Solver, and Verifier; the Verifier is an EMA copy of the policy (α = 0.999) that receives no gradients, needs no labels or critic, and inherits the Solver's gains slowly.
2. Verifier-gated rewards. The Verifier screens each generated question–answer pair for image grounding, answer correctness, and difficulty before it can reach the Solver; it separates genuine difficulty from invalidity and credits reasoning only when the answer is correct.
3. Dispute resolution over references. When at least 75% of formatted Solver answers agree on a value that contradicts the reference and all initially scored incorrect, the Verifier re-solves the problem from the image, and if it sides with the Solvers it overwrites the reference, recomputes correctness, and revokes the Questioner's correctness credit.
4. Gains on two backbones. Improvements on mathematical visual reasoning and on scientific benchmarks with biology content for Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct, with reward dynamics that behave as a healthy self-play curriculum.

Main Findings

• Strong gains on Qwen3-VL-8B-Instruct trained on Vero-600K images alone. Trinity improves the base model on every benchmark: MathVerse +12.8 (58.2 → 71.0), MMMU +12.6 (57.0 → 69.6), SciVQR-Bio +8.6 (56.7 → 65.3), EMVista +2.3 (44.8 → 47.0), and MME-SCI +1.3 (19.0 → 20.3). Full row: MathVision 57.8, MathVerse 71.0, MathVista 79.6, MMMU 69.6, EMVista 47.0, SciVQR-Bio 65.3, MME-SCI 20.3.

• Released self-evolving baselines on the same backbone fall short. MM-Zero scores 39.6 on MathVision, 45.1 on MathVerse, 67.2 on MathVista, 58.3 on MMMU, 45.0 on EMVista, 62.5 on SciVQR-Bio, and 17.6 on MME-SCI — below the base model on all three math benchmarks. DPE scores 53.9, 57.2, 76.2, 69.1, 44.1, 60.8, and 20.4, matching Trinity on MMMU but trailing it by 13.8 points on MathVerse and dropping 0.7 on EMVista.

• OmniScience variant is competitive and science-only. Trained on roughly 1.5M figures (1,520,179 rows across 371 shards), Trinity (OmniScience) scores MathVision 58.2, MathVerse 70.6, MathVista 79.6, MMMU 70.7, EMVista 47.0, SciVQR-Bio 62.5, MME-SCI 21.9, improving every benchmark and statistically indistinguishable from the Vero-600K variant on science benchmarks.

• Smaller but consistent gains on Qwen2.5-VL-7B-Instruct. Trinity improves the base across the mathematical benchmarks: MathVision +1.1 (26.6 → 27.7), MathVerse +1.5 (50.0 → 51.5), MathVista +1.2 (70.7 → 71.9), plus EMVista +3.0 (43.9 → 46.9) and SciVQR-Bio +2.8 (38.3 → 41.1), with MME-SCI at 11.7 versus the base's 11.1. The paper attributes the smaller margins to the Verifier being the backbone itself: a weaker backbone screens and grades less accurately.

• Reward dynamics look like a working curriculum. On Qwen3-VL-8B trained on OmniScience (steps 0–299), averaged over first versus last 50 steps: question factuality R_fact rises from 0.27 to 0.57, Verifier-rated difficulty R_diff from 0.11 to 0.23, Solver correctness R_correct from 0.53 to 0.63, and reasoning quality R_reason from 0.76 to 0.93. Challenge reward R_chal stays below 0.09 because it is assigned only to the selected candidate per image (selection rate rising from 0.16 to 0.23, with a selected-only mean of 0.36 in both windows, giving products of 0.057 and 0.083) and because it is a non-stationary target.

• The 7B trajectory behaves the same way with smaller magnitudes. R_fact 0.19 → 0.45, R_correct 0.64 → 0.71, R_chal 0.029 → 0.055.

• Protocol caveats reported by the authors. The 8B starts from a format-warmed checkpoint (300 examples, supervised warm-up), which must be accounted for when comparing to an unmodified Instruct checkpoint. The three science benchmarks are single-run (K = 1). Open-response items in SciVQR and MME-SCI are scored by a local Qwen3.5-27B judge using the official SciVQR judge prompt (the official judge is Qwen2.5-72B-Instruct). VisPlay, iReasoner, and V-Zero have not released checkpoints, so they could only be compared on the mathematical benchmarks reported in their own papers.

Methodology in Plain English

Trinity keeps two copies of one VLM. The policy answers under two different prompts: as a Questioner it proposes G = 4 candidate question–reference-answer pairs per image; as a Solver it produces K completions for the chosen question (K = 6 for the 7B run, K = 4 for the 8B runs). The Verifier is a slowly updated average of that same policy — a Polyak or EMA copy with α = 0.999 — used only for inference. Because it lags the policy, it cannot be gamed as easily as the policy grading itself, and it needs no separate training.

The Verifier intervenes three times. First, it tags every candidate question on factuality (are all premises visible in the image, and is the question unanswerable without it?), answer correctness, and difficulty from a four-level rubric (observable, single-step, multi-step, conceptual — scored 0.1, 0.4, 0.7, 1.0) that caps lookup-style questions. Format-valid candidates that pass the gate are ranked by difficulty, with ties broken at random; if none pass, the image contributes Questioner rollouts only. Second, each Solver completion is scored for correctness (a three-tier rule-based match, a numeric hard-mismatch guard at 1% tolerance, then a text-only equivalence check) and reasoning quality from the image and the chain of thought (0, 0.25, 0.5, or 1), penalizing visual misreads and shortcuts. Third, if at least 75% of formatted Solver answers agree on a value that differs from the reference and all initially scored incorrect, the Verifier re-solves the problem itself and picks a side; if it sides with the Solvers, the reference is overwritten and every correctness score is recomputed.

The Questioner's reward combines a format gate, a factuality gate, the a-priori difficulty, and a Solver-validated challenge term (the empirical failure rate of the current Solver), with coefficients (α_Q, β_Q, ρ_f) = (0.3, 0.7, 0.5) — so an ungrounded question earns nothing, a grounded question with a wrong reference earns at most half, and empirical failure dominates. The Solver's reward is w_c·R_correct + w_r·R_reason·R_correct + w_f·R_fmt, with (w_c, w_r, w_f) = (0.5, 0.3, 0.2), meaning a fluent but wrong chain of thought earns nothing for its reasoning. Rewards are standardized into group-relative advantages as in GRPO, and Questioner and Solver rollouts are pooled and optimized with GSPO's sequence-level clipped surrogate. The Verifier is refreshed after each outer step: φ ← αφ + (1 − α)θ with α = 0.999. At deployment only the Solver policy is used.

Why This Matters

Impact on research. The paper challenges the dominant reward signal in self-evolving VLMs — answer agreement — by showing that grounding, reference correctness, and reasoning quality can each be evaluated by a lagged copy of the model itself. It also connects self-improvement to scientific imagery, arguing that a self-evolving multimodal agent can strengthen scientific reasoning from unlabeled images alone. The authors are explicit that the reward decomposition (verification instead of consensus) is the contribution, not new benchmark architecture.

Real-world applications (as framed or motivated by the paper):

• Life-science and biological discovery agents that read figures, plan multi-step analyses, and improve from their own experience without proportional growth in expert annotation.
• Electron-microscopy understanding, which the paper calls central to structural and cell biology — the target of the EMVista benchmark where Trinity improves EMVista by +2.3 on the 8B backbone and +3.0 on the 7B backbone.
• Scientific figure reasoning from open-access journals: the OmniScience ablation trains only on figures cropped from papers across biology, chemistry, physics, medicine and more, and still improves every benchmark.
• Medical and STEM image reasoning inherited from the training corpus, whose STEM sixth includes science diagrams, geometry, pathology and radiology images (PathVQA, VQA-RAD).

Industry relevance. The loop consumes raw images and never reads a corpus's question–answer annotations, which is attractive wherever scientific images are abundant but expert annotations are scarce. The costs are real and reported: three co-resident roles require eight A100 80GB GPUs, one 8B outer step takes on the order of ten to twenty minutes, and 300 steps take several days, with wall-clock time dominated by generation rather than the gradient step. The authors note the loop is more expensive per training example than supervised RL on annotated data, which they describe as the price of not needing annotations.

Future Directions

1. Stronger backbones as drop-in Verifiers. Because the Verifier is the backbone itself, its accuracy is bounded by the model's own perception and reasoning; the authors propose dropping in any stronger open VLM, pairing a fixed Solver with Verifiers of different sizes to measure how much gain is attributable to Verifier accuracy, and calibrating Verifier decisions against an independently annotated sample.
2. External anchors at dispute resolution. Retrieval over the paper a figure comes from, a symbolic checker for numeric answers, or — in the laboratory setting envisioned by the workshop — an experimental readout could be inserted at the adjudication stage without changing the rest of the loop.
3. Cheaper Verifier instantiations and lower memory cost. A quantized EMA copy, a Verifier that shares the vision encoder with the policy, and batching Verifier calls across the G candidates are proposed to reduce memory and compute.
4. Joint versus alternating optimization, and broader scale/backbone coverage. A controlled comparison of the joint update against an alternating schedule that freezes one role while the other trains; varying backbone size within one family to test whether gains grow with scale; and applying the loop unchanged to architecturally different VLMs such as Gemma and Molmo2. The paper also lists measuring evidence-grounded biological judgments and downstream scientific utility directly as a requirement before claiming use in biological discovery.

Target Audience

Researchers and practitioners in multimodal machine learning, reinforcement learning for language and vision models, and AI for science — particularly those building self-improving or self-play training loops who need a reward signal stronger than majority agreement, and those interested in agents for biological and scientific image reasoning. It is also relevant to engineers assessing the compute and memory footprint of three-role self-evolving training (the paper reports eight 80GB GPUs and multi-day runs for 300 steps at 8B scale). Readers should be comfortable with policy-gradient terminology; the paper's own framing of the problem — agreement is a weak proxy for truth — is stated plainly enough to be useful to a broader technical audience.

Authors’ abstract

Self-evolving vision-language models (VLMs), a form of self-improvement in which a model generates its own training data from unlabeled images, are a promising route toward agents that expand their reasoning capability in an unsupervised manner, without relying on ever-larger annotation budgets. Existing methods pair a Questioner that proposes problems with a Solver that answers them, but reward both roles mainly by agreement among sampled answers. Agreement is a weak proxy for truth: it cannot tell whether a question is grounded in the image, whether the proposed reference answer is right, or whether a confident majority is wrong in the same way. We present Trinity, in which one VLM plays three roles, Questioner, Solver, and Verifier, and the Verifier is a self-verifier: an exponential moving average (EMA) of the policy itself, requiring neither labels nor an external judge. The Verifier screens every generated question for image grounding and answer correctness before it becomes supervision, scores Solver reasoning against the image, and adjudicates disputes between the reference answer and a strong Solver consensus, correcting the reference and penalizing the Questioner when the consensus is right. Trained on images alone, Trinity improves Qwen3-VL-8B on mathematical visual reasoning and on science benchmarks with biology content, for example, +8.6 points on the biology split of SciVQR and +12.8 on MathVerse, and its reward dynamics behave as a healthy self-play curriculum should. These results suggest that a self-evolving multimodal agent can strengthen its scientific reasoning from unlabeled scientific images alone, with the model itself serving as the verifier.