PaperGym: Rubric-Centered Evolution for Research-Plan Generation

AI summary

Overview

Research area: Natural language processing, specifically LLM post-training and reinforcement learning for scientific reasoning; sits at the intersection of AI-for-science, rubric-based reward modeling, and research-plan generation.

Technical level: Intermediate. The core data-construction idea is easy to grasp, but the training schedule assumes familiarity with GRPO, on-policy self-distillation, KL-divergence objectives, and LLM-as-a-judge evaluation.

Scope: A single paper introducing PaperGym, a framework that converts arXiv papers into supervised/reinforced training environments for generating research plans, plus the PaperGym-20k corpus, two held-out benchmarks, and a two-stage rubric-centered training recipe evaluated across three Qwen3 model scales.

What This Paper Is About

AI systems that aim to automate scientific discovery all begin with a research plan — the proposed hypotheses, methods, and experiments — but unlike math or code, a research plan has no checkable ground truth, so standard reinforcement learning with verifiable rewards has no environment to train against. The authors address this by mining scientific papers themselves: every paper already contains a realistic research problem and an expert-authored solution, so the paper can be restructured into a task plus a grading rubric. Their goal is to build that environment correctly (so the reward actually measures planning ability) and to exploit the rubric fully during training (so the reward signal is not wasted as a single scalar).

Key Contributions

1. A structure-aware pipeline that turns papers into training environments. Papers are decomposed into four stages — Research Goal, Background, Research Method, Experimental Design — and the research question is synthesized only from Goal + Background while the reference answer and grading criteria come only from Method + Experimental Design. This disjoint sourcing cuts criterion leakage to 3.7%, versus 11.90%–34.10% in existing datasets.

2. PaperGym-20k plus two targeted benchmarks. A 20,000-instance corpus spanning computer science (~50%), physics (~25%), and economics (~25%), with ten atomic binary criteria per instance split across methodological innovation and experimental design (63.8% Method-oriented, 36.2% Experiment-oriented). Two held-out benchmarks, PaperGym-Innov and PaperGym-Design, evaluate those two dimensions separately on papers released after the training window.

3. A rubric-centered two-stage training schedule. The same rubric is used twice: first as privileged context for an on-policy self-distillation (OPSD) teacher that provides dense token-level guidance, then as the reward signal for GRPO that verifies complete plans. This "widen-then-narrow" schedule outperforms SFT, either stage alone, and the reverse ordering at every scale.

4. Empirical validation that the data, not just the recipe, drives gains. With the training recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons against 28.2% for models trained on RubricHub Science and 13.7% for the untrained base model.

Main Findings

• Question/criteria decoupling is the central lever. Drawing the question from Goal + Background and the criteria from Method + Experiments produces a leakage rate of 3.73% (PaperGym-20k), 4.71% (PaperGym-Innov), and 4.97% (PaperGym-Design), roughly 3–9x lower than ResearchPlanGen-ArXiv (34.10%), ResearchPlanGen-ML (31.29%), ResearchQA (19.22%), RubricHub Science (17.39%), and HealthBench (11.90%). Low leakage means a model cannot farm reward by paraphrasing the question.

• The two-stage schedule beats every alternative. On Qwen3-1.7B/4B/8B, OPSD followed by GRPO improves five-benchmark averages by +5.56, +5.04, and +4.81 points over the base models — the top score on every test set at every scale. SFT on reference answers yields marginal or negative gains (as low as −0.96 average at 8B), because it collapses output diversity onto one reference plan.

• Stage order matters, and the reason is entropy. OPSD alone peaks near step 200 then degrades from overfitting to privileged context; GRPO alone starts weaker because exploration from a cold start is noisy. OPSD raises output entropy by absorbing new knowledge, and GRPO then lowers it sharply while converging to high-reward regions. Reversing to GRPO→OPSD loses on every benchmark at every scale (−2.04, −0.99, −1.31 average), with the largest penalty on PaperGym-Innov, the benchmark most dependent on retaining a broad prior over valid plans.

• Rubrics are better privileged information than reference answers. In the OPSD ablation, conditioning the teacher on the rubric scores 14.16/10.45 on Innov/Design, versus 13.08/9.57 for the reference answer and 13.81/9.17 for both combined. A single realized solution narrows the set of valid continuations the teacher preserves.

• Both rubric sources and both dimensions are necessary. Removing either question-derived rubrics (−0.97/−0.21) or answer-grounded rubrics (−1.87/−1.11), or restricting to the innovation dimension alone (−0.03/−0.52), all degrade performance relative to the full pipeline.

• Rubric quality is the bottleneck, not extraction quality. Swapping the extraction model to a weaker one costs only −0.97/−0.67, while swapping the rubric generator costs −2.00/−1.18, indicating that nuanced scientific judgment in criterion writing matters more than the document-processing step.

• A small trained model surpasses a much larger one. Qwen3-8B after PaperGym training reaches 73.48 on ResearchQA, above Kimi K2.6 (73.19), and outperforms scientific-task baselines Intern-S1-Mini and Rebicon-Preview on both PaperGym-Innov (24.47 vs. 7.48 and 20.88) and PaperGym-Design (21.88 vs. 4.48 and 20.42).

• Performance scales with data volume. GRPO-trained Qwen3-4B improves monotonically from 0.5k to 15k training instances, advancing from 16.12 to 19.21 on PaperGym-Innov and 13.89 to 16.96 on PaperGym-Design.

• Reward mixing favors specialized criteria. A specialized-to-general reward ratio of 7:3 (α = 0.7) scores best (17.17/12.35), ahead of 8:2 (14.63/10.86) and 6:4 (16.40/11.54).

• Judge reliability is adequate. Four scoring models are highly self-consistent across five temperature-0 runs, with stronger judges scoring more strictly; the small Qwen3-8B agrees with Kimi K2.6 on nearly 80% of binary verdicts.

Methodology in Plain English

The approach rests on one observation: a research paper already contains both halves of a training environment, but they live in different sections.

Step 1 — Restructure each paper. The authors pull plain-text LaTeX from arXiv and use a map-reduce procedure with a large model: split the paper into natural sections, extract per-stage content, then merge duplicates into a clean four-stage summary. Concrete numerical results are deliberately excluded from the Experimental Design stage to prevent the model from memorizing numbers rather than reasoning.

Step 2 — Build the task and the critic from disjoint sources. The research question is written from only the goal and background. The reference answer and the grading rubric come from the method and experiments. Because the two never see the same text, the rubric cannot restate the question, so the model cannot earn reward by paraphrasing.

Step 3 — Write the rubric. Two complementary sources of criteria are generated in parallel: question-conditioned rubrics (derived from the question alone) and answer-grounded rubrics (derived from question plus reference answer). These are merged, deduplicated, ranked by importance, and the top ten are kept. A second, instance-agnostic general rubric is reused from prior work to enforce completeness, specificity, soundness, efficiency, and ethical safety. During GRPO, the reward combines the two sets as a weighted average, with 0.7 weight on the specialized criteria.

Step 4 — Train in two stages. First, OPSD: the model generates its own rollouts, and the same model acting as a teacher — with the rubric as privileged information it will not have at inference time — scores those prefixes, with the student minimizing their divergence. This builds a broad prior over what valid plans look like. Second, GRPO: the model samples a group of candidate plans, a frozen copy of the base model grades each one criterion by criterion into binary verdicts, those become a scalar reward, and group-normalized advantages update the policy with a KL penalty to prevent drift. The two stages form an explicit entropy curriculum: OPSD widens the distribution, GRPO narrows it.

Step 5 — Evaluate. Three Qwen3 scales (1.7B, 4B, 8B) are trained with LoRA on the CS subset (~10,000 instances). Evaluation uses two in-domain held-out benchmarks (Innov and Design) plus three external ones (ResearchQA, ResearchPlanGen-ML, RubricHub Science), scored by an LLM judge, alongside a three-way win-rate study that isolates the contribution of the data from the contribution of the recipe.

Why This Matters

Verifiable-reward RL has transformed math and code, but has stalled on open-ended scientific reasoning precisely because no automatic checker exists. PaperGym shows that a large, cheap, and structurally sound critic can be mined from the scientific literature itself, and that the critical design choice is not which model generates the rubric but which sections of the paper each component is drawn from. The finding that SFT on reference answers actively hurts, and that scalar rubric rewards waste most of their information, has direct implications for anyone building training pipelines for subjective, generative tasks beyond science.

Real-world applications:

• Automated research assistants that propose concrete, falsifiable experimental plans a human researcher can act on, rather than generic summaries of a field.
• Grant and proposal drafting, where a rubric-aligned model can generate method-and-design sections that map onto reviewer criteria such as novelty, soundness, and feasibility.
• R&D ideation in industry labs, using domain-adapted variants of this pipeline to help teams scope experiments before committing expensive wet-lab or compute budgets.
• Peer-review and evaluation support, since the same rubric machinery produces interpretable per-criterion verdicts that can triage submissions or flag plans with weak experimental design.

Industry relevance: The release of the pipeline, the 20,000-instance corpus, and both benchmarks gives labs without access to expensive expert annotation a ready-to-use environment for training research-planning capabilities. The win-rate design also offers a template for evaluating data quality independently of training recipe, which is useful for teams deciding whether to invest in data construction or in better optimizers. The result that a trained 8B model matches or exceeds far larger proprietary systems on ResearchQA suggests domain-specific rubric supervision can be a cheaper path than scaling parameters for structured scientific tasks.

Future Directions

• Whether the leakage reduction transfers beyond arXiv. The corpus is dominated by computer science, with physics and economics as secondary; the same section-mapping logic may need adaptation for fields with different paper structures, such as law, clinical trials, or humanities scholarship.

• Whether rubric quality can be raised further, since it is the identified bottleneck. Replacing the rubric generator with a weaker model costs roughly twice as much performance as replacing the extraction model; the paper does not explore whether stronger or human-in-the-loop criterion writing extends the gains.

• Whether the two-stage schedule generalizes to other subjective generation tasks. The entropy-curriculum argument — widen with distillation, narrow with reward optimization — should in principle apply to creative writing, legal drafting, or strategic planning, but this is untested.

• Whether reward hacking remains a risk at longer training horizons. The authors note that confining rubric evaluation to GRPO reduces verifier calls and that careful privileged-information separation avoids leakage, but they do not report an extended study of verifier blind spots as training continues past their 100–400 step budgets.

• Richer signal extraction from rubrics. Per-criterion verdicts currently collapse into a weighted scalar before reaching the policy; using criterion-level structure directly in the advantage estimate is an obvious unexplored direction.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning with learned or rubric-based rewards, and AI-for-science systems. It is also relevant to practitioners building evaluation harnesses for subjective, open-ended generation, and to those who need to judge whether a data-construction investment is worth more than a training-recipe change. Readers should be comfortable with GRPO, distillation, and LLM-as-a-judge protocols; the paper's distinctive insight — that separating the source sections of the question and the criteria eliminates reward hacking at its root — is accessible even to those who skip the equations.

Authors’ abstract

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.