Reward Modeling for Scientific Writing Evaluation

AI summary

Overview

• Research area: Natural Language Processing, specifically reward modeling and automated evaluation of scientific writing (related work sections, peer reviews, novelty assessments, and paper revisions).
• Technical level: Intermediate. Readers need some familiarity with large language models, supervised fine-tuning, reinforcement learning from preference feedback, and reward models, but the paper explains its training design in largely conceptual terms.
• Scope in one sentence: The paper introduces two open-source reward models, SciRM and SciRM-Ref, trained in two reinforcement-learning stages to evaluate diverse scientific writing tasks by reasoning over explicit, dynamically supplied evaluation criteria and scoring rubrics.

What This Paper Is About

Scientific writing tasks such as generating related work sections, writing peer reviews, judging novelty, and revising papers all have their own task-specific requirements, which makes evaluating them hard. Existing LLM-as-a-judge systems and reward models are built for general-purpose benchmarks with fixed rubrics, so they struggle to reason over sparse domain knowledge and changing criteria, and fine-tuning a separate evaluator for each task is costly or impossible when annotated data is scarce. The authors' goal is a single, cost-efficient, open-source evaluator that generalizes across scientific writing tasks without task-specific retraining.

Key Contributions

1. Cost-efficient, open-source reward models for scientific writing evaluation. The authors introduce SciRM and SciRM-Ref, tailored specifically to scientific writing evaluation rather than general-purpose judge benchmarks.
2. A two-stage reinforcement learning training regime. Stage one optimizes scientific writing evaluation preferences; stage two refines reasoning so the model reflexively reinterprets the supplied constitution to correct and stabilize its own judgments. Both stages use the GRPO algorithm.
3. Multi-aspect, constitution-conditioned evaluation instead of a single aggregated score. The models are conditioned on an explicit evaluation constitution (structured criteria and label space, with example evaluations per possible score) at both training and inference time, and they produce aspect-level judgments with separate reasoning and score components, improving reliability and interpretability.
4. Joint training across diverse tasks and datasets for robustness and generalization. Data is curated from related work evaluation and the RevUtil review-utility dataset, yielding 65,357 data instances (58,712 train, 6,645 test), and the resulting models are tested on seen and unseen tasks and aspects.

Main Findings

• Strong performance on seen tasks. On review utility evaluation and related work evaluation, both SciRM and SciRM-Ref consistently achieve the highest accuracy across all evaluation aspects compared with baseline models. SciRM-Masked, trained without the Actionability and Grounding aspects, still outperforms baselines except for Qwen3, GPT-5.2, and o3-mini on the Grounding aspect.
• Near-perfect accuracy on Positioning Consistency. Our models reach 0.995 (SciRM) and 0.996 (SciRM-Ref) accuracy on Positioning Consistency, while Coherence is the more challenging related work aspect, reaching 0.73 for both models because it requires comparing citation sentences with the cited papers.
• Verifiability Extraction is the easiest aspect. All models reach their highest scores on Verifiability Extraction because it is a binary claim-detection task rather than a 1–5 scale judgment; SciRM and SciRM-Ref both reach 0.92 there, and the highest baseline scores are Skywork at 0.90 and GPT-5.2 at 0.89.
• Review utility is the hardest seen task. Overall task accuracy on RevUtil is 0.71 for SciRM and 0.69 for SciRM-Ref, versus 0.83 for both models on related work evaluation; the strongest baseline overall on RevUtil is o3-mini at 0.60.
• Gains on unseen tasks. On the novelty alignment task, SciRM-Ref reaches 0.74 accuracy compared with 0.61 for SciRM, while o3-mini is the highest at 0.78; on the revision task both models score 0.82 on Relatedness, 0.73 on Correctness, and 0.78 overall.
• Second stage mainly helps reasoning-heavy tasks. SciRM and SciRM-Ref are nearly identical on related work (0.83 for both), review utility (0.71 vs. 0.69), and revision (0.78 for both); the difference comes from novelty alignment, where SciRM-Ref scores 0.74 versus SciRM's 0.61.
• Best average across the four tasks. SciRM-Ref achieves the largest average score of 0.76, ahead of o3-mini (0.75), GPT-5.2 (0.74), SciRM (0.73), Qwen3 (0.68), Qwen2.5 (0.67), Selene (0.63), Skywork (0.63), Granite3.3 (0.60), Llama3.1 (0.58), DS-Qwen (0.57), Prometheus (0.56), and DS-GRM (0.54).
• Reasoning traces matter. Ablating reasoning at evaluation time reduces average performance by 0.07 for SciRM and 0.08 for SciRM-Ref, with the largest drops on novelty alignment (-0.21 and -0.25 respectively).
• Qualitative error differences. Baseline models tend to make speculative inferences beyond the criteria, while the trained models produce grounded reasoning; SciRM-Ref improves rubric alignment, whereas baseline models can overlook changed text in the revision task. Manual analysis found the models' reasoning traces are fully consistent with their final verdicts.
• F1 numbers are not shown in the provided content. The paper states full accuracy and F1 results with means and standard deviations are in Appendix D, but the truncated content only includes accuracy tables.

Methodology in Plain English

The authors frame evaluation as a generation problem. A model is given a prompt consisting of a task query, a constitution (criteria plus a scoring rubric), and example evaluations for each possible score. It must produce reasoning wrapped in <reasoning> tags and a final score wrapped in <score> tags, which makes the score easy to extract automatically across different tasks and rubrics.

Training happens in two sequential GRPO stages on top of Qwen 2.5-7B, chosen after preliminary experiments on scientific writing evaluation. In stage one, the reward depends on output formatting and score correctness: -0.5 if no valid score tag is present, 0 if the parsed value is not a valid number, 0.25 if the value is a number but outside the allowed score set, 0.5 if the value is in the score set but wrong, and 1.5 if correct. A quadratic length penalty is subtracted to discourage both truncated and runaway generations, which the authors say was needed to counter reward hacking and training instability. Because the reward distinguishes formatting errors from semantic errors, it gives the model a smoother learning signal.

Stage two targets reasoning. The model's initial reasoning is kept but its score is removed and replaced with a reflection prompt that asks it to revisit the query and criteria before committing to a final score. The reward now depends on both the initial and final scores: 0.5 for staying correct, 1.0 for correcting a wrong initial judgment, -0.5 for staying wrong, and -1.0 for backsliding from correct to wrong. This rewards self-correction and penalizes degradation. SciRM is the model trained only in stage one; SciRM-Ref is trained through both stages.

For data, the authors combine aspect-level related work evaluation (coherence, positioning type, positioning consistency, all binary) with the RevUtil review-utility dataset (actionability, grounding, verifiability extraction, verifiability, and helpfulness, on a 1–5 scale except the binary verifiability extraction step). The Verifiability aspect is split into claim detection followed by scoring only the instances that contain claims. Human-annotated and synthetic splits of RevUtil are merged. Training and testing use LoRA with rank 64 via the Unsloth framework, vLLM for rollouts with 4 samples per instance using nucleus sampling at temperature 1 and top-p 0.95, and run for roughly three days on a single NVIDIA A100 GPU (80 GB).

Evaluation repeats inference five times (three times for GPT-5.2 and o3-mini) at temperature 1 and top-p 0.95, reporting mean and standard deviation.

Why This Matters

• Research impact: The work argues that scientific writing evaluation deserves its own treatment rather than being folded into general-purpose judge benchmarks, and that a single trained evaluator can be reused across tasks instead of building task-specific models for scarce annotated data. It offers a route to inference-time adaptability to changing or even contradictory rubrics, which the authors contrast with Constitutional AI's fixed, internalized constitutions.
• Real-world applications:
  • Automated feedback on draft related work sections before submission, with aspect-level judgments on citation coherence and positioning.
  • Peer-review quality assessment and triage, including flagging whether review claims are verifiable and whether comments are actionable and grounded.
  • Novelty screening that compares LLM-generated assessments against human assessments for alignment in novelty verdicts.
  • Checking instruction-following in automated paper revision, separately scoring whether a revision addresses the instruction (relatedness) and whether it improves the text (correctness).
• Industry relevance: The models are open source and small enough to train on a single 80 GB GPU, which the authors present as a practical alternative to proprietary LLM judges. The paper also argues that aspect-level rewards, rather than a single objective, could guide scientific text generators during reinforcement learning, which matters to anyone building writing assistants for technical domains.

Future Directions

• Scale beyond 7B parameters. The authors limited experiments to 7B-scale models because of GRPO compute requirements, and hypothesize that larger models would capture task-specific evaluation nuances, generalize better, and reason more accurately.
• Build more fine-grained scientific writing evaluation datasets. The authors note that dataset scarcity forces most scoring rubrics to be binary, and that richer datasets would enable more comprehensive analysis.
• Use the models as reward providers for reinforcement learning over scientific text generation, steering generators along multiple evaluation dimensions instead of one aggregate objective.
• Test further transfer to additional unseen tasks and domains, given that the unseen-task evidence here rests on novelty alignment and revision evaluation, and that the reasoning ablation hurts novelty alignment most.

Target Audience

Researchers and practitioners working on LLM evaluation, reward modeling, and reinforcement learning post-training, especially those focused on expert-domain or scientific text. It is also relevant to developers building automated reviewing, revision, or scientific writing assistance tools, and to anyone who needs open-source evaluators that can follow supplied rubrics without per-task retraining.

Authors’ abstract

Scientific writing is an expert-domain task that demands deep domain knowledge, task-specific requirements and reasoning capabilities that leverage the domain knowledge to satisfy the task specifications. While scientific text generation has been widely studied, its evaluation remains a challenging and open problem. It is critical to develop models that can be reliably deployed for evaluating diverse open-ended scientific writing tasks while adhering to their distinct requirements. However, existing LLM-based judges and reward models are primarily optimized for general-purpose benchmarks with fixed scoring rubrics and evaluation criteria. Consequently, they often fail to reason over sparse knowledge of scientific domains when interpreting task-dependent and multi-faceted criteria. Moreover, fine-tuning for each individual task is costly and impractical for low-resource settings. To bridge these gaps, we propose cost-efficient, open-source reward models tailored for scientific writing evaluation. We introduce a two-stage training framework that initially optimizes scientific evaluation preferences and then refines reasoning capabilities. Our multi-aspect evaluation design and joint training across diverse tasks enable fine-grained assessment and robustness to dynamic criteria and scoring rubrics. Experimental analysis shows that our training regime strongly improves LLM-based scientific writing evaluation. Our models generalize effectively across tasks and to previously unseen scientific writing evaluation settings, allowing a single trained evaluator to be reused without task-specific retraining.