From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks

AI summary

Overview

Research area: Empirical study of how research propagates into practice — specifically, how reinforcement learning (RL) post-training methods published on arXiv reach the code of open-source training frameworks for language models.

Technical level: Intermediate. The paper sits between machine learning research and open-source software studies; readers benefit from knowing what RL post-training is (PPO/GRPO-style training, loss terms, clipping) and how GitHub commits, pull requests and issues work, but no mathematics is required.

Scope: The paper traces seven open-source RL training frameworks through their commit histories and links them to 6,939 screened arXiv papers from January 2025 to June 2026, measuring which proposed methods were actually implemented, how quickly, by whom, and what happened to the rest.

What This Paper Is About

Most practitioners use a new RL method for language models only if the framework they train with ships it, so a method's practical reach depends on whether it lands in a framework's code. This paper measures that step — framework inclusion — which the author argues has not previously been measured, and compares the papers that got in with those that did not. It also reads the public record (issues and pull requests) and the technical reports of large open-weight models to see how non-included methods were treated and what large training runs actually used.

Key Contributions

1. A measurement of framework inclusion with a defined unit and tier system. The paper defines inclusion as an implementation reaching the main branch of a tracked framework, records the tier of each inclusion (default, conditional default, core option, switch, experimental code, example recipe), maps every option name to a catalog of 65 methods, and traces 426 addition/removal/default-change events across seven frameworks.

2. A screened supply of methods with validated labels. It harvests 139,135 computer-science arXiv metadata records through OAI-PMH (identifiers 2501 to 2606), screens them to a pool of 6,939 papers on language models and reinforcement learning, and labels each with one of seven categories, with three described checks on recognition, stability and the definition of a "general" method.

3. A public-record audit of proposals and merges. It downloads all 26,960 issues, pull requests and discussions of the seven frameworks and codes 48 threads on 40 papers, plus 88 merged landing pull requests, to describe how proposals for methods were reviewed, merged, declined or left open.

4. A released tracker linking failures to implementations. The data and code are released as a tracker whose index links training failures (entropy collapse, rollout–training probability mismatch) to framework implementations and settings, alongside each framework's contribution rules and the outcome of earlier proposals.

Main Findings

• Inclusion is rare. Of 1,598 papers proposing a general RL training method, 37 reached at least one of the seven frameworks, a rate of 2.3 percent. Only 1 is an unconditional default, and 7 are included by three or more frameworks.

• Intake did not keep pace with supply. Across five quarterly cohorts, the number of general methods included within 180 days stayed between 5 and 7 while the cohorts grew from 72 to 386 papers, a 5.4-fold rise. The share of each cohort included within 180 days fell from 6.9 to 1.3 percent.

• Large-scale training is the biggest correlate of inclusion. Policies of 30B parameters or more were trained in 41 percent of included papers against 10 percent of matched non-included papers. The gap persists when model reports are excluded (12 of 29 against 13 of 139) and within papers with an industry author (10 of 25 against 11 of 90).

• Changes shaped like a configuration field are more often included. A coefficient, clip, mask or weight in the existing loss was the change in 29 percent of included papers against 9 percent of others, while a new training stage or model was the change in 26 against 53 percent. Frameworks expose the widely included methods as one or two fields, such as clip_ratio_high in verl and epsilon_high in TRL for clip-higher.

• The gaps are not explained by companies or code release. Included papers more often come from a company that maintains one of the frameworks (32 against 15 percent) but only 5 were included by that company's own framework; all 18 papers with an outside committer landed in a framework outside their own company. Training code was linked in 44 percent of included and 44 percent of non-included papers.

• Inclusion is fast when it happens. Among the 54 catalog methods with a source from 2025 or later, 49 have an exact first landing, with a median of 30 days from first arXiv version to a main branch (30.5 days over 42 papers when counted once per source paper). GSPO reached TRL 3 days after its first version and all seven frameworks within a week; in the same month, 43 other papers proposed a general method and only 2 reached any framework.

• Authorship changes with breadth. Of 38 recent methods found in exactly one framework, 13 first landed through a commit by a paper author; methods in three or more frameworks have 70 landings, 40 of them from named people outside the author list.

• Most methods were never publicly rejected — they never appeared. Of 1,561 general methods no framework included, maintainers declined or redirected only 12. 96 percent have no thread that cites them, and a search by method name lowers this only to 95 percent.

• Proposals that do appear often stall. Of 28 pull requests on those papers, 17 were still open after a median of 196 days, and no prior merger had replied in 13. A random sample of 250 unscreened issue threads contained no training-failure report, and the 7,075 threads opened since January 2025 contain 48 that report a training failure, 23 of them an unnamed collapse or divergence.

• Users name different failures than papers address. Sparse reward (218 general methods) and loss of diversity (156) appear in no screened thread, while the rollout–training probability mismatch is named by 12 threads and addressed by 8 catalog methods in the frameworks. Method requests are thin: 59 threads request a method, the most requested (DAPO's dynamic sampling) drew 8 requests across seven frameworks in 21 months.

• Large model reports state few methods. Of 13 reports of open-weight models with 100B parameters or more, 8 of the widely included methods are stated somewhere; 6 reports state at least one, 2 are the source of one, and 3 describe a variant under their own name. One of the seven frameworks is named in 4 reports.

• Component choice matters for inclusion. Clipping/trust-region, off-policy correction and loss aggregation make up 187 general methods, 10 of which reached a framework; rollout sampling and credit assignment account for 554 general methods, 4 inclusions and none in three or more frameworks.

• Inclusion rarely reverses. 4 of the 114 landings of recent methods were later removed: TRL dropped experimental trainers for GFPO, DPPO and PAPO, AReaL dropped the AEnt recipe, and NeMo-RL reverted its first GSPO commit a day later and re-added it four days after that with test fixes.

Methodology in Plain English

The author picked seven frameworks that are built for RL post-training of language models, run their own training loop with several selectable algorithms, are developed in public, and have at least 2,000 GitHub stars: verl (with its recipe repository), TRL, OpenRLHF, NeMo-RL, ROLL, slime and AReaL. For each, the author identified the few places where algorithms are registered or selected, parsed those places at every commit on the main branch, and checked the parse against each framework's algorithm documentation. Because frameworks name the same method differently, every option name is mapped to a single entry in a catalog of 65 methods, and the build fails if an unmapped option appears.

Parsing can miss implementations hidden under unmapped names or placed elsewhere, so a citation audit searched each framework's full main-branch history for arXiv identifiers of pool papers. Framework files cite 51 general methods, including all 28 that parsing had found; for 9 of the remaining 23, a citation sat next to an implementation and they were added to the catalog. Commit dates are used as the date a change became available, and dates from root commits, bulk releases or renames are treated as upper bounds.

For the supply side, the author avoided keyword search through the arXiv API (which misses LaTeX-typeset names), harvested complete metadata through OAI-PMH, and screened for papers mentioning language models and reinforcement learning. Each paper was labelled from title and abstract under fixed rules; three checks covered recognition of known sources, re-labelling stability (140 papers labelled twice, agreeing 129 times), and the "general method" rule against a manually reviewed sample of 200 papers, which agreed on 188 and implied about 1,590 general methods. A separate read of 70 randomly drawn model reports estimated that abstract-only labelling could shift the denominator by at most about 13 percent.

To see how non-included methods fared, the author downloaded all 26,960 issues, pull requests and discussions of the seven frameworks and matched them to pool papers by arXiv link, identifier or quoted title, then coded each matched thread from its own text. Finally, the RL sections of 13 technical reports of open-weight models with 100B parameters or more were read to record which widely included methods each stated it used.

Why This Matters

Impact on research: The paper reframes "adoption" as a measurable code event rather than a citation count, and shows that the bottleneck between a published method and the code practitioners run is not peer rejection but non-appearance: most methods are never proposed to a framework at all. It also documents the characteristics associated with inclusion, so authors can see which kind of contribution the ecosystem currently absorbs.

Real-world applications:

• Authors choosing how to present a method: the paper's implications section pairs each observed association with a concrete step — name the configuration field a change touches, publish the failure signal (entropy curve, rollout–training divergence), open a short public issue, and provide a configuration and seed maintainers can rerun.
• Framework maintainers designing contribution rules: the tracker collects each framework's stated rules, the concerns its reviewers raised, and the outcome of earlier proposals, giving maintainers a comparison point (TRL's demand-based gating, slime's verifiable-change rule, verl's recipe collection).
• Teams selecting a training stack: the tables of which methods appear in which framework, at what tier and how fast, help a team judge whether a method is one configuration change away or a separate project.
• Reviewers and meta-researchers: the failure-to-implementation index connects reported training failures to code and settings, and the issue analysis shows which failures users actually report in public.

Industry relevance: Included papers more often have industry authors (90 against 64 percent) and more often train policies of 30B parameters or more, and the largest open-weight model reports are where the same stabilisers appear. The records show that companies rarely push their own methods into their own frameworks, and that inclusion crosses company lines mainly through outside committers — relevant to anyone deciding whether to contribute upstream or maintain a fork.

Future Directions

• Widen the framework set. Four other frameworks meet the same criteria — open-instruct, SkyRL, prime-rl and RLinf — and at their latest commits implement 2 general methods none of the seven include, which would raise the inclusion rate from 2.3 to 2.4 percent. Whether that near-flat result holds under a larger set is open.

• Measure what the parser cannot see. The study covers framework options only; methods that live in user code outside the tracked places, and proposals handled in chat groups or internal repositories, leave no public trace. Counting those channels would test whether the 96 percent without a public record reflects invisibility or rejection.

• Test the associations causally. Scale, the form of the change, and industry authorship are reported as associations. Whether presenting a method as a single configuration field, or demonstrating it on larger policies, actually raises the chance of inclusion is not established.

• Track downstream effects of inclusion. The paper measures whether methods get into frameworks, not whether inclusion changes what practitioners train with, how results replicate, or how methods are cited; linking the tracker to usage and to re-evaluation studies is a natural next step.

Target Audience

Researchers who publish RL post-training methods and want their work to be usable; maintainers and reviewers of open-source training frameworks; engineers choosing a training stack and wondering why some methods are convenient and others are not; and meta-science or software-engineering researchers studying how research enters production code. Readers looking for new algorithms or benchmark results will not find them here — the contribution is measurement and a released dataset.

Authors’ abstract

Practitioners use reinforcement learning methods for language models through training frameworks, so a method that a framework includes is one configuration change away for its users. We measure how often this happens. We trace seven open-source frameworks for RL post-training through their commit histories and link them to 6,939 screened arXiv papers from January 2025 to June 2026. Of 1,598 papers that propose a general RL training method, 37 reached at least one framework. Across five quarterly cohorts of papers, the number included within 180 days stayed between 5 and 7 while the cohorts grew from 72 to 386 papers. Compared with papers from the same quarters, included papers more often trained policies of 30B parameters or more, at 41 against 10 percent, and more often changed a coefficient, clip, mask or weight in the existing loss, at 29 against 9 percent. Both gaps hold without papers from the companies that maintain the frameworks, while code release was equally common in both groups. The frameworks' issues and pull requests show that few methods are turned away in public. More than nine in ten of the general methods no framework included have no thread there that cites or names them, and many proposals that do appear stay open without a reply. Two further records, the frameworks' issues and the technical reports of 13 large open-weight models, show that users rarely report training failures in public and that the reports state 8 of the 13 widely included methods. We release the data as a tracker that links training failures to framework implementations and lists each framework's contribution rules and earlier proposals.