Hardcore Reviews
Hardcore Reviews

Picking an Open-Source Agentic RL Framework: Five Contenders

Five open-source Agentic RL post-training frameworks compared on positioning, ecosystem and engineering shape rather than benchmarks (self-reported harness figures cannot be compared): verl (23,650-star upstream HybridFlow with a 465-star Xiaomi fork carrying five production environment suites), TRL (19,398 stars, the official Hugging Face on-ramp), OpenRLHF (10,045 stars, Ray-based PPO/DAPO/REINFORCE++), AReaL (5,796 stars, Ant-lineage asynchronous agent RL), and NeMo-RL (2,033 stars, the NVIDIA enterprise ticket). Four scenario verdicts: pick TRL for LoRA-scale trials; put verl and OpenRLHF into your PoC for large-scale verifiable-reward RL; look at AReaL for long-horizon asynchronous agent training; pick NeMo-RL inside NVIDIA all-in stacks. All Apache-2.0 (LICENSE files are the final authority); star counts are a 2026-09-27 snapshot and self-reported performance figures are not trusted.

Published September 27, 202610 min read
<!-- agentic-rl-frameworks-comparison-review | review | Picking an Open-Source Agentic RL Framework: Five Contenders -->

On September 22, while shipping MiMo-V2.6, Xiaomi quietly did something most coverage ignored: it forked verl. Within days that fork (XiaomiMiMo/verl) collected 465 stars, while its upstream, verl-project/verl, sat at 23,650 stars at the top of the open-source RL post-training chart. This was not a random fork. Put TRL, OpenRLHF, AReaL, and NeMo-RL on the same table and a clear picture emerges: the infrastructure for Agentic RL — training agents on verifiable task rewards — has grown into five visibly diverging engineering tracks. By Xiaomi's official figures, MiMo-V2.6 went through roughly six days of live RL over about 750,000 trajectories, and the framework-selection question behind numbers like that is now unavoidable for any team training its own agents.

The confusion is real: all five are Apache-2.0, all are Python, every README claims high performance and scalability — yet star counts span 2,033 to 23,650. If you look only at stars, you will conclude "just pick verl." But a large share of that gap is simply a head-start differential (TRL was created in March 2020, NeMo-RL in March 2025 — a five-year spread) plus differences in community positioning, not a clean ranking of capability.

First, the scope statement. All star counts, fork counts, licenses, creation dates, and last-push dates in this article come from a GitHub API snapshot taken on 2026-09-27. Performance numbers self-reported in each project's README are not used here — the evaluation harnesses differ, so benchmarks are not comparable across projects. What this article compares is positioning, ecosystem, engineering shape, and fit. A note on how this piece divides labor with existing coverage on this site: our open-source flagship model comparison evaluates the models themselves, while this piece evaluates training frameworks; our LLM fine-tuning SOP covers the general SFT/LoRA route and complements this framework comparison. For Xiaomi-side launch details see our MiMo-V2.6 release coverage; for a close reading of the fork itself see our Xiaomi verl fork resource. Neither is repeated here.

1. The Five Contenders at a Glance

The raw specs first (snapshot 2026-09-27; licenses as reported by the GitHub API license field):

FrameworkRepositoryStarsForksCreatedLicenseOfficial positioning
verlverl-project/verl23,650n/anot sampledApache-2.0HybridFlow: a flexible and efficient RL post-training framework
TRLhuggingface/trl19,3983,0212020-03-27Apache-2.0Train transformer language models with reinforcement learning
OpenRLHFOpenRLHF/OpenRLHF10,0451,0262023-07-30Apache-2.0An easy-to-use, scalable, high-performance Agentic RL framework based on Ray (PPO/DAPO/REINFORCE++)
AReaLareal-project/AReaL5,7966122025-02-24Apache-2.0The RL bridge for LLM-based agent applications, made simple and flexible
NeMo-RLNVIDIA-NeMo/RL2,0335742025-03-16Apache-2.0NVIDIA's scalable toolkit for efficient model reinforcement learning

Three first-pass observations:

  1. The top two are close in stars but five years apart in age. TRL is the oldest of the five (created 2020-03-27), more than three years ahead of OpenRLHF (2023-07-30) and nearly five years ahead of AReaL and NeMo-RL. We did not sample the upstream verl creation date in this snapshot, but 23,650 stars puts its community mass in the same league as TRL's. An ecosystem veteran that started in 2020 and a fast riser that caught up from behind have very different growth logics — that distinction matters in section three.
  2. Zero license differentiation. All five are Apache-2.0 (subject to each repository's LICENSE file as the final authority), so licensing drops out of the decision entirely. The weight shifts to engineering shape and ecosystem fit. The only footnote: mimoagent, a companion repo under Xiaomi's fork umbrella, uses MIT — equally permissive.
  3. All five are actively maintained. Most recent pushes within the snapshot window: TRL, AReaL, and NeMo-RL on 2026-09-27, Xiaomi's fork on 2026-09-26, and OpenRLHF on 2026-09-17. Nobody is dormant, so the "is it alive?" filter eliminates no one.

2. Positioning and Architecture: Who Synchronizes, Who Goes Async, and Who Backs Whom

The real differences among these five are not "can it do RL" but three axes: synchronous versus asynchronous design, single-machine versus multi-machine priorities, and which ecosystem each project is anchored to. Descriptions below stay at official-README levels of caution — features not documented in our sources are not claimed here.

verl: the largest community, now with production-grade validation from Xiaomi

verl's official positioning is HybridFlow — "a flexible and efficient RL post-training framework." It is a ByteDance-lineage open-source project now maintained under the verl-project organization. Its hold on the top star spot matters because when "training agents on verifiable rewards like executable tests and rule checks" turned from a research demo into an industrial requirement, the community's default answer was verl.

Xiaomi's fork adds a layer of production validation. On 2026-09-21 Xiaomi created XiaomiMiMo/verl (465 stars and 49 forks in our snapshot, Apache-2.0, last pushed 2026-09-26). Its README describes it as the Agentic RL training code for MiMo, built on verl 0.9.0.dev and adding reproduction code for five RL environment suites:

DomainTask familyVerifierLaunch script
CodeSoftware engineeringExecutable testsscripts/code/train.sh
CyberVulnerability reproductionRule checksscripts/arvo/arvo.sh
GeneralKnowledge workRubric reviewscripts/general/general.sh
VisualWeb developmentVisual scoringscripts/design/webdev.sh
MusicSymbolic music compositionRule checksscripts/design/music.sh

Two companion repos were created the same day: uni-agent (14 stars, Apache-2.0, a framework for training long-horizon agents) and mimoagent (30 stars, MIT, a 100-line agent that solves GitHub issues). With the MiMo-V2.6-RL-oss training dataset openly downloadable on Hugging Face, the stack forms a complete training bundle. Read alongside Xiaomi's stated figures — roughly six days of live RL, about 750,000 trajectories, 1M training context, single-step updates of 2.7-3.7B tokens — the fork looks like genuine production infrastructure, not a launch-day publicity move. One caveat: GitHub compare shows the fork at ahead_by 0 and behind_by 16, so divergence details cannot be verified; this article sticks to README-verifiable facts. Training-recipe details belong to Xiaomi's technical report; for integration-level guidance see our MiMo-V2.6 integration SOP.

TRL: the Hugging Face official default with the broadest ecosystem

19,398 stars, 3,021 forks, created March 2020 — the most senior of the five. The official positioning is one line: "train transformer language models with reinforcement learning." That is the broadest framing here; it does not presume you are training agents. RLHF and preference-alignment work — the "make the model behave better" category — all falls within its scope. With Hugging Face's official stewardship and native fit with transformers, TRL is almost always the first name individuals or small teams meet when trying LoRA-scale RLHF. The trade-off: once your goal escalates from alignment to large-scale verifiable-reward Agentic RL, whether its abstractions still fit your cluster size is something to verify with your own workload — your proof-of-concept will answer that.

OpenRLHF: the Ray-based multi-machine workhorse

10,045 stars, created 2023-07-30. The official description compresses the key facts into one sentence: "an easy-to-use, scalable and high-performance Agentic RL framework based on Ray (PPO & DAPO & REINFORCE++)." Unpacked: building on Ray delegates distributed orchestration to general-purpose infrastructure rather than a homegrown scheduler — a direct plus for teams already running Ray in production. Listing PPO, DAPO, and REINFORCE++ in the repository description advertises the supported mainstream RL algorithms. For teams running large-scale RL post-training across machines without wanting to marry a single vendor's stack, OpenRLHF is the middle path: more training-focused than TRL, lighter on conceptual overhead than verl's HybridFlow abstraction.

AReaL: the Ant-backed flag bearer for asynchronous agent RL

5,796 stars, created 2025-02-24, official motto "the RL Bridge for LLM-based Agent Applications, made simple and flexible," with an Ant Group lineage. Our source material tags it as emphasizing asynchronous design, and that tag deserves unpacking. As agent trajectories grow longer — Xiaomi's stated training context is 1M tokens — and environment interactions grow more expensive, a synchronous generate-then-train loop leaves compute idle waiting for the environment. Asynchronous designs decouple generation from training so they run in parallel, which is widely treated as the dividing line for agent RL frameworks. Xiaomi's fully asynchronous architecture in MiMo-V2.6 training corroborates the same trend. If environment interaction dominates your training workload — long-horizon tasks, real tool calls, multi-turn exchanges — AReaL's positioning lands exactly on that point.

NeMo-RL: NVIDIA's toolkit, the enterprise-stack ticket

2,033 stars, created 2025-03-16 — the youngest and smallest by stars of the five, but backed by the NVIDIA-NeMo organization and described officially as a scalable toolkit for efficient model reinforcement learning. Its value logic differs: teams pick it not for community rankings but because their infrastructure already lives inside the NVIDIA family, where the fit among training, inference, and operations tooling is hard to replicate. A low star count in this context means a narrow-and-deep user base, not weakness. If your stack is already committed to NeMo, shortlist it; if not, it naturally ranks lower.

3. Scenario-Based Verdicts

Compressing the analysis into four directly actionable answers:

  1. Individuals or small teams trying LoRA-scale RLHF: pick TRL. Officially maintained by Hugging Face, the broadest ecosystem, seamless with transformers workflows, and the lowest trial-and-error cost of the five. Get "where do rewards come from and how does data flow" working in TRL first, then decide whether you need something heavier.
  2. Large-scale verifiable-reward Agentic RL: put verl and OpenRLHF into your PoC. Both grew up for exactly this scenario. verl brings the HybridFlow abstraction, the largest community, and now Xiaomi's production validation plus five environment suites worth copying; OpenRLHF brings the Ray foundation and an algorithm lineup of PPO, DAPO, and REINFORCE++ written on its front door. Run both on your own tasks and your own cluster, then decide on measured throughput.
  3. Long-horizon asynchronous agent training: AReaL. Asynchrony is its reason for being. The longer the trajectories and the costlier the environment interactions, the larger the payoff. Xiaomi's publicly stated fully asynchronous architecture lends further weight to the direction.
  4. NVIDIA all-in enterprise stacks: NeMo-RL. If your infrastructure is already inside the NeMo ecosystem, keeping the training framework in the same ecosystem minimizes operational and version-compatibility costs.

Two cross-cutting reminders:

  • TRL and verl/OpenRLHF are not rivals so much as stages. A common team pattern is two-phase: use TRL for day-to-day alignment iteration and hand large-scale verifiable-reward Agentic RL to verl or OpenRLHF, switching by project stage.
  • The value of Xiaomi's five environment suites is not benchmark numbers but copyable homework. How verifiers plug in (executable tests, rule checks, rubric review, visual scoring), how task families are organized, what launch scripts look like — it is all there to study. Even if you ultimately do not use verl, reading that directory structure will save you a week of design from scratch.

4. Licenses and Risks

All five are Apache-2.0 (per the GitHub API license field), which means commercial use, modification, and redistribution face no extra licensing hurdles — unusual tidiness for this space. Three red lines still need drawing:

  1. Treat each repository's LICENSE file as the final authority. The GitHub API license field is auto-detected and may diverge from the actual LICENSE file; before commercial deployment, read the original text, especially patent and trademark clauses. The same applies to the MIT license on Xiaomi's companion repo mimoagent.
  2. Stars are a snapshot, not a trend. The 2026-09-27 snapshot indicates accumulated popularity, not maintenance quality or responsiveness. With creation dates up to five years apart, cross-time star comparisons are inherently skewed; judging momentum requires at least two time points.
  3. All performance claims are self-reported. Speedup ratios and throughput figures in each README were measured under the project's own harness and task selection. They are not comparable and should not drive your selection. The only reliable method is running your own benchmark on your own cluster.

Conclusion

The biggest takeaway from this comparison is not who won but that five distinct tracks have matured: TRL owns alignment on-ramps, verl and OpenRLHF own large-scale verifiable-reward training, AReaL owns asynchronous long-horizon agents, and NeMo-RL closes out the enterprise stack. Xiaomi forking verl adds a verifiable data point to the question of whether open-source frameworks can carry production-grade Agentic RL — the 750,000-trajectory claim is on the record, and the fork's five environment suites are open for inspection. Map the four scenario verdicts onto your situation, then test with your own workloads. Stars will move; the division of labor between these tracks will not.

FAQ

Q1: Can these five frameworks be benchmarked against each other directly?

A1: No. Each project uses its own evaluation harness and task setup, and all performance numbers are self-reported in their respective READMEs, so they are not comparable across projects. This article deliberately avoids benchmark comparisons and evaluates positioning, ecosystem, and engineering shape instead. For selection decisions, run your own tasks on your target cluster.

Q2: Are star counts a valid basis for choosing a framework?

A2: Only as a reference. The star counts here are a GitHub API snapshot from 2026-09-27, and the five projects' creation dates differ by up to five years, which inherently favors earlier movers. A steadier approach is to look at recent push activity, community responsiveness, and fit with your specific scenario.

Q3: Does Xiaomi forking verl mean verl is the only right answer?

A3: No. The fork is one company's choice for training its own model, and the README itself describes it as the Agentic RL training code for MiMo. It demonstrates that verl can carry production-grade training, but it does not diminish the value of OpenRLHF, AReaL, or the others in their respective scenarios.

Q4: I just want ordinary RLHF alignment — do I need these heavy frameworks?

A4: Usually not. LoRA-scale preference alignment starts and ends with TRL for most teams. Only when your workload involves large-scale trajectory collection, executable verifiers, or multi-machine clusters do verl, OpenRLHF, or AReaL become relevant. For the general fine-tuning route, see our LLM fine-tuning SOP.

Q5: What legal risks should I watch for in commercial use?

A5: All five are Apache-2.0 under the GitHub API's classification, keeping the commercial barrier low, but each repository's LICENSE file remains the final authority. Before commercial deployment, verify the license text directly — particularly patent grant and trademark clauses — and consult legal counsel when needed.

This article is AI-assisted and human-edited. Last updated: 2026-09-27

FAQ

Can these five frameworks be benchmarked against each other directly?
No. Each project uses its own evaluation harness and task setup, and all performance numbers are self-reported in their respective READMEs, so they are not comparable across projects. This article deliberately avoids benchmark comparisons and evaluates positioning, ecosystem, and engineering shape instead. For selection decisions, run your own tasks on your target cluster.
Are star counts a valid basis for choosing a framework?
Only as a reference. The star counts here are a GitHub API snapshot from 2026-09-27, and the five projects' creation dates differ by up to five years, which inherently favors earlier movers. A steadier approach is to look at recent push activity, community responsiveness, and fit with your specific scenario.
Does Xiaomi forking verl mean verl is the only right answer?
No. The fork is one company's choice for training its own model, and the README itself describes it as the Agentic RL training code for MiMo. It demonstrates that verl can carry production-grade training, but it does not diminish the value of OpenRLHF, AReaL, or the others in their respective scenarios.
I just want ordinary RLHF alignment — do I need these heavy frameworks?
Usually not. LoRA-scale preference alignment starts and ends with TRL for most teams. Only when your workload involves large-scale trajectory collection, executable verifiers, or multi-machine clusters do verl, OpenRLHF, or AReaL become relevant. For the general fine-tuning route, see our LLM fine-tuning SOP.
What legal risks should I watch for in commercial use?
All five are Apache-2.0 under the GitHub API's classification, keeping the commercial barrier low, but each repository's LICENSE file remains the final authority. Before commercial deployment, verify the license text directly — particularly patent grant and trademark clauses — and consult legal counsel when needed.

Related

Hardcore Reviews

TrnRT, Kindi, Baseten: One License Forbids Managed Service

Three inference engine repositories, TrnRT, Kindi and Baseten, on one comparison table, with license terms and hardware bets as the axis rather than a buying recommendation or benchmark ranking. Apache-2.0, Elastic License 2.0 and AGPL-3.0 coexist; one LICENSE says verbatim that the software may not be provided to third parties as a hosted or managed service, and GitHub reports NOASSERTION. Hardware claims from NKI megakernels on Trainium2 to GB300, B300, Quadro K420 and RTX 3080 are labelled as README self-reports. Stars are cumulative totals, with no growth score or single-day delta.

Nov 16, 20269 min read
Hardcore Reviews

qwen-code Preview, OpenBot, free-claude-code: Three Compared

Sets the qwen-code v1.0.0-preview beside OpenBot v1.0.2 and the rolling v6.x of free-claude-code in one table, comparing version states and access boundaries rather than picking a winner. Five rows cover models and providers, where AG-UI and MCP attach, permissions and approvals, stability during Preview, and license points. The free-claude-code column states AGPL-3.0-only, the NOASSERTION value from GitHub, the README disclaimer that it is not affiliated with Anthropic, and that provider terms govern integrations. Star counts are cumulative totals.

Nov 15, 20269 min read