An implementation of the GRPO trainer in the OpenRLHF framework, replicating DeepSeek-R1's core alignment algorithm without requiring a separate reward model.

Stars

99

7-day growth

No data

Forks

0

Open issues

0

License

No data

Last updated

2026-07-15

AI repository intelligence
FR-AI / ANALYSIS

Why it is worth attention

It brings DeepSeek-R1's GRPO to OpenRLHF with measurable gains: 35% less GPU memory, 1.8x faster training, and 96.2% AlpacaEval win rate, making RLHF more accessible.

Who it is for

  • Researchers exploring novel RLHF algorithms
  • Practitioners seeking memory- and time-efficient alternatives to PPO
  • Developers aiming to reproduce DeepSeek-R1-style training

Use cases

  • Aligning LLMs with group-relative advantage estimation
  • Running RLHF on limited GPU resources
  • Quickly iterating on group size and KL penalty hyperparameters
  • Evaluating GRPO vs PPO on downstream tasks

Strengths

  • Eliminates the reward model, simplifying the pipeline and reducing computational overhead
  • Reduces GPU memory usage by 35% compared to PPO
  • Speeds up overall training by 1.8x
  • Achieves 96.2% win rate on AlpacaEval 2.0, close to PPO's 100%

Considerations

  • Slightly lower AlpacaEval win rate than PPO (96.2% vs 100%)
  • Implementation is coupled with the OpenRLHF framework, limiting standalone reuse
  • Performance may vary across different models and datasets not tested in the README

README quick start

快速开始

Description

GRPO alignment algorithm trainer based on OpenRLHF

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo
Featured
lopopolo GitHub avatar

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents
2,390
slvDev
Featured
slvDev GitHub avatar

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models
1,960
littledivy
Featured
littledivy GitHub avatar

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI & Machine Learning
1,482