Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization

• : frozen model at the end of SFT
• : RL model at the start of each iteration
• : currently updating RL model

Step 1: Group Sampling

First, we sample a batch of prompts from the RL training prompt set. With each prompt, we generate completions using the current RL model . Each completion is a sequence with tokens.

Step 2: Reward Calculation

For each sequence, we compute the reward with a reward model or a rule-based heuristic, obtaining rewards .

From these rewards, we compute their mean and standard deviation, obtaining and . Finally, we compute the advantage, a very popular concept in RL, which is just z-scored rewards: .

Step 3: Credit Assignment

Outcome supervision: The naive way would be to assume each token’s advantage to be the same as the sequence’s, i.e. for . However, this doesn’t let us know which token would lead to a good outcome.

Process supervision:

1. Group the tokens in a sequence into reasoning steps: , where each is a contiguous span of tokens.

2. For each reasoning step , compute its cumulative reward up to and including that step: .

3. Compute mean and standard deviation across all steps and sequences:

4. Normalize each step reward:

5. Assign each token an advantage equal to the sum of normalized rewards from its step onward (letting denote the step index containing token ): . The intuition is the same as a return in RL: a token gets credit for all the reward that follows from its step, not just the reward of its immediate step.

6. Calculate the token-level importance sampling ratio:

It measures how much the currently updating policy has shifted from the old policy at each individual token, conditioned on the prompt and all preceding tokens .

This feeds directly into the clipped PPO objective, which is the next natural step:

where:

• The min-clip prevents the policy from updating too aggressively in a single step
• The KL penalty against (the SFT model) keeps the RL model from drifting too far from the base A few natural additions that would make the exposition complete:

The is typically computed as a per-token estimator rather than exactly, using:

This is non-negative and an unbiased estimator of the true KL.

The outer training loop

After each iteration, set , sample a new batch, and repeat. remains frozen throughout all iterations.