Policy gradient and PPO: learning the policy directly

A stochastic policy as a network, REINFORCE and the policy gradient theorem, the baseline and the advantage, actor-critic, PPO with clipping written from scratch on the cart-pole, what happens without clipping, and how PPO fine-tuned language models.

12 minutes Matěj Žalmánek 2026-10-16

Series AI basics Part 49 of 52

Full series syllabus 52 of 83 topics published

What you will take away

How a policy is learned directly from outcomes, without a table of values.
Why subtracting a baseline helps and what the advantage is.
Why clipping matters so much in PPO and how it relates to RLHF.

Useful to know first: Q-learning and DQN: learning the value of actions, from a table to a neural network, Markov decision processes: states, actions, rewards and the Bellman equations

The DQN from the previous part learns how good each action is, and the policy follows from that only indirectly: pick the action with the highest value. The second big family of methods goes straight for it: the network gets a state and returns action probabilities, and we shift those probabilities so that good actions become more frequent. This family gave us PPO — the algorithm that taught robots to walk, beat professionals at Dota 2 and was behind the fine-tuning of ChatGPT. In this part we write it from scratch and compare it with its simpler predecessors.

It builds on the parts Q-learning and DQN, Markov decision processes and Softmax.

1. Learning the policy directly

Definition — parametrized policy

Let (S, A, P, R, γ) be an MDP (notation from the part Markov decision processes) with a finite set of actions A and an initial distribution p(s₀) on S, and let θ ∈ ℝ^m be parameters. A parametrized (stochastic) policy is a family of policies (π_θ)_{θ ∈ ℝ^m} where for every θ and every s ∈ S, π_θ(· | s) is a probability distribution on A and π_θ(a | s) is a differentiable function of θ. For a finite A one takes a network f_θ : S → ℝ^|A| and a softmax of its outputs:

π_θ(a | s) = e^f_θ(s)_a/∑_{b ∈ A} e^f_θ(s)_b;

for continuous actions the network returns, for example, the mean and variance of a normal distribution. The objective is the expected return as a function of the parameters

J : ℝ^m → ℝ, J(θ) = E_{π_θ}[G₀],

and we look for θ with the largest possible J(θ) by gradient ascent θ ← θ + α ∇J(θ) with a learning rate α > 0 (∇ denotes the gradient with respect to θ).

In plain words

DQN keeps a table of “how many points for which move” and plays the move with the most points. A policy instead says directly “in this situation push right with 70%, left with 30%”. The advantages: it works with continuous actions too (how hard, at what angle), randomness is built in, so the agent explores on its own, and a small change of the parameters means a small change of behaviour.

Example — the return of one attempt on the cart-pole

The reward is +1 for every step and the discount is γ = 0.99. With T − t steps left until the end of the episode, the return is a geometric series:

G_t = 1 + γ + γ² + … + γ^T−t−1 = 1 − γ^T−t/1 − γ.

At the start of an episode that lasts the full 500 steps: G₀ = 1 − 0.99⁵⁰⁰/0.01 = 99.34. Twenty steps before a fall: G = 1 − 0.99²⁰/0.01 = 18.21. The step just before a fall: G = 1. So the return is always positive and never exceeds 1/1 − γ = 100 — this will come in handy in the section on the baseline.

2. REINFORCE: reinforce what worked

Definition — the policy gradient theorem and REINFORCE

The policy gradient theorem. Let (S, A, P, R, γ) be an MDP with finite S and A and γ ∈ [0, 1], in which every episode s₀, a₀, r₁, s₁, …, s_T ends after at most T_max steps (T ≤ T_max), and let π_θ be a parametrized policy with π_θ(a | s) > 0 for all s, a, θ. Then

∇J(θ) = E_{π_θ}[ ∑_t=0^T−1 γ^t ∇ log π_θ(a_t | s_t) · G_t ], G_t = ∑_k=t^T−1 γ^k−t r_k+1.

For γ = 1 the factor γ^t disappears and we get the form in the breakdown below. Practical implementations (ours included) drop it even for γ < 1; the result is then no longer an unbiased estimate of ∇J(θ) (in general it is not even the gradient of any function; Chris Nota and Philip Thomas, 2020), because later steps of the episode get more weight than the theorem gives them — in practice it works well all the same.

REINFORCE (Ronald Williams, 1992) is the algorithm:

choose an initial θ ∈ ℝ^m (random network weights) and a learning rate α > 0;
play episodes with the policy π_θ (here at least 2,000 steps in total);
in every episode compute, from the end, G_T = 0 and G_t = r_t+1 + γ G_t+1 for t = T − 1, …, 0;
compute the gradient estimate ĝ = the average of ∇ log π_θ(a_t | s_t) · G_t over all steps played;
θ ← θ + α ĝ, throw the data away and continue with step 2 until the budget of steps in the environment is used up.

For a softmax π = softmax(z) with network outputs z ∈ ℝ^|A| the gradient with respect to z is ∇_z log π_a = 1_a − π, where 1_a is the vector with a one in the place of action a and zeros elsewhere.

In plain words

Play, look at how it turned out, and make every move you made a little more likely — the more so, the better things went after it. Moves from successful games get reinforced more than moves from unsuccessful ones. Nobody has to say which move was right; the outcome is enough.

∇J(θ) = E_{π_θ}[ ∑_t ∇ log π_θ(a_t | s_t) · G_t ]

∇J(θ): the direction in which to turn the parameters so that the average return grows fastest; it cannot be computed directly, because the return depends on θ only through random play and the unknown physics of the environment
E_{π_θ}[ ∑_t … ]: an average over episodes played by the current policy — which is why it can be replaced by an average over games actually played; and why data from a different policy (older, someone else’s) is not enough without a correction
∇ log π_θ(a_t | s_t): the direction in parameter space that makes the played action more likely; ∇ log π = ∇π/π divides by the probability, so a common action, which shows up in the data often, does not get extra reinforcement just for being common
G_t: the weight: how well things went after the action; only rewards from step t on, because an action cannot influence rewards that have already come

The gradient is a weighted average of the directions “make this action more likely”, weighted by the return that followed it.

Theorem — where the formula comes from (the log-derivative trick)

Let the assumptions of the policy gradient theorem hold. The policy π_θ produces an episode (trajectory) τ = (s₀, a₀, s₁, a₁, …, s_T) with probability

P_θ(τ) = p(s₀) ∏_t=0^T−1 π_θ(a_t | s_t) P(s_t+1 | s_t, a_t)

and let G(τ) = G₀ be its return; there are finitely many trajectories, so J(θ) = ∑_τ P_θ(τ) G(τ). Then

∇J(θ) = ∇ ∑_τ P_θ(τ) G(τ) = ∑_τ P_θ(τ) · ∇ log P_θ(τ) · G(τ) = E_{τ ∼ π_θ}[ ∇ log P_θ(τ) · G(τ) ],

because ∇P = P · ∇ log P. Proof (sketch): the logarithm turns the product into a sum, log P_θ(τ) = log p(s₀) + ∑_t log π_θ(a_t | s_t) + ∑_t log P(s_t+1 | s_t, a_t). Only the middle sum depends on θ, so we do not need to know the physics of the environment P: ∇ log P_θ(τ) = ∑_t ∇ log π_θ(a_t | s_t). Finally, for each a_t we drop the rewards before step t: they do not depend on a_t, so their contribution has zero mean (see the theorem in the next section). Of the sum ∑_k γ^k r_k+1, what remains at step t is ∑_{k ≥ t} γ^k r_k+1 = γ^t G_t, which is the statement of the policy gradient theorem. ∎

Example — one REINFORCE step with a softmax

Two actions, network outputs z = (0, 0), so π = softmax(z) = (0.50, 0.50). The agent played action 1 (“right”) and the return from that step was G = 10. For a softmax, ∇_z log π(a) = 1_a − π, so

∇_z log π(1) · G = ((0, 1) − (0.50, 0.50)) · 10 = (−5, 5)

(colours as in the breakdown of the formula above; the result is an estimate of ∇J from a single step).

An ascent step with α = 0.1: z ← (0, 0) + 0.1 · (−5, 5) = (−0.5, 0.5). The new policy is π = (e^−0.5, e^0.5)/e^−0.5 + e^0.5 = (0.269, 0.731). The action that worked is now more likely. (In a real network the weights θ change and the gradient reaches z through backpropagation; the direction is the same.)

θ = random weights of the policy network
repeat:
    play whole episodes with the policy π_θ until there are at least 2,000 steps
    for each episode, from the end:         # G = 0 past the end of the episode
        G_t = r_(t+1) + γ · G_(t+1)
    g = mean over all steps of  ∇ log π_θ(a_t | s_t) · G_t
    θ = θ + α · g                           # ascent; here an Adam step
    throw the data away — next time it comes from a different policy

3. Variance and the baseline

Definition — baseline and advantage

Let (S, A, P, R, γ) be an MDP and π_θ a parametrized policy.

A baseline is any function b : S → ℝ that depends only on the state, not on the action. The gradient estimate with a baseline is ĝ = ∑_t ∇ log π_θ(a_t | s_t) · (G_t − b(s_t)); by the theorem below it has the same mean as the estimate without b, but its variance can shrink considerably.
The advantage of a policy π is the function A^π : S × A → ℝ, A^π(s, a) = Q^π(s, a) − V^π(s), where Q^π and V^π are the value functions from the part on MDPs (the letter A with arguments denotes the advantage here, not the set of actions). The best common choice of baseline is b = V^π_θ, in practice estimated by a second network; then G_t − b(s_t) is an estimate of the advantage.
Actor-critic is a method that simultaneously learns a parametrized policy π_θ (the “actor”) and an estimate of the value function V ≈ V^π_θ (the “critic”) and uses the critic’s estimate in the policy gradient estimate.

In plain words

On the cart-pole the agent gets +1 for every step, so every return is positive — REINFORCE therefore reinforces every move, only some more than others. The signal “this was better than usual” gets lost in the noise. Subtracting the expected level turns it into “this move went 20 points better than expected in this situation” — and worse moves get a negative sign.

∇J(θ) = E[ ∑_t ∇ log π_θ(a_t | s_t) · (G_t − b(s_t)) ], A(s, a) = Q(s, a) − V(s)

∇ log π_θ(a_t | s_t): the direction “play this action more often”, just as in REINFORCE
G_t, Q(s, a): what actually came after the action (G_t), or how much it yields on average (Q is the mean of G_t); on its own almost always positive, so it does not tell a good action from an average one
b(s_t), V(s): the bar: how much is expected in this situation regardless of the action; after subtracting it, worse-than-average actions get a negative sign. It may depend only on the state, not on the action — otherwise the gradient would be biased (theorem below)
A(s, a): the advantage: how much better action a is than the policy’s average action in s; exactly the information we want to reinforce

Subtracting the bar does not change where the gradient points on average, it only reduces how much the estimate jitters from batch to batch.

Theorem — a baseline does not bias the gradient

Let A be a finite set of actions, π_θ a parametrized policy with π_θ(a | s) > 0, s ∈ S a state and b : S → ℝ a function. Then

E_{a ∼ π_θ(· | s)}[∇ log π_θ(a | s)] = ∑_{a ∈ A} π_θ(a | s) ∇ log π_θ(a | s) = 0;
E_{a ∼ π_θ(· | s)}[∇ log π_θ(a | s) · b(s)] = 0.

So subtracting b(s_t) from G_t does not change the mean of the gradient estimate (it suffices to condition on the state s_t).

Proof: ∑_a π_θ(a | s) ∇ log π_θ(a | s) = ∑_a ∇π_θ(a | s) = ∇ ∑_a π_θ(a | s) = ∇1 = 0. For a fixed s the term b(s) is a constant, so it can be pulled out of the sum. ∎

Example — same mean, different variance

One state, two actions with π = (0.5, 0.5). Action 1 yields the return 12, action 0 the return 10 — both positive, as on the cart-pole. The component of the gradient estimate with respect to the output z₁ is, from a single sample, (1_a=1 − 0.5) · (G − b) (colours as in the formula above).

Without a baseline (b = 0): if action 1 comes up, the estimate is 0.5 · 12 = 6; if action 0 comes up, it is −0.5 · 10 = −5. The mean is 6 − 5/2 = 0.5, the variance (6 − 0.5)² + (−5 − 0.5)²/2 = 30.25.

With the baseline b = 11 (the average return): 0.5 · (12 − 11) = 0.5 and −0.5 · (10 − 11) = 0.5. The mean is still 0.5, the variance 0.

The true gradient is π₁(1 − π₁) · (12 − 10) = 0.25 · 2 = 0.5. Both estimates point the right way on average, but without the bar a single sample misses by ±5.5, with the bar not at all. In reality the variance does not drop to zero (the returns are random and there are many states), but the principle is the same.

We measured it at the start of learning: 40 independent batches of 1,000 steps of the same random policy, a REINFORCE gradient from each. Just subtracting the average return (a constant baseline) reduces the sum of the variances of the gradient components from 0.177 to 0.131, that is by 26%. A learned value V(s), which tells good situations from bad ones, helps more.

Definition — the sum of the variances of the gradient components

Let K ≥ 1 (here K = 40) and let ĝ⁽¹⁾, …, ĝ^(K) ∈ ℝ^m be gradient estimates from K independent batches (vectors over all m network weights). For j = 1, …, m let ḡ_j = 1/K ∑_k=1^K ĝ_j^(k) denote the mean of the j-th component. The sum of the variances of the components is the number

∑_j=1^m 1/K ∑_k=1^K (ĝ_j^(k) − ḡ_j)².

It is the trace of the (sample) covariance matrix of the estimate — a single number for how much the estimate jitters. The drop: 1 − 0.131/0.177 = 0.26, that is 26%.

PPO below needs a learned V as well. It does not estimate the advantage by the difference G_t − V(s_t), though, but by a compromise between it and the one-step estimate from the part on Q-learning:

Â_t = ∑_l≥0 (γλ)^l δ_t+l, δ_t = r_t+1 + γ V(s_t+1) − V(s_t)

Â_t: the advantage estimate for step t (GAE, John Schulman et al., 2015) — what ∇ log π is multiplied by
δ_t: the TD error: how much better one step went than V expected; a one-step advantage estimate with low variance, but biased if V is wrong
r_t+1: the reward that actually came — the only piece of reality in δ
V(s): the learned bar; in s_t+1 it replaces the rest of the return with an estimate, in s_t it is subtracted as the baseline (after a fall V(s_t+1) = 0)
(γλ)^l: how quickly to forget more distant steps. λ = 0 gives just δ_t; λ = 1 gives, after summing (the V terms cancel out), exactly G_t − V(s_t) — unbiased, but with high variance. We use λ = 0.95

In code it is computed from the end in a single line: Â_t = δ_t + γλ Â_t+1.

4. PPO: small, safe steps

REINFORCE has one more weakness: it uses each batch of experience for only one gradient step and then throws it away. If we took more steps on the same data, the policy could drift away from the one that collected the data, and the gradient estimate would stop being valid — learning collapses. PPO guards against this.

Definition — Proximal Policy Optimization (PPO)

Let π_θ be a parametrized policy, V : S → ℝ a learned estimate of the value function (the critic), ε > 0 (typically ε = 0.2), γ ∈ [0, 1] and λ ∈ [0, 1]. For x ∈ ℝ let clip(x, 1 − ε, 1 + ε) = min(max(x, 1 − ε), 1 + ε) denote the number x clamped to the interval [1 − ε, 1 + ε]. PPO (Proximal Policy Optimization; John Schulman et al., OpenAI, 2017) repeats these steps:

Set θ_old = θ, π_old = π_{θ_old} and collect a batch of steps (s_t, a_t, r_t+1, s_t+1) with this policy.
For every step compute the TD error δ_t = r_t+1 + γ V(s_t+1) − V(s_t) (after a fall V(s_t+1) = 0) and the advantage estimate by GAE (generalized advantage estimation) Â_t = ∑_l≥0 (γλ)^l δ_t+l, where the sum ends with the end of the episode.
For several epochs (here 10), maximize with gradient steps with respect to θ the clipped objective L(θ) = E_t[ min( r_t(θ) Â_t, clip(r_t(θ), 1 − ε, 1 + ε) Â_t ) ], r_t(θ) = π_θ(a_t | s_t)/π_old(a_t | s_t), where E_t is the average over the steps of the batch (minibatch) and r_t(θ) is the probability ratio (not to be confused with the reward r_t+1); at the same time train the critic V by reducing the mean of (V(s_t) − y_t)², where y_t = Â_t + V_old(s_t) is the target computed in step 2 from the critic’s values before learning.
Throw the batch away and continue with step 1 until the budget of steps in the environment is used up.

Once the probability of an action changes by more than a factor of 1 ± ε in the direction that improves the objective, the gradient from that sample vanishes.

In plain words

After the match the coach says “do this more often”. The player repeats it in training — but only until they do it a fifth more often than before. Then they stop learning until they play the next match and find out whether it still works. Steps big enough to improve quickly, small enough not to go off the rails.

The PPO clipped objective as a function of the ratio r = π_new / π_old (ε = 0.2; the band 1 ± ε in green) advantage A = +1: above 1 + ε it adds nothing more advantage A = −1: below 1 − ε it adds nothing more

L(θ) = E_t[ min( r_t(θ) Â_t, clip(r_t(θ), 1 − ε, 1 + ε) Â_t ) ]

L(θ): a surrogate objective maximized for several epochs on the old batch; its gradient at θ = θ_old is exactly the REINFORCE policy gradient with the advantage
r_t(θ) = π_θ/π_old: the ratio of the new and old probability of the played action. The data was collected by the old policy; multiplying by the ratio lets us estimate from it how the new one would do (importance sampling). At the start r = 1
Â_t: the advantage from GAE: its sign says whether to reinforce the action (+) or suppress it (−)
clip(r, 1 − ε, 1 + ε): the ratio clamped to the band (green in the figure); outside the band it is constant, so no gradient comes from it — this sample can no longer push the policy further
min: the worse (pessimistic) of the two estimates: the clipping applies only where the improvement would look suspiciously good. When the action got worse instead, min keeps the unclipped term and corrects the mistake

Reinforce advantageous actions, but only until their probability changes by more than a factor of 1 ± ε; then the sample falls silent.

Example — three samples and clipping with ε = 0.2

The old policy gave the played action probability 0.5. After a few learning steps it is 0.65, so r = 0.65/0.5 = 1.3 and clip(1.3, 0.8, 1.2) = 1.2 (colours as in the breakdown of the clipped objective above).

Advantage +2: min(1.3 · 2, 1.2 · 2) = min(2.6, 2.4) = 2.4. The clipped term won, and it does not depend on θ — the gradient from this sample is zero. The action is already 30% more frequent, and that is it.

Advantage −2 (same ratio): min(1.3 · (−2), 1.2 · (−2)) = min(−2.6, −2.4) = −2.6. The unclipped term won: the policy made a bad action more likely by mistake, so the gradient keeps pulling it back.

Advantage +2, but the probability dropped to 0.35: r = 0.35/0.5 = 0.7, min(0.7 · 2 = 1.4, 0.8 · 2 = 1.6) = 1.4. Again the unclipped term — a good action that the policy suppressed comes back. Clipping only brakes steps that would improve the objective “too much”.

repeat:
    π_old = π_θ;  collect 2,000 steps with the policy π_old, store π_old(a_t | s_t)
    for each episode, from the end:                  # V(s_(t+1)) = 0 after a fall
        δ_t = r_(t+1) + γ · V(s_(t+1)) − V(s_t)
        Â_t = δ_t + γλ · Â_(t+1)                      # GAE, γ = 0.99, λ = 0.95
    target_t = Â_t + V(s_t);  normalize Â to mean 0 and variance 1
    10 times (epoch):
        shuffle the steps and split them into minibatches of 100
        for each minibatch:
            r_t = π_θ(a_t | s_t) / π_old(a_t | s_t)
            θ: a step up along the mean of min(r_t · Â_t, clip(r_t, 0.8, 1.2) · Â_t)
            V: a step down along the mean of (V(s_t) − target_t)²

5. Experiment: the cart-pole

The same cart-pole as with DQN, reward +1 for every step, the episode ends with a fall or after 500 steps. The policy is a 4 → 32 → 32 → 2 network with a softmax, the state value a second network of the same shape. All four methods get 100,000 steps in the environment, in batches of 2,000. REINFORCE takes one gradient step per batch, PPO ten epochs over minibatches of 100. The “without clipping” variant is PPO with its clipping switched off.

Average episode length during learning (thick: mean of three runs, thin: individual runs) REINFORCEREINFORCE with a baselinePPOPPO without clipping

method	first time the average ≥ 475 (run 1 · 2 · 3)	at the end (run 1 · 2 · 3)
REINFORCE	— · — · —	463 · 333 · 348
REINFORCE with a baseline	90,000 · 80,000 · —	488 · 500 · 404
PPO	30,000 · 20,000 · 20,000	500 · 500 · 500
PPO without clipping	100,000 · 30,000 · 50,000	494 · 44 · 500

In all three runs PPO keeps the pole up for the full 500 steps after just 30,000, 20,000, 20,000 steps and never lets go again. REINFORCE improves slowly and does not make it by the end; with a baseline it is somewhat faster and reaches the goal in two runs out of three. Careful, the comparison is not entirely fair in the amount of computation: PPO takes ten times as many gradient steps per batch. It is fair, however, in what counts most in reinforcement learning — the amount of experience the agent has to collect in the world.

And what happens when we run the same repeated learning on a batch without clipping? Episode lengths jump up and down, run 2 ended at 44 steps. Why becomes clear from how far the policy drifts in a single update from the one that collected the data:

Distance of the new policy from the old one after each batch (KL estimate, run 1, logarithmic) PPO without clipping

For PPO the average distance is 0.014 nats and never exceeds 0.06. Without clipping it is 5.5 on average and at times up to 25 nats — after a single update the policy does something completely different from the one it learned from. The data then says nothing about it, and improvement is a lottery. Clipping is a small tweak to the formula, but it turns an unstable algorithm into a reliable one.

Definition — the KL estimate between the old and the new policy

Let π_old and π_θ be policies on a finite set of actions A with π_θ(a | s) > 0, and let (s_i, a_i), i = 1, …, N, be the steps of a batch collected by the policy π_old. The KL divergence from the part on information theory, here with the natural logarithm (unit: the nat; 1 nat = 1/ln 2 ≈ 1.44 bits), averaged over the states the old policy visits, is

D_KL(π_old ‖ π_θ) = E_s[ ∑_{a ∈ A} π_old(a | s) log π_old(a | s)/π_θ(a | s) ] = E_{s, a ∼ π_old}[ log π_old(a | s) − log π_θ(a | s) ]

and its estimate from the batch is

D̂ = 1/N ∑_i=1^N [ log π_old(a_i | s_i) − log π_θ(a_i | s_i) ].

The actions a_i were played by the old policy, so the average over the batch is an unbiased estimate. It is the average logarithm of the ratio, and so e^−D̂ is the geometric mean of the ratio π_θ/π_old over the collected actions.

What it means in numbers: for PPO at most 0.06 nats, e^−0.06 = 0.94 — actions from the old batch are typically a few percent less likely, and the data still tells us something about the new policy. Without clipping up to 24.9 nats: e^−24.9 ≈ 1.5 · 10⁻¹¹, the new policy would almost never play the actions it has just learned from.

6. PPO and language models

In the part on RLHF we mentioned that InstructGPT and the first ChatGPT were fine-tuned with reinforcement learning. It was exactly PPO: the policy is the language model, the state is the text so far, the action is the next token, and the reward at the end of the answer comes from a reward model learned from human preferences. The penalty for drifting away from the initial model (KL) plays the same role as clipping — it does not let the policy run off to where the reward cannot be trusted.

Definition — the reward when fine-tuning a language model with PPO

Let x be a prompt and y = (y₁, …, y_n) an answer (sequences of tokens) generated by the language model being tuned π_θ, where π_θ(y | x) = ∏_k=1ⁿ π_θ(y_k | x, y₁, …, y_k−1). Let r_φ(x, y) ∈ ℝ be the output of the reward model, π_ref the frozen model after fine-tuning on examples (SFT) and β > 0 the strength of the penalty. PPO then gets the reward

R(x, y) = r_φ(x, y) − β · log π_θ(y | x)/π_ref(y | x)

(this is how InstructGPT writes it, Long Ouyang et al., 2022). The mean of the second term over answers is exactly β · D_KL(π_θ ‖ π_ref). An episode is one answer, the actions are tokens, the reward r_φ comes only after the last token; the rest is PPO as described above.

For large models, though, PPO is expensive: besides the model it needs a value network of the same size. That is why simplifications appeared. DPO from the part on RLHF avoids reinforcement learning altogether. GRPO (DeepSeek, 2024) drops the value network and uses the average reward of several answers to the same question as the baseline — exactly the constant baseline we measured above. This is how models that “think” about mathematics and code are trained, where the correctness of an answer can be checked.

Â_i = r_i − mean(r₁, …, r_G)/sd(r₁, …, r_G)

Â_i: the advantage of the i-th answer; all its tokens get it, and it then goes into the PPO clipped objective
r_i: the reward of the i-th of G answers to the same question, for example 1 = the result is right, 0 = it is not
mean(…): the baseline instead of a value network: how well the model usually answers this question — an easy question has a high bar, a hard one a low bar
sd(…): the standard deviation of the group, sd(r₁, …, r_G) = √∑_i=1^G (r_i − r̄)²/G, where r̄ is the mean; it evens out the scale so that questions with large and small spreads of rewards pull equally hard (like the advantage normalization in our PPO)

GRPO (Zhihong Shao et al., DeepSeekMath, 2024): the advantage of an answer = how much better it is than the other answers to the same question.

Example — GRPO advantages for four answers

The model answers a task four times, correctly only the first time: rewards (1, 0, 0, 0). The mean is 0.25, the standard deviation √0.75² + 3 · 0.25²/4 = √0.1875 = 0.433.

Advantages (colours as in the formula above): 1 − 0.25/0.433 = 1.732 for the correct answer and 0 − 0.25/0.433 = −0.577 for each of the three wrong ones. The sum is zero: the correct one is reinforced, the wrong ones suppressed. If all four answers were correct (or all wrong), all advantages are zero and the question teaches nothing — learning comes only from questions where the model sometimes gets it right and sometimes not. (We divide by G; some implementations divide by G − 1, and the numbers then differ slightly.)

7. Summary

Policy gradient learns the action probabilities directly; it works for continuous actions too and explores on its own.
REINFORCE reinforces actions in proportion to the return that followed them; it is simple, but its gradient estimate has high variance.
Subtracting a baseline reduces the variance without changing the mean (here by 26% even with a constant); with a learned V(s) you get actor-critic.
PPO uses each batch several times and, by clipping the probability ratio, keeps the policy close to the one that collected the data.
On the cart-pole: PPO manages 500 steps after 20–30 thousand steps in all runs; without clipping the policy drifts by up to 25 nats and learning swings.
PPO fine-tuned the first ChatGPT; its simplification (GRPO) trains today’s “thinking” models.

Want to get a feel for it? Have an AI agent run PPO from the Stable-Baselines3 library on the LunarLander environment, vary ε in the clipping (0.05, 0.2, 0.5) — and plot how the speed and stability of learning change.

Do you see it differently? Write to support@coden-app.com.

Policy gradient and PPO: learning the policy directly

1. Learning the policy directly

2. REINFORCE: reinforce what worked

3. Variance and the baseline

4. PPO: small, safe steps

5. Experiment: the cart-pole

6. PPO and language models

7. Summary

An Agent on your computer, driven from your phone