Until fairly recently, most of the successful machine learning algorithms were based on supervised learning, where you have some input x and some reference output y, and you train a neural network to successfully predict y given x.
Recent breakthroughs such as large scale pre-training of LLMs and supervised finetuning builds on top of this. In the case of pre-training, the input x is all prior tokens preceding a token y, and the task is to predict y given those tokens. Because no explicit annotation occurs in this setup and the input and labels are directly derived from the raw data itself, this process approach is called self-supervised learning and it is typically done over trillions of tokens. Supervised finetuning is closely related to this but is more similar to conventional training paradigms. In this stage, you have a set of questions Q and corresponding answers A, the LLM is trained to take the question as input and correctly predict the full sequence of output tokens A. This process teaches the llm to follow instructions and learn how to chat. Notably, the ground truth A in this setup is either human written or derived from a more powerful LLM. This is a form of imitation learning. It works but has a number of challenges, which has necessitated the use of RL to substantially advance the capabilities of modern LLMs. In this post, we will review the challenges with supervised finetuning and then discuss how RL solves these problems.
SFT is OFF-POLICY
What does it mean for an algorithm to be off-policy? It means there is a divergence between the data that we are using to train the model and what the model would have generated itself. In a simpler term, the model is being optimised against some gold set of trajectories which it might never generate by itself. This is in contrast to on-policy algorithms where a model generates a trajectory and is optimised on the correctness of its trajectory. We would make this even more clear with a simple example.
Let’s imagine a single training
Question: What is (2 + 6 ) 3
Answer: add 2 and 6, to give 8, then multiply 8 * 3 = 24
To learn this with SFT, we will perform cross entropy loss on the next token in the sequence.
Step 1:
Input: User: What is (2 + 6 ) 3 Assistant:
Predict: add
Step 2:
Input: User: What is (2 + 6 ) 3 Assistant: add
Predict: 2
Step 3:
Input: User: What is (2 + 6 ) 3 Assistant: add 2
Predict: and
…..
Step N:
Input: User: What is (2 + 6 ) 3 Assistant: add 2 and 6, to give 8, then multiply 8 * 3 =
Predict: 24
The above is simply unrolling of the full input and target sequences. It basically says, for each full context of the question + answer tokens up to each point in the answer, we use that context to predict the next token in the answer sequence. There are a few issues with this. Let’s unpack what is going on here.
Take step 1. We had the input “User: What is (2 + 6 ) 3 Assistant:” and we use cross entropy loss to optimize the LLM to predict “add”. At this point, we are doing fine, but let’s move to another step.
At step 2, we feed in the question + the first generated token “User: What is (2 + 6 ) 3 Assistant: add” as input to predict the next token “2”. A problem starts creeping up here, we are basically performing what is called teacher forcing, because, the sequence “User: What is (2 + 6 ) 3 Assistant: add” has become off-policy since the continuation “add” was not generated by the model, but we are making a grand assumption that after training, the model would learn to generate that first token. In reality, due to multiple reasons, the model at inference might generate a different token instead of “add”, and since we are optimising the context + “add” as input to learn to predict “2”, this scenario might not occur at inference, and the actual scenario that occurs at inference might be context + “if”. Basically, as the steps grows, we keep utilizing gold continuations that were generated off-policy by a human or a different model as context to predict the next token in the sequence. For example, at the final step, we have the input, “User: What is (2 + 6 ) 3 Assistant: add 2 and 6, to give 8, then multiply 8 * 3 =”, and are optimizing the model to predict the next token “24”, but all the tokens “add 2 and 6, to give 8, then multiply 8 * 3 =” are off-policy here, and might never be sampled from the model in that exact order after training. For example, if you sample at a temperature greater than zero, the continuation after the question might be completely different, which will greatly reduce your chances of actually predicting “24”. Even if you sample at zero, there is always a chance that your llm is a bit random. See https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/.
In simple terms, as your output sequence grows, with SFT, you are optimising the next token with off-policy data, and the distribution you are optimising is not the same as the llm output distribution at that point. This is a major reason why off-policy methods such as SFT and DPO struggle on complex domains such as maths and reasoning (see SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training. As these are domains where cloning and memorisation can only get you so far. As soon as you start running OOD evals, the performance drops. If your test scenarios is similar enough to your training data, this is less likely to be a problem, but in the real world, your model will struggle.
In theory, if your SFT data is arbitrary large in the order of several hundreds of billions of tokens, the diversity of those samples should compensate for the off-policy nature of SFT. Doing SFT at such scales is often called mid-training.
Sometimes, this is not a problem, as we showed in step 1, the learning is on-policy at your very first token. So, if you are only ever predicting a single token in your training for some reason, SFT might be all you need. This could be true for example in basic classification tasks, but there are caveats. First event if you are simply predicting single class labels, dependending on your tokenizer, your single class label might be multiple tokens in the llm vocab. Also, if you are training a reasoning model, which needs to perform chain of thought before predicting your class label, then you will be very off-policy before your class label is predicted.
Data Efficiency of SFT
For the reasons stated above, it becomes obvious that when learning off-policy you need a whole lot of highly diverse data at mid-training scale to achieve meaningful performance, hence SFT is not a very sample efficient learning process.
REINFORCEMENT LEARNING
RL algorithms such as PPO and GRPO are on-policy algorithms that works by letting a model take a query, sample a full response and the response of the model is graded by some reward functions which tells us if the model output is good or bad. This reward is then used to update the model using some variant of REINFORCE such as PPO and GRPO.
RL has a long history, but it became widely known through deep RL systems that played games like Atari, Go and Dota. DeepMind and OpenAI did a lot of pioneering work here with deep neural networks trained by RL. In the field of LLMs, RL powers all the modern models such as GPT-5 and Gemini 3. In this section, we will unpack how RL works, and why it leads to models that generalize stronger than SFT trained models.
How RL on LLMs Work
Let’s review our earlier sample, you have a question,
”What is (2 + 6 ) 3”
And we have some labels that tell us what the final answer is, “24”,
In RL, rather than trying to clone a human written answer token by token, we will instead do the following.
Pass the question to an LLM
Sample a full response from the LLM. (not just one token at a time)
Use some reward function to grade if the response from the LLM is correct.
For example, if the LLM response is, “To solve this, simply add 2 and 6 to give 8, and then multiply 8 * 3 to give 24”. Our reward function will mark this as a correct answer and we use this reward to update the weights of the llm using policy gradients.
However, if the LLM generated a response “To solve this, simply add 2 and 6 to give 7, and then multiply 7 * 3 to give 21”, this will be marked as a wrong answer and we will update the weights of the llm so it avoids doing this the next time.
One notable difference. The exact output sequence is not predefined, the model could output any answer as long it is correct, the exact sequence of tokens is not cloned. In SFT, we define a single gold answer sequence and use cross entropy loss to clone that answer, but in RL, we generate full answers and grade if the answer is correct or not.
During the lifetime of the RL training, the model will likely generate multiple different correct answers, and it will learn from it. This makes RL very sample efficient, since a single question - answer example will lead to multiple answers generated at training time.
A key difference in RL is, at each point in time, every single sequence of tokens in the models answer is generated by the model itself, hence it is on-policy. In practise, the model answers are sometimes sampled at a high temperature during RL training. The on policy nature of RL means each time the model is operating within its token distribution. And during inference, the distribution will not diverge. In SFT, a model might do well during training at predicting some final answer since the full context that led to that answer is fed as input from the ground truth GT, however, at inference, since it needs to generate the answer sequence token by token, the model can easily find itself in uncharted territories. This is less of a problem with RL, as each time it gets a good reward, its answers are derived from a sequence of tokens the model generated by itself.
A good analogy for SFT vs RL is; with SFT, you are reading a book and trying to make informed predictions about what happens next in the book, for example, you are reading the answer to a problem in a book and you are making a guess on what the final part of the answer is. However, during RL, you take a question, and you are working out the answer yourself and thinking about what the next part of your answer should be. In SFT, you are following another persons reasoning process while in RL you are following your own reasoning process and optimising that to be optimal.
A natural question that arrises is, why don’t we just sample full responses in SFT? The core issue is that the sampling step is non-differentiable, so you cannot backpropagate gradients through the sampled tokens in the usual way. RL using policy gradients provides a means to do this.
If you are unfamiliar with RL, you can watch the part 1 of my RL fundamentals series.
Generalisation and Sample Efficiency
For all the reasons mentioned above, RL is very sample efficient compared to SFT. The fact that you do not need to actually predefine all the ways an answer can be written and instead, you let the model try out a large amount of ways a question can be answered correctly and the model gets good reward each time it answers the question correctly. This also leads to better generalisation, as the model moves from cloning gold answers to actually learning sequence paths that lead to good answers.
For more on the differences between on-policy and off-policy algorithms, refer to the excellent blog by thinking machines.
If you found this useful, you can also subscribe to my youtube channel , connect on twitter , linkedIn or reach me directly via email johnofenwa@gmail.com