Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Reinforcement learning remains more art than science because of fundamental challenges in balancing exploration and exploitation, designing effective reward signals, and achieving sample efficiency. Unlike pre-training, throwing more compute or data at RL does not guarantee improvement, and the hidden costs of hyperparameter tuning are often underestimated. The biggest open problem is creating reward systems that generalize beyond narrow, carefully crafted environments.
Key points
RL requires balancing exploration of new actions and exploitation of known strategies.
In RL for language models, training data is generated during training, creating overhead.
Sample inefficiency means RL often needs millions of failed attempts to learn useful behaviors.
Poorly configured rewards can lead to reward hacking, where models exploit loopholes.
RL hyperparameters are highly sensitive, adding hidden costs to training.
Techniques
- exploration vs exploitation
- on-policy learning
- off-policy learning
- reward design
- reward hacking
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Saying that RL sucks is not really a hot take or unique take. Many people share a similar sentiment when it comes to reinforcement learning or RL, but RL is also really cool. It's what helped AI
complete simple games like Super Mario and eventually AlphaGo beating Isador, other games like Dota and Starcraft and domains like robotics currently undergoing a transformative moment with
advancements made in RL. And even in language models like coding, we've seen how RL can improve an LLM significantly during post training. And yet, RL remains more art than science and can
often lead to hacky results. And there are many reasons why that's the case, more than what I can summarize into this video. But understanding some of the bottlenecks and limitations that exist
in RL will help you better equipped to prepare for what kind of advancements we should be looking for, and why just throwing more GPUs at the problem doesn't really solve the underlying
problem. Take a look at this simple simulation where this humanoid tries its best to stand up. And currently it has a really difficult time doing the most basic tasks like standing up. Now we
could use RL to reward the humanoid to actually stand up. But even that is not really a trivial task. What you're seeing here is two different outcomes depending on the RL algorithm that was
used where one stands up a little bit and then falls and repeats the cycle while the other one just freezes without fully standing up. This is one of the common problems that occurs in Rall
called exploration versus exploitation. Exploring different actions like this might lead to a better strategy in the end where clearly in this example leads to the humanoid actually standing up but
eventually falls over and over again. Whereas exploitation is something like this where the humanoid just tries to maximize what it already learned without taking big risks in exploring for
potentially a better solution like standing up. And the cost that's involved here is that exploring can end up wasting a lot of compute on actions that doesn't really go anywhere. While
too much exploitation can also lead to this mediocre strategy where it doesn't really go beyond suboptimal strategy like this where it just settles there thinking that that's the best it can do.
Now, when you extrapolate this simple analogy like this humanoid into much more complex environments like Super Mario, robotic simulation, gaming, and more, you start to realize that the
challenge is really how to balance between the two. Now, the two attempts that you just saw in the humanoid example are just two timelines, so to speak, out of many different timelines
that could exist in the humanoid actually learning from random to being trained in the end. And there's so many different ways, as you can see, that the humanoid could have actually learned to
actually stand. So, there's a huge overhead involved here when it comes to actually training the model itself. But let's not get ahead of ourselves just yet because we haven't really talked
about the scarcity of data, which is arguably one of the biggest bottlenecks in RL in comparison to pre-training. I'm sure you've heard many times by now how LLMs are trained using carefully sourced
internet data, and that's often what happens during pre-training. But in RL, especially in language models, it takes an already pre-trained model and then goes into additional training on top of
that. And the data that we need to do this very part simply doesn't exist. Well, to some degree, instead of pouring a bunch of data into the model in RL, the data is actually generated during
training and that generated data is then fed back into the model to be rewarded accordingly after being graded. And this is a huge overhead because data is not readily available in language models.
For example, let's say that we're trying to improve the model to solve a very complex coding tasks given this repository. We have to wait until the model actually finishes many tool calls
back and forth. And once the agent thinks that it's done, then we can grade the result and assign the proper reward. And while we can parallelize this by duplicating this instance across, let's
say, eight concurrent RL environments and grading them relative to each other, there's still a huge inference cost in actually generating eight rollouts that run in parallel. When we look outside of
language models and look at robotics and simulation, while parallelizing RL environments like we saw in coding might seem feasible, that's not really the same case in real life. When it comes to
complex simulations and robotics, they can actually be pretty costly. And this can often lead to what's known as sample inefficiency problem in RL. Essentially, how much interaction or experience does
a model need to actually learn something useful? If making the humanoid stand up requires millions and millions of failed attempts, how much compute do we really have to waste to actually learn? So, how
do we really optimize for the right amount of sample so that the model learns as much as possible from every interaction that it has? And this is where we get to the discussion around
what you might have heard of already in the past called on policy or off policy in RL. But first, let's talk about Firecrawl sponsoring this video. Your agent needs context from the internet to
stay relevant. But there's a problem with internet scrape data because it contains all this hypertext and additional information that takes up time and context. But also, your agent
might need to sort through the metadata around them. Firecrawl recently introduced Alexandria as a place of ground truth for your agent that has curated information from websites like
World Bank Data, Hugging Face, Wayback Machine, Skyscanner, and more. Sort of like the library from Alexandria in 2000 BC containing tens of thousands of scrolls. But now your agent can go
through Firecrawl's library of Alexandria for highly credible and highly sanitized data to better drive what you're working on. I can just pair up my coding agent with firecrawl mcp to
use their services and also use firecrawl in code to selectively target specific sources from Alexandria like census data in the US to do further analysis. Try fire crawl for free
without an API key. Link in the description below. Because in the real world, a failed attempt isn't really just resetting the scene back to square one, but the actual parts might be
damaged. or even in complex simulations, there's a huge load time in resetting the scene back to square one's original state. Off policy method often tries to store past experiences into a replay
buffer to essentially compound its learning instead of an on policy method, which collects new experiences after each round of training, which is more often used when simulation is cheap to
reproduce. For reference, most language models today tend to lean towards on policy or near onpolicy methods with algorithms like these that you might have heard of before. And that's exactly
what we see when we look at the progression of different labs in what RL algorithms they use to train their models. But whether the data was generated for limited training updated
in on policy or instead stored in a replay buffer in an off policy, how much learning we can actually extract from the generated data is a much different problem. Contrary to what you might
think, having more data or having more compute doesn't really guarantee that RL will be better, which is a fundamental different framework of thinking than in pre-training where you can speed up
learning or even have more intelligent model by throwing more compute or more data or even making the model bigger. And this kind of strategy doesn't really apply in the same way in RL in most RL
algorithms. Essentially, everything really boils down to reward, which is how do we really define what a good behavior looks like and then how do we translate that into a signal that the
model can actually learn from. This is arguably the biggest question that is easy to solve in one domain, but hard to generalize into a scalable reward system that can just work in an open-ended
tasks that we want the model to actually do well in real life outside of narrow domain tasks that we ask them to do well in during RL. Even a simple game like Super Mario, you could design a reward
to be awarded positively when it finishes the game and negatively when it dies and nothing more. This kind of sparsity in reward can make it really inefficient for the model to actually
learn how best to play Super Mario. In reality, you want to set up based on input to punish when it stays still, reward positively when it moves ahead and negatively when it moves back and
stomping over a gloomba as a positive reward and so on. You also want to have periphery rewards like maximizing points and taking less time to finish the game. And all of this play into factor of the
game. So, not only is it important to set up the right reward for the model to explore, we want to tune the hyperparameters for the job at hand for learning to be optimized, we also want
to avoid badly configured rewards where the model might actually cheat and hack the reward system, which can be shown in this example of how in a boat racing game, the boat starts out driving on
track and finds a region in the game where it can keep racking up points by circling this region over and over again without really finishing the game while at the same time scoring higher than
anyone else in the player technically. So, if you think about the budgets that's involved in RL training, while Frontier Labs often release the amount of money spent on running RL from
beginning to finish, what we often don't see is the amount they actually spent in tweaking the hyperparameters in RL. Because RL can be so sensitive to hyperparameters, you have an additional
overhead in actually configuring the environment for the best outcome. As you can imagine, simply dropping the model into the environment and resetting the environment each time doesn't really
guarantee success and can often take up a lot of time and money. And given that Frontier Labs compete where now models are being dropped every few weeks, it means that we need RL to also be
efficient and fast at the same time. Now you'll find that when we dig into the specifics of each RL algorithms like GRPO and PO they're actually more nuance problems beyond the framework of reward
but actually translating them into updating the model's weights and the discussion around this does get a little bit more technical here as we talk about training stability and different
mechanisms that we put in place which some of them I covered in my previous video. But all in all, despite all the challenges I described in this video, like exploration versus exploitation,
reward system, hyperparameters, narrow versus generalization, reward hacking, training algorithms, and budgeting are all certainly shouldn't be thrown out with a baby water because it's still
what powers a lot of actual improvements, especially in LLM. We saw the rise of what reasoning models in 2024 could do in pushing the capabilities of LMS much further. And
later in 2025, it was advancements in RL that actually still helped push LLMs even further than we've ever anticipated what a simple language model can do. But we still have yet to figure out how to
make RL more transferable beyond narrow tasks and environments, but actually making the model more generalizable into complex open-ended situations that doesn't need carefully crafted system
that tells the model what success looks like at each step of the A.