Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
LLMs can potentially discover new knowledge through reasoning, not just regurgitate training data, according to Anthropic researcher Jeremy Berman. He explains that reinforcement learning (like GRPO) trains models to reason via trial and error, enabling them to build new abstractions. This is distinct from pre-training and instruction tuning, which only teach autocomplete and assistant behavior.
Key points
Neural networks are universal function approximators that can represent any function with enough layers and parameters.
Pre-training trains a model to predict the next token on internet-scale data, producing an autocomplete engine.
Instruction tuning uses human-written assistant responses to teach the model to follow a chat format and stop generating appropriately.
Reinforcement learning (GRPO) trains models to think by having them generate multiple answers, scoring them, and updating the model to favor correct ones.
The speaker predicts that models can discover new knowledge through reasoning, as they build new abstractions.
Tools mentioned
Techniques
- backpropagation
- pre-training
- post-training
- instruction tuning
- reinforcement learning
- GRPO
- chain-of-thought
- rejection sampling
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
This is Jeremy Berman from Anthropic. He's a researcher there and no, we are not related. And we sat down and he explained exactly how large language models get built. Everything from
pre-training to post-training, reinforcement learning. And so, if you ever wanted to learn about this stuff, here's the right video for you. And this is for members only. So, I just wanted
to say an extra thanks for being a member. I hope you enjoy this video. I hope you learn a lot. And of course, if you want to see other videos like this, let me know down in the comments. If you
have any questions, also just drop a comment. I'll try to answer them. >> My goal is really to give an intuitive highlevel overview on how we're able to build intelligence and intelligence
systems really from just a file with numbers and a relatively small Python program. So, um I'm going to break this up into seven what I hope to be intuitive parts. um what are neural
networks, how they're trained, what are large language models, um what is pre-training, what is post-training, how did we get them to think, and can they discover new things? And this is all the
same techniques or basic versions of what the modern large language model landscape looks like. So, Chat, GPT, Claude, this is all kind of what they're doing to make these incredible models.
>> Yes. Yes. Exactly. Um, I think you know there are a lot of um each in each one of these sections you could do a whole deep dive for hours. Um, I'm going to try and stay high level and intuitive
but of course you know ask any questions and maybe we could do you know a follow-up video um for for the ones that are most interesting. >> And and I just want to give you your
props for a second because you are self-taught. you are a self-taught AI researcher and so like this is the starting point for anybody who wants to follow that same
journey. >> Yes, I would say really just follow your intuitions like you know uh there are a few parts to training these things and you can go as deep as you need to in
each one you know kind of find what you get obsessed about and then just follow that into a rabbit hole. Um and you know I think this is like very very selfeachable even even today. Cool. All
right, let's do it. >> To start off with, what are neural networks? I want to uh start by giving a real world example. So, let's say we have a sheet of apartments in New York.
Um they have inputs and then they have the price. So, an input would be the location. So, Soho, the square footage and the number of baths. Very simple example, toy example. And so the idea is
we need to find the pattern the function that maps these inputs location square foot and and number of baths to the price in New York. And so as you can see we have a set of inputs and then we have
the actual price to the right. And so a classic neural network um is really like a way to build a function. So you can imagine you could build a function to do this, right? Like you could have a
function that takes in the square footage, the bath. um you could code it up and then you could get pretty close to the right answer for the price. But uh is there a way that we could teach um
the computer to build this function for us? And for a toy example, this you know seems okay, whatever, who cares? We can code up our own example. But you'll see as we start to model really really
complicated things, it's very useful and in fact, you know, we couldn't do it otherwise to actually have uh the model build the function for us. And Jeremy, when you say function, you literally
mean kind of the old technique might be, okay, if square footage is between a,000 and 2,000 and you know, if the baths is between one and two, price it around here. And so you you basically have to
hardcode all of this logic yourself, which you know, if we have seven examples as shown here, maybe it's not so bad. But what if we have thousands, tens of thousands or millions?
>> Exactly. Millions. And then what if instead of three inputs, we have a thousand inputs, right? Uh a real estate broker has, you know, hundreds of inputs in their head, right? The finishings um
when it was built. Um and what the real estate broker is doing is they're pattern matching, right? They're saying, "Okay, I I have a pattern match. I have, you know, a general idea of what each of
these mean, and so I can now map this property to how much it should cost." Um but what we're trying to do is um do this have the computer build this function for us automatically in kind of
a black box. So what does this really mean? Well here you can see this is a very basic formula and let's just say we have inputs you know of the location, the size, the baths, the location and we
have our our answer. So what we can do is we can have these knobs and the way the knobs work is you multiply the size by a certain parameter by a certain value. Uh let's let's randomize it at
first. It's randomized to 0.25, right? We multiply the bathrooms by 0.05. We multiply the location by 0.15. We do all of this. We add it up together and we should have an output and this output
gives us a price. Now we just randomly initialized these uh these variables here, right? So what we get out and you can see in these examples for SOHO it predicts 942K
uh 742 860. These are all wrong because really what we're get after is the green. But we could actually adjust these dials, right? And eventually we can get actually get pretty close
uh to >> Can you can you explain in basic terms what the actual kind of what you're multiplying by the knob? What does that represent?
>> Yeah, so the knob is actually really just a random number uh in this case uh from 0 to one. And uh it's like a black box. You are, you know, taking your inputs which are you're turning your
inputs into numbers and then you're multiplying it by these random numbers or knobs. And there is some combination of knobs that will get you really close to always getting the right answer. Now
let's say we can't possibly do it with three knobs, right? With with three numbers. So, when I'm turning this knob, the the number is moving, right? As you can see here, and it's adjusting the
output. Let's say we can't get perfect with these three knobs. Well, you can actually just add um more knobs or more layers or more parameters to build an even more complex function till
eventually you reach the green. Um, and tell me if this is a right way to describe what these knobs are doing, but it's it's kind of assigning a almost an importance value to each of those
inputs. So maybe the size of your apartment is super important to the price output or maybe actually how many baths you have is really really important. Maybe the location is less
important. You're is that a good way to describe it where each of these different knobs is kind of adjusting how important our little formula, our little algorithm is taking into account each of
these factors. Yeah, that's a great way to put it. Now, this is an extremely simple example where the number of bathrooms isn't in some way intertwined with the location. And so, let's say
Soho, it might be not desirable to have a large place. Um, this is a very simple example uh just for demonstration purposes, but you're exactly right. In this simple example, it's basically
saying how much do we weight each one? How much do we multiply each one? You can imagine we can generate we can add more terms such that you also divide by the size in a new uh knob assuming the
location is a certain thing. So we can build uh arbitrarily large complex functions just by adding more knobs and changing the ordering of the knobs. And uh this would be an example of basically
just one layer of a network because um you know we just have this simple um simple function. But you can imagine that this this output all of these knobs produce one number and th that number
gets fed into a whole new set of knobs and a whole new set of knobs. And so you have this function inside of a function inside of a function which is ultimately what neur neural networks are. And
again, at each uh layer, you have a whole new set of knobs that you can twist and turn. And the point is really to minimize this number right here. So this is the average uh on average each
prediction is off by. And so what we're trying to do is minimize this number. So you can see that if we're just doing this by hand, you know, in this simple example, we can get relatively close.
And um and so this is really the intuition. Yeah. behind neural networks. It's really just turning knobs uh that that create arbitrary functions that get you closer and closer to minimizing what
we call loss, which is um the difference between what the output is and or what what your model predicted and what actually exists in the world. >> Right? So, if the actual sales price of
that apartment in Soho is $2 million, and we predict or the model predicts it's $1 million, that's way off. And we want to continue to adjust those knobs until we get closer.
>> Exactly. Yes. Um, and you just want to keep adjusting. So, um, we will come back to this example. Actually, let me now go back to the slides. This is just a general quick level intu intuition
behind what what exactly are knobs and and what are they doing. Uh now here's uh the actual example if this were a neural network we don't really need to focus into what all these numbers are
but if you wanted to actually see we would turn the location into numbers as an input layer. We would turn the size into a number. We would turn the bathrooms into a number. And then each
of these connections is the actual knob. It's the number. It's the weight. And so uh a latitude has a weight that goes into uh and in this case there are three um there are three intermediate layers.
So we can construct a function. Um and what you're doing is you're basically you're filling in this is a number, this is a number, this is a number, this is a number. You're multiplying this number
by this number plus this number by this number plus this number by this number. And I know it gets very complicated which is why this is really um yeah >> this this looks intimidating but I think
the way to think about it is what we did on that previous slide with the knobs it it's just a much more sophisticated version of that. >> Yes. Yes. Exactly. At the end of the day
we are just turning knobs and we are building these functions and as you can see these are functions that are now you know relatively complicated even though we're just turning knots. Um, now, uh,
as you can see, it would get very, um, timeconuming to have to go in and individually tune each knob. You know, okay, we tune these knobs and then we run these examples and then we see that
we're off by this amount. Okay, well, let's adjust this knob. Let's adjust this knob. You know, there's there you can do it by hand. But, um, the great the great invention really of neural
networks is we can actually do this. we can have the computer calculate a lot of how we should tune these knobs automatically and I will now get to that but just before I know uh this seems
relatively simple right our neural networks really just they're just numbers and then it's a Python script generally that really just describes um how these numbers uh are ordered and
wired and that is really it so uh you know what is a language model a language model is really just a download from hugging face which has the numbers and uh the numbers are the brain it's
actually the parameters. So when we say you know there's a um a two trillion parameter model that means they're they're basically two trillion numbers and then we have a python function that
describes the wiring of all of these numbers which will ultimately build the compute graph which we will then run the inputs through and then we will get our output. But it is really uh really as
simple as just tuning knobs. I I said before that there's a better way than just handtuning these, right? So if you've heard of PyTorch or Jax or or TensorFlow, these are called autograd
engines. And at the end of the day, really what they're doing is they are keeping track of all of the parameters, all of the numbers and all of the uh computations that those numbers are
involved in. And then they are keeping track and then they are you know at the end you get your value which is how much you are off by. So in this case we were off by you
know around 0.2 million. And then what the autograd engine what pietorch is doing it's keeping track of all of the calculations and it knows what each parameter uh it actually knows how much
each parameter if you were to wiggle it. If you were to update it, you know, by 0.1 or by 0, how much it would affect the ultimate loss. So how much if we wiggle this size, how
much should it ultimately affect this position? So the way it does this is that ultimately we are just creating you know a function. We are creating a very complicated function that ends up
outputting the loss but it's a function and um what we can do is we can take the derivative of each parameter. Uh so uh with respect so we can take the derivative of this ultimate loss how
much we are off by with respect to each single parameter and so you can imagine we have this size weight which is multiply >> Jeremy sorry let me interrupt for a sec
there was a lot of terms thrown out um maybe you can just use the most basic terms and then go deeper. >> Yes. Yes. Taking a step back, we are ultimately
um just building a function that takes in the inputs. It multiplies these inputs by a variety of numbers which are the knobs and then we get an output and we can tell how different that output is
from what we expected. With just this information, we can then um determine how much basically if we what the effect of changing one of these knobs is on the final loss. In other words, if we were
to shift this this this parameter this this knob, how much what is the effect on the final loss? And we can do this per parameter. And so let's say we shift this and the
effect actually on the loss is something like um three. So for every one change we make in the size the loss is uh let's say increased by three. So now we actually know well if we want to
minimize the loss right well we can just make this number a bit smaller and if we make this number a bit smaller we are going to decrease the amount we're off by and we can do this per parameter and
so what pietorch is really doing is it's saying for each of these knobs what is the effect of each of these knobs on the final result such that we can now know how to slightly adjust each
knob to get closer to the actual number that we're trying to get to. >> And PyTorch is traditional software. It is a piece of software that kind of manages this entire pipeline for us.
>> Exactly. Right. PyTorch is tracking the effect of this knob on ultimately on the outcome of the prediction. And so at the end of the day it will tell us how much a change in this knob will affect the
loss. And once we know how much what what is the effect of this parameter on the final loss we can then inch it either in a positive direction or negative direction right we can add 0.1
or subtract 0.1 um and that will get us closer to the actual prediction. And so this is and we do that for every single parameter. So every parameter, every knob is moved slightly such that uh if
you were to run the inputs again, you would get closer to the right answer the next time. And >> and it's it's not guessing at this stuff. It is it actually has to run the
calculation and figure out what ho how the different knob turns will affect the final output. How will how will it it will affect the loss, >> right? And u the way it does this is is
ultimately it uses the the chain rule from calculus. I think it's probably we shouldn't get into this and there are videos that can describe this. Um >> yeah, a little too deep.
>> I think it's good to just take for granted take for granted that PyTorch will basically give you this information. Uh so you Yeah. >> No, I'm sorry I cut you off. Please
finish that. >> No. So, so if we take for granted that PyTorch will give us this information, what we have, we have the amount we were off by given our current knobs and then
we know that if we just adjust the knobs um according to how much PyTorch says they will affect the loss. So in the right direction. So if if this knob if we increase this knob the loss will go
down, we should increase it. And if we decre and vice versa, right? If if if the loss if PyTorch tells us, oh actually uh when this when this gets larger, the loss grows as in our
inaccuracy grows. Oh well, let's make it a bit smaller. And the amount that we inch up or down by is determined by the learning rate. >> And somebody who's just learning this
for the first time doesn't necessarily need to know what's happening under the hood of PyTorch. >> Totally. Uh it does it automatically. And I'll actually I I have some code
very simple code that will show you the few lines the only lines where where this is all done automatically for you. And uh what this looks like in practice I just have a quick so I'm going to hit
autotrain here. And as you can see it's uh automatically updating. Let's reset autotrain. So it's automatically adjusting those knobs right so that we are minimizing the
amount that we are off by. And so it looks like >> possibly the best we're going to get is around being off by you know uh 0.07 million. Um, and that's that's actually
because we only have a simple linear really uh uh function here, right? This is a very simple function. But if you add more layers to your neural network, as in you have functions inside
functions inside functions inside functions each with their own knobs and we have activation functions which change it so that you can uh represent more complicated relationships we won't
which we won't get into. You can actually um you can represent any function with a neural network uh which is a really cool property. It's called a universal function approximator because
you just add more layers, you add more knobs and you can get to any function. So in theory we can actually get this loss to zero if we uh added a ton of knobs and a ton of layers.
>> Awesome. and and and you're like what you have just described is the basic unit of all AI today. Yes. Is that >> fair? Exactly. This is called back propagation. Yes. This is how we are
able to um update our knobs in a directed fashion to minimize the difference between um what is actually in the world that we want to model and what our model gives us. Now the
important ingredients going into this is you need the inputs. You actually need to know, okay, where's the home located? What's the size? How many baths? And then you also need to know the real
world answer. What did that apartment actually sell for? Yes. Exactly. Right. And um you know there are various uh there's a lot to cover in this. I think for now um yeah this is this is the
intuition behind it. Um and we could of course go deeper. And the as a final point to this, I'm going to show some code very simply that actually does this in PyTorch. So um you don't really you
don't need to know I I don't want people you know no one should be scared off by this code. This is just turning these inputs into numbers. Uh these are the labels. So this is the the home prices.
And then this is the training. This is it. This is the training. And so uh here is our prediction. So we run uh we run the model. We we run the the the function. We calculate what the
predictions are. This is our loss that we were just describing. This is how off we were. And now all we this uh means that we're just uh zero uh we're starting a new step. So uh forget
everything that we've um we've done before. We're going to lossbackwards which means uh we're going to calculate for each of the parameters what is their effect on the loss and then we are going
to take a step according to our learning rate which says given that now we know the directions that we should be um making parameters larger or smaller and the amount let's do that and now we can
test and so I'll just run this quickly again you really don't need to know what's happening but this is just an example of literally it's it's this much this is what pietorch gives us um this
would be a lot of code >> it If and if this seems a little bit complicated, what's really nice is Claude and Chat GPT, these systems are extremely good at
writing and explaining code. So you can literally have it write this code for you, adjust it for you, explain each line in detail. So if you want to go a little bit deeper, that's a great way.
>> Yes, definitely. Yes. So this ran and here is what we are seeing. Again, this is not how you would want to train this in production. This is just a toy example. And so for each of our
optimization steps, right? Each tuning of the knob. And actually in this case, we are taking training steps. So a thousand steps, a thousand tunings. Um, and we are getting results for every 100
just so that we don't have a thousand outputs. So at step zero, right, randomly initialized knobs, our loss is is eight. Um, and our prediction for an upper west side. And this loss uh is
using a mean squared error. Um, the larger the loss, the worse it is. Um it is basically taking the literal difference in prices and and turning into uh you know a loss which we won't
get into here but but the larger it is the the worse the prediction was. >> And so how does it how does it know to head in the right direction? So like it could just test a bunch of those in uh a
bunch of the knobs randomly but it seems like it's kind of you know heading towards the correct answer. How does it know to continue towards that correct answer? Yeah. So um when we do loss
dobackwards what PyTorch is doing is it's saying for each parameter I know where it was involved in calculating the final result and I know how off the final result was
from uh what I wanted to get right which is actually calculated here. So here's the predictions what we actually said here is minus the actual labels. uh we're going to square it, take the mean,
uh which is which is generally how we I won't get into this, but this is um a way of turning those into, you know, uh uh numbers that are are are useful for PyTorch. And so what this loss aggressor
is saying is for each of these parameters, what effect if I were to make them larger, if I were to increase the knob, what effect would they have on the loss? Would it make the loss larger
or would it make the loss smaller? And uh now that we've calculated this so what actually happens in each of these parameters which is a compute graph we have a dotgrad which tells it the effect
of that parameter moving up on the loss and we do this step which then nudges that parameter either makes it larger or smaller uh according to the direction of what is called the gradient but really
in terms um we are making it smaller or larger to decrease the loss if we were to run this So um we are nudging it in the direction that will minimize the loss. And again
it knows that because um it knows how each parameter how each knob is related to the final loss. It's kept track of this the whole time. And so it can tell yeah if we make it larger what is the
effect on the loss. We've now uh kind of have have an overview of how a general non- language model neural network works which is you have specific inputs they are of specific you know numbers and
then they output a specific number and we can basically calculate given this output how how close are we to the expected output. Language models are totally different beasts. they are much
more general. But what's interesting is we can take the same document, the same data and we can uh build a language model that will also predict the price but we do it with language. And so the
inputs oops the inputs are now the actual words on this document. Um really in reality they're tokens which are fractions of words or or multiple words put together but for now simplified
let's just say they are words. And so the model the training data as input will be um words as a sequence. So at first it will just use Soho it'll predict this comma soho comma it should
predict 1200 soho >> and so sorry real quick. So, we're looking at the same data, right? Soho, 1200, two bath, which before were individual inputs, but now we're looking
at it as just like pure text, which it's just like here's a string of text. Let's continue and look what the next potential token or word is. >> Yes. Yeah. And fundamentally at a high
level neural networks are finding patterns in your data and they are really becoming your data. So the idea is they are learning to output given the data that it's already seen and the
patterns it has ingested what is the most likely next word such that I am being faithful to the data that you presented me with. So so in essence neural networks really are just the data
that you are putting into it. Right? So if we only feed it this type of document and let's say this document is a million lines right um it's it knows how to only make sense of these types of words but
it should at the end of the day actually understand the patterns in the data such that it can also output um a next word which in this case will be the prediction for the uh price of the of
the unit >> and and Jeremy when we're so the previous model we're running is very different from what a large language model is. Uh
first can you remind me what the previous model is called? >> The previous model uh was just a uh neural network. >> Okay. Can neural networks or are neural
networks more accurate when you're looking for a very specific answer versus like a textbased answer? >> Right. So the real answer is with enough data in compute they should converge to
the same thing which is finding the correct function for to map your data and finding the patterns. So um in the limit these should converge. However with finite resources if there's a
problem where you're just wanting to output the price of something you would uh certainly prefer to just have a normal uh neural network. A language model is is a you know in the subgroup
of of neural networks like you know a language model is a deep neural network but um the point is you know neural networks will get the gist of the thing. These are two flavors of neural networks
but fundamentally what is happening under the hood is knobs are adjusting to map inputs to outputs and uh if you do it for a long enough time with you know the same data they should converge to
the same answer. Now, as you can tell, language is a whole new beast, right? Because there are so many more degrees of freedom with language than there are with numbers or, you know, with specific
numbers that mean a certain thing. So, uh, compute for compute, you will always prefer to do just a normal neural network. Um, but the the underlying, you know, the the truth is they will both
converge to the the correct answer, which is to find the function, the the correct function, the correct pattern from from input to output. And because there's so much more flexibility with a
language model, that makes it take much more data, much more input data to converge to the same level of accuracy as a as kind of the previous neural network we were just looking at would
be, right? >> That's generally correct. Yes. >> Okay. >> Um, yeah, there's a there's a lot of nuance with what you said, but but
absolutely right. There are so many degrees of freedom. There are so many complicated relationships now that we're in language space that right um yeah you're going to need a lot more
parameters and a lot more compute uh to to make sure that it really grasps the pattern. >> Cool. >> So um yeah but ultimately you know
language models are are just a new flavor of neural networks but much more general. And so what's really cool is okay we care about predicting the next price. Sure, we can use a language model
for it, but the the beauty is we can dump any text, train any uh uh language model on any amount of text, like the entire internet, and now it'll soak up the patterns of the internet, and
ultimately, if you train it well, it will be able to predict the next word as an autocomplete of a good of a given text, and it will reflect the data that you gave it. So um if you train it on
the entire internet and then you ask it to just you know start uh generating next words it will generate the most likely words given the input which is the internet and so it is finding all of
these patterns and it is really what its data is ultimately and uh I think you know one uh obvious question is well it was easy to calculate the loss the the how off we were when we were just
outputting numbers at the end but with language you're outputting a word at the end. How are we supposed to compare 1.2 million to 1.5 million? These are two words. Well, um the trick is language
models don't output the final uh layer of the of a neural network when we're predicting just the price is just one. It's just the price. It's the number. With language models, you actually um
the final layer are the number of words that you could possibly predict next. And each of them is assigned a probability of how likely the model thinks that token that next word should
be. And now you can compare the probabilities. So let's say the right answer is 1.2 million. And uh as we're training the language model thinks this is only 10% likely. But the real answer
is it's 100% likely because it actually happened for the the the real world says 1 1.2 million is the right answer. So now we tune our knobs to turn what was 10% likely, we want to inch it towards
100%. So we want to make this more likely, 1.1 1.2 million more likely, slightly. We're going to nudge all of the input knobs to make 1.2 just a bit more likely, which by definition makes
everything else a bit less likely. And this is how we're training these neural networks. Here is a real example of using Quen to uh 0.5B which is 500 million um uh knobs numbers.
And we have two models here. We're going to take a base model which is a pre-trained model from Quen and we're going to start with this. And so what the base model says is again we've taken
all of the internet and we um now have this model and we can now generate completions from it. We have now created a complete an autocomplete from the internet. So all it knows to do is is
complete texts that are similar to those that it has seen in in training. So we can actually just take it for a spin here. This should already be loaded. Cool. Okay. So what I'm doing here is
just two examples. I'm giving it an input of zero. So it's basically saying I'm asking it and then I'm asking it to generate 100 new words. And so what we would expect is it's going to generate
words that it's uh that are in what it's been trained on. It thinks it's most likely randomly. So we can tell that this model Quen has been trained on a lot of code because the first thing it
outputs are code, right? Just autocompleting from the internet. And then the second thing is this weird Chinese uh thing. So it's obviously also been trained on text that looks like
this. So we can it's safe to assume that a lot of the training data in the Quen model that it's been trained on this base model looks like this. Now it's autocomplete. So I can actually say
Matthew Berman said and now uh it's going to generate 100 tokens that follow set and we get to just see what's in this weird mind of this model that has seen basically all of the internet.
Matthew Burman said in his interview that a good book should be a a a good story with main character. I don't think you said this. >> I don't remember saying that.
>> I don't think you said anything. But the point is this is a an autocomplete. It's it's it's repeating what it thinks what given all of the inputs that it's been trained on. What are likely things that
should follow? And that is really all that this is doing. But it's fun to play around with. So again, it's autocomplete. You can put anything you want in here and then it will output
something interesting. And if you remember, yeah, >> real quick, Jeremy, so I think one interesting thing to point out is that this Quen model was trained probably a
while ago and maybe or the reason why it's making something up about what I said is because it has no information about it. Nothing in the training data told it what I have said. Now, if as
part of the training data, it took some of my YouTube videos, transcribed them, and put them in the training data, it would know what I said. That is true. However, I'll note that a pre-trained
model might not have been trained to output necessarily things that are coherent or correct or not hallucinated, but it will definitely sound more like you if it has been trained on data that
you've actually said. it it might not give verbatim things that you've said but so it's definitely been trained on Einstein right and this these might be uh correct quotes but also what you'll
see is this is a bit incoherent right the answer to this question is in the text my understanding is this is like incoherent right this is this is autocomplete uh it's just it's just
producing what it thinks is most likely but this is actually not very useful it might be useful you know for this amount maybe if you want to write an email it's seen a lot of emails it can give you the
gist of what a good email might Um but this is also not very useful. So if you remember you know the pre- chatgpt days uh generally these were were base models and the base models
aren't trained to be helpful. They aren't trained to necessarily tell the truth. They are just trained to autocomplete whatever is given to them. And so this is pre-training.
pre-training is okay, we have this incredible neural network where we can um we can train it on anything we want, any text we want, and it will have the gist, the patterns of this text, and it
will predict things that it thinks are likely given the text that it's been trained on. Um with no guarantee that it's correct. Um it's just what's what's more likely? What's what's likely?
What's likely? What's likely? Uh given the text that it's been trained on. So um the question is how do you turn this into a chatbot or something useful? So one idea could be well what if we
actually actually if I say like user what is 1 + 1 assistant. So people figured out that because it's just autocomplete, what we can do is we can actually like figure
out interesting things to prefill it with such that then it starts out outputting completing more of what we want it to. So what I just did is I kind of tricked it. I said okay there's a
this is a text we want you to complete. A user says what is oneplus 1 then the assistant colon and we're kind of tricking it into becoming an assistant, right? It's taking all of what it knows
about assistants online and then it's going to autocomplete it. And so user, what is 1 plus 1? Assistant 1 plus 1 equals 2. Okay, so now you know it goes on to something incoherent, but okay, we
actually have an answer and this is really cool. We've just turned this um you know this autocomplete engine into something that could possibly be useful. And this is something that researchers
found. And so they're thinking, wait, okay, so if we come up with some sort of um format, we can trick the model into being an assistant by basically like having complete and if it were to
complete the document, it would do what we want. And so here, this is the actual Quen 2 formatting. So they come came up with something called M start, im start. And we can actually
just run this and it will autocomplete as an assistant, but it hasn't been trained to know when to stop. So, it's just going to continue and continue. Um, it's not
necessarily a helpful assistant, but it is now assuming the role at least right here of the assistant. Like, this is a correct answer. >> And when you say continue and continue,
it's literally just indefinitely predicting whatever the next token is, whatever the next word is, >> right? And this is I I gave it 200 tokens. So for the next 200 tokens,
predict what you think is the right thing to do. >> Right. And it doesn't >> if you said 10,000, it would continue till that number as well. Okay. Got it.
>> Yes. Yes. And as with language models, uh remember it's predicting it's predicting a sequence. So for each new word, it it um adds that word to the sequence and then predicts the next one.
Adds that next word to the sequence and then predicts the next one. So, we've noticed this. Researchers noticed this and they're like, "Okay, so we can now trick it into thinking it's
an assistant." So, what if we just create a bunch of data for how we want the assistant to behave with our formatting and then the model again which will will learn the formatting
that we're telling it to learn and it will become an assistant. And so this is what we do. And this is the start of post- training, which is we're taking this base model, which is kind of a
model of the world. It can assume all of these various personas. It doesn't know it's an assistant. It it hasn't been trained to be an assistant. And we're we're filling it with a bunch of data.
Maybe we start by having humans literally like write user queries and then write assistant queries. And then we're training on it. And then we're going to say we we generate um a 100,000
examples of uh an assistant autocompleting. And we're also going to include in these examples a stop token, which means this is when the assistant stops. And now the assistant learns,
okay, my role, it's starting to gain an ego here. My role is an assistant. And when I'm done responding to the user in a good way as seen in the data, I should emit a stop token. And when the program
sees the stop token, it stops generating. And so now I'm going to use the instruction model which has been trained with this data to know that it's an assistant. It now has an ego and it
should know how to stop. And as you can see here uh why is the sky blue? The sky appears to be blue when sunlight however when light so when we look at the sky the clouds different colors and refract
options in the air. Great. So this is a pretty pretty good response. And so comparing this this response to this response is really the difference between an instruct model and a
pre-trained or base model. And this probably in the quen 2 series was mostly really done with uh generating a bunch of data of good assistant messages with this formatting. And the model learns to
uh learns the patterns and learns to be an assistant. And what is key here is every message. So this is what you might be familiar with when you actually like write to a language model, right? You
have a ro user contentwise sky blue. Under the hood, the model is actually just generating this format and giving it to the model. So it's trickery, right? Like uh you are just prefilling
the model with what it's things that it's been trained on and then it can respond in a similar way. So these are this and this are equivalent. These are the exact same.
Um, so we've turned this autocomp completion engine into an autocomp completion engine that now understands has been trained to see a lot of assistant messages and knows uh now what
good assistants sound sound like and can do it. Um, and if you ever want to see what is actually being sent to the model, the actual tokens that the model is taking as input, you can just um you
can use the tokenizer, you can input the messages and it will give you literally what the what the model sees. So with the when you write why is the sky blue what's what the model's really saying is
a system by default message and then it is why is the sky blue and it starts the assistant so it knows to complete after assistant >> and I just want to interject if people
want to try this it is effectively free to do what you're doing you can use Google collab for free quen is an open weights model you can download for free um So, uh, and we'll drop all the links
to all this stuff down below. >> Yep. This is all free. I don't have a special plan. I'm using a T4, uh, for free on Google Collab. >> Cool. Yep. Uh, okay. So, this is the
start of post training, which is really exciting. Uh, we have an assistant model and this is, you know, GPT3. There are a lot of things I'm glossing over, but this is the general gist is we've turned
an autocomplete engine and we've tricked it with more data into thinking that it's an assistant and it's now developed this assistant persona. And um this is really cool and interesting. But um now
there's kind of a separate problem which is the models are uh you know can produce interesting things but they are uh making obvious errors right if you asked it I don't know if you remember
asking it math right it will seem very confident it will return to you the answer without thinking right so here's an example uh what is you know 398 * 923 plus 2901 right um and the result of
multiplication of this is is uh 37,000 which is wrong. The right answer is 370,000. So this model totally got it wrong, right? And the um but it it wasn't assistant, right? It generated it
correctly. It knew knew to stop. Um so now there's this new issue which is the models are just blurting out whatever they they think, right? Uh they're not actually thinking. And when if I were to
ask you this, you would, you know, think about it, think step by step, actually solve it. um you know thinking and so um one thing that researchers discovered is maybe all you need to do is first
prefill the assistant with let's think step by step and then ask it again and then all of a sudden you actually don't necessarily always get the right answer but it is now actually thinking step by
step and it's getting closer. So now, yeah, >> by the way, the discovery of let's think step by step seems like very obvious, but it was kind of a
recent discovery. It was, wasn't it post chat GPT becoming really publicly available, publicly known that people figured out, oh, literally just ask it to think step by step and you get better
answers? Cuz I remember I when I was making videos in the early days of model reviews, I would manually type think step by step to get better answers from the models.
>> Yeah, this um it's unclear what was happening at the labs at this time, but in public this is definitely true. And there are multiple ways to do it. There's one to like first force the
assistant to start with this text, which is what we're doing here. There's also one where you prompt it to think step by step, which also yields better results. And yes, this is relatively recent. Um,
again, I think probably the labs had figured out pretty quickly, but I I agree publicly this was this was relatively new. And now all of a sudden, we're getting quite close. Um, okay.
Now, I'm going to transition >> and and the first public publicly available model to use this new thinking was 01. Is that right?
>> Right. Uh, it was 01. Yes. Um, >> by OpenAI. Yes. So, uh you know we were living in this non-thinking paradigm and part of why the non-thinking was the case is because on the internet uh you
get a lot of the outcomes of thinking right you see blog posts which had a lot of thinking but uh you know uh by humans to build the blog post but you don't see their thinking with code right you see
the final finished code but you don't see the thinking all the thinking that came into writing these these pieces of code and so the internet generally lacks thinking data and and not only that even
if it had thinking data This is the model learning to think but not for itself. Kind of imitating what it thinks thinking should be. Now in the limit if you had enough of this data and a large
enough model it's very possible it would you know grasp fundamentally the patterns of thinking and you could tell it to think like Einstein and it would think for itself. But um practically
what we found is that um we can actually train the model to think for itself to actually try out problems and learn from trial and error which is called reinforcement learning and this is what
led to 01. Yeah. So 01 was the first public model that that had thinking and it was this I remember when it came out it was just incredible. It was doing really well on things like arc. It could
code uh you know much better. It was just a a whole new way of being able to use these language models. It was hallucinating less. Um, and we didn't really know how it worked publicly until
Deepseek came out with R1, the famous DeepSeek uh R1 paper, which showed how you can do this with reinforcement learning. So, I'm going to gloss over quickly uh around the the next stage,
which is using uh reinforcement learning to train the model to think for itself and and to uh through trial and error update itself. So um how uh deepseek was able to train this model I think for
itself is called uh gpo um is the method they use group group relative policy optimization but what this really means is okay so we have this math problem right um let's have the model try to
answer this math problem a 100 times and actually let me take a step back maybe the naive approach is let's have the model try and answer a problem 100 times and let's tell it to think step by step
and every time it gets it right. Let's add this to the training data and train on this. So reinforce the times when it thought well and we'll just throw out the bad examples. So this is a naive
approach which actually does work. It's called rejection sampling. There are a few problems with this. One is that you don't learn from negative examples. And then two um there's a lot of noise in
every roll out in every response right there's ofs random tokens that don't actually matter. They're kind of noisy but you're also upvoting them. again you're training on them and
go is a way of learning from negative examples as well and also with scale canceling out a lot of the noise so that you have this stable training of the model trying out problems for itself and
learning. So the way it works, the model um answers a problem a 100 times and then you score each problem based on how well it did. One, how well it followed, you know, your desired formatting for
thinking and then two, whether it got to the right answer and then you um you take the average score of all of these groups and then you subtract its score from the average and this is called the
advantage. This is basically how much this uh solution was better than the average solution the model was producing. And then >> and Jeremy just to just to clarify this
is all within the category of post- training. This is not happening within kind of what people think of chain of thought or kind of the thinking aspect of the model. This is separate.
>> Yeah. This is this is training the model to learn to think on its own from from its own synthetic data. Yeah. >> Got it. And so um with this now each each time you answer a problem you know
its relative score how well was it relative to if it you know answered 100 times and then you use this number to update the model to be more or less like that response over time. So for every
token in a in a role in an answer that was really good. So we have a really good answer right? It was like a one of one. It it was it was great answer. uh we make every token in that response
more likely based on how good it was relative. And same thing if it's a bad example, we make every token in that uh roll out in that in that answer a bit less likely. And so we are just tuning
the dials to shift the model to make it more likely to answer correctly and less likely to answer incorrectly. But if you think about this on a high level ultimately right this is just changing
the data that we are using to train these models to be more correct and it this actually has an added benefit. So this is called on policy training. Policy is the model and this is the
model um generating its own training data on policy and reinforcing. And this is um in many ways better than taking someone else's thinking and updating on someone
else's thinking because the model generated this this um answer itself with its own circuits. And so if you want to make this answer more likely, well the you know the model's already
capable of answering like this. So you can just twist the knobs just a little bit to make it just a little bit more likely to output this token where next word um versus if it's someone else's
thinking you are making all of that other person's thinking more likely but you don't know what um words you're making less likely by doing this. um you might start to forget or or degrade in
other capabilities in a way that with reinforcement learning uh you don't as much because the updates are generally nicer when you are they are um you are being less destructive to the knobs when
it is your own thinking because you generated it you just need to tweak it slightly. Um so this is uh this was a uh you know breakthrough um and it allowed the models fundamentally to learn from
trial and error and to teach itself uh new things. And now we're in this regime where if you can build an environment such that you can have the the model um answer many times and you can verify
whether the answers are correct then you can build this regime and the model will start to hill climb. will start to improve on this task with limited degragation or degradation to um the
other things that the model knows. Of course, there's still degradation. Yeah. >> Well, uh Jeremy, I wanted to ask you what why are these pre-training and post-training phases separate? Because
it sounds like reinforcement learning is so powerful. Why can't that just be done from the beginning? and and like maybe you can just kind of from a very high level explain why there
needs to be two phases, >> right? So there are papers that try and merge the two and I think it is totally possible that they they will be merged and I'm sure many people are working on
this but I think maybe just uh on a high level fundamentally when you're pre-training you're taking off policy traces as in you are taking the world's data right and you are building a next
token a next word predictor from the world's data once you give the model an ego a personality now if you give it data it will start to
imitate and become that thing. And so you can't if you were to then go back to pre-training data, let's say, oh, I have this really great code corpus uh set of code and I want to train the model after
it's already been uh assistant uh post-trained. What will happen is the model will start to adopt the style of the data that it's training on. And so it will start to lose the ability to be
an assistant. It will start to respond in code maybe. and then it will also take it for granted. Uh so once you have to be very careful that you are making sure that once you give
it an ego, you give it the assistant model, you really can't at least uh you know publicly you there's no um it is difficult to then train it on data that it did not produce. Um so yeah, if you
were to train it on a code corpus, it would start to then talk like code. It would start to take that code for granted. Um but there are a lot of interesting um there are a lot of
interesting experiments going on to yeah to to uh build in reinforcement learning to all all parts of the stack. >> Very cool. Finally, I'll just end with um you know, I think for for a long
time, first of all, we're in a very interesting moment, which is um language models, you know, to my knowledge, haven't come up with, you know, super important novel discoveries that have
been worldchanging at this moment, but they are now starting to crack unsolved human math problems, for example. And one of the main criticisms of language models is well if they're just next word
predictions and they've been trained on a distribution and by definition there's nothing new in this distribution because we've taken things that we've you know already know right uh to train these
models. You know are they really going to be fundamentally capable of escaping the model that they've been trained on and developing new uh new knowledge? I think um you know the answer to this is
yes because ultimately reasoning is the process of making these unlikely true and interesting things more likely. And so if you think you know Einstein um when he's coming up with general
relativity at first the idea of general relativity wasn't a likely thing for him to just come up with because he hadn't thought through and built up built all the assumptions into abstractions to be
able to deduce general relativity. But it was over time through reasoning that he was able to mold the you know his thinking and ultimately the data itself that he's been trained on such that he
was able to get there and um models I think are the same way right uh as long as we continue to train them to reason well over long horizons I don't think there's a reason why they shouldn't be
able to develop new knowledge and um escape the distribution because fundamentally when they are reasoning for themselves and we are now teaching them to reason for themselves Um what
this means is they are slowly building new abstractions on the fly until they're able to do deduce new things. So um you know time will tell but my money is on on the fact that that models will
continue to get much better and they will end up um discovering new things. >> Well Jeremy I want to just say thank you so much for sharing all this. Um I I learned a lot. It it really shows that
it, you know, it seems a little bit intimidating, but if you start from the fundamentals and you build off of that, you can really learn it. You can apply it. You can experiment with this stuff
yourself. And if you want to make this into a career, you can absolutely do that. It does not require a PhD. Certainly a lot of work, a lot of hard work, putting your head down, learning
it, but it is very, very possible. So, I just want to say thank you again. >> No, thank you.