We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

0:12 Hey um hi everyone. Uh thanks for being here. Uh yeah, I'm super happy today to talk about uh automated eye research and uh especially uh all those like frontier model uh perform at uh automated eye

0:27 research task. Um so I'm Elie. I worked at Prime Intellect as a research engineer and uh yeah I will go through our work on on this subject. So first I want to basically explain a bit why we

0:40 are doing that and why we think it's super important to do that in the open. Um so first uh I think we we all agree that uh we've heard about like big labs saying that this bad thing called

0:55 recursive self-improvement is coming very soon. Uh so recursive self-improvement is like model training models uh without uh human intervention basically. Um but uh we don't have any

1:09 benchmark to basically quantify if this is true or not right. Uh and even less we don't have like a third party benchmark by non- big labs to to to see if it's something coming soon or not.

1:23 And the other part is that we think that uh it's super important to understand all those model uh do research because we think that a lot of the scientific research that will come into the coming

1:35 years uh will be based also on AI tools. So it's super important to understand how those model do research not just only AI research. So we try to build kind of this environment to test the

1:50 capabilities of the model to do so. So it all started with uh Andre Karpati uh that's basically had fun by doing this video where he trained uh GPT2 from scratch in like 90 minutes like GPT2

2:07 training takes like weeks and uh no in two years ago I think it only took like 90 minutes. So what does it mean to reprod reproduce uh GPT2 in 90 minutes? It means that in 90 minutes you achieve

2:20 this target loss. Um and yeah and that's at this point when you have the same loss than GPT2 you consider that your model is somewhat of equal performance.

2:34 Um then what happened is that the community took this repo uh this GitHub repo and create another one called moed nano GPT and this effort was leaded by someone

2:46 called Keller Jordan. And what happened is that they basically took this 90 minutes then 45 minutes and then now we can train like GPT2 validation loss model in less than two

2:59 minutes which is honestly crazy and it took like two years to to achieve this. So it's a very strong benchmark where uh a lot of very talented researcher iterated on um yeah so we decided to

3:13 take this environment of speedun so it's kind of a game so the goal of the game is to achieve this loss in the fewest in the shortest amount of time so this is the nano GPT1 and you can uh you

3:29 don't have almost any constraints the only constraint that you shots that you need to use the same validation and training data, right? Um there is a new speedrun called the optimizer speedrun

3:41 that was released uh a few months ago and here it's slightly different because uh you can only change the optimizer related parameters. So for instance nano GPT you can change the architecture uh

3:55 do uh attention whatever uh optimizer sp you can only change like Adam to m shampoo or whatever optimizer is your favorite

4:08 um yeah and so this is a bit more researchy because uh it's less about optimizing the program to be as fast as possible but more like finding the best method possible. no matter the the the

4:22 time you put into the computer, right? So, um yeah, why take speedrun as an environment for automated AI research? First, uh we think that it's a good evaluation. We'll see later why. Uh and

4:38 this is kind of the main focus of this talk. But we also think it's probably a good training environment because uh it's a way to give the model a reward. So the reward is positive if the model

4:50 bit the speed run and beat the last record sorry and the reward is zero or negative if it didn't manage to to do it. So it's a good uh environment to train model. It's also quite fast like

5:02 as you see uh previous record were around 2 minutes for the optimizer one. uh each run take about like 15 to 20 minutes and uh yeah and there is like clear rules basically and we also think

5:17 it's like a good environment to make discovery so like kind of breakthrough in our research because uh there is those clear rule that you can verify or not. Um yeah. So yeah.

5:32 Um so what we did uh so the release was like about two months ago and uh there was this optimizer speedrun and we decided to basically compete with the community by launching two AI agents. So

5:47 Codex and Cloud Code. Codex was like GPT 5.5 with XI and uh cloud code was Opus 4.8 with XI. Um and yeah, we decided to basically let the agent free on our cluster uh and uh and just iterate on

6:04 it. So we have like V1, V2, V3 is just basically us stopping the agent and then restarting. V3 uh was like one or two day before the release because we saw that our agents no longer have the best

6:18 record. So we were like okay take all the the human uh record in the last few week and just try to to improve upon it and and and it worked. Yeah. And we also have this novelty track where the goal

6:31 is to uh beat the record with only novel ideas. Um and we'll see that this this was more complex for the the models. So our RS is very simple. Honestly, we could have just replaced it with

6:48 slashgo, but they there was no SLG goal at the time. So, we made our own goalm. It's actually quite fun that we choose the same name and we had the goal. MD and kind of agents that MD that define

7:00 the rules and we let the agent propose ids and then it can submit a job with sbatch on our slum cluster and uh basically the way it works is that it can submit on nodes that are available

7:16 but only under a certain permission which means that if someone want to use this node uh the model just like cancel the job it's called preemptable permission. So yeah, then it measure the

7:28 it read basically the training logs then decide if it's a record or not. To validate a record you need to basically pass a statistical threshold to make sure that it's just not see the

7:38 optimization and is just not random. Right? So yeah a few results from this experiment. The first one that was honestly very painful to work with is

7:49 that code clothes code keep stopping every nine or 10 hours and basically said yeah I cannot improve the record it's too hard for me there is no way to to go beyond it and then I was just like

8:03 okay continue explore new direction hey just go again for 10 hours and then say yeah I cannot beat the recall and so on. So basically onethird of the time the cloud code agent was idle because I had

8:16 no way to basically monitor it and codeex totally the opposite just worked for all the all the time and uh yeah almost never idle never asked for question and and and very impressive in

8:30 that way. Um we also give the option for the model to basically write uh a bunch of stuff into what we call a scratch pad which is basically the active memory of the

8:43 model. Uh we observe that basically codeex writes a lot on the scratch patch. So each plot that I will show are kind of normalized by the number of active order. So this is not only about

8:57 codex working more it's it's really different behavior. So yeah, you see that uh writes a lot more to to this scratch pad to this memory and uh the shape of the like the

9:10 the I don't know the tone of the the each file was also super different like CL was super excited about getting new record with a bunch of emoji and so on and CEX was just like here is what I do

9:24 here is the decision I take what I will do next like super robotic kind of um Yeah, we also have this plot where basically we saw that codex was spawning much more sub aents than cloud. Uh we

9:39 saw that codex burn much more token than code. So I think in total it was like kind billion of token but it's like there is obviously this input tok uh input caching that make it it's not like

9:54 one billion output token. Uh so yeah we also see that codex did a lot of compaction because it only had like 250k context window and cloud only do it like one per hour and codex is more like

10:08 no it's even less than one power for I mean one for the full run for code and codex was like one uh was 20 every one hour. So yeah um yeah here is the main results. So

10:23 what this plot shows is that basically we so in in white you see that the human record progression right and in red you see cloud I mean it's supposed to be orange but whatever and in blue uh you

10:40 see codeex right and you see that at almost every time uh cloud and codex are better than the human record and code is super good at the beginning very very fast to achieve very good

10:53 Um yeah and one thing that is super important is that the model have the ability to basically fetch the human records at any time and that's what codex did that's what cloud did sorry

11:05 because when I restarted it it basically fetch the new record from human and improve upon it um yeah so the result is that uh I think at the time the best uh record was like uh 2,990

11:22 step and we beat it by like uh uh 50 or 60 step for code and codeex was like 20 step above. So I think it's both impressive and and yeah um so we so this is like not released yet.

11:38 This is something that we are working on currently and basically the idea is that this is a cool experiment to do but it lack of structured right. uh if you want to do a real benchmark, you want to do

11:50 multiple seed, you want to do uh yeah proper uh thing where you you you you basically put all the model and earnestness in the same condition, right? So this is what we are working on

12:02 right now and basically um the idea is to do three different track uh one without any access to really like measure the capability of the models to do AI research based on only the model

12:16 weight knowledge one with only archive paper and one with like full access. So it also have access to the the like the latest record by human. And for this we plan to do both uh the nano GPT track

12:31 one which is the original one and the optimizer speedun where we we only launch uh we only constrain the the optimizer to be to be novel basically. Um yeah so I will present some result on

12:45 the optimizer speedrun. Uh this is basically what we got. So we let the agent iterate for six day almost five days let's say and we see that uh codeex k and clothes uh are super effective so

13:02 for GLM this is not finished run right so the model is actually still iterating on the cluster right now but we see that cloud is once again very good at it and we see that surprisingly Kim is also

13:15 very competitive and kind of have this breakthrough on day four where he kind of beat Codex with a new record, right? It's also interesting to see that uh Claude is much more like progressive in

13:30 the way it improved the record and Kim has really this step function where I kind of do a breakthrough and so on. Uh so this is an interesting plot because I mean six day is quite a lot for anal uh

13:43 uh but you you can change this uh axis by also the number of output token and then kind of tell a different story because in max mode consumes so much more token than codeex and kimmy and you

13:57 also see that Kimmy is actually super efficient uh for the number of token that uh it uses. So it's scheming K2.7 code. Um so yeah uh we also see that they have a different approach to uh

14:13 using the literature and papers. Um so for instance like code is doing a lot of search on papers and actually include found a paper that no other model found and it actually lead to the best record.

14:27 So it's kind of funny and uh yeah um one of the main issue of all of this is that uh when I when I launched this this agent and I think that's something important that I want you to to kind of

14:44 uh remember for this co this talk is that when I launched this these different agents I was expecting them to come up with some crazy ideas on optimizer that's like no one of discover

14:56 but honestly it wasn't the case. Uh they did some clever trick where basically they combine different papers. uh they kind of do plus one improvement over a bunch of method but there was really

15:08 like no novel optimizer or mechanism that was uh coming from those model and I think that's kind of telling that even on something that is not simple but I'd say that it's kind of accessible for

15:23 people right for like human researcher uh spending like days and weeks for the the model like cannot like find new uh optimizer and mechanism. So we believe that there is a way to basically make it

15:41 more um make it better for discovery instead of evaluation and this is coming from uh this is very inspired from alpha evolve by Google and also a bunch of papers that have been released since

15:53 then. It's kind of this multi- aent system that interact together uh bunch of generator. You have closed model but you also have open source model here that are super effective for the cost

16:05 right. Uh they can suggest ideas then you run the speedrun so you get the reward then you have a judge that basically give a quality feedback can also be like the judge also have this

16:17 taste. you can kind of have like the judge have a taste about the the method if it's good or not uh if it's outside the loop and then you can uh basically decide which method you want to scale to

16:33 a larger number of parameters and number of token. Um so this is kind of the scale part of the speedrun because some a lot of method in the the speedrun community uh people are often saying

16:47 that they doesn't work at large scale. So I think it's very important to also put scale elements in this loop. Uh and I think also that uh human are super useful here to basically judge the ID of

17:01 agents kind of steer them in the right direction and so on. Um yeah so we didn't try it yet I mean we are kind of trying it right now and uh we hope that this will lead to to to new discovery in

17:15 AI research at least and also a way is that you can define multiple speedrun so this is the next slide if you like it's from safe bank slides but if you if you don't have the reference good for you

17:29 means that that you're not too online uh but the idea is that uh by changing the object objective and the constraints of the speedrun you can basically create a lot of diversity and constrain the model

17:41 to go into a certain direction and uh yeah and make those discovery. So uh at HIM in Ind. Uh there is a bunch of stuff here that we I mean most of it we didn't release

17:55 yet but we are working on GPU sandboxing to allow model to iterate into sandbox because you need GPU sandbox for this kind of stuff. We are working on our own agents that are very efficient for like

18:12 RM framework. So it means like you have a file system and you can write information read from it. Uh and you also do like this programmatic tool coding thing. We also training a model

18:22 to be good at it on top of like open source model. And uh the thing that we already released is that we have those set of liberation and product called verifier primaril training where you can

18:34 basically train evaluate any environments on any RS and the model that you can train can be like GNM 5.2 too which is very big and and yeah we have like we work a lot on making those

18:46 li very efficient to to ship the best quality for for our clients. Yeah. Uh I mean yeah super excited about this domain. Once again I think it's super important to have uh a part of like this

19:01 recursive self-improvement to happen in the open because there is actually a lot of people working that are not on big labs. So you need to basically uh yeah make it easy for people to understand

19:13 all those model work to do research and so on. So that's kind of our goal and uh yeah thanks a lot.