Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline

0:17 Uh, hey guys, this is Era. I'm a founding engineer at Klein and today I'm going to be talking about recursively self-improving coding agents. So, this is a very interesting conversation in

0:30 the sense that everyone wants to understand what RSI is, how we accomplish it, how we go through it, and how it affects our career. So I'm going to start with giving you just three

0:38 goals of this entire conversation. Just three simple goals and that would help you understand like what what we get out of it and what the future uh for this looks like. So the goal number first is

0:49 that understanding the intent of frontier labs right everyone who's outside of the frontier labs is always trying to speculate like hey what what is the topic in open or what is deep

0:57 mind going to come out with or what what are they trying to make and what's the future going to look like who has cracked the RSI. So like this kind of helps you get get a rough outline of

1:06 what their road map looks like. And the second thing is that like if the models are improving so fast like how do you think about your career or the next few years right that is really what we want

1:16 to understand. And we want to understand like okay things are changing so fast like the ground underneath us is shifting so quickly right now like it's like uh

1:26 there used to be a time when you could like demonstrate your side projects on your personal website or GitHub and now it's just like you could literally oneshot them using any Frontier coding

1:37 engine. And with that it's like it's like oh how how do we demonstrate proof of work? How do you demonstrate, you know, that that that we know what we're doing and how do we think of our career

1:47 for the next coming years, right? So, it's a very interesting times that we're living in. And then the third and the most important goal of the conversation is escaping the permanent underclass,

1:56 right? So, I know this seems like a joke, but this really is a very valid possibility that people are contemplating. Everyone's like, "Oh, well, if everything that we want to be

2:06 doing can potentially easily be done by a frontier model, like what are we what are we good for?" And like where does software go? Where where where does the future of technology going to go? And

2:14 what are we going to do as as things evolve so fast, right? So I think that there's like a lot of like fear-mongering but also there's like some very valid merge where people are

2:25 concerned that like if all the revenue of the planet is going to go to um a few hardware companies that make frontier GPUs and then a few few uh AI companies that that make the best models like how

2:38 are we how are the rest of us going to thrive and what what are we going to do here? So, I think I'll I'll walk you through a few examples and a few ideas that will like really really help you

2:47 reconsider the career choices and and and the life choices you're making and what your problems you're picking up and then and then that will help you like realign

2:56 what it is that we should be working on and what it is that you you know would would that you could work on that will stand the test of time and help you both just not just like on a psychological

3:06 but like financial and other levels as well. So uh to give you a very precise example, let's start with something very precise that I had done. So a couple months ago I uh came up with recursive

3:18 shampoo improvements on a coding harness. So this was the first public example of uh an automatic eval improvement. So to give you an example um we used to do this thing where we

3:30 would measure the quality of a coding agent using evaluation scores. So on terminal bench 2.1 Kim K3 with client would score um usually like you know um 60 70% and then depending on like how

3:45 good the model is like you get better scores on on terminal bench 2.1. So we noticed that when Kim K3 came out, our score was 77.5%. Which is a good score, but we were like, "Oh, is this is this

3:55 truly is it truly the best score that this model can get on this benchmark? And also like what does the model release paper claim about what their best score is?" So the model's release

4:05 paper said that they could score up to 85% or even more. So we were just like, okay, okay, so we are not getting the sort of score. And often what we do with model releases in client is that we try

4:15 to like optimize your score and try to get the best out of the model. So when we started hill climbing on the on the model like we we realized this we realized that like you know there's a

4:27 lot of steps that we do says that we do it gracefully says that we pursue the right kind of problems and do it the right way. And we started thinking like okay if we there's a lot of ground work

4:37 here how do we automate it and this is a demonstration of like how we automated it. So I'll I'll just leave that as a placeholder for now and we'll get get back to it later where we talk about in

4:45 detail of like how does like this automatic improvement loop of a client harness work. So this basically means that like you have a coding agent harness that could just like basically

4:54 improve itself and become better on on the same set of problems. So let's start with the ground terms. What is recursive self improvement? Recursive self-improvement is the idea that an AI

5:03 system that helps [snorts] build a more capable successor which repeats the process you know until you hit artificial general intelligence right you have better systems you have better

5:12 improvements you have a better system and then you know you have singularity so a classic example is like a self-reing coding agent right self-reving coding agent is just like a

5:21 coding agent that can improve itself so you know this this is all like a pie in the sky idea like it's like it's like you know yeah sure this looks cool but how how did this even come out to be

5:30 like how did we even get to here? So if you notice in topics road map in 2021 in 2021's road map they very specifically said that uh uh they wanted to develop a general AI assistant that can interact

5:43 conversationally with a user to create edit and explain software. It would display all the capabilities of a software engineer colleague. It would write functions. It would identify bugs.

5:52 It would design an API discuss style and best practices. Right? So we don't expect to reach this goal right away but it's in the direction which we're striving. It's interesting how this was

6:01 more than five years ago. More than five years ago, people had predicted that there exist a day where you will have like a full-blown journal assistant that can basically act like a colleague,

6:10 which we do have right now. Um, then in 2022, Chad Deputy got released. We all know how that happened. It was it was a fun moment. It was like the first time things got sparked for us. Um, then in

6:22 2023, 2024, there was the first time chat models could call functions. These functions would be simple functions like hey what's the weather or you know what's the what's you know what's this

6:33 computation or what this computations result in python um and then finally in 2024 when s 345 was really easy that was when the agents could like truly call like actual tools the way we call like

6:45 think of tools so it's like reading a file editing a file you know reading searching something on web uh listing listing a folder searching and grapping through you know uh a file system things

6:55 like that. So, and in 2024, we had the intuition that like chatbot, what if chatbots could actually talk to each other, you know? So, then that's was the first time when we had like, you know,

7:06 uh multi-agent conversations. Now, at the time, these things were very janky, right? like they were just like taped together in this like weird weird uh bizarre ways where you have just like

7:16 these classes and these links and then like these hierarchies of classes and it was like it was all like kind of like a very janky attempt to like make us think that like these these models could like

7:26 have these pipeline of flows of responses that can all communicate with each other but a [snorts] lot of it was like mostly deterministic and it was like you know it was not as free flowing

7:34 as it is right now. So, and you know, we started getting the intuition. So, this was the autogen project, but like in only in 2025, we we [snorts] started getting the like what if the agents just

7:45 kept getting better and faster and cheaper, right? What if that just kept happening? And then in 2026, uh we we had the intuition that like what if what if the agent could improve itself, which

7:56 is like the final, you know, the final part of the whole process. So, let me give an example like what does it mean for an agent to improve itself, right? So like it's like a lot of times when we

8:05 think of like evaluating an agent, we initially used to think of evaluating an agent in the form of like just a single turn conversation. So we would tell we would tell like um um agents that like

8:16 ask a simple question like how many toes does a dash animal has like it could be let's say how many toes does a cat have and then al would go through and the element would like prepare response and

8:26 then we would get the response. So like a cat has maybe 10 toes or 18 toes, however many. I don't I actually don't know how many toes the cat has. But anyway, um yeah. So then we create the

8:37 response as like a yes or no. But like when you work with an agent, there's a ton of things that you're doing. So let me give an example. Let me give an example. So for example, you're going to

8:45 ask an agent like hey set up this MCP for me. So now it's not just like it's not just like you know a trivial oneable like thing like it will like search online and go oh what's the MCP? what is

8:56 the config? So, I'll like edit and download some soft files. Like maybe I need to add database update the MCB tool. Maybe I need to read and edit some files. Maybe I need to run some bash

9:06 commands. Right? So, you do all of that and then the agent response like, hey, I did it. And then you grab that with a bunch of tests to gauge like, hey, did the agent actually accomplish the thing

9:15 that it said it would do. So to do that thing the second part which is like more intricate where you're actually like truly you know testing agents like uh to their best extent like it's like you

9:27 know um we use this thing called terminal bench terminal bench is a benchmark for you know AI agents in terminal environments and that is something that we we have a lot of like

9:37 experience with at client and uh it's like a it's like one of the best ways to like really gauge like is your coding agent accomplishing uh in general the task that you're doing uh that you want

9:47 to be doing with a hudded model. So there's a couple there's a many different parts of a validation suit. So ideally you want that to run in um [snorts] you know you you want that to

9:58 run in um individual sandboxes. Then you have like independent graders that like check everything like how many turns it took, how many tokens it took, how much latency it took, how many how many how

10:08 many tool calls did it make, what was the trajectory looking like and then grading all the results for things like hey did you not reward hack? Did you not cheat through the solution? do not just

10:16 like look up the solution online things like that. Um so we we basically like a lot of these uh data sets of evaluation problems have a set of tasks. So when here's a terminal bench 2.1 they have 89

10:29 tasks. So what we do is like we spin up 89 parallel containers where they all run online and they run in parallel and they're sandbox and then they each try to solve a problem. So then they're all

10:41 uh in parallel and independent of each other and then at the end we get get a score. So our score could like give us measures like is is a is this is the model performing well and what kind of

10:52 failures it has right? So like you run the evaluation you get a score. So you get an aggregate score and there aggregate score could reveal a set of failures. Okay failures could be of any

11:01 kind. Failures could be hey the read file tool is not working. Sometimes the entrance is so slow that like you know the agent's working it's doing it's making progress maybe it's running a

11:10 circle but it's so slow that it literally like runs out of time it runs out of like 45 minutes or where many that were given for the task and just like stop working. So once you do that

11:19 once you understand hey this is what we ran this is what the failures we got we bucket the failures in different categories. So like 10 failures of you know this kind of tool called 10 10 20

11:28 failures of like say inference not working or something else right and then we once we bucket the failures into different categories where like we try to make an improvement to the harness

11:36 because like what if there's like something wrong with the harness that we could change such that the model score would improve right and on and on and on and you do this iteratively it takes a

11:44 while it's just like you know hill climbing is like a very groundworky kind of process and you know usually usually we we took we used to take a lot of time trying to do this but it was a great way

11:55 for us to like learn how to improve models. So [snorts] one way to hill climb on public eval is that like you know we we could have a thing where you know we started doing

12:06 this thing where we could just like what if we could have like a longunning agent that runs for a while and automatically improves the model score and that's what we did. So we ran GPD 5.6 soul that led

12:18 the work and that was like the main agent and then uh Ky K3 was just running eval. So what it was doing is that like GBD 5.6 was like hey run these events get these scores let me look at the

12:29 failures let me look at where where it went wrong and then I'll make changes to client's harness and then I'll run these eval again and then I'll keep trying. So again, you can see how like very quickly

12:39 this can like go off the rails because this is supposed to be like the running and evaluation itself takes like an hour. Then you have to read the feedback. Then you have to like see if

12:47 you're properly interpreting the feedback to make reasonable changes to the harness that are like not reward hacking that are just like actual genuine improvements and then run the

12:54 eval again. So we ran that for 17 hours and uh when we ran that for 17 hours there was like a legitimate hell climbing and what we found that was very interesting. This happened with a very

13:05 detailed prompt that accomplished this is that like we had baseline score of 27% and then you know we we found issues like loop detection. Loop detection is when a harness just runs in circles

13:17 around a problem right uh we found like background process grinding where background processes would run too long. Uh we found some file index issues and other things as well. I mean when I say

13:27 we what I actually meant is GP 5.6 solve right? So that found those issues that found and bucketed those issues. It's like hey how about I make those changes in the harness and I run the eval again

13:38 and it iteratively built the hill climbing process completely autonomously on its own without my interference and 13 hours later when I come back to it it like it has improved the harness's

13:47 performance in so many ways. So see what does that mean? What does that mean? It just it just basically means that like we're at a point where the experience could make the harness better. That like

13:58 there's like a two-level learning loop where the harness does some action, interacts with the environment, understands whether or not it did the right thing or the wrong thing and gets

14:07 some reward and and like almost and then has there's like a meta agent that's observing the main harness and finding the changes to it and just like incorporating those changes and

14:17 improving. Now it's I think in my personal opinion it's much harder to do in real time because if you do it in real time you overindex on solving certain problems or you're just like too

14:28 cartel to a specific thing. But I do think that this is something that works like on a broader scale of like working on a wide range of problems and you want to be able to like you want

14:38 you want to be able to justify the changes you're making on a broader act applicability so that the changes you make here will apply to a broader range of problems as well. So what does that

14:49 mean? Like what am I trying to get at here? The point I'm trying to get at is that the models are so good right now that they can if they can improve the harness, they can improve a lot of other

14:58 things. So then what else can they improve? So this is this is a this is a post uh primarily from OpenI where GP2 5.6 autonomously rewrote and optimized its own production kernels and those

15:10 production kernel improvements remove the end to-end serving costs by 20%. you know it also had 15% higher to token generation efficiency and this was like completely done by 5.6 soul on itself,

15:24 right? So what that means is that like inference is becoming much cheaper and that means that like if on if if for a company like OpenAI if you're having 20% low lower serving cost for a frontier

15:35 model that could be a billions of dollars of improvement in price and you could you could do that as well for your own models. Um another interesting thing is that like when people have kernel

15:45 writing uh competitions so core auto is a company which has a kernel writing competitions and you'll notice that like they they mentioned that like the person who won the kernel writing competition

15:56 the explic said that I have never handwritten a GPU operator it's all ALM and agent driven so notice that like how like the best uh inference improvements kernel improvements are at this point

16:08 completely exclusively done by a swarm of agents or generally like agents doing some magic to you know magically improve like the kernel kernel code and have faster inference cheaper inference. Um

16:20 yeah so basically like the improvement surface that you think of is not just like the harness improvement it's literally the whole surface stack right so the whole stack from like evaluation

16:31 to harness improvements to training data to post training to world interaction to inference like literally the entire pip part of the pipeline like all the way from like electricity going through your

16:43 machine to like making a network request going through the model models doing inference um making a response and getting back to you like every single part of the pipeline we can optimize it.

16:53 So the whole RSI loop has like um there's a couple different parts so it's like pre-training um uh neatronic examples of like you know AI curated uh pre-training data that was good um

17:03 there's like post- training examples which are you know where there's like constitutional AI examples of like post- training uh improvements then inference I gave you the example of GP 5.6 soul

17:14 for eval and eval generation. I give you our own example of how we improved ourselves on our own eval. Um in general what you want to think of is that like anytime a model is trying to do

17:24 something and it's like not able to do it or not able to accomplish it. So for example like imagine you're having a long conversation where you ask the model something obvious. So like it's

17:33 like or something that you think is not necessarily obvious but it's like it's hard but you would assume that the model could potentially do it. So to give an example, let's say you want to make uh

17:44 add a new column to a database and then fill that column, right? And then put a cache in front of it. So what happens is it does the whole thing, but for some reason the way it pushes the cache in

17:54 front of the whole uh database like it kind of doesn't do it right. So at that point you're like, "Oh no, set this cache differently. Set this as the TTL." And then you give like a correction

18:03 signal and then that kind of like fixes your problem and you kind of go about your day. The way I want to think of it is that I want to think of like every single time

18:11 like a human correction to the digital landscape is a part of a delta that needs to be covered. And what that means is that like every single time you're finding like the edge of model

18:24 capabilities of like how far it can push for infants, how much it can push for this, how much it can push for, you know, solving your problems. You want to find that delta and we want to like push

18:34 that frontier a little bit further. a little bit further and that's what post training does, right? So post training is is this is in that post training we find these like really interesting and

18:45 really hard long horizon problems for uh for coding agents or for other kinds of problems and then just like slowly hill climb their models on on those kind of problems. So anything that has a

18:54 verifiable loop can be optimized but is this is not just limited to software like literally like the physical world itself could have a verifiable loop. So to give an example like let me let me

19:06 give you a very funny kind of bizarre example where like it's like you you look at this and u you um um look at this uh like robot and it's it's typing something and it's like

19:20 really really like interesting what the robot is typing. So notice how like notice how the robot is itself typing like how it wants to like solve robotics and imagine if the

19:39 robot itself could like interpret the results of of like the answers and just like keep improving then that means that like this is you know basically like an ASI robot. Another example that I could

19:50 give you from my uh personal life is that like um I have this um chip that has the ability to see and has a speaker and has a microphone inside of it. So I was trying to set up this uh with my

20:03 machine and it was um a gift that I had for my girlfriend. And uh what I found very interesting about this is as I was trying to set it up, it was very slow. It was slow to respond. It had higher

20:14 latency. It had an agent in and of itself. Right? So what I did was I connected the step to my um computer and I let um I let um coding agents run on this hardware and basically run on this

20:29 hardware automatically make tests to it. So like my computer would talk to my hardware give it some tests like hey do this do that and just like what do you see right now? So then it would notice

20:42 the end to end latency of like the inference response of like how fast uh my hardware is responding and then um what I found is that like it was able to do the breakdown of like every single

20:53 case and scenario where there was like high latency um slower response or or just the algorithm wasn't right and I could then set up a recursive loop where both the firmware could be updated the

21:07 um the tool calls the inference model used um the speech to text model used all of those could be improved automatically with like an open router key which I'd given to a coding agent

21:17 and then overnight with self-running experiments with the hardware uh my um this tiny little piece of beautiful hardware that I have was like deeply optimized so was this also like even

21:28 hardware could be a part of the RSI loop so what happens when you connect these loops right that's the next question like you could have cheaper inference you could have like more experiments

21:36 budget you have like you know better data and better models. So, is it singularity? It's a very question to ask and I think that the part of the parts of the loop are already there, right?

21:46 Like we're we we are really we really are very close to parts of the RSI loop, right? And what do we do like what do we do about this and it's like how do we think of that in terms of like our

21:57 career and how do you think of that in terms of you know our lives and what do we want to do next? So here's my advice to everyone who is watching this like please please please please please move

22:06 closer to the RS right so it's like right now like people are working on all these like different kinds of software which is like you know a agents for you know construction agents for this agents

22:16 for that and it's like I think that you're you know what the way I see those things is that like I think of those as like you're missing the forest for the for the trees you really are and it's

22:25 like you have this like very interesting godlike technology that can change parts of humanity. And then we're not ambitious enough to pursue the best problems. We really are not. And we're

22:37 just like I think that it's not just like the problem isn't just that we're not ambitious and we're also like what we're pursuing. A lot of those things are just like one model release away

22:47 from like being completely like wiped out, right? So like it's also like it's also like it may not be that intellectually simulating because like all you're essentially all we're

22:56 essentially doing is like sitting all day. It's like hey cloud do this for me. Hey cloud do this for me. And I don't know if that's like I don't know if that's like the most intellectual

23:02 stimulating or like or like um thoughtful use of a person's time like maybe maybe you know we're supposed to be solving harder and more uh more frontier problems and I think that like

23:12 it's like the ceiling is like our ambition. So instead of like working on these kind of things my suggestion to people is to like you know do things that of course like goes without saying

23:22 but you should say things you believe in and you know I'm just a guy on the internet but do things that help you move closer to the RSI loop right? So like anything that just like the end

23:31 to-end pipeline of like hey this is where like the models are lacking like it's like whether that's inference whether that's post training whether that's literally like physical

23:40 interaction whether that could be you know send sending end to-end pipelines of truly any kind you know whether that's hardware problems whether that's um networking problems whether that's um

23:50 building a sense of community things that just you know um things that just like help um you as an engineer get closer to the benefit benefits of the improvements um in uh in u um in in the

24:04 model. So this is a very interesting slide from Sasha topic. So um um they mentioned that like you want to have like this you're in this space where it's like you know you only you only

24:17 have like 1 to five years left to contribute to luxury to the world what do you do right so it's like um they they give great suggestions where like prefer ambitious targeted and past

24:25 collaborations right and you know in general like the time scale like all the timelines that you give to yourself like they should be just like completely squeezed out should be completely

24:34 squeezed out because it's like a lot of times and you also have to think through that like if if the model if the problem that you're working on is too ambitious like it's like is it probably is it

24:43 probably better to wait for just like the next next frontier model to come out or not like that's that's something you also have to weigh through right and uh you you ideally want to work on problems

24:53 that benefit from scale and you know you also at the same time want to avoid projects that can be solved by sale alone because then you know if even if you like it's like if you're working on

25:01 a problem and then the next model release comes out it solves your problem so that's not that's probably not a good thing to do so it's very like you have to like think philosophically of like

25:09 what is the endeavor that benefits uh greatly from auto improvements but like isn't also completely wiped out from it and then ideally like it's like if the if the endeavor could also uh be a part

25:20 of the the like the improvement pipeline uh that could be amazing. So you see a lot of companies raising on say um making data pipelines and there's a lot of money there. There's a a lot of

25:34 companies working on physical hardware and stuff. So I think if you're a software engineer watching these, it's like I think that if you're if you feel like you're like here the delta of your

25:42 intellect and ability in your day-to-day, it's like not contributing as in like you have the intellect, you can make a difference. But like if cloud is just like oneshotting all those

25:52 problems like in your day-to-day and you're just like kind of like becoming a me proxy, you should work to think through. Take a step back and think through. like okay where are the

25:59 problems where where you're actually truly uh contributing and that they will stand the test of time and then that kind of reflection and realignment right now would I think like have like very

26:09 interesting consequences for you um in the long term um so yeah so thank you so much you know thank you so much for your time like I think that uh I think that if you're watching this and if you've

26:20 gone this far I think that you're definitely contemplating like one of the most interesting improvements in the course of humanity And uh you know there's many philosophical arguments

26:30 here. Some people might say hey you know I I think I'm maybe I'm off on the timelines or maybe maybe I'm I'm just being too pessimistic or maybe I'm being too optimistic. But I would love to hear

26:41 all those thoughts. So this is you know my QR code. Please feel free to scan it and let me know all your thoughts. Uh thank you so much. Very kind of you to give me your