Agent Frameworks Considered Harmful — Rémi Louf, .txt

0:01 [music] Hi everyone. So originally I thought I was going to give a very technical talk but I saw was in the leadership track which I'm not sure what it means but I

0:21 was like okay I'm going to do half high level and half technical. So it's more a story about what I you know what I did in January because in around December um agents kind of became really good uh you

0:36 know there was a step function something happened uh I think it was opus 4.6 six and that's when I realized and I work in AI where I was like okay this thing is really happening and so I took two weeks

0:47 out I took two weeks away so I'm the CEO of text uh you know which is 15 people company I just told my CTO I was like okay I'm just going to go away for two weeks and I'm just going to dive in this

0:58 thing and try to understand what we can get out of it and how good it is and so the story is you know it is the story of me scratching my own itch for two weeks and trying to figure out how we can use

1:11 actually use agents and what are good primitives to build agents and whether you know it already exists. Um this was a really clickbait title but actually it turns out to be a good title even for

1:23 this talk. Um so what you can see here on the left the castle is my office. Uh that's true. I do rent an office in Castle and the small thing with an arrow uh that you can see is like this robot

1:38 mower which kind of works unattended all day every day. Uh it just does its stuff in the background without anyone having to use a remote control or think about it or anything. And I kind of wanted the

1:53 same thing for my morning because my mornings are always the same thing the first couple hours. It's browse market news review of like linear could be Jira my CRM and also I like to walk for an

2:06 about an hour in the morning and then the next hour is spent trying to process the really long voice note that you know was recorded while walking and you know all I wanted was my morning briefing

2:19 with my coffee and that's kind of what we've been told for a couple of years like what the future would be. Um, but then when you really start working with it, even if you're not coding, all you

2:33 get is a TUI today. Uh, so it's amazing. You can code. You can actually, you know, I started doing things that were not coding in it. They're great for this. Uh, agents are great for this, but

2:44 it's kind of the equivalent of having a robot like a tractor mower that you still have to stay on even if it's driving by itself, right? It's kind of very frustrating because you have to it

2:55 can do many things but you still have to be on and so of course the labs didn't stop there and they came up with apps uh which I call basically SSH with vibes that's great but in this situation when

3:10 that came up I was like wow that's awesome I don't have to use like a ter like SSH on my phone anymore codeex is great however I noticed I just started you know I was on my walk and I was just

3:21 instructing the agent to do things while I was walking and so I wasn't thinking you know very clearly anymore. I just started running agents on my phone during my morning walk. And this is not

3:31 great because this is the equivalent of this is you're kind of midway uh you know it's not the tractor that you have to stay on. It can actually do something without you being right next to it. But

3:41 you still have this remote control that you know you kind of have to change the trajectory every now and then. Uh that's useful. It's kind of absurd uh when you think about it and actually when you

3:50 look at people like on their phone all the time just doing this, it's kind of absurd and it's clearly transitional. Like surely we're not it's not it's not going to stop there. Um and so I did a

4:03 very dumb thing as a co which is I started coding uh don't tell my board and I started to build the dumbest thing that could possibly work. And of course it became a really a crazy rabbit hole.

4:16 Uh the repo is there if you want to take a look at it. The code is not amazing. Uh but it works. Um so the first thing is that you know I started using frameworks. Uh I mean there are great

4:30 frameworks. I'm not going to name any frameworks because they're all good in their own way and they all have flows in their own way which is fine. Uh but I spent all my time actually editing the

4:40 prompt within the code and I was like this is actually not very useful. So I'm like everyone here, I hate YAML u like the next guy, but I still found that this was actually a lot easier to start

4:52 implementing agents without code. Uh you can version it, you can diff it, uh you can review in the PR. Um but it's just and it's just so easy. You can just, you know, write your file, you drop it in a

5:04 folder, and then it just magically appears once you have the runtime and it just magically works. Um, and you know, then I needed like my market watch to run every morning while I'm, you know,

5:17 while I'm walking in the fields. And for that we have things that, you know, uh, have been around for a while, which is chron jobs. uh and schedules specify you know when the agents need to be run and

5:31 also we'll see it's very important later uh they publish uh they publish events and you know markdown and chron obviously you know it's much more complicated than under the hood but the

5:44 interface is this you don't write code and that's the whole product so far and honestly just mostly worked at this point I'll I'll come back on mostly uh later And so this is actually a real

5:57 picture of my one of my morning walks. And so what I do is I record voice notes while I'm walking. Uh but chron you know chron jobs I mean people would use chron jobs for this because that's what's

6:09 available in codeex today but they're not ideal because they cover when but this is just one point in time. It doesn't cover because this happened and you know things that happened in our

6:21 system like automatically when you drop the voice note now in the system it will emit an event and an agent will react to that event and it's the same thing when you have a new email a new entry in the

6:34 CRM I mean anything a new PR that's open a new PR that's merged etc just reacts to events it's not just a chrome job and that and that means that you know agents of the Voice note processor is just you

6:47 know not a chron job but here you have accepts and returns. So it just declare what it accepts and what it returns as an event and here it accepts a voice note transcribes it turn it into durable

7:00 notes on the right and it emits a new event and for that it uses structured outputs. We'll come back to this. And you know, now we finally have the future we're promised because that voice note

7:13 agent emits voice note processed. And then I have my daily brief agent that actually will take the output of the chrome job. We'll take the output of the voice note agents and we create my daily

7:24 brief which is posted as a slack message. So the slack message.post post event is actually uh is actually like um a process actually subscribes to this and emits uh and sends a slack message

7:40 to me. It's actually this is a real this is a real thing. It's working. I can show you after on my phone and you know there are frameworks that are going to sell you the fact that you need graphs

7:52 for this in code. Uh you do not need graph uh in this case. All you need is events. You have no edges to maintain. Agents sub simply subscribe to events. Anyone can come in and edit this. You

8:06 don't need to. Yeah, you don't need to know how to code. You just need to know what events exist in the system. Fun and fan out are free. No code and it's just drop a file and the topology emerges

8:16 whatever the log says happened. And you know then of course I I tried to run it. So the first version took about I mean you know I cheated I cheated I used

8:30 uh I used codeex and it took about like a day to write like the first thing uh out of my week but of course I tried it and it broke uh so these are real examples actually the dates know but

8:43 it's real examples it's like the first day delay brief was posted to Slack twice um on Wednesday one of my voice notes completely vanished and then you know towards the end of the week. I I

8:55 kind of like played with the prompts all week and the market brief was garbage, but I didn't version uh I didn't version my changes and I couldn't remember actually what I changed in the prompt

9:06 that made the thing completely useless. Now, if there are distributed or existed engineers in the room, you probably know this shopping list already. uh there is nothing new under the sun and you know

9:24 each failure so each of these failure modes that you found actually led to building one piece of what turned out to be a runtime. So the lost note actually turned into a log. I just wanted

9:37 everything to be saved forever so that I could go back to it and look into uh into what happened. the duplicates. It was because I was not following, you know, it did several attempts and I was

9:48 not following them. I didn't have a proper queue. I wasn't counting the attempts, etc., etc. And then probably the most interesting part is the last prompt. I got into a really deep rabbit

10:00 hole in there and I just ended up uh building a content like a content addressed system for this uh content addressed system. You can think of git uh you can think of nyx and any other

10:11 build system. And you know that was and I didn't do this because I wanted to design a runtime. I mean by that point I still just wanted my agents to work and I also like the distraction. Uh and I

10:25 just paid off debt as it appeared like errors as they appeared. I hope my board won't see this talk. Uh so the log the log is the systems memory. Uh nothing is lost and

10:40 everything is observed. uh you can you know you only have one append only events table on the left it's a real common line uh in like command za events and you get all the events they are

10:53 causally linked as well like you know which event triggered which event which happens to be super useful when you're debugging and you know even with three four agents you start having like major

11:06 debugging headaches so that was super super helpful Um and everything is querable which again for debugging. The second thing is you know okay we

11:19 have a log so we can trace back things etc. But it's still really hard to know what went into the like what went to the model what prompt was sent to the model again because what you see when you're

11:33 using codecs is kind of a lie. like you kind of have like a live chat session with the model and so you tend to think that oh that's what the model saw and you know that's exactly so I can

11:43 understand what happened the truth is that's not exactly what the model saw um there are many reasons for that one is I mean compaction obviously is a big part is a big thing but also you know there

11:54 are just quirks also you know openai doesn't share or enthropic for that matter don't share the thinking with you the thinking traces so you have no idea I mean kind of have an idea of what went

12:04 in but not completely either. And so you need something different. Uh you need something different and that was the big rabbit hole which is trying to find a way or build a system where you can

12:17 trace back to what the model saw internally. And so what I did was basically built I mean nothing new. This is basically how build system works. Uh so you have different parts for a

12:28 prompt. You have your system prompt. You have a description of your first skill of a second skill. Then you have the description of your tools. You have your user message which is the question to

12:38 the model. Each one of those is stored and addressed and you know stored somewhere as a identifier which is a hash. And so when we build a prompt instead of building a piece of I mean

12:50 before rendering the text we actually represent the prompt as a list of these u of these hashes. And so what that means is that down the line when I have a model answer, which by the way is also

13:05 stored in the same way, we can trace back to the prompt very easily. And then from that prompt, we can know exactly what went into the model's context, which actually matters a lot. I mean, it

13:16 matters a lot for debugging, but it also matters. I mean, it makes compaction a lot easier. You're just manipulating a graph, right? You're not manipulating strings. It's just a lot easier. And it

13:26 makes KV cache management a lot easier as well indirectly. And but I think that when you know I guess probably the main advantage that's when you use that scale is really auditability. It's

13:41 like you can know exactly what happened with that agent and why it returned what it returned. And so you know I'm just going to go pretty pretty quickly over this. Uh what you get once you have this

13:55 graph is you get diffs like you can say okay what changed between these two runs like which com which components changed was it just my message did I like give the model a different skill did I give

14:07 it a different tool so you can just yeah you can just run this function and it will show you you know the difference between uh the runs so here you have you [snorts] know three components that were

14:18 identical there's one which is you know the user message changed And then you had all these other messages that were actually you know that were continuing. It's continuation of a single session.

14:29 Uh then you have another thing for free which is replace. Uh replays turned out to be really useful for me because after a while I mean when I saw the cost ramp up like the thing when you have

14:42 observability is that you do realize that cost increase very quickly. I wanted to try with open source models and so I wanted to rebuild old request for to eval. and see if I got the same

14:53 thing out, if I got something satisfactory, if I need to change anything. And turns out that once you have uh you know this content addressing system, you can rebuild the request from

15:04 the graph and you can just replay it exactly the same and you can you know resend, you can use a different model. Uh you can use a different request if you want, you can uh you can change it

15:14 and so yeah, you get actually a lot of things I mean for free. You need to implement a thing. Um and so this is kind of different um from what you find I mean what I found when I started doing

15:26 this it might be different today because it was a couple of months ago is that out there you had a lot of libraries so it's just frameworks and frameworks just call code uh your agents leave inside

15:37 their abstractions um and I don't like analogies with you know operating system okay everyone has used that analogy but okay let's say a kernel

15:47 like runs processes and your agent kind of is a process. It doesn't matter what it does actually. Uh but the system can schedule it is built to isolate it. It can isolate it and journals it with the

15:58 log and the agent definition. So the markdown is user land like you don't need to use it with that system if you don't want to actually have a front end that doesn't use this markdown uh this

16:08 markdown format at all. And okay, here's a very important point and you know that's kind of a takeaway and it's also what justifies me working on this because disclaimer structured outputs is

16:19 our specialty and we've been working on this for three years and it just ended up being a big dog fooding project and the reason why I did this at the beginning was not because I absolutely

16:30 wanted to use our software. I didn't necessarily want to you know forklP to other software etc. It's just because Infropic was terrible at structured outputs and so like 20% of my events

16:42 were wrong and were rejected by the system. So that's why I ended up doing this. And the goal, you know, the job of the kernel is actually to make bad actions impossible, not just unlikely.

16:52 And so you have this two boundaries with between agents and the external world. The first one is type tool calls. The two tool calls uh you don't want, you know, you don't want to call tools that

17:03 don't exist, etc., etc. and also the boundary with other agents which is typed events and this is non-negotiable I found like you can get a lot of errors just from this uh I wrote a really long

17:14 blog post about this uh it's if you follow the QR code you'll find it and yeah and the result of that is I actually deployed it uh within the company after I built this and now today

17:26 after a month of deploying it we have 20 agents on the left that are not just contributed by technical people by the way which is kind of what markdown uh what Maron gives you and then on the

17:37 right is you know we deploy it's called the internet there's the briefs there's a bunch of I mean there's a bunch of things as you can as you can see kind of like a few you know as a

17:48 conclusion a few lessons uh the first one is that well executed background agents are really magical uh they feel like this you know robot mower that I had at the beginning is I really just

17:59 sit down and I come back and I have this morning brief that is probably even better than what I would have had just doing it manually and it just appears in my inbox every day and processes my you

18:10 know random thoughts. Uh the difficulties that you meet doing this kind of thing is just good old engineering problems. I mean there's really nothing new under the sun when it

18:20 comes to orchestrating these things. It's just good old software orchestration. Uh open source models are there. Uh they're good enough. I replaced so I don't have any third party

18:30 APIs anymore. Now I just use open source models and even on my laptop I use a local model. Uh so it's good enough for what I do with it. For coding I don't know but for what I do with this it's

18:42 good enough. The infro category is definitely unsettled. Uh I tried a few things before I started building myself. And I would advise that today like definitely start building before you

18:54 buy. Uh so if you're a small company, if you're a tech CEO, it's kind of an advantage because you can just do this without, you know, tasking engineers to do this and get them off track. But I

19:05 would definitely try to build before I buy just to know exactly what I need and you know the limitations of what exist. Uh also I will say that to people building uh frameworks for this is

19:19 please eat your own dog food. Sometimes it's pretty clear that people are building you know agent orchestration frameworks etc but not eating their own dog food. So please do and the other

19:30 thing is I'm really glad I took this two weeks off to play with the field because that completely changed I mean that changed the trajectory of the company. I know we're an AI company we should be in

19:39 it etc but you know business is such that you're always thinking about the next thing the next thing the next thing and it's the same everywhere. But what I'm urging you to do is to stop and

19:49 actually immerse yourself in this and try to see how useful it can be for your company. Uh so you can steal the code. Uh it's not a product that we sell and we don't intend to sell this. Uh you can

20:01 read our blog as well. Uh so I haven't explained uh this yet but I will publish something about it. And thank you for your attention. >> [music]