Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

0:01 [music] Hello everyone. My name is Amaya Bhavadkar and I am the field CTO at Brain Trust. Uh Brain Trust is a eval observability platform that helps AI

0:22 teams build and improve their AI with confidence. So, I'm sure all of you, if not, you know, I I'm sure everyone here has built some

0:36 application over the last couple of years that has a model at the center of it, right? Some sort of a chatbot or a AI agent or some system that's doing batch processing using AI at the heart

0:49 of it. And I'm sure all of you over that time span have done significant uh changes to that application. You have either rewritten that application entirely or

1:02 you have like done some pretty complex surgery on your application and the way it looks now compared to how it looked when you started is likely very very different. And I think everyone's

1:15 probably uh experienced the same pattern which is like how building a demo with AI is really easy but making it production quality is really hard. The same way when you're evolving your

1:29 AI application and making significant changes to it, it can be very very challenging. Right? And uh the challenge is not because you built it the wrong way. The challenge is because the system

1:42 around you is evolving and changing so dynamically, so rapidly. You know, the models are changing, the way your users use your application changes, the data that your application works with

1:53 changes. And all of those things require you to continually make changes to your applications. And so if you look at, you know, the rate at which the models have evolved

2:06 over the last couple of years, it's truly astonishing. like every few months there's a new release and that locks unlocks a you know a ton of new capabilities a ton of new features that

2:18 were not present in the previous generation of the models right we have started seeing like models that got really good at working with tools models getting really good at handling very

2:29 long context uh we started seeing models generate code that can be reliably and safely executed in uh sandboxes. We've seen memory systems becoming very

2:43 sophisticated and practical. And so each of those was not a minor upgrade. It was not an incremental change to the previous state-of-the-art. It was a step function change, right? And so now we

2:54 are moving from this era of like re uh sort of iterating on improving our applications to replplatforming our applications because everything is changing so dramatically.

3:07 So why can't you just drop in a new model and ask expect your system to work? Um well the models the previous system that you built was built with some assumptions around the existing

3:24 limitations and the constraints that the models had. Right? Your previous systems were built to account for the fact that your models weren't really as good at tool calling. for example, and so your

3:37 system implemented a bunch of logic to make it work with those limitations. And so when you drop in a new model, right, uh you are not able to tap into the new capabilities, the new state-of-the-art

3:49 without really restructuring your systems in a pretty dramatic way, right? And so in order to capture that kind of capability, the new unlock, you have to rearchitect.

4:04 And so as you rearchitect right um what happens is u models evolve. So you've got to go in and change your application architecture do a lot of work on on getting it to work with the new models.

4:17 But that means that you also now have to update your evals. The way you ensure that your system is going to operate reliably, right? Because every new um uh unlock is potentially also giving

4:32 you new surface area where things can go wrong. And so your evals now have to adapt and evolve to your new architecture. And so you know architecture follows model updates and

4:44 your evals have to follow your architecture. So as I talked through the various generations of the AI systems architectures and how you do uh you know

4:56 what that architecture is and how the eval to evolve with those architectural changes. I want to ground it in a real example and so what I want to talk about

5:05 is um on the subsequent slides I'll share a bunch of notional evals but I want them to be grounded in a real agent. In this case we are going to look at this S sur agent. the SR agent is

5:17 able to not only read information but it's able to take actions on and update systems so it can you know roll back a deployment or uh escalate it to a human page someone uh so it has access to read

5:31 tools and write tools so let's see how this system would have evolved through the various generations of AI architectures so let's start with the simplest case

5:42 right this is how a lot of AI applications started about 3 years ago. This is a single prompt, a single model call. You have one input, one model call, one output. And so the focus of

5:56 evaluations was on the final answer quality, right? Did you get the correct answer in terms of uh accuracy and factuality? Um or did the uh model hallucinate something? Did it make up

6:10 stuff? or did it reference uh old non u the the previous knowledge that it had been trained on and not the latest uh information related to that subject. Um so in this case um you were really

6:25 focusing primarily on the final answer that was your unit of evaluation. And so the approach was you would put together a golden data set. You will create a bunch of various scores that were

6:37 looking at um encoding your definition of what good looks like that then you could evaluate the answers against. And this was great. This was a good way to get started. It was narrow because

6:48 there's no tool calling. There's no orchestration. there's no uh retrieval, no other steps. It's just a simple call to the model. Uh but the next iteration of this was the chain. This is where you

7:00 started doing a set of steps before you actually made the model call, right? Uh the typical rag application looked like it took the user input. It parsed some information from the user output input.

7:13 It then used that to retrieve information, then generate the context and then hand it over to the model. And then the model synthesizes reasons on that information, synthesizes an answer

7:25 and you evaluate the answer. But there's a number of other places where things could go wrong. Yeah, your parser could extract the wrong information. It could retrieve the wrong context. The model

7:37 could struggle with the context. Like in the early days, even though the model windows were the context window sizes were increasing, the models struggled to um reason over large context. So context

7:49 stuffing could be an issue for the model performance. And so now you had multiple uh areas of failure. And so you needed to eval. But this was kind of very um what I

8:07 would call very limited like it did things a very specific way all the time, right? And so in late mid late 2023 early 24 the React paper became really popular.

8:22 And so folks were looking at building um model um in a loop running a model in a loop where it could uh reason and act uh in a step-wise way. So the model could make tool calls. It could then

8:38 understand what the tool calls returned uh reason on that data and then figure out what the next step was so that it could then continue to run this in a loop till the user intent was finally

8:51 satisfied or the model ran out of the iteration budget. Right? And so this was great because it now gives you gives the model the AI system a lot more flexibility. It's not pinned down to

9:03 operating in a very specific workflow. It now is able to reason on the various intents and it's able to self-organize, self-chestrate and complete the user tasks. Unfortunately, the models of that

9:16 era were not as robust as they needed to be. So, you know, models struggled with tool callings. They got the arguments wrong. The models struggled with orchestration. So, they called the wrong

9:27 tools. The models still had challenges with reasoning. they weren't necessarily doing a great job of, you know, dealing with long context. So you had things like context collapse. And so while the

9:40 idea was like really really exciting, um, it fell short of delivering on the actual promise. And so what does what do you do when your model can't be controlled, right?

9:51 You take the control and you bake that control into the system that you're building around the model. And so teams started moving towards these kind of workflow graphs, right? Um they started

10:03 building the orchestration and the execution and planning logic into the the system itself either as a graph or as a state machine. And so you took control of the orchestration while you

10:20 allowed the models to operate at the node level. And that way you got a lot more uh reliability and predictability in how your AI was going to operate across those various intents.

10:35 But then the problem is you are now building a system that is designed to work for a specific set of intents for a specific types of use cases. And as you start hand, you know, the system starts

10:47 interacting with with instances that are outside that distribution, the system starts struggling with that, right? You expect um you know a certain set of applications or u user interactions to

11:00 work well because they can be fulfilled by the orchestration that you have designed. But when your the user intent needs to be requires other things to happen beyond what's specified in the

11:12 orchestration, the system can start um you know breaking at the seams. And uh in order to do that, folks were now building a lot more complexity into their orchestration logic. And so you're

11:23 building these special uh branches and way you handle special intents in the complex graph that described your system. And so what that means is like you had now a ton of different surfaces

11:38 for failure. So you now had to deal with uh you know uh dealing with uh branch consistency and branching logic failures. You had to deal with things like the contracts between the nodes not

11:52 working out well. Uh you had to deal with the limitations of uh the nodes that were you know built for a specific set of use cases. So you know there were classifier nodes for example and they

12:07 could make mistakes and so you could now have a significant amount of um you know areas where you could uh where the system could fail. And so your evals now have to not only look at u you know the

12:23 overall orchestration but they now have to you have to have node level evals. you have to uh make sure that you have evalu uh you know how you do retry loops.

12:41 There's a lot of complex behaviors of the system that now need to be evaluated in addition to all the other things that you were evaluating before. So

12:56 the graphs were kind of popular like in in late 24 early 25 and so a lot of systems were now implemented using certain frameworks and they were now in production. Uh but then um Anthropic and

13:11 OpenAI launched some amazing new model capabilities mid late 25 and what that was like tool calling became extremely reliable. We started uh seeing uh much better orchestration control. Uh the

13:27 models were able to plan a lot more effectively accurately. They were able to manage long horizon tasks. they were able to do a much better job of introspecting and course correcting. And

13:39 so like as things went a little off track, the models were able to, you know, understand that and bring the execution back on track. And so what that meant was a lot of these u u

13:54 graph-based systems were not able to take advantage of these new capabilities. they were still running into some of those like brittleleness issues that the new model state-of-art

14:05 had unlocked and so um we started looking at building out um the react loop again that's that started working and so now you had this new AI systems that could effectively reliably work in

14:21 a loop they could make those tool calls they could figure out the next step and then they could u essentially go in and um fulfill the user intent. But the way they worked was very it had a high

14:35 degree of variance. So every trajectory for the same input if you ran it a couple of times you would see you know dramatically different trajectories while yielding the right answer. And so

14:45 now there's a lot of variance that you have to deal with. So now instead of just focusing on a specific eval the unit of eval was no longer just one eval now you're looking at doing an analysis

14:58 of the distribution of the evals you're taking the same eval you're running it multiple times you're running it k times and you're ensuring that uh you get a statistically relevant signal from that

15:08 eval so now new metrics like uh pass at k and pass raise to k or pass wedge k these were the new metrics that certainly started to make a lot of sense. pass at K is like if you take the

15:22 same that eval and you run it K times does it succeed at least once and that is a measure of its capability and pass wedge K is like if you run that eval multiple times how many times of those K

15:36 instances does it run successfully that's a measure of its uh reliability and so now you can understand whether your system with a high pass at K uh you know is reliable by seeing seeing how it

15:51 you know by measuring the pass wedge K metric for example. So [snorts] this gives you a lot more um you know u understanding of like how your system is working what the failure sources are and

16:04 how you work on those right and then more recently what we've seen is um there's a big shift from it's your system is not just a model running in a loop right it becomes a product system

16:17 it's that there's a model in the loop that's augmented by a lot of peripheral components you know you have a memory system that is able to provide robust memory storage and memory um retrieval

16:30 capabilities uh within a session cross sessions. Uh models can tap into this memory to you know improve upon their runs in subsequent instances by learning from previous runs for example. You've

16:43 got robust code execution uh sandboxes now and so you can run model generated code reliably robustly on uh uh during uh execution. You've got um MCP and skill uh directories that the model can

17:01 now tap into and you can you know weave in extensibility. You now have things like a skills repository or a skill systems that can be used to continually augment the the the capabilities of

17:14 models through you know symbolic instructions. And so uh now you know like uh these systems are getting pretty complex and as a result uh you know if you are continuing to to use the eval

17:30 from the previous generation you're going to get sort of a partial coverage of your system. you're not going to see uh how your system is fragile in ways because of the unlock because of the new

17:45 surface that you have uh you know uh unlocked in your new system. So what that means is um just reflecting back on the pattern

17:57 is like you know all of these model innovations resulted in in you know corresponding shift in the architectures and so so you've seen these waves of architecture and then what's needed is

18:08 like your evals to be congru congruent with that architecture right uh because ultimately it's the eval that are sort of your durable asset that describe how your system is supposed to work. And as

18:21 you go through these generational shifts, that's a good way to ensure that you know your system your user users experience your system in a way that things that were working are not broken,

18:31 but it's unlocked a bunch of new capability. And so everyone's seen this, you know, diagram of this flywheel. Everyone's sort of like bought into it

18:41 conceptually, right? the idea of harvesting data from production to inform your eval so that your evals are reflective of the real world. I think that all makes sense, right? And and

18:52 this is the way that you know teams that are doing a great job at building and shipping and improving their AI systems, they they they follow this workflow pretty religiously.

19:03 Um so I've talked to a lot of teams and I think while there is a general acceptance that yeah you need to run that workflow um in practice a lot of teams don't do that their eval are

19:13 somewhat static and even if you're not changing your AI agent architecture you're you know by not really being disciplined about running that that workflow that flywheel you are now

19:27 getting stagnant evals that are not being as effective in helping you measure and improve the quality of your AI. And especially as you go through this generational shift, it's really

19:38 important that you need a mechanism to not only harvest data from production in a way that shows you failures that you are looking out for because you defined what good looks like as part of

19:51 your evals. But you also want something to shine a light on the new failure types, right? the system is going to fail in new and novel ways in ways that you might not have anticipated and you

20:04 now need to start harvesting that data in a meaningful way. And you want to do this again as as part of the flywheel. And so this is where you need systems to come in and u shine a light on things

20:19 that are broken in ways that you had anticipated, but also broken in a way in ways that you had not anticipated. And this is really important. So I'm going to quickly talk a little

20:31 bit about like how we do this in brain trust. So brain trust provides all the components that you need to run this flywheel. We've got evals, we've got observability. We have ways in which you

20:42 can get insights from your production data to harvest u new eval cases that you can then pass off to the to the team that they can then use to hill climb and improve your AI system. But topics is a

20:57 really cool feature. What topics does, it does a cluster analysis on all of your production data. And so the idea over here is now you are able to find new categories of failures that you had

21:11 not anticipated. So your system is now able to look at all what's going on in production and it's able to now start surfacing these new failure modes that tell you here's a new new failure uh you

21:24 know um uh situation that you hadn't thought about and you didn't have any guardrails in place. didn't have any eval in place and so now it's really easy for teams to expand the set of

21:37 their evals to now cover those kind of new failures. And so this is this is a pretty exciting uh capability in brain trust that enables these teams to continually not only get new failure

21:53 examples for known failure modes but more importantly as they make these systemic architectural changes they're able to also understand the new ways in which your system is going to fail and

22:06 build out effective data sets from production data. So I think the takeaway for today's talk is that the models will keep on changing. Uh I I

22:19 don't think we're going to see any slowdown. I don't think we have hit a plateau yet. I think there are lots of unlocks that are coming down um this road. Um and as a result you will be

22:31 making significant changes to your AI systems. you know, you'll be doing a lot of surgery on your AI agents in the coming months, years. And so it's really important that you

22:44 have a robust workflow system in place to ensure that as you make those changes, as you incorporate these new models into your systems, that your systems continue to get better at doing

23:00 new things, but also continue to work well for the things that they were doing before. And so building out like a robust eval discipline uh with the right tools and

23:12 the right automation and the right systems becomes paramount to manage these generational changes. And so ultimately what you want is um to really uh index on that flywheel and make it

23:27 part of your workflow so that uh you know the ability to improve incrementally when the changes in the system are incremental and the ability to improve your system in a in

23:42 sort of a step function way are both supported by your evals. So with that, I want to say thank you. [applause]