Model Whisperers How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, Google

[music] Hi everyone. Uh sounds like everybody came back from lunch. So hope everybody is recharged and not sleepy at all. It's always interesting to do a talk right after lunch because you never know. It's a mixed crowd. Uh but we're very happy to be here. Happy to see you all. Our talk is going to be about eval. Of course, we're in the eval track. We're going to talk you through what are some things that worked for us while we were building eval uh especially for YouTube ads. We work on the YouTube ads team as part of the we do image and video models for YouTube ads. So, building an agent is hard. I think anybody who's here in the audience probably has built an agent as a side project or as part of production systems. It's a very hard thing to do. It's laborious. It takes a lot of time. Uh making it reliable is harder. So having it do things that you actually want it to do in production, understanding the different kind of things that it can play with, how it's going to react when you launch it to your end users, that's always a very hard thing to do, which is why eval are pretty handy way to manage that. >> Uh yeah, and then um basically uh the first step when you're doing this is of course you need to have your agent foundation. So um when you're building uh your agent, you know, you will want to have a focused and strong set of LLM friendly tools to give your agent a very good foundation. Um so yeah, I would say it's important to first optimize these tools and make sure they're the best they can be before just jumping onto um larger agent evals. Um so once your tools are optimized uh you can also take some other steps like making an independent critique agent right with a remediation loop and this can fill more gaps as far as um having a self-correction mechanism and filling those gaps where uh maybe your base tool set has limitations. Uh and then once your base structure is defined uh you can have you can then go to um having an eval and having a strong eval is very important as this gives you um like a way of proving the value of changes you make as well as running ablation experiments on any changes you make. So I would say this is a very essential tool for climbing the quality ladder. Uh but again it's very important to have that uh good foundation to begin with. Um so yeah the the reliability of your agent is basically a function of the capabilities of the agent uh the guard rails and the evals. Um so understanding uh what your agents do in the real world. uh basically uh generative AI outputs as I'm sure you're all familiar are not exactly deterministic right so it can often fail in certain areas or uh one time it can succeed one time it can fail so we can't really guarantee how it will behave in the wild and for some use cases this is extremely important right and we need a way to uh measure at scale and make sure that it is uh getting the output we want uh despite the non-determinism of these models So we need to define um what's good here and eval behave in the real world by defining what good looks like. Uh it's basically just setting this is uh you know our target output. Uh so to build evals that actually scale they really need to be strict and measurable. And uh so an interesting uh thing here that I think might be somewhat counterintuitive is that early on vibing can actually be kind of good for you. Uh and what I mean here by vibing is basically um doing things that are not exactly scalable to begin with. Um so when you're first uh starting out it may be that uh you you know you could uh take a track of basically just going ahead and making the super comprehensive eval right um but we found it actually works better to first do intuition based approach where you kind of um first see the capabilities and look at the outputs and at this stage it's pretty easy to tell what the issues actually are right uh so even though this is non-scalable it will still give you like a very good idea of when you change this what happens um and like uh it allows you to more quickly iterate as well. So at this stage prompt tweaks can also have like large performance gains you can make a radical change to the architecture um and your eval is not kind of like hindering you in this way. So it's like a very good way kind of you know like an early stage company of just like first you know doing something making more radical changes quickly. Um so yeah this way I think you can also get very familiar with what you're building what the failure patterns are and uh it gives you more of a a sense of depth and understanding it uh which allows you to hill climb in a targeted way and these will basically be very useful learnings when you're actually like building the more comprehensive EVO. Uh yeah, and then there's a chart here showing kind of uh you know if if you uh jump to scale to um these scaled raiders like too early uh it can cause you to kind of have like very big ups and downs as you might be iterating and calibrating the eval as you are struggling with uh changing the model radically. Um so another thing here is um you can you should start early and start small. So um you don't as I said before you don't need to have like a massive golden set on day one. Um you can just kind of start with a few core tasks. So you can look through your agent and define what are the primary things that you want to target, right? Um and just uh basically start with those like highle things and can slowly come to a more detailed representation as you move on. Um, and so here it's important to also test the negatives. Checking if the model like didn't do something as bad, uh, something bad is just as critical as checking if it did the task. Um, so yeah, and uh, yeah, there's a funny visual here about writing the evals can be a very small point and humans arguing over what the rubric should be is, uh, is kind of like a very large task here. Uh cool. So we've established that we want to start small and we want to wipe code at the start. Not wipe code but like why eval at the start and try to get a sense of like what our models doing good where the agents are falling. Find out all the patterns. Eventually you'll get at a stage where you will try to involve more teams. So if it's just a core team of like PM and GX working at the start then you'll bring in more team. You'll have a bigger golden set a bigger data set that you want to test out. So you'll think about scale raiders, LLM raiders, all of that. So we'll get a little bit more into what that looks like. So just a couple of things on like working with scale raiders and things that worked for us. Uh one was that providing them with a clear rubric of what they were actually rating with very clear examples. So we had a lot of situations, especially early on when you're building things. of course like there are so many edge cases and difficult cases that we've not tested out that a raider might encounter. So they're coming back to you saying oh what what what should I do in this case and then sometimes we as a team are like disagreeing on like should this be a pass should be should this be a fail things like that. So I think that's very important to do early on as much as clarity and examples you can give the raers that would be super helpful. So yeah to that point like human human agreement should be strong within your team of what you consider a good use case and a good past case for an eval. Uh the second things that we noticed that helped us a lot was getting explanations from raider. So when you do have your teams or other scale members rate eval if it's a pass or a fail, that doesn't really tell you much about where should the agent improve, what was the thinking that went behind coming to that conclusion. So it's helpful to get explanations of why they're rating something a certain way. And this is true for like if you do single side evals or sideby-side eval like when you're testing two models at the same time having explanations of why one thing failed or one thing worked can be super helpful. Uh other things to keep in mind is that you could also do like in our case it was multi output. So we were asking scale raiders um when we were building ads like are these ads accurate like did we do the right things for it? Is it brand safe? Is it like something that we expected it to be? Things of that nature. So we had like almost like a multi-turn eval system. If you're building those kind of cases, it can get a little tricky because it's not exactly a pass failure. Your raiders could be like, "Oh, well, it does very well in well in brand safety, but it does not do really good in like accuracy or things of that nature." So explanations really help you like get to the bottom of like where is it that the agent's actually like missing things. And then you can also use that input to train your agent better. Okay, so now we talked a little bit more about involving cross functional teams and human agents, but what about if you're using LLM raiders, auto raiders, LMS judges. So we ended up going down that path. Also few things that we tried to do to kind of set this set a more uh comprehensive structure for ourselves is one we we monitored disagreements or in some cases teams monitored agreements. So basically if you can have a sample pipeline of sorts that is monitoring how a human raider or some expert would rate an eval versus how an LLM would rate it. You can get a sense of like how it's trending and if the agreement rates are in the ballpark that you would expect it to be. Uh the second thing was we went a little bit beyond pass and fail. So we also looked at agent traces which we'll get to later. But when we were doing pass fail eval and trying to understand how things were getting rated, we did a couple of spot checks to understand the reasoning behind those logic so we could really see what was going on. How did it come to the conclusion that something was a final pass or no? Uh and again like just reinforcing the high quality ground truth point that Daniel made earlier. So we want to give a golden set that's like super expansive. It covers a broad range of use cases and it also has very high human human agreement within your team. Okay. So this is a quick example of what we had seen in the agent. I'll walk you folks through it. So uh it basically says if you if you want to know what it's doing look at it at its thinking. Uh we had given the agent one of the prompts and it was that for legal reasons disclaimers can never be removed. And we had mentioned that to the agent a couple of times in the prompt. We had like trained it on that and it was all going fine. But then we started seeing that there were edge cases in which the agent was seeing the prompt and it was seeing that there's a disclaimer present in the ad and then still removing it. And we could not find that if we were just doing a categorical like the this x% pass rate or not. So we really had to look at the traces to see what was going on. And in this example, you can see in the initial trace, it actually detects that there is a disclaimer in what it's searching for and it says, okay, I found a disclaimer and now I'm going to go ahead and remove it, which was not what we asked it to do. Uh, this is a sample image that I created that I ran through the agent. It says, "America, we can do better. It's a public parks ad." And if you'll see at the bottom right, it says paid by the community of parks of keep parks clean. and we sent it to the agent and it just removed it when we told the explicitly not to. So those kind of things will happen. So it's really important to like check the reasoning and how it's like getting to the things that you care about. Uh cool one other thing. So like all ML systems some things still are applicable which is my favorite part. So agents will not generalize very well depending on the kind of like data sets that you've trained on. It's usually a good idea to have some sort of data set to test for like edge cases like broader capabilities. Uh also a very good practice to have a test set of sorts. If some of you have worked in like traditional ML systems, it was always good to have test validation sets of that sort. Same concept applies here. Uh if you have a test set, use it sparingly and also refresh it with prod data. Daniel. >> Yes. Um so then um yeah this slide is basically uh talking about hill climbing and how it can be quite rewarding. So um yeah you might find that uh after human eval right uh you can uh this this diagram basically shows like the uh kind of stack of what you can do. So after um human eval if it meets the bar obviously you can finish there but um it probably won't and then uh when it doesn't meet the bar uh you can first review your eval set uh and find numbers like precision and recall um and then you can um iterate and basically uh you know make changes to the eval or adjust your uh rating guide things like this uh or just adjusting the model itself or the agent uh adjusting its tooling. Um, and then through all of these things, you can both iterate on your eval as along with um, iterating on the agent. And this loop, uh, once you have a very good eval defined, it works quite well for iterating on your agent um, and doing quality hill climbing as I mentioned before, doing ablations and whatnot. And then um so as far as uh launch readiness you uh basically need to understand um regressions. So identify where and why the model performance is uh degrading so that you can distinguish between acceptable trade-offs and critical failures. Um so yeah this is basically just saying like it's very important to understand from these evals right what is the exact issue that you're having and figure out um the trade-offs here and then um you should also this is a very important point so you should focus on patterns rather than isolated runs so a tempting thing is to hyperfixate on very small examples from the model right so you might have uh one run that you do with your agent and then you find uh it fails on this case and you might think to yourself, okay, well, I should update the prompt based on my eval and uh the the trace of the agent, right? But um if you do this, this is kind of a trap because as I said before, these are non-deterministic systems. So what's more important is that you actually rely on patterns. So in your golden set, it's important to have multiple examples that can cover these kind of uh patterns that you might want to see. And you basically want to look at the entire picture of how often is it failing on that pattern, not that specific individual example. Um, and yeah, it's important here also to invest in online evals uh and and making sure that your data is matching the real world representation. >> Okay, awesome. So, we talked a bunch about like what worked for us while we were building evals. Of course, your mileage may vary depending on your application. Things can uh things can differ. Uh some of the things that we wanted to recap here was uh it's like what what we think makes a good eval system generally is like it should be representative of what you want your product to be great at. And that will differ depending on the state at which your agent is when you're building MVP cases, it would look differently versus when you're doing production rollout. So that would differ. Uh but it still needs to be very much uh centered around what do you want your product to be good at and optimizing for that. Uh important to of course keep it evolving. That's why we talked about having your online eval having test sets that are refreshed with production data, having sampling pipelines, all sorts of things. Uh highly curated golden sets which will also evolve as your use cases evolve. So training teams whether it's scale raers or your cross functional teams on how to rate things how what are you expecting out of them that's also very important that's uh I think now it's getting more mainstream so hopefully it's less less controversial but like six months ago our teams were like still figuring out okay how do we like do this what's expected out of it so I think investing in those trainings can be helpful uh and then raider templates and rubrics with clear set of examples so you don't have scale raers coming back to you saying I'm not sure how to rate this. Um lots of like things getting marked as like unknowns or I don't know things like that. Also choosing the right launch metrics. So some of the launch metrics launch slides that we showed you. This is a very high level of generally how you would do a launch readiness. You would like check it. You'll do bunch of iterations on the model. You'll do an AB diff or ablation and you'll try to see okay where is the regression happening? What's an acceptable regression versus not? Things like that. As you're doing these systems, it's important to like uh get some clarity early on on what is your gatekeeping rule like what's your launch criteria. Is there a certain precision recall number that you're looking at? Is there some other metric that you look at? If you're doing a model eval, then probably that metric looks different than just the usual precision recall. So those things can also be important to keep in mind. Uh yeah, those are all the tips that we have to build production grade eval. Thank you. [applause] We're on time. >> Awesome. Thank you very much. Uh do we have time for questions? Staff one. Do we have time for questions? Just one. All right. You went up first, sir. Go ahead. >> Are uh all your eval judgments being performed by humans or are you also using LLM as a judge? Um, and if so, what's your calibration process look like for calibrating that judge to provide good evaluations? >> Yeah, I think that's a good question. I think I wouldn't say all. I think it depends on very use cases like depending on like what kind of systems you're trying to build. We have of course like a plethora of use cases. So, I won't say all. I co can't go into details about what the benchmarking and all of that system looks like, but some of the things that we talked about in terms of disagreement rates and ma monitoring like sampling pipelines, those things hold true generally. Yeah, >> happy to chat more offline. Yeah. >> Awesome. Great. Thank you very much. Uh Daniel and Pratika, please give them a round of applause. You can always ask them questions in the back. Thank you. >> [music]