AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

0:01 [music] Good afternoon everyone. Thanks for uh coming for a post lunch uh talk. Always appreciate that. Um my name is Farup and here's my teammate Nachiket. Uh we are

0:25 uh here behalf of the Door Dash Genai platform team. Um and we kind of wanted to share our eval journey. Uh it started as uh uh you know eval is another engineering thing but then it slowly we

0:40 realized it evolved into a cross functional effort and we kind of want to share our story here. So what is this team? This team is a gen platform team. Uh we are a horizontal team that helps

0:52 all other product teams. So product teams at Door Dash build on top of the infrastructure and the primitives that we provide. Um and we see our uh USP and the value that we provide is that we

1:05 help product teams balance these three forces which is accuracy, latency and cost. Um initially we applied this in terms of models but if you think about it it also applies to agents. Um and the

1:18 way we achieve this is we have uh primitives and building blocks. Um so for example we have an LLM gateway where you can easily switch between different models uh and try the latest and

1:29 greatest. Uh we have an agent gateway where you can connect to tools uh and other agents uh and we help solve authentication uh agent identity and other things in a central place uh which

1:43 our security team can bless. Um similarly we pair the LLM gateway with open weights models hosting. Uh, of course cost is a number one concern these days. Uh, and we uh kind of

1:55 invested in open weights models uh and have seen significant impact uh already. Um, and maybe we'll talk about that in a future conference. Uh, the fourth pillar is eval and that's the part that we

2:07 would want to share today. Um when we started talking to product teams internally at Door Dash uh there were varying uh distinct needs across teams. We had a consumer discovery and shopping

2:22 assistant team. Uh for those who attended Ragago talk earlier today uh you will uh see the need for session level quality judgments. um uh then the personalization ML then you needed uh a

2:35 way to scale up human judgment and with multi- aent systems we needed trajectory based evals now the question is how do you cater to all these different needs under a common platform

2:50 and uh as we spoke to these teams we realized like um we needed to empower the people who are the domain experts and in our case that was strategy and operations folks it as product managers

3:02 uh it was even labeling partners uh and not only engineers so we kind of started with like okay we have to be UI first and this was the guidance we had from Andy Fang our co-founder as well um so

3:14 we had UIs for non-engineers to contribute uh then we kind of evolved to also being API first so that engineers can also build and not be blocked on the central platform and they can build

3:26 their own uh uh uh systems uh and Then of course with the coding agents now we have become workflow first where we kind of empower SNO and PMs to also being able to uh navigate uh the platform and

3:41 uh run operations as well. Um so with that context I'll hand it off to Nachig to talk about uh how we went about delivering this. Cool. Thanks Harup. Um and thanks everyone for joining us. I

3:54 know France is playing right now and I promise you this will be better than that. I'm kidding. Um so as Faroo was saying uh Evals is not just an engineering harness it is a cross

4:05 functional effort across different pillars across different uh teams uh that actually helps us add all the domain specific knowledge into our uh into the quality of the AI itself. So

4:18 from your traces to your data sets uh from you know scoring mechanisms uh this is all basically a team sport. we all have to play uh and help improve the quality of AI.

4:33 So going a little bit deeper into the same aspect uh we have different uh teams uh at Door Dash who help us actually improve the quality of AI. So you're going to have your strategy and

4:43 operations folks who are going to set priorities, set the quality bar that you want to aim for. You're going to have your product people who are going to translate uh these requirements into

4:53 rubrics workflows. You're going to have your operations teams running uh annotations. You're going to have your engineering teams like us uh providing APIs, telemetry, data sets, judges, all

5:04 you know the the cool things. Um and combining all these together is is what a recipe is for actually making sure that you are shipping quality AI products through an eval platform.

5:19 So we've tried to boil this down uh into sort of you know like a a continuous iteration loop. Uh so right from tracing uh you know having a tracing solution viewing your sessions your traces to

5:33 sampling them down you uh you know to a very small uh set that you actually want to look at uh annotating these with the domain specific expertise that you bring in with the different teams I mentioned

5:45 reviewing those uh then creating those golden data sets which are going to be you you know your uh golden data sets and that that you want to measure or calibrate against uh and then of course

5:56 like you know monitoring this over a period of time and then you know rinse and repeat uh go through the whole loop again. So this is in our experience has been you know like a good sort of

6:06 continuous loop uh for you know shipping quality AI at the plat on on the platform level uh we we have two surfaces uh so we have the telemetry layer uh where we have all

6:19 our traces our scores uh observations that is also sort of the plane where users are able to access these traces using an MCP using an SDK uh using our APIs and then we have the workflow This

6:33 is where a lot of our strat ops, our product teams operate on the platform. So this is where all the annotation tasks are set. Uh you know this is where they review their golden data sets, uh

6:43 create their judges, calibrate their judges and so on. So maybe today we'll go through you know these sort of four different uh modules or pillars of our platform uh step by

6:55 step. Uh so again first one uh tracing and sampling uh which is actually capturing what your agents what your LLMs are actually uh you know outputting for the lack of better words uh and

7:08 actually viewing those. Now in order to also power this uh whole platform we have I think as far mentioned we have gone in an API first uh approach. Uh what that has allowed us

7:20 to do is have these table APIs that actually uh you know and then you know build UIs uh on top of that. Uh so all our scores our data sets uh these are all powered by very stable APIs uh that

7:34 our team owns. Uh so all your API access uh including you know like an SDK access is basically powered by this single uh plane. Um again

7:47 going back uh and you know like just refreshing your memory. Uh step one capture your traces uh capture your sessions uh measure your scores. Uh then you want to start uh almost you know

7:59 like adding all your judgment your context your domain knowledge uh and then calibrating your judges is what we have seen as the whole uh life cycle. Step two is on the annotation side. Uh

8:15 so you obviously are capturing a lot of your uh agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where

8:26 things did not go well. This is where you can actually titrate your your you know and actually look in inside what's actually happening uh at the session level and annotate these data sets. Um

8:39 and as Surup mentioned, we have a lot of use cases. we have we we talked to multiple different teams who have uh various uh ways of annotating uh their data sets uh and it's it's almost hard

8:52 for a platform team to you know build like a UI specific uh for each use case uh and and you know to give you an example uh it's usually going to be an annotator who's going to annotate these

9:05 data sets so the platform team is you know in charge of the APIs we have a strategy and of person who's actually deciding what to annotate and then you have an annotator who's actually going

9:15 to annotate uh your data set. So we took this approach uh everybody uh has uh you know access to coding agents uh and we actually doubled down on that API first approach. So because we had these APIs

9:27 we were actually uh able to enable our statops teams to use something like a codeex or a claw code and v code their own annotation UIs. Uh so we had different use cases. Uh I think we had a

9:41 talk from Ragav before. Uh we had image annotation use cases. We had some uh you know manual testing use cases. What stood out to us was the underlying patterns were similar. So if we are are

9:54 API first uh we can actually enable our our our partners to simply v code these UIs for annotation. So it's it's like a very simple example then you know of of a vibe coded UI looks pretty clean does

10:07 the job uh and you get you know the annotation that you eat this is basically like a menu from a restaurant uh it's it's you know nothing crazy uh but the point I want to make here is

10:18 that what helped us was to give this workflow in the hands of the operators so that they can actually build their own vcoded annotation UIs. Uh so moving on once you have these annotation UIs

10:31 you obviously want to you know calibrate your your your judge prompts you obviously have some LM as a judge uh metric that you're tracking you want to now start improving that with these

10:41 golden data sets u in order to do that uh you know we have a pretty simple process uh you you're going to start with you know some judge prompt take a look at you know

10:51 what exactly do you want to measure from the output uh g you know have have us have something simple you're going to have your baseline scores uh where you're going to simply run those LLM

11:02 judges on your traces and then you're going to have that optimization loop. Uh so we use uh the JPEA library which is a pretty commonly used library out there for prompt optimization. Uh and once you

11:14 know the the iteration loop is complete uh our partner teams are happy they're going to then elevate that judge prompt as their LLM as a judge. Now even while doing that uh LLM as a judge as a

11:26 concept the whole prompt calibration concept might be uh straightforward to a lot of folks but it is still like a pretty new and evolving field. Uh and what we wanted to do was really reduce

11:37 the friction of back and forth with an engineering team. So we tried to really remove all the complicated logic and make this into a self-s serve UI. So the screenshot that you actually see is what

11:48 actually exists. uh so uh you know like a product manager or an operator is going to come to our UI. They're going to set some of these configs uh on the platform and then actually run the

11:59 calibration loop themselves. So they don't have to worry about the different settings that they need to worry about what are the different uh you know tweaks that they need to do and they can

12:08 actually like you know run a calibration loop using any model of their choice. I think in this example I have Gemini they can use run it using uh you know any of the claude or the openi models too. The

12:19 other important piece was actually uh making this reviewable. Uh you know again a lot of this uh is a closed box where you can't really it's hard to see what's actually happening. Uh so the

12:30 second piece that we built was actually giving them vis visualization and visibility into what's actually happening. So on the left you can see we and this is like one of the good

12:39 examples where we saw like a significant amount of improvement in the judge prompt. uh and we actually show the you know the the previous the original system prompt and the calibrated prompt

12:51 to our partners so that they are also able to gain that trust uh why as as as we build this [clears throat] >> yeah just want to add to that is this

13:02 enables different configurations in different teams in some teams you have seen the strategy and operations folks own the prompt uh you have seen some teams where the product manager owns the

13:11 prompt you have seen some teams where engineering owns the prompt so this gives gives the flexibility for teams to design and evolve because we are all learning. So the even the org uh design

13:20 is improving and we are enabling that. >> Yeah, that that's a good point. I think the overall idea was to you know build something which is as self-s served as possible so that uh you know people

13:31 aren't always necessarily blocked by our team helping them out. Um and then finally you know uh the quality loop in practice. you know as we've been going through this exercise we've seen a lot

13:41 of improvements happening to our product as well. So you know for example we sort mentioned we started with the UIs we are you know now API and workflow first uh we're trying to reuse a lot of the

13:53 existing infrastructure that already existed at Door Dash uh and that's helped us uh get a long way. Now some of uh we we we've seen obviously like you know really good results. I think a very

14:04 good result that we we do like to call out is we actually did see a lot of reduction in the spend uh at per annotation cost as you as you all can imagine we do have you know thousands of

14:16 rows that need to get annotated every week uh and it can get pretty expensive at doash scale uh and having this selfserve uh annotation platform really helped us reduce increase the velocity

14:29 and reduce the cost that we were actually spending with these annotators. to to annotate the data for us. Uh obviously uh this resulted in faster loops. Uh teams were able to iterate

14:40 faster. They were able to uh you know calibrate their own judges in a completely self-s served way. Uh and thus it has resulted us in in in moving with a very very high velocity.

14:54 So uh finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop uh which is you know you you have your traces you want to look at your traces

15:05 your sessions you want to sample it down to a size which is which you are comfortable with uh you want to start annotating your data sets you really want to start uh making the data better

15:17 with the human knowledge that exists and the domain knowledge that exists and then calibrate your workflows was calibrate your agents, calibrate your LLM judges with this golden data set and

15:28 then repeat this whole cycle uh over a period of time to you know to ship reliably and ship with high quality. Um yeah, we have 4 minutes left. Thank you once again. I think that was the last

15:40 slide. Uh thanks for attending and if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now. >> [applause]