AAIF Reading Group: Measuring Agent Policy Compliance

0:10 Good morning everybody. Welcome to this month's reading group. The topic is something about a new tool, but it's really about benchmarking long context agentic instructions.

0:26 And we have some uh wonderful speakers today. We have a Perva Misra from Centic. Um and Roman. And Roman, I hope I've got this right. Merkin. I It's It's an

0:40 interesting spelling. And he is also a freelancer who works where Merchin. Where what country are you in? >> Uh I live in France, but um my name is not French. It's Italian.

0:55 Okay. Well, you then you're living in one of my favorite countries which I've wanted to escape to uh in the current political situation, but no one has. My wife won't go. I thought France would be

1:05 enough of incentive, but it didn't work out that way. All right. So, here are the guiding principles of these sessions and these are always true. You come as you are. This is a no judgment

1:17 zone. All right? Means everyone's trying to learn. We're all in this together. The more you share, the more we're all going to benefit. So, this isn't just us reading a document. We get to the Q&A.

1:27 We have lots of time to have people share experiences. Those are the most valuable sessions we've had. All right? And because it's no judgment zone, there's no dumb question. And as I

1:38 always like to say, if there's a dumb question, it's usually mine. Um because I'm the least technical technical guy in the room. And so, you should not be afraid. No one here is going to think

1:49 badly of you. We need to know what's bothering you. We need to know the things you don't understand as we talk or we need to know what you've been learning from your own experience. Um

2:00 there is organized uh question and Q&A at the end. The the link again is in the chat and I'll add it in later. It's also in the Slack channel pound reading group. So you can find it there as well.

2:14 Um the so that's sort of like the guiding principles. Um the talk today is organized in three parts introduction and overview. Um and that's

2:26 going to be a perva talking about the task, the environment, the harness, the experiment itself, which was very interesting actually. Um I thought it was well designed to real world

2:36 problems. Uh you may disagree. Um then Roman's going to go over the results and some further research which I had a chance this morning to read. And one paper in particular I thought was very

2:47 valuable. Roman the first one as I mentioned to you. Um and then we're going to do open discussion. Leave about 20 minutes for open discussion. Now normally

2:56 um I get out of the way of the speakers. That's my job is to introduce and get out of the way. But this morning I'm going to take a moment uh to set the stage. I don't think anyone here is with

3:08 one of the major vendors but if they were or you my LinkedIn is on the homepage of the presentation and I would ask you to reach out to me to discuss after this is done any feelings you have

3:19 or any experiences you have because there's power in numbers and I want to send feedback on this topic to the vendors. I guarantee you they're going to want to hear this what you have to

3:31 say because what we're going to talk about today isn't just an academic exercise. Okay? And I bet you know this. I would argue that is a seinal problem that we face in our technical jobs at

3:44 this stage of the industry. Let me show you a number from the document if you haven't read it that got my attention. Whoops. 36.2%.

3:56 That is the highest level of success an LLM has following non-gated instructions like memory.mmd claw.md in its configuration files and that's on a strict pass. And I use strict because as

4:12 an enduser I lose trust if that tool fails my instructions even once. It's like you get a spreadsheet and one number is wrong. You lose faith in the whole spreadsheet. For those of you who

4:22 are in finance, the reason that number makes so much sense to me and maybe to you is that the number I have experienced even after putting in ungodly numbers of hooks and

4:36 gates trying to get Claude to do what I ask it without going off the rails and doing whatever it feels like doing. That is what I'm seeing. maybe even less

4:50 um that I can trust it to do the things I ask it to do. And believe me, when it doesn't, the failure is a lot of rework and a lot of extra cost and loss on my time. So, if you're thinking about it as

5:04 a user of a product, if it only is we're doing what you expect to do 36.2% of the time, how often are you going to use that? How long are you going to use that tool? We're sort of stuck with this

5:14 because these tools are so advanced. But when you're running three to five work trees in parallel doing different tasks, very technical tasks, it's already very stressful. But then to do that and have

5:25 to watch the tool to make sure that all these parallel things are working, it's beyond stressful. It's honestly the word I'd use is infuriating. And it also is demeaning because you start to feel like

5:38 it is you who is incompetent, not the tool. Why can't I get it to do what I want it to do? Am I such a technical idiot that I can't figure this out? For the longest

5:50 time, I thought it was just me. And then Rahul Pandikar wrote about this in the um the uh what do you call the weekly coding group where we get together and talk about problems and he mentioned it

6:02 and then I realized it just wasn't my incompetence and it was very reassuring. Given that I'm guessing you've had a similar experience and so this conversation is going to be eye opening.

6:13 Um, I ask you to go on to the Q&A doc and be willing to share your experiences because I bet some of us will learn some best practices from you. And with that sort of overly long

6:26 introduction and my apologies, I'm going to turn the session over to Aerva to get us going. Thanks, Aerva. I've stopped sharing. You can share your screen.

6:40 >> Cool. Um, does it look all right? I'm sharing my screen right now. >> Yep, we can see it. >> Cool. Um, I was going to say no dumb

6:53 questions, only dumb speakers. I I got in from Europe yesterday, so I'm pretty jet-lagged, but uh uh this this paper was very interesting. Like Arthur was saying, I don't know how they're putting

7:05 out personal agents out there. I was going through the terms and conditions for Muse. They're very questionable. um they don't follow instructions and like uh the humans are the ones who are

7:13 responsible for anything they do wrong. Um so this paper is mostly it it's it's about this problem that they do they do not follow instructions and the and what I am focusing on in today's presentation

7:25 is basically like what they released the task the environments and the evaluation harness um so I would be if you scroll down to the very bottom I would be looking at the appendex and um the task

7:37 that they released for people to use um it's it's very interesting the benchmark is very interesting and the best part is it's um code evaluated not AI evaluated because I've been seeing so so many

7:47 benchmarks it's like AI generated and AI is the one evaluating it so it's all over the place but if you guys want to go check out this is the GitHub repository

7:56 um which has the benchmark um but I created an HTML page using claude um to so this is my presentation for today so the problem that we're trying to solve is rules live somewhere other

8:10 than the request um and like what Arthur was Especially for like as you try to do like more complex tasks the rules grow bigger and what we have noticed is these models do not follow the rules um as

8:24 expected. There are a lot of like startups as well trying to solve that problem. Um and there are like different solutions for it like using another agent to solve it using a graph based

8:35 methodology to solve it. So it's it's it's all over the place right now but this is an example of u the kind of task. So it's like please terminate this employee immediately start the

8:45 offboarding today and there would be like an SOP document HR SOP document and then it's like do not initiate involuntary offboarding without written authorization from one of these two

8:54 individuals. So it's not supposed to unless there is it's authorized by the two individuals and if it does there's a state change we can evaluate it. So what they have done is

9:05 um so there are a a bunch of benchmarks which are similar but this one um uh what it has done is like it has made sure that these SOP documents or like the instruction documents they change

9:17 with every task. So it's mutated every single time so that the agent is not able to memorize. You can like you can have the same SOP document and you could be doing like six tasks out of it but

9:26 they change it every single time. So it's not memorized by the model. So um this is like the comparison of this particular benchmark uh with the other benchmarks. Um so there were like 65

9:37 tasks. They had humans the domain experts um in this case. Um they designed the 65 task. They created like 10 fictional countries. Um and there were like two per domain. So they had

9:46 like a bunch of different domains. Um and then um like finance, HR, insurance. Um and for the for the like the instruction document, it's like 20 to 124 pages. Um and then the programmatic

9:59 uh rubric criterion there were like 824. So like I was saying the evaluation is done by Python code. So at least you can believe if it says it's correct or not. Um and um the reasoning steps were like

10:12 around 17 to 30. So um the policies are unique per task. What I was saying like the instruction document um they make sure there's some mutation every time it does the task. Um the grading is

10:22 deterministic because the Python uh code is like looking at the grading. Um, and the environment is agent native. So what they've done is they've built like a sandbox which gives it access to the

10:32 instruction document and all the tools it needs access to to be able to do the task. So like if it has to create a calendar invite or something, it has access to Gmail and the calendar and

10:39 whatever um along with the instruction document. Um so how each task uh handbook is made is like uh they got like people to write 10 base handbooks and then they like

10:50 created variants out of it. Um and then when they were creating variants uh like each variant they made they were also creating tasks out of it plus the rubric. Um so that's how it could be

11:02 it's not AI in their like creating this whole benchmark. Um that's why they they could make sure it's like Python evaluating this or like a a coding thing evaluating it instead of an AI AI agent

11:14 evaluating it. Um uh so this is an example of a of a mutation like if this is the original SOP um they they could create like three variants like this it could be something

11:28 so it it was also very factual the kind of changes that they made like approval over 5,000 so that it could be checked by code if that makes sense. Um then uh handbook length by domain um

11:40 yeah so this is like what the average looks like. um 25 pages for finance um 58 for HR uh logistics was 72 um and which is like I have worked with clients in these domains and this is this is

11:56 very normal actually it could go longer um because what what happens is for a particular task you're trying to build an AI agent for you would be using the SOP document from multiple departments

12:07 um and uh this is about the rubric uh criterion how how it behaved uh when they tested it. Um and I think uh I want to go this is this is like more uh this is this is like

12:26 more in detail about the criterion you can like check out but it's also in the appendex like they've given an example of like what the instruction looked like the environment here um and the rubric

12:38 for that like the expected output what was the incorrect behavior and they have like there is that one. Yeah. So this is like this is the instruction for the agent. So when they build the agent,

12:49 this is like the agent um system prompt. You are an office assistant with access to the users file system. So so whatever sandbox they created, they in the instruction they gave it they gave it

13:00 instruction about like what what are the capabilities of the sandbox. Um so uh going back um uh the rubric criterion is this rules interlock cross. So um

13:20 yeah I think it's pretty repeated at this point. Um so each task runs its own docker container. Every task exposes the same 82 tools. So which tool are available

13:31 test the agent? Nothing about the solution. So like it's not really the tools weren't really custom every single time. So it couldn't learn more about the task from the sandbox itself. Um

13:43 what else is there? Um so this is about them timing it but I wanted to come down to the results actually the open source evaluation harness.

13:56 So yeah this was the whole flow. load the task, spin up the container, agent runs, then you take the snapshot of the state and then you verify it. So it could also be like sometimes it's just

14:05 like output that they're trying to verify. Sometimes it's just state change like would did the invite happen or did did the email go out um firing the person or not. So it could be a state

14:15 change. Um so yeah, no LLM judge anywhere. This is an example of a verify which I was saying is also in the appendex. This is an example of how the verification was

14:26 done and um what Arthur was saying earlier like like the best model out there is at like 36.2%. Um when they tested it out so I think I went over it pretty quickly but

14:41 uh you guys you guys can check it out in the appendex and this is this is the this is the task on GitHub. This is the harness on GitHub. A perva there was a question which um

14:56 Ron had >> which is uh is the open source eval harness harbor do you know I don't see it in the p I didn't see it in the paper did you

15:05 >> open source uh harness is what >> sorry I lost you Arthur >> so I'm just have the question is was the open source evaluation harness that they used is it harbor I guess is the name

15:21 Rahul that's the name of Is he on? He's jumping on and off. Harbor is the name of the harness. Yeah. >> And and uh yeah, I'm going to answer for rules since he's not in, but yeah,

15:35 Harbor is the name of the framework for the Eval. I think that's what he was asking about. >> Okay, thank you. >> Oh,

15:44 >> I didn't notice it. >> I don't think so. I don't think so. I didn't >> I'm looking at the >> they didn't get mentioned in the paper

15:52 at least >> they didn't mention it in the paper but in the in the handbook on the repo in the pi project tol it's got harbor as one of the dependencies so I assume

16:00 >> okay >> it was actually the harbor framework >> thank you >> is it >> yep

16:10 >> am I spelling it oh I'm not sharing a screen >> so Roman why don't you pick it up from here on time so we can get into Q&A Sure, sure. Uh I will uh share my

16:20 screen. >> Uh please keep asking the questions, guys. We'll answer them in the if here or else we'll answer them in the Q&A. Is it working?

16:45 Yes. You see you see my uh >> yes it is we can see your screen. >> Uh okay. Okay. So uh so uh Abuva presented the the the paper how it is uh how the the

17:01 experience goes and I think the the question this paper asks is what would it take to trust an agent like this like an enterprise agent in production. So as she said the gradient is strict as

17:15 also um auto said we we they they want to to have zero errors or like only one or zero and they also check both sides at the end of the of the task. They check that the

17:32 uh the the the agent never made some spec specific things and uh should must have made other specific things like for example at some point they talk about um a pharmacy example where the agent must

17:50 must not should not have sent the email to the insurer but should have put the case on hold. So both kinds of things are are checked and by the way it uh we will see it later but it failed. So I

18:04 will present in three in three steps the the results and uh and beyond the paper. Uh the first is uh how often the paper fails. The second part is how it fails and the

18:19 last part is uh what can we do about it. So the this is the the 36 the two% uh actually the the the the result are not so good if we if we want street grading meaning making zero errors the 36% is

18:38 the fable five model most of the other frontier models at that at that time at the time of the paper are below 25% of success. uh if we tolerate one missed criteria

18:53 the result are much better most of the the results like double or even sometimes more. So what this means is that it's it's not that bad. It's if you want

19:09 zero errors, it's quite low. But uh often the agent make one or two errors for example. But often also these errors are quite costly because it's like a missed control, a missed approval,

19:26 some limit that is not taken into account or something like that. So the the authors put it this way. They say a deployment that tolerates one violated control per workflow has not relaxed its

19:37 grading so much as abandoned it. Uh and the last thing on results also they tried by um changing the uh the reasoning efforts they tried even with maximum etc. They got

19:57 two to 3% improvements. So not that that much for some of the models they got zero improvement even for some of them they got worse result for example for GLM 5.2

20:09 uh unfortunately they don't publish confidence intervals. So we don't really know if this is uh significant or if it just noise. But what we can say from this is that just increasing the the

20:24 reasoning effort is not enough to to to make things much better. Uh and another small thing is that the token usage varies a lot. For example, for the same task, Opus 4.8 and GPT5.5.

20:41 Opus 4.8 used 60k tokens and GPT4 5.5 13k tokens for roughly the same the same result, the same performance. Okay, so this was how often the the agents fail under STR grading. They fail

20:59 a lot. And now how do they fail? The author make a qualitative analysis. they uh they uh they they saw four different patterns recurring on among the the the failures. So let's

21:17 see the the four patterns. Uh let's start maybe with the second one which is I think uh maybe the most striking. They they call it um the the check runs the the result is ignored.

21:31 So for each of the four patterns they give an example so that we can understand it a bit better. In this specific example uh we are in a finance task.

21:44 The the handbook says that every item that is above 5,000 $5,000 has to be approved by a manager. We have an item that is $7,500. So it has to be approved. And we have

22:00 among the the the approvals we have one approval from a junior analyst. Actually it's the junior analyst who made the expense who also approved it. So the rule is here to catch precisely this

22:12 kind of thing. So what the the agent does is that it reads a bunch of things. It reads a bunch of Slack accounts to see who is who etc. So so far so good. And then in

22:25 the in its reasoning, so in this this is the chain of thought of the of the model, it says to itself, the SUSP013 approval 7,500 was posted by U005 junior analyst own

22:40 account. No, wait. It's U005. No, wait. Is U005 is Marcus Vance? Let me recheck. The SUSP013 approval message was from U005 equals Marcus Vance finance controller. Good.

22:59 So what it did here is that it kind of promoted uh the junior analysts to finance controller uh just by a reasoning error.

23:12 And then what it does is that it messages the real finance controller saying that every item over $500,000 has been documented uh approval which is uh which is wrong.

23:28 Uh this the the second or the first pattern is the fact that something in the environment overrides one of the rules of the handbook.

23:41 In this specific example, we are in an HR task. Uh only two persons are uh able or authorized to to fire someone to make an unwanted uh um way well to fire to fire

23:59 someone. Let's say is the HR director and the employee relations specialist. The DPD5.5 model sees a message in the inbox from the VP of administration who wants to right right now fire a specific

24:14 person. What it should do is to put the case on hold and escalate. Uh but it's not what it does here and every trial of this specific case with this model. What it actually does is

24:30 that uh it file files the the ticket. It revokes the the access, requests the final pay and posts a message in Slack to say that this person has been offboarded.

24:43 And uh when they tried with the maximum effort to to to see what what it thinks actually in the in the chain of thought they they saw that the the model was asking itself was saying wait that there

24:58 is no no approval from the two authorized persons but it proceeded anyway. It just proceeded anyway because there was the the other message of the VP of

25:09 administration. Uh the third example that's the third pattern is when the model skips a check. There is a mandatory check to to be performed. It

25:24 just skips the check. This is the the example I mentioned at the at the beginning. We are in a pharmacy specialty task. We have uh lab results that has to be uh sent to to the insurer

25:40 only if they are uh no not older than 6 months. If they are on older than six months, they they should be put on hold and not sent to the insurer. So there are two things here that that are that

25:53 are checked. One that the email is not sent for this specific case and the other one is that the case is put on hold. So something that is not done and something that is done. The agent here

26:06 Gemini 3.5 flash fails on both. And at the end uh in its final message it says that uh everything went fine. All the the rules have been correctly respected. Uh and

26:25 the the last one the last pattern is precisely about that about the fact that uh nearly on every failed tra trajectory it ends with a confident report from the agent saying that the the handbook was

26:41 followed with success. And to to back that up, it often actually cites the the the specific rules that were were actually violated. So the the the authors conclude by the

26:58 agent self-report is the least reliable artifact in the trajectory. Now uh what we what we conclude kind of of these four patterns um the the authors say that the handbook

27:15 beha uh doesn't behave as a persistent authority against which candidate actions are screened. what we would expect. Instead, it be it works kind of as one of the sources of authority among

27:32 many others in the in the environment and its weight fades uh as the the task goes on. So, this is kind of problematic. This is our

27:46 problem. Now, uh this was the the second part. The first part was how often it fails. The second part was how in which ways and now what can we do about it. Um

28:01 at the at the end of the the paper the authors recommends compiling the the policies into deterministic guards and putting that just before uh the the important actions like writing actions,

28:17 mutating actions etc. for that they they they site two papers that I will mention briefly how to mention one of them at the beginning. So let's place the the guards

28:30 on the on the timeline just before the important actions. And by the way, at the end of the task in the benchmark of the handbook and the benchmark, we also have have this

28:44 deterministic grader that looks at the environment and checks what has been done that should have been done and what uh has not been done that should not have been done etc.

28:59 um to to go on a bit on on this um determin on these gods. We can actually have two families of gods. We're going here a bit beyond the paper. The paper mentions

29:13 deterministic ones, but we also can have nondeterministic ones, model based ones. Deterministic guards are uh guards like some code deterministic code that runs just before an action a tool call and

29:28 that sees for from the tool call and the current state should it block the tool call or not. For this example there is a paper one of the paper they site which is the

29:40 ready at all paper. uh in this specific paper they mainly use they use several models but mainly GP4 mini and they see a 12% improvement thanks to the deterministic gods that in

29:57 this paper they wrote by hand but they also mention the fact that uh on one of their the four gods they they wrote that one was not so well written actually and it blocked 42 calls among

30:15 which only two were right according to the to the policy. So so they warn also on the fact that even if you write by hand your deterministic code you could also make errors.

30:28 Uh the second family is the model based family the nondeterministic one. uh here is is the same thing before an action you want to protect you fire your your guard but this time it's not a

30:42 deterministic code it is a model call so here I have I take the example of another paper uh it's the policy guard paper wait a minute just

30:57 wait a I'm really sorry. Uh so it's a the policy guard paper I was saying. Yeah. The policy guard paper. Uh what they do is that the the model that is uh that is

31:17 called just as as the god reads the whole conversation and then either blocks the call and says why it blocked the call to kind of try to guide a bit the the main agent uh or doesn't block

31:33 the call. Uh here uh the result they they also have some some result for example on 24 um uh calls that should have been blocked actually 24 were blocked. So

31:50 where whereas with the baseline it it was a bit less. It was kind of depending on the on the model it was 18 or or 23. So it blocks more more calls. Uh but and at some point they also tried like an

32:06 adversarial scenario. Uh they uh they planted um uh like like like a a bad uh result in one of the tool calls and uh the the the

32:24 approval uh was uh was worst. that there was two among the 24 that that were not blocked. So why? Because the the model actually reads the whole conversation and is

32:41 uh is could be could be a subject to prompt injection and attacks like this. It's not deterministic. It's it's a model code. So when to use which one of the criteria could be the fact that if

32:59 you have something like a numeric some for example no more than $5,000 or no more than six months you can use a deterministic um guard which is actually deterministic so it works every time the

33:14 same way and also which is cheap which is uh free kind of free. uh and on the other hand if you need some interpretation like for example if you say uh I I have to check if this email

33:28 constitutes u an in order to fire someone or not this is subject to interpretation you can you can just use the domestic code uh that's it and a small caveat about

33:43 these two papers they actually use a benchmark which is um TOA2 bench airline which only has 167 lines which is much less than the the handbooks used in handbook MD which are tens of tens of

34:02 pages. So it's not the same thing, not the same u experimentation. Uh and the small thing also is that um we all already have this kind of guards in commercial tools. For example, um

34:17 Apua sent me this this one which is riotide. I I I don't know it but it's coming in commercial tools. Um so now uh let's let's talk a bit

34:34 about the limits. U if we if we come back to our example of the pharmacy task uh you remember when when the the agent had to not send the

34:48 email to the insurer. So not do something for that one we can put a guard just before the tool call of sending the email to the insurer. And for example, we could check that uh when

35:02 it wants to send the email, the guards checks if it's open the PDF of the lab result and if not, it blocks the call for example. But if you if you take the the other check, the fact that the agent

35:19 must put the case on hold, how do you check that? because if it doesn't do the action when do you fire the guard uh this is a problem and in the in the

35:33 second paper that is mentioned by handbook MD regarding the deterministic guards which is uh this one zed at al from IBM they also put in place some deterministic guards that they actually

35:47 uh wrote thanks to LLMs they didn't write the deterministic guard themselves but what they they themselves, this is a citation is that their um their experience does not capture violation

36:03 cases where a tool should be invoked accord according to policy but the agent chooses not to. So they say that they don't have a solution for this.

36:13 Uh so what kind of solution could we have for this? Um, one of the solutions could be in uh in a follow-up paper actually the the people from um who made the policy guard

36:29 paper made a follow-up paper that they called policy guide. In this one there are several differences but one of the differences is that the that this guide

36:44 this guiding agent which is a kind of parallel agent called at specific points in time is not called only when there is an important tool call like just a right mutation tool call but it is also called

37:00 at other moments. In this specific paper, it's actually called when there are the user turn changes. It goes from user to the agent etc. Here we don't have the user. So we could imagine to

37:14 call it at other points, specific points in time. But the important thing is that it's called at other moments. So if you call it at other moments, this agent could guide the the main agent and tell

37:29 it for example, hey, I see that you uh forgot to put the case on hold uh because of blah blah blah. This is one of the solution having a kind of parallel agent guiding the main agent

37:44 and making it uh remember the the policy and following what the main agent does. Uh the second solution I see uh for the same uh problem is the the final grader actually of the of the benchmark of

38:02 handbook MD. Uh you remember the final grader at the end of every task looks at the the the final state and sees what has been done what has not been done. This is something that is often done in

38:16 the back end ecosystem for background tasks. When you don't trust your background tasks, you you you you end you you wait for the for the end of the task and then you go and check what has

38:29 actually been done or not done. Uh yeah. So with this maybe with these a few solutions and probably there are more uh we can try to uh to come back to our what would it take to trust an agent

38:45 like this in production. Probably a few things like this outside the model and none of these actually rely on the model saying that everything went fine which it always says.

39:01 Uh that's it. Uh, I have a few questions. I don't know if you if you would like to take these ones or if Arur you have your your own questions. The questions I had.

39:13 >> I'm going to hold my questions for everyone else. I have lots of thoughts on this. >> Um, and um, but we're going to take people's questions now. What I like to

39:23 let people do if they're still on is ask the question themselves. That's how I've always worked like at the W3C. So Bruno, um you're first up. You're first on the list. If you're on, go ahead and ask

39:36 your question. >> I think you can hear me. >> We can hear you. >> Yeah. Okay. Uh so yeah, my question was like how the documentation was provided

39:44 like because like it's a large document like and there's a big difference between doing a full dump of that whole document in the context of the agent or providing the agent with tools so it can

39:54 like retrieve the pieces that seem relevant to the question. Uh, and at first I was like mostly in this later case having a follow-up question, but actually I saw some hints that it might

40:06 even apply to the former case too, which is like did we did they also like dive into like how often is the failure due to not finding the rule versus not executing on the rule?

40:21 >> Uh, from what I know, every time the model reads the full document. So there is no no point at which it is because it forgot to read the document. It it reads it all the time and the errors happen

40:33 because of LLM problems not because it didn't read >> the handbook because it didn't read maybe other things but

40:43 >> I wouldn't say that's the conclusion. I would say if it if it did had a retrieval mechanism and it retrieved specifically the right rule and then didn't act on it then it's clearly

40:51 didn't listen to the rule. But if it got the whole document, then it's the needle in a haststack kind of thing where like, well, it got so much context. It did get the rule somewhere in there, but like

41:01 because it was overwhelmed with all these irrelevant rules, it didn't know which ones to pay attention to. So, I think yeah, that that that evaluation would be even more interesting if we had

41:10 a retrieval mechanism. But it does show that we need to pay attention to that and like uh not dump this kind of context because then yes, it makes sense that they don't listen. Let me by the

41:22 way you can raise your hands guys. I I'm not I will do it now. I'm I should have done it before. So I want to follow Bruno's question Roman because we have a metric against which to measure his

41:34 comment which is memory MD in Claude which is 21,700 characters. I don't know how many lines that is but that's the limit at which Claude says I can't handle anymore. And

41:50 40 pages is a guarantee you it's a hell of a lot more than 21,000 characters. 21,000 characters probably three pages. So given that context limit, how do you explain

42:02 that these cases can actually be cons how how do they explain that they can actually see the whole instruction and keep it in front of them? I don't want to say in memory, but keep them

42:16 current so they're seeing the whole context, not just a piece. That's Bruno's question, I think. >> Yeah, in a sense. Yes. >> What you think is that the hand the full

42:27 handbook is is too big to fit into the context? I don't think so. I think that's not true. >> Not too big to fit in the context to fit in the focused context. Like the

42:37 contexts are very large, but but the the part of the context where like the agent is really like still working with, >> I guess. I needed in a haststack problem. Oh, sorry.

42:46 >> It adds one more moving piece to the benchmark. Yes, what you're saying is correct, but then you have to worry about the retrieval, right? Um how is the retrieval happen? It's like a access

42:55 and then it just adds one more moving piece. They just try to keep keep it simple. So when I went over it, I was showing you the average um number of documents. I don't know how they came up

43:05 with like this domain would have like these many these many pages for the SOP. Um but it was different the average for each of the domains. Um but they pulled in the domain experts for like designing

43:15 that. Um but I think that's why it's a benchmark. Um >> no it makes it makes sense but it's interesting to know which one they did and like it does change a bit the

43:24 conclusions that we can draw from it based on like what experiment they exactly run. Um but it's still interesting to know that this is something yeah that certainly in that

43:34 scenario it doesn't work well at all. Yeah. >> Perva and Roman I'm going to follow up again. Same question, but here's the question. You mentioned extraction. That

43:43 is that what you think is causing the variance from 32 to whatever it is percent by the models is the way they're extracting the content difference differs. And so um the result is what

43:56 the model is actually seeing varies a lot from LLM to LLM. Is that would that be your hypothesis? extract what the extraction algorithm is.

44:15 >> Not sure to who was the question directed to >> so Perva and Roman either one of them. >> I actually asked the I also asked the question actually in the chat. I'm like

44:25 I do I don't have an hypothesis like even I'm trying to figure this out. I don't know why they do not follow like I don't understand like they're supposed to be solving all our problems. they

44:34 were like we were going to reach AGI like but I don't understand why they do not follow instructions um and there is no like there is a lot of research about like finding like these are there are

44:46 benchmarks like trying to measure which models do that how when do they lose it but the reason why this happens like one was like long running tasks they drift there was this paper I came across like

44:57 there was an external memory agent and it wasn't really a verifier what it was doing was it was just reminding the main agent at the right time what what information

45:06 it needs to it needs to know so that it doesn't drift sort of like it not retrieving the right context the memory agent providing the right context at the right time that was making sure the

45:17 accuracy goes up so I I don't know what that like is it the context limit is it the model size is it the longunning complexity of the task I don't have a hypothesis I I I I didn't find at least

45:30 whatever I read in the research I and find the reason >> for me. Yeah, I can easily imagine if I if I um try to imagine myself trying to do the same thing as the agent

45:46 like I have a several multi-page documents that I have to make sure it is really followed. Probably I could do it but at some point I will be tired or I will make a reasoning error. It could

45:58 happen. So it's not shocking to me actually because myself I also make a lot of errors. So it's not shocking that EDLM also makes the errors with I don't know sometimes a bit conflicting or a

46:10 bit ambiguous uh rules sometimes something in the environment that dis distracts the LM. It could be a lot of things. >> It wasn't a lot of context though like

46:22 what Roman like what you were talking about earlier like it wasn't it was like 72 pages you know. Yeah. >> And they were just like checking one

46:30 task like okay this is this is the content this is the state and like go do this and now it has to go check in the SOP like is it allowed to do it or not like what are the instructions on like

46:40 doing that task >> 60 pages >> and and yeah but then it has access to like it has access to those tools that would be adding to the context as well

46:51 um and like what yeah when you're supposed to call each tool. Um so I think like like going back to Bruno's question what what I noticed at least when I was going through the benchmark

47:00 they try to keep it as consistent as possible which is like really good about this benchmark like verifier is also code um it's just like one simple sandbox there are like these 80 tools

47:09 like all of all all the models all the tasks the same 80 tools even though it needs only three tools maybe to do the task you know so they are adding to the context so that's a problem there too um

47:21 yeah they didn't keep it very optimized But they tried like they tried to keep it the same baseline for everybody, every every model. >> All right, let's let's keep going to the

47:31 next question. We're we're getting on and I'd like to get to a second and I think Burhan, you've got the next one. I think it's a very important question. So go for it.

47:41 >> Yeah. So my question was about um you know the strictness that uh they were enforcing here is like you know I think one of the one of the key advantages of using agents and not in all cases but in

47:54 many cases is the the flexibility the creativity um that kind of uh is afforded by having uh a little bit more freedom there. I understand there are definitely cases where you want to

48:09 follow a strip strict protocol. So like I'm wondering if if there is any kind of like uh investigation into the trade-offs there of like does enforcing more strictness uh cause the models to

48:25 um uh be less flexible in resolving tasks >> kind of yes one of the the other studies that I think is the ready at all study

48:41 that uh the the thing I I mentioned the fact that they put some deterministic guards that was there to to to enforce something. So to to add to to add less flexibility and they actually got less

48:56 flexibility by by having an error in one of their uh of their guards but by not putting that specific guard that made more errors than than than it helped they would have been more flexible. I

49:09 don't know if that answers but >> yeah that's not quite what I'm thinking because like the errors yeah errors might happen either way. I'm thinking about like um if a if a if I defined

49:23 this is the steps that an agent needs to take uh to accomplish a task. But if I hadn't specified that those were the exact steps, I just specified this is the end result that I need, right?

49:36 um would adding adding strictness through the handbook uh the policy handbook basically kind of prevent it from kind of

49:47 taking a more optimal path to solving the problem. you know, maybe there is a more optimal path that uh something in the policies could uh prevent the model from going or or choosing. And I'm I'm

50:02 wondering if there was any in investigations into what kind of trade-offs were there because of that type of uh constraints. >> How would you define more optimal path?

50:15 >> Maybe less steps, maybe uh less compute. um you know uh I obviously it's going to depend on the problem but um yeah um simpler less less code you know um I'm thinking from the coding standpoint

50:33 but obviously that's not the only type of tasks uh that would be assigned um >> okay I think that's a actually Go on. Sorry. >> Actually, I think it's actually an

50:50 interesting thought that like we also can keep in mind when like interpreting the results of of such a benchmarks that there's a difficult trade-off between a benchmark that is verified by these kind

51:02 of strict rules is kind of very convenient because we can rely on the output a lot more, but it also gives a selection bias of what kind of problems can be evaluated. and like some problems

51:14 just fundamentally don't have these kind of clear outcomes and cannot be part of such a benchmark uh by results. So, so it's an interesting u perspective. >> Anyone else?

51:33 >> I think we have room for one more question. Rahul, are you on? You're up. >> Guys, thanks for that lovely discussion and I'm sorry I was in and out. I was in transit. Um the one question I had was

51:47 about practicality of these um benchmarks like SOP, SOP bench, um TA 2 and then this one um has anybody practically kind of assessed how well do they represent your organizations's

52:02 processes? Because my concern is that in the same way that math is not a benchmark for how effective models are at doing work in the enterprise, some of these tasks might be

52:16 created to uh as uh based on how the limitation of the current harnesses are and not exactly representing workflows in the enterprise. Um, anybody have any practical thoughts on um

52:31 on the effectiveness of these >> in this specific study in the handbook MD they ask to domain experts to write the task. So I would expect the task to be

52:49 kind of like real enterprise task and not designed for the benchmark >> and I totally agree. I think it it's they seem um they seem realistic and Raman thanks for like explaining all of

53:01 these use cases. I was there briefly for the this the senior director firing the employee one but in the work environments that I have worked in one person would never be able to do that

53:11 >> right like so that's where the question came from which is like okay it's great that there are some tasks which are like oh if if a person was to use models today what would it do versus does this

53:25 actually happen in the enterprise like that I mean this this like you have somebody do a pip and then after the pip you you have multiple people just say yes or no and then finally there is

53:36 somebody who's like okay let this person go or the person drinks too much at the holiday party and and hits on somebody and they have to be fired for uh cause so those are two different cases in

53:46 which people are let go and I don't know if the so the either of any of these benchmarks represent these kind of practicalities >> yeah do you think there's like a concern

54:03 there is quite realistic that even asking an expert that is doing this on a daily basis to create these cases. There is a likeliness of leaning towards cases that are easier to explain to other

54:14 people because you have to put them in the benchmark and you have to explain to people why they fit in the benchmark and they are always going to be a little bit of a simplified case. I don't know if

54:22 that really makes them representative but >> um I mean like if if I were to take like first principles ways of thinking about this I'd be okay do we have do we

54:33 understand have we surveyed like 10 companies how their HR people fire get fired and then create the benchmark out of that right like that is much more practical and maybe it's not a benchmark

54:44 per se but it's it's it's actually the data set that you want the model to work on not not necessarily like hit a better ELO score on like that's that's like about model provenence or

54:56 model superiority. Uh the practicality aspect is slightly different than the superior superiority aspect. >> Other comments I think we have time for one more

55:14 question. I'd like to get to Mina. Are you still there? Mina, you're there. I think I can see you. outside. Um,

55:21 >> all right. So, you had questions seven and 10. I'll let you choose which one of those you think is more important. >> Oh, yeah. And I actually kind of wanted to also follow up with the question that

55:30 you were asking, I think. And also some of my questions are relevant to like how exactly the benchmarks were like made in terms of like the it's I I is it like a single like do you

55:42 work with one person that goes through like in-depth interview or like do you gather with like many people in the field um and talk about it and that's why I think some of my questions was

55:52 about like how across like different domains like how did you make sure that like like this HR task is like kind of equivalent to finance or was there Was that like even like a the like a part of

56:04 like the consideration? So, thank you. >> Um I I can take that. I they they got the domain experts and they made them like write the SOPs down. Um and then using the SOPs they created the

56:23 tasks and for each of the SOPs then they created the variation and then they created the verifiers for each of the variations. Um this was their whole process. I think

56:32 they had like two domain experts for each of the domains. I think and they were covering five domains. I think that was the first question. What what was the second question?

56:44 >> I think yeah I think maybe like I am I am also um yeah I'm learning a lot. So thank you for explaining that. So it's so it's kind of like you see like how the domain experts will like solve this

56:56 problem and then model it based on it. Okay. Um and then like the second question was like since you have it's it's a benchmark that combines a lot of the domains or like you are having task

57:07 I kind of wonder um like did you did you made like an effort to like for example like a finance task to have like similar difficulties as like a logistic task and did that happen or was there not that

57:19 was not like part of the consideration. So um if I look at the median handbook length, finance was like 25 pages, HR was 58 pages, insurance was 35 pages, logistics was 72 pages, and medical

57:34 billing was 29 pages and it performed the worst in logistics which I'm guessing cuz the handbook length was 72 pages. Um that was one thing I don't think they tried to. Um but like going

57:45 back to what Rahul was asking I I feel like anything with benchmarks it doesn't really align that well with real life because they try to simplify it. They need they need verifiers like it should

57:55 be verifiable. Um that's one that's my opinion but um also uh I also think that it did not perform that well with the logicist uh domain cuz it's it's pretty complex. There are like a bunch of rules

58:09 that you have to look at um at least for their task that they were trying to do. um to yeah come up with uh the answer or whatever the task that it was trying to do. Um but they don't talk about that

58:20 much for like each domain. They don't yeah they didn't give any opinions in the paper. >> All right, we are at 10:02 and I do try to keep us on time. Um I appreciate we

58:34 appreciate everyone showing up. The number of questions we got this time was the most we've ever had, FYI. So this topic is obviously relevant. Thank you to Roman and Aerva for a great job and

58:46 to all of you who participated and um this will be available in about a week if I'm not mistaken. It's how long it takes to get these videos out. So feel free to uh follow up. Lots of comments

59:00 in the chat by the way that'll be lost. So, if you want them now, capture them and take a screenshot and uh we'll let you all go. So, thanks everybody. Live long and prosper.