Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

0:05 Good morning everyone. My name is Hershel Jan and I have here with me Pearl. Today we are going to talk about saying no. It's called as admission control. How our system decides which

0:20 request to get in when your coding agents overwhelm the LLMs you host yourself. Uh before we move on a quick intro about us.

0:32 So I am a senior software engineer at Audible where I lead some of the MLAI infrastructure. Uh I am also writing an open-source handbook called as LLM inference at

0:43 scale. It has received over 300 plus stars by now. I am also maintaining an open-source framework called agentship which helps you ship reliable and secure agentic

0:55 harnesses in an ultra fast speed. Uh perv would you like to introduce yourself? >> Sure. Hi everyone. I'm Perv. I'm a senior software developer at SAP. I work

1:06 on like cloud application and LLM infrastructure. I wrote an LLM fine-tune guide. I contribute to agentship and I'm like building a task tracker which is a local post task board for you and your

1:20 agent. Uh apart from this if you would like to connect with us our LinkedIn QR codes are right there on this slide. Now back to you Hershel.

1:32 Great. So here is the plan. We are going to start with understanding the motivation why any serving stack can admit so much of a traffic.

1:46 Second, we try to dive into admission control where the decision to say no lives and what it needs to see. Third, we will go through some of the proven strategies, the different ways a system

2:00 can actually make the admission decision. And fourth, we will try to dive into the benchmarking where you will learn how to prove that your policy for admission is working or it's not

2:13 working. So let's get started with the motivation. Um and for this everything today uh let's try to

2:28 understand what is inference. Everything today happens during this inference itself. So inference is simply a moment when someone sends a prompt to the model and gets back an answer.

2:43 So the step one is like a user prompt comes in. It's a sequence of words. Uh for example, it could be like failing a fix a failing test. The model which is running on your GPUs writes the answer

2:57 one token at a time. A token is roughly like a word as I mentioned. Uh it writes the first word then uses that word to pick the second uh then it uses both to pick the third and it keeps on

3:12 continuing like that and it cannot write all of them at once because every word depends on the previous one. So those once all the words are generated they stream back to the user uh who ask for

3:28 the result and that volt prompt and the answer out is like that one request. Now I want you to hold on to one small detail like model has no idea how long this answer is going to be. It only

3:44 finds out whether it writes when it only finds out when it writes the first last token. Keep that in mind. It will matter in the few minutes.

4:01 So a model never sits on the internet itself. Every request travels through a short pipeline. It starts in your app or your agent. Then it reaches your API gateway

4:16 which checks your API key counts how many requests you are sending. Then a router picks a replica which which is just one running copy of the model and finally the requests land on the

4:28 inference engine. The common some of the common inference engines are the VLMs, SGLAN, Tensor RT etc. engine is what actually hosts your model. It actually serves your model uh and it

4:45 loads the model weights into your GPUs. It groups many requests into one batch so the GPU stays busy and for every request it keeps something called as a KV cache. In plain words, the KV cache

4:58 is the memory the request builds up when it try when its answer is being written. Longer the answer, bigger this memory gets. Keep this picture in your head. Every

5:10 box here makes some decision but one decision is missing and we will find it out in the few minutes. First, how much an engine can actually hold.

5:26 Every deployment has a ceiling and it comes from the three targets that pull against each other. Quality is how big your and precise your model is. Latency is how fast the first token arrives and

5:40 every token after it. Throughput is how many requests your GPUs can serve at a same time. You know the old saying good, fast, cheap, but you have to pick two. It's the same here. You cannot pick all

5:53 the three. You cannot prioritize all the three. So you can't So every product has to pick two. For an example like an interactive chat, you would want to go with the quality and the latency because

6:06 users are more sensitive to that. It's quite obvious to the users if the quality is down or the latency is high. For something like an async agents, you would want to go with a higher

6:17 throughput as well as quality. If quality is less, you will have to redo the task and redoing is very very expensive because these are the background and the asynchronous tasks.

6:30 And the high volume real-time systems such as the autocompletes they would try to favor latency as well as and the throughput in favor of the quality because you don't need an expensive

6:41 model for doing the autocompletes for you. But whichever corner you give up there is still a maximum load you can carry while hitting your SLO your service level objective. Let's say

6:54 that on this deployment it's eight requests at once. In a normal minute, you never notice that limit. Now, picture the 40 requests arriving in the same two seconds.

7:08 It's such a high load. And this is not like hypothetical. It has happened to like biggest providers in the world. Here are the two public postmortms two years apart.

7:24 On November 8, 2023, OpenAI after a day of unusual heavy traffic, maybe not 40 requests, it would be some million requests, their routing servers ran out of the memory. They started failing

7:37 their health checks. They added more replicas, then had to roll them back. The way out was to cut incoming traffic and ramp it up slowly. In January 21 2026 enthropic the sonnet 4.5 tire gear

7:55 got slow first then it started returning errors over about 90 minutes. The cause they gave was simple not enough compute for the demand. And now let's look at what each team

8:10 decided as their next set of the action item. Neither of them says we need a faster engine. Open AI's followup is graceful load shedding. Enthropics is detecting

8:24 capacity limits earlier. It's the same idea in the different words which is decide on your request sooner to make sure which request gets in. And

8:38 there are more incidents which we have linked in the appendix. If you are interested definitely go and read through them. So

8:52 if you walk backward from those follow-ups, they both point at the same place, the front door. Here is the pipeline again with the missing step filled in.

9:05 We have a client, we have an admit layer, the Q router, and then the engine. It's never the client talking directly to your engine or client talking to your

9:17 queue directly. There is an admission phase in between. Admission is a decision made before a request takes any GPU memory and it only has three possible answers.

9:33 Admission decides whether to start now or you want to hold the request. You want to accept it but make it wait. Third is the reject. That means not right now. It can throw

9:50 different amount of errors here such as 429s and we will dive into this bit more deeper in the second part. this that is the decision that was missing from our pipeline.

10:06 So why does it have to happen before the engine and what would happen if you will actually accept the request what would be the cost of that yes

10:19 so let's try to take a look at this like two phases of the inference like prefill and the decode. So first comes the prefill. That's the engine reading the W prompt, building the request KV cache.

10:31 And then comes the decode. That's the engine writing the answer one token at a time. With every token, the cash grows a little more. All of that memory stays reserved until the request finishes.

10:45 So a yes is not a moment. It's a commitment that lasts the request whole life. And here you can look at the example. There is a request A, there is a request

10:57 B. Once you admit a request A which is a short answer request and then you admit a request B which is a long answer request. Now when you admit a request C which is again a short answer request,

11:09 we don't know how longer it would take because there is a longer request which was already in the memory. Should we even take that request C or should we not even take the request C. So there is

11:20 a cost associated of like admitting every request because admission could still result in missing out your SLOs's. Another impact apart from the SLOs's is your memory.

11:40 Here is what it does to your GPU. So every running request it keeps growing your cache as it writes. So your total memory keeps climbing. When the memory gets to the full engine has three ways

11:51 to cut. Either it will aait or like throw away your request. Either it will put it to your waiting state that means it will swap or it will just say okay I won't be able to serve and I

12:05 I will probably return you an error. Every one of those throws I mean every one of those causes a certain delay in processing of the request. and missing out your SLOs's again while also wasting

12:21 your memory. So why is it like our problem like we have con servers from open AI anthropic bold teams which are serving their own infrastructure

12:41 for the inference. So it's a problem for us because there are teams who want privacy. They want to have their own models hosted on their own GPUs

12:57 and they want it to be hosted on their own GPUs due to the cost perspectives but they have to also protect the burndown on those GPUs. So they need to effectively decide which

13:11 request to serve across their inference ecosystem. On top of and on top of that like the traffic arriving at the door has also changed.

13:30 Earlier there used to be a person that was chatting. He would send one request at a time. There would be a seconds of reading in between and then there would be a slow answer. But now coding agents

13:40 they have taken over. There are tons of calls that these coding agents they send along with the retries, pauses, um like the undo calls as well and it there

13:54 is such a overwhelming traffic that your cluster has to handle and what that means is cluster has a responsibility to take a wise decision of which requests to accept and which uh requests to

14:08 reject. And this is exactly that visualization. You ask your agent that fix an o test. It runs all your it tries to run multiple tool calls that is reading of

14:27 the files like around there could be like 14 files. There could be like around 10 tests that it has to run and it would combine together to make 40 50 calls for just one request and all these

14:42 40 requests they require like reaching out to the LLM getting back the response. Uh and in between some of these requests would be long running some of the requests would be short

14:55 prompt request. So we need to decide effectively where which one to pick and which one not to pick. Yeah. And that was the like reason why we need to worry about

15:15 admission control. Why we need to worry about handling the traffic right way across our infancy ecosystem. Okay. So this brings us to the admission

15:36 control. Uh now moving on to the next slide. Uh that is slide number 18. So in this slide we will be covering three questions. Where does this decision

15:50 from? which request to admit and which to reject sits back at your stack and the another question is what does it need to see to make a good call or

16:03 judgment on the request and what does it returns back to the client. Moving on to the next slide. So here uh we see the pipeline again which each layer's jobs is written out. The agent sees its own

16:21 call and decides when to retry. The gateway sees API key and request count and enforces a rate limit rate limiter for each and the router which sees replicas or healthy and which one

16:37 to pick. Now the engine sees its KV cache and the current batch and decides the order of request to run and what to evict and print. So every layer decides something. Now

16:54 ask each of them one question. Is the room for this request? The agent says not my job. The gateway says I cannot see it. And the router say not my job and the engine says I only

17:09 find out when I have already said yes. The admission box is missing in in between. So now moving on to the next slide. Um so where to put this decision change

17:28 what it can see and what it cost you to run. So there are three possible moments. The first is an invoice right in front of the engine. Here the policy sees the request itself plus whatever

17:43 metrics the engine exposes. All you need is proxy or gateway. The engine stays exactly as it is. The second one is the Q boundary. Now the decision happens inside the engineuler

17:58 as it picks the next batch to run. The third is during decoding when the scheduleuler can pause a request or take back its cache halfway through an answer. Those later movement can see

18:14 more but they mean changing the engineuler. That means keeping your own modified copy of BLLM or SG lang and updating it every time original changes forever. So

18:28 an ingress is the only movement that works with the engine exactly as it ships. So that's where we will stay for the rest of the talk. So now moving on to

18:39 the next slide. Uh so what actually sits in ingress, right? Uh so you you you can think of it as a function. It can take in what it knows about the request and current

18:54 state of the system and it returns one of the three answers like if we have to admit the request. So the admit is the request enters the engine now and memory is committed for

19:09 this moment. If it's deferred with a time we can call toao hold on like uh hold on and come back after some time and there is another decision which is with uh reject with a code that no here

19:26 is the reason why and until the policy says admit the request cost you nothing. It holds no GPU memory at all. Now moving on to the next slide. uh to make the call well the policy

19:46 needs two views at the same time from the request and from the engine. So from the request how long the input is, which client or tenant sends it, which agent session it belongs to, and a deadline.

20:03 And from the engine point of view, how long uh KV cache is, how long the oldest request has been waiting in the queue, how often the prompt is already cached, and how how loaded each pool of replica

20:19 is. Now look back at the ST. The gateway has left side and the engine has the right side. Nobody has the both. And that's really why admission has to be its own layer. Now moving on to the

20:36 next slide. Here are those eight signals turned into simple rule. Each one is compared to threshold and when it crosses it pushes towards the outcome in the colored pale

20:51 from the request side. If the input is too big to fit in the memory headroom just refer it. If a client or tenant is over it share reject it with 429.

21:06 If one agent session has too many calls in flight defer with extras. If waiting would make a request miss its deadline anyway reject it from the engine side. Uh you can see at the bottom half of

21:23 bottom half side. While KB cache is under high water mark admit it. When the oldest request has waited longer than your target defer it. If the prompt is already cached admit it first because

21:39 it's cheap to serve. And when every pool is full, reject it with 529. One rule of thumb, pick the signal that changes in second, like a QH, not a one minute average. A coding agent sends 40 calls

21:55 in 2 seconds. A minute average won't even notice. So that's what the policy reads. Now what does it say back? Now moving on to the next slide.

22:08 We will take the three answers one at a time and look at each from both sides. What the server sends and what your agent should do with it. First, yes, the server sends back a 200 and the response

22:24 starts streaming. At that moment, three things happens inside the engine. It commits KV cache from the whole prompt and the request joins the running batch and decode one token per step alongside

22:37 everyone else and it holds that memory until it finishes or until the client disconnects. That's the commitment from part one just

22:49 in the form of an HTTP response which is why the admission decision has to happen before 200 never after it. Now the client side if we are writing the agent here is how to handle 200.

23:06 Watch the stream not the status code. First time is properly start your timeout clock when the first token arrives not when you send the request under the heavy load prefill alone can

23:21 take few seconds. Second, if you decide to give up, actually close the connection. The server only frees the KV cache when the request finishes or the client disconnects. An agent that times

23:34 out but leaves the connection open is still holding the GPU memory. And the third one, uh, a stream can fail halfway through, right? And that that's no status for that, right? and you're

23:49 you're left with partial answer and you have to decide whether to retry or carry on. In short, 200 promise to start. It's not a promise to finish. Right? The second answer is dipper which

24:05 basically means not yet. The server sends back 503 with retry header. The header tells the client how many seconds to wait. The policy picks that number from the engine. For example, how

24:18 long until the running batch finishes and freeze the enough cash and you differ when capacity is coming back soon? It's a burst, not so not a trend, right? And for example, like if

24:30 you think about it like Q age is rising but the system isn't saturated. If the capacity isn't coming back anytime soon, a dipper just kicks kicks the can down the road, right? And in

24:48 that case, we can just reject it with the honest answer. Moving on to the next slide. On the agent side, handling a defer is simple. Sleep for retry after time and a little

25:02 random jitter, then retry again. The jitter just means a small random extra delay. So client don't like don't call or like don't at all come back at the exact same moment. They will have to

25:16 wait. Look what the server sees from 10 retries without retry after every agent picks its own delay. And that's usually that's usually zero. So they all come

25:32 back at once. That's how 40 calls turns into 80. With retry after and jitter the retry spreads out across the window serves asked for.

25:50 Okay. So if if we see the uh reject reject returns 429 or 529 the third answer is no. Uh and there there are two different codes for it because there are two

26:05 different reasons. A 429s means too many requests. This particular client or agent session is using more than its share. The policy saw something like per client load, the

26:18 number of concurrent calls or a kota. The problem is the caller and then the 529 means the provider is overloaded. Enthropic uses this code. Every replica is sold. So nobody is getting in right

26:34 now. The problem is pool not the caller. Now uh handling a reject. So how should agent react on 429? Reduce your own load. Cap how many calls that agent has

26:51 in flight at once. And on 529 retrying harder only makes this worse. Waiting longer switching to different model provider or park the task for later. and one either and on

27:07 either one keep a retry budget. Use expansion exponential back off with jitter which means waiting longer after each failed try and after few tries tell the user what's going on because a

27:21 reject that gets retrieded at full speed is how you end up in one of the postmodm from the talk at we described at the starting. Now over to you Hershel

27:40 three. Thank you P. Uh so this is the module three where we will try to learn about the different strategies uh that have been rolled out for the admission

27:50 control. So far we have seen that no layer in the today's stack asks where uh whether there is a room and that layer is ours to add and now we give it a brain. So we will look at why counting

28:03 requests isn't enough. Breaking a policy down into four different choices. See what four different research paper chose and finish with a decision tree that you can use onto your own ecosystem.

28:20 So let's start with what most team already have which is counting. Yes. Counting the number of the requests. A simple idea is to rate limit at the gateway which gives API key a fixed

28:35 allowance set in advance. It's great for keeping the clients fair, keeping your tenants fair with each other, but it does nothing to protect your GPUs because it doesn't change with the load

28:50 that is happening onto your GPUs or your cluster. A fixed cap at the proxy gets closer. It counts how many requests are in flight on each replica. The problem is that the quick crep and a long

29:05 planning to call count exactly the same while the real capacity of the GPU changes with every batch. So in practice it mostly cues and clients timeout ends up doing the

29:20 refusing. Neither one knows whether our 40 calls are cheap or expensive. The cost of air policy does though and it estimates each request will actually

29:35 cost what each request will cost in the KV memory and the tokens using the live state from your engine. And it says no before the request actually takes up the memory. So the short version is count is

29:50 not a cost. How do you build your own policy? So there are four choices here. The first is the unit which tells you what you are actually

30:04 admitting. Are you admitting a single request? Are you admitting an agent session or are you admitting a tenant? The two second is the signal which means what is that the policy is

30:20 watching for taking an admission decision. This could be as simple as your memory pressure which is your KB pressure. It could be your Q depth. It could be your deadline slack. That means

30:32 whether your request is able to meet your SLOs's or not. And whatever it is, it has to actually move when the system gets busy. So three is the rule which means how does the

30:46 signal gets turned into a decision. So rule is used for that and that can be anything from a simple threshold that takes microscond to a full optimizer that takes seconds.

31:01 Fourth is the action. Once you have the rule that gives you a decision what do you do with it? What does your how does your client gets impacted through those actions?

31:14 That's the admit defer and the reject that per just talked about in the module two. Each choice limits the next one. For example, a deadline signal only works if your request actually carry the

31:28 deadlines. So let's go through the first three. Choice one. Do you admit a single request or do you admit a poll sessions? Say you have two agent task 20 calls

31:46 each room for almost half of them. You can uh point you can see the top row. So if you admit per request about half of each task gets in but neither task can finish because each one is

32:03 stuck waiting on the calls that didn't make it. You have used all the capacity and completed nothing. If you admit per session, session A gets all of its call and finishes. Session B gets one clear

32:17 answer. Come back later. And this picture is illustrative, but the logic holds. Our 40 calls are one session. Coding agents send sessions and yet most

32:30 published policies admit only the single requests. So which metric do you watch for what particular problem? This is the

32:46 question that we are trying to answer. It depends on the problem that you are trying to prevent. If you are if you want to keep the cash from overflowing, you definitely have to watch your KV

32:59 pressure. If you want to avoid your wasted recomputee, you have to watch out the KV used together with the cash hit rate. A full cache is fine if it's being

33:12 reused. If you want to keep your weights to be shorter, you have to watch for your queue depth and you have to take actions based on that. If you want to meet your

33:25 latency targets, you need to watch your deadline slack. Take actions based on that. If you need to plan for your long answers, you need to predict for the output length.

33:37 And look at the last column fails when it tells you when each signal stops working. Look at the first row. KV pressure fails when the cache never fills up.

33:52 Remember that row? It comes back to bite someone in the part four. So what are the four ways to take that decision? That is how do you convert your signal

34:09 into a decision from the cheapest to the most expensive. Here are the ways a threshold just compares one number to the one limit. Example is a policy named clued weight that was uh created

34:27 somewhere in 2026. It takes tens of microsconds per decision. Second is a dual gate which needs two signals to agree before it acts. Example

34:40 is a paper called concour uh which checks like KVUs and the cash hits under 100 microsconds. Third is an optimizer which solves like one big planning problem across all the

34:54 cues at once. Example is a QLM. It's thorough but it takes about 5 seconds per decision. And the fourth is the formal bound uh which does the math on the worst case.

35:08 Can this request still meets its deadline? The example is the Kronos on which uses the like time to first token and it uh provides you like a microcond level

35:21 latency. And now hold that cost column up against our agent. It sends 40 calls in 2 seconds. A 5-second decision shows up after every party is over.

35:38 So what the four papers did as the complete policy fluid weight admits single request uses the threshold on your KV use and reports 10 to 25% improvement. Conqueror is the one that

35:53 admits your VO agents. It uses your dual gate and reports four to 4x uh the throughput of the SG lang but it was tested offline. Uh QLM admits the request plans across

36:09 the W feed and reports 40 to 90% gains at about 5 seconds per decision. Kronos admits requests against their deadline slack with a formal bound and takes deadline misses from 99% down to zero.

36:25 Definitely very amazing accomplishments. So each purper used a different model engine and a workload. So please don't read this as a leaderboard. The question that matters for this room

36:39 is does any of them fit a coding agent and the answer is no. We placed every paper on the two axises. One axis is what it optimizes deadlines or the throughputs. Other is what it admits

36:57 single request or the VO agent sessions. A coding agent needs a policy that admits bold sessions, works online with live traffic and runs engine you haven't fought

37:11 and nothing sits there cleanly. Conc doesn't need a for but it was only tested offline. So in practice you will pick the closest match and build just build your own. And

37:23 here is how you can choose which polic like which kind of a policy you want to try it out. So here is that decision tree. First

37:36 the question that you have to answer is does your request carry any deadlines? If yes, set up a some sort of a formal bound on that deadline. This will take some microsconds level of latency.

37:50 If there is no deadline requirement, the second question that you can ask is does the KV cache fill at the peak load? If your KV cache fills at the peak load, then

38:07 you would want something like a way to detect the cache prefixes used. So you can use some sort of a dual gate mechanism or a KV threshold mechanism. Um and then if your KV cache gets filled

38:24 at the peak load, it does not gets filled at the peak load, then you what you could do is you can set up a Q level of threshold on it. And that gets us to the benchmarking

38:38 like once you have a policy, how do you benchmark? So I will be handing it over to pop. Thank you. So now moving on to part four which is benchmarking. As we just saw no

38:57 publish policy fit a coding agent cleanly. So you are going to adopt one or you you will have to build your own and then you have to prove if it's actually working or not. So to now

39:11 moving on to the next slide you would have to test your policies right so for that you need four things first is your cluster your policy and trace and score. So let's dig deeper

39:26 into each. So your cluster of the first measure it it is capacity with no admission control at all. The load your replica can sustain on their own. Every load level at after this is described as

39:42 multiple of your number like multiple of that number and uh about the policy u which signals it reads what limit it uses and what it does when it says no. The third one is trace which is a record

39:58 of real traffic that you replay using code coding agent traffic ramping from half your capacity up to twice your capacity. Four is a score. Then they run the same

40:13 trace twice. Once with the policy and once without and then you can compare your result. The score is where the most people go wrong. So let's look look at that. Now moving on to the next slide

40:28 which is uh how how you score. The score we recommend is called goodput under admission. So you can see the formula on the screen. The top of the fraction is the request that met their SLO their

40:44 latency target. That means the answer started on time and it never stalled for too long between tokens. The bottom of the fraction is every request that arrived including the one

40:59 which got rejected. Why does the bottom matter so much? So try this. Uh refuse 95% of your traffic and serve the remaining 5% perfectly. If you only count what you

41:15 admitted, your SLO attainment is 100%. Looks amazing, right? Just ship it. But no like good put under admission says 5%. Shedding looks three when you only account the request you kept. It

41:33 isn't. So moving on to slide 42. The second thing to get right is the workload. A benchmark only tell you something if it looks like your real

41:47 traffic. So replay agent traces with their fan out retries session not uniform random arrival. Describe load as multiple of the

42:00 capacity you measured and sweep from half of it to double it. Set your guideline as multiple of your latency with no load. And remember a benchmark that never goes past capacity

42:17 never actually tests the policy because the policy never has to say no. So you have to make sure that you reach at that point. Now

42:29 move moving on to the seven uh seven things every admission result should disclose. Signal liess is one of the seven things every admission result should disclose.

42:42 Here is the full full list that you can see on your screen. The load as a multiple of measured capacity deadline sets relative to latency with

42:54 no load. Signal liveless which is no paper report. How many times you repeated the run and how much the result varied not one lucky

43:07 run. Where each parameters comes from, turned on this setup, borrowed from somewhere else or left at the default. Whether the rejected request count in a good put denominator and the full

43:22 configuration and the full configuration is model, engine, version, hardware, workload, SLOs's and baseline. None of this ask anyone to change how they evaluate. It

43:38 just ask them to say what they did. Use it as a checklist when you read on on a paper and when your benchmark and when you benchmark your own policy. Doing all seven by hand is painful. So we built a

43:54 tool that records all of this uh as as a default. So uh the tool is admin perf and it's it it's two things in one repository. So in production if you see it's a library you

44:13 put in front of VLLM or SG lang it decides admit defer or reject for every request that's ingress layer from part two. So if you also see we on the right side

44:31 uh it's uh in in a benchmark it's a harness that runs any policy against a real engine on a real GPU and reports what the policy cost both sides calls the same function

44:45 policy decide. So the policy you benchmark is exactly the policy you deploy. We build it because across those 16 paper no two share the same baseline

45:00 engine workload or SLO admitf makes a head-to-head comparison possible. Now moving on to the next slide. So using its pive command. So you can see on the screen we have five commands.

45:18 So infra up starts a real VLM on a real GPU. Infra smoke checks that it's actually serving. Then we have bench runs every policy several times each. Then we have bench computer tell you

45:34 which policy won and how by how much. And then we have infraown which which basically means it should shuts all all off so you stop paying. And that's my favorite one.

45:48 Uh and so now provisioning and benchmarking are separate commands on purpose. Loading a model takes minutes and every policy has to face exactly same deployment.

46:00 If you are reprovisioned between policy, you would change the very thing you are trying to hold a constant. The deployment is constant. The policy is variable.

46:15 So now uh here a policy must declare the signal it reads. This is how admit perf deals with the trap I just showed you. Every policy has to declare the signal it needs. This KV cache threshold policy

46:31 says I require KV used correction which is the share of the KV cache in use. If the engine cannot report that signal, the run stops at the startup with an error. You can actually read without the

46:48 check a missing signal reads as a zero and KV cache threshold that always sees zero admits everything. It's quietly turned into the no admit baseline while your results looks completely normal. So

47:04 policy that cannot see its signal should fail loudly and not just fail silently. Um so here what we did is every decision is recorded with the state it saw. Every

47:22 run leaves behind a folder you can open inspect. There would be four files as you can see on the screen. Those are manifest JSON summary JSON decision JSON and uh out outcome JSON. So

47:39 what each does is uh you can take a look deeply once you run the results. Now we we will be looking at the shading cut. So shade

47:51 shading 37% cut tail latency. So here are the four results how they will look like on a real GPU. The setup small model we used quen 2.5 uh and it's a half billion

48:06 model on a A10 GPU. The engine was kept out at four sequ sequences at a time. Eight request at 14 per second two repeats. If you look at the first row with no

48:23 admission control, the P95 time to first token was about two two and two and a half around 2 and a half seconds and P 45 sorry P95 means 95% of the requests were faster than that and it swung by

48:40 about 700 milliseconds either way. Now the last row if you see with a queue up policy that allows at most two request waiting 60 63% of requests were admitted and the P95 dropped to about

48:58 around 1 second that's 2.5 times better with the exact the good same put now if we look at the signal matter more than the threshold but this This is the result I love most.

49:15 Same run. KV cache usage peaked at half a percent. The queue got 24 requests deep on a small model. KV cache is huge compared to the four short sequences. So it never comes close to filling. All of

49:31 the pressure showed up in the queue. Instead, a KV cache pressure policy would have botched a flat line and admitted everything. that that's the row of choice from two. So

49:48 now here and you you can try it today. You don't need a GPU. There is a small fake engine that speaks the three endpoints admit puff uses and it gets busy under load just like a real one. So

50:02 you can test the whole setup on your laptop. If your right policy is your own package, admit perf discovers it automatically. No change to the repo and contribute

50:13 contributions are very welcome. New policies, workload, engine adapter or matrices. Okay. So if you want to remember like one

50:32 slide just make it this one that when you are accepting the request in your inference clusters you need to have a mechanism to say no early something that works at the

50:47 ingress something before saying yes to the request before that request goes into the GPU before that request takes up your memory. and per added clearly. There are a

51:03 couple of error codes that your admission policies return. You need to handle those requests and those errors. And with that we would like to come to the closure. We would like to thank you

51:20 all for your patience, for your time. And here are like two of our projects. It all started as part of the LLM inference at scale handbook. We and when we were doing the survey, we found the

51:35 gaps and we prepared admit puff. So here is the source code for it. Feel free to go to it and like experiment with it. Raise the issues, contribute to it. idea is to

51:52 build your admission policies, benchmark them and be confident that it uh helps you keep your cluster stable. Yeah. >> Thank you everyone. >> Thank you everyone.