Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean

0:01 All right. Hello everyone. So, my name is Rohan and this is a quick demo on the optimization agent that what has been recently built. So, a quick intro into what the purpose is and what the goal is

0:14 of this agent and then I'll get into a brief demo and results that from a previous run that we just did. So, what the goal is here is to build an agent that helps

0:27 find the best VLM serving config for any given model. The purpose is to reduce the time it takes to manually find the best configs and the VLM

0:38 serving recipe and if we can automate this and have it run in the background and have the agent find the recipes and the optimized parameters faster then it will

0:48 save a lot of time. So, that's the goal of the agent and a little bit more into how the agent actually does this. So, the strategy that the agent uses is called Bayesian optimization

1:00 and Bayesian optimization is a technique that's used to find the most optimal numbers in any given situation. So, in our situation it's to find the optimal numbers in these VLM

1:13 recipe. It uses mathematical steps that essentially minimizes the search space. So, by minimizing the search space it reduces a lot of the time that it takes

1:23 in finding the optimized numbers in this recipe during the run. So, BO Bayesian optimization is a strategy that is used but Optuna is the tool that uses a strategy in the search loop. So, Optuna

1:39 uses Bayesian optimization. It feeds it the current recipe, the starting recipe. Bayesian optimization runs on that and over many iterations or many trials it eventually finds the optimized recipe

1:51 and Optuna basically takes each recipe after every trial, assesses it, gets a score, uses a strategy to change the recipe, improve and then it reruns it within the

2:02 uh search loop um until the trials is done. So let's go a little bit into how we do this. So um currently in a droplet here and this droplet for example has GPT OSS

2:14 120B model uh loaded up. So what we can do is uh we can look at the current

2:24 um recipe. So what we see here is four regimes, right? That we're testing it on. So there's chat short, there's chat long, there's rag and batch classify. Uh they have their own set of ISO OSL

2:35 tokens that we're feeding it through. Uh all these um can be changed depending on what uh we want to see. But essentially this is the starting config, starting recipe that you would get when you

2:47 download the weights of the model. Um so moving on, once you have the model downloaded in your droplet and you are ready to run this agent, you paste in

2:58 this command uh which essentially tells you you know where the where where the weights are coming from, um activate a virtual environment and have um

3:08 uh or not have but run the script called tuner.py. Uh this script essentially tells you or it it has a couple of things you have to tell it. So you you

3:16 want to give it the name of the study. You want to tell it the starting YAML file which we just saw above here. You want to tell it the hardware that it's running on and you want to tell it the

3:27 number of trials that you want. So uh in this example we have 24. Um briefly I'll show you another example of a past run that was run uh that had only nine trials. So

3:39 um anyway uh this is the command that's that's run. Uh typically these commands take um a couple hours to run. I think for the

3:49 GPT OSS it took about four or five hours and then for the other model it took a couple of hours as well. So we can go into the HTML report that's generated after the

4:01 uh agent finishes its work. So this one is for the GPT-OSS model that I mentioned. Uh what what we see in the HTML report is which trial number was the best trial

4:12 out of the total of how many trials. So in this case it was 50 trials. Uh you get so at the end of the run you get an optimized YAML file which includes all the numbers that we

4:21 currently look for in the BLM serving recipe. So in this case we found or the agent found that 256 was the optimized number so on and so forth. And the way that it's getting the accuracy is the

4:33 amount of good put tokens that's being generated. So overall we found that from the first config we had an improvement of 17.5% in terms of tokens generated. And

4:46 this one as I mentioned is for GPT-OSS. We have another HTML report here which is for the DeepSeek R1 Distill model. In here we found that or in here we had nine trials that was run and out of the

4:59 nine the ninth one was the best. Uh we found that token throughput doubled and ITL was improved by almost half aka it was reduced by almost half. So this is a quick intro and brief demo

5:12 on the commands you run and what the output looks like. In in in general you get a HTML report that tells you um how each trial went and then you also get um a YAML file that has the optimized

5:24 recipe at the end of the day. So hopefully that was helpful. Thank you.