# Searching for a better model-serving recipe

Source: https://www.youtube.com/watch?v=y9JtCCZP2MQ

Machine-generated transcript; recognition errors remain. Source-recording timestamps. Boundary-crossing subtitle ends are clipped to this session.

[02:13:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8017s) All right. Hello, everyone. So my name is Rohan, and this is a quick demo on the optimization

[02:13:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8023s) agent that has been recently built. So a quick intro into what the purpose is and what the

[02:13:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8030s) goal is of this agent, and then I'll get into a brief demo and results from a previous

[02:13:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8035s) run that we just did. So what the goal is here is to build an agent that helps find

[02:14:03](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8043s) the best VLM serving config for any given model.

[02:14:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8047s) The purpose is to reduce the time it takes

[02:14:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8050s) to manually find the best configs

[02:14:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8054s) and the VLM serving recipe.

[02:14:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8056s) And if we can automate this

[02:14:17](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8057s) and have it run in the background

[02:14:19](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8059s) and have the agent find the recipes

[02:14:22](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8062s) and the optimized parameters faster,

[02:14:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8064s) then it will save a lot of time.

[02:14:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8066s) So that's the goal of the agent

[02:14:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8068s) and a little bit more into how the agent actually does this.

[02:14:31](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8071s) So the strategy that the agent uses is called Bayesian optimization.

[02:14:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8076s) And Bayesian optimization is a technique that's used to find the most optimal numbers in any given situation.

[02:14:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8085s) So in our situation, it's to find the optimal numbers in this VLN recipe.

[02:14:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8090s) It uses mathematical steps that essentially minimizes the search space.

[02:14:55](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8095s) So by minimizing the search space, it reduces a lot of the time that it takes in finding the optimized numbers in this recipe during the run.

[02:15:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8105s) So Bayesian optimization is a strategy that is used, but Optuna is the tool that uses a strategy in the search loop.

[02:15:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8114s) So Optuna uses Bayesian optimization.

[02:15:16](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8116s) It feeds it the current recipe, the starting recipe.

[02:15:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8120s) Vision Optimization runs on that, and over many iterations or many trials, it eventually

[02:15:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8126s) finds the optimized recipe.

[02:15:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8128s) And Optuna basically takes each recipe after every trial, assesses it, gets a score, uses

[02:15:34](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8134s) the strategy to change the recipe, improve it, and then it reruns it within the search

[02:15:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8139s) loop until the trials is done.

[02:15:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8141s) So let's go a little bit into how we do this.

[02:15:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8145s) So I'm currently in a droplet here, and this droplet, for example, has GPT OSS 120b model loaded up.

[02:15:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8154s) So what we can do is we can look at the current recipe.

[02:16:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8162s) So what we see here is four regimes, right, that we're testing it on.

[02:16:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8165s) So there's chat short, there's chat long, there's rag and batch classify.

[02:16:10](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8170s) They have their own set of ISL OSL tokens that we're feeding it through.

[02:16:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8173s) All these can be changed depending on what we want to see.

[02:16:18](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8178s) But essentially this is the starting config, starting recipe that you would get when you download the weights of the model.

[02:16:26](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8186s) So moving on, once you have the model downloaded in your droplet and you're ready to run this agent,

[02:16:33](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8193s) you paste in this command, which essentially tells you where the weights are coming from,

[02:16:39](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8199s) to activate a virtual environment and have, or not have, but run the script called tuner.py.

[02:16:48](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8208s) This script essentially tells you, or it has a couple of things you have to tell it.

[02:16:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8211s) So you want to give it the name of the study.

[02:16:54](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8214s) You want to tell it the starting YAML file, which we just saw above here.

[02:17:00](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8220s) You want to tell it the hardware that it's running on.

[02:17:02](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8222s) And you want to tell it the number of trials that you want.

[02:17:05](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8225s) So in this example, we have 24.

[02:17:08](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8228s) Briefly, I'll show you another example of a past run that was run that had only nine

[02:17:13](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8233s) trials.

[02:17:14](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8234s) So anyway, this is the command that's run.

[02:17:20](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8240s) Typically these commands take a couple hours to run.

[02:17:25](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8245s) I think for the GPT-O, it says it took about four or five hours.

[02:17:28](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8248s) And then for the other model, it took a couple hours as well.

[02:17:32](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8252s) So we can go into the HTML report that's generated after the agent finishes its work.

[02:17:41](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8261s) So this one is for the GPT-OSS model that I mentioned.

[02:17:44](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8264s) What we see in the HTML report is which trial number was the best trial out of the total of how many trials.

[02:17:50](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8270s) So in this case, it was 50 trials.

[02:17:52](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8272s) So at the end of the run, you get an optimized YAML file, which includes all the numbers that we currently look for in the BLM serving recipe.

[02:18:01](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8281s) So in this case, we found, or the agent found that 256 was the optimized number, so on and so forth.

[02:18:07](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8287s) And the way that it's getting the accuracy is the amount of good put tokens that's being generated.

[02:18:12](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8292s) So overall, we found that from the first config, we had an improvement of 17.5% in terms of tokens generated.

[02:18:21](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8301s) And this one, as I mentioned, is for GPT-OSS.

[02:18:24](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8304s) We have another HTML report here, which is for the DeepSeq R1 distill model.

[02:18:29](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8309s) In here we found that, or in here we had nine trials that was run, and out of the nine,

[02:18:36](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8316s) the ninth one was the best.

[02:18:37](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8317s) We found that token throughput doubled, and ITL was improved by almost half, aka it was

[02:18:43](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8323s) reduced by almost half.

[02:18:45](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8325s) So this is a quick intro and brief demo on the commands you run and what the output looks

[02:18:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8331s) like.

[02:18:51](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8331s) In general, you get an HTML report that tells you how each trial went, and then you also

[02:18:57](https://www.youtube.com/watch?v=y9JtCCZP2MQ&t=8337s) get a YAML file that has the optimized recipe at the end of the day. So hopefully that was helpful.