1 02:13:37,820 --> 02:13:43,640 All right. Hello, everyone. So my name is Rohan, and this is a quick demo on the optimization 2 02:13:43,640 --> 02:13:50,780 agent that has been recently built. So a quick intro into what the purpose is and what the 3 02:13:50,780 --> 02:13:55,680 goal is of this agent, and then I'll get into a brief demo and results from a previous 4 02:13:55,680 --> 02:14:03,720 run that we just did. So what the goal is here is to build an agent that helps find 5 02:14:03,720 --> 02:14:07,140 the best VLM serving config for any given model. 6 02:14:07,940 --> 02:14:10,360 The purpose is to reduce the time it takes 7 02:14:10,360 --> 02:14:14,440 to manually find the best configs 8 02:14:14,440 --> 02:14:15,600 and the VLM serving recipe. 9 02:14:16,040 --> 02:14:17,860 And if we can automate this 10 02:14:17,860 --> 02:14:19,160 and have it run in the background 11 02:14:19,160 --> 02:14:22,140 and have the agent find the recipes 12 02:14:22,140 --> 02:14:24,460 and the optimized parameters faster, 13 02:14:24,620 --> 02:14:26,040 then it will save a lot of time. 14 02:14:26,320 --> 02:14:28,060 So that's the goal of the agent 15 02:14:28,060 --> 02:14:31,480 and a little bit more into how the agent actually does this. 16 02:14:31,480 --> 02:14:35,840 So the strategy that the agent uses is called Bayesian optimization. 17 02:14:36,300 --> 02:14:45,400 And Bayesian optimization is a technique that's used to find the most optimal numbers in any given situation. 18 02:14:45,400 --> 02:14:50,580 So in our situation, it's to find the optimal numbers in this VLN recipe. 19 02:14:50,860 --> 02:14:55,800 It uses mathematical steps that essentially minimizes the search space. 20 02:14:55,800 --> 02:15:05,140 So by minimizing the search space, it reduces a lot of the time that it takes in finding the optimized numbers in this recipe during the run. 21 02:15:05,840 --> 02:15:14,000 So Bayesian optimization is a strategy that is used, but Optuna is the tool that uses a strategy in the search loop. 22 02:15:14,200 --> 02:15:16,480 So Optuna uses Bayesian optimization. 23 02:15:16,580 --> 02:15:20,520 It feeds it the current recipe, the starting recipe. 24 02:15:20,520 --> 02:15:26,100 Vision Optimization runs on that, and over many iterations or many trials, it eventually 25 02:15:26,100 --> 02:15:27,780 finds the optimized recipe. 26 02:15:28,400 --> 02:15:34,520 And Optuna basically takes each recipe after every trial, assesses it, gets a score, uses 27 02:15:34,520 --> 02:15:39,540 the strategy to change the recipe, improve it, and then it reruns it within the search 28 02:15:39,540 --> 02:15:41,440 loop until the trials is done. 29 02:15:41,940 --> 02:15:45,600 So let's go a little bit into how we do this. 30 02:15:45,600 --> 02:15:53,400 So I'm currently in a droplet here, and this droplet, for example, has GPT OSS 120b model loaded up. 31 02:15:54,120 --> 02:16:02,000 So what we can do is we can look at the current recipe. 32 02:16:02,160 --> 02:16:05,840 So what we see here is four regimes, right, that we're testing it on. 33 02:16:05,960 --> 02:16:09,960 So there's chat short, there's chat long, there's rag and batch classify. 34 02:16:10,280 --> 02:16:13,680 They have their own set of ISL OSL tokens that we're feeding it through. 35 02:16:13,680 --> 02:16:18,360 All these can be changed depending on what we want to see. 36 02:16:18,960 --> 02:16:24,840 But essentially this is the starting config, starting recipe that you would get when you download the weights of the model. 37 02:16:26,360 --> 02:16:32,920 So moving on, once you have the model downloaded in your droplet and you're ready to run this agent, 38 02:16:33,160 --> 02:16:39,240 you paste in this command, which essentially tells you where the weights are coming from, 39 02:16:39,840 --> 02:16:48,060 to activate a virtual environment and have, or not have, but run the script called tuner.py. 40 02:16:48,360 --> 02:16:51,740 This script essentially tells you, or it has a couple of things you have to tell it. 41 02:16:51,920 --> 02:16:54,080 So you want to give it the name of the study. 42 02:16:54,420 --> 02:16:59,900 You want to tell it the starting YAML file, which we just saw above here. 43 02:17:00,080 --> 02:17:01,980 You want to tell it the hardware that it's running on. 44 02:17:02,360 --> 02:17:04,920 And you want to tell it the number of trials that you want. 45 02:17:05,080 --> 02:17:07,600 So in this example, we have 24. 46 02:17:08,560 --> 02:17:13,860 Briefly, I'll show you another example of a past run that was run that had only nine 47 02:17:13,860 --> 02:17:14,260 trials. 48 02:17:14,880 --> 02:17:20,320 So anyway, this is the command that's run. 49 02:17:20,980 --> 02:17:25,120 Typically these commands take a couple hours to run. 50 02:17:25,300 --> 02:17:28,640 I think for the GPT-O, it says it took about four or five hours. 51 02:17:28,900 --> 02:17:32,020 And then for the other model, it took a couple hours as well. 52 02:17:32,020 --> 02:17:40,780 So we can go into the HTML report that's generated after the agent finishes its work. 53 02:17:41,120 --> 02:17:43,780 So this one is for the GPT-OSS model that I mentioned. 54 02:17:44,240 --> 02:17:50,240 What we see in the HTML report is which trial number was the best trial out of the total of how many trials. 55 02:17:50,340 --> 02:17:51,600 So in this case, it was 50 trials. 56 02:17:52,780 --> 02:18:01,440 So at the end of the run, you get an optimized YAML file, which includes all the numbers that we currently look for in the BLM serving recipe. 57 02:18:01,440 --> 02:18:07,200 So in this case, we found, or the agent found that 256 was the optimized number, so on and so forth. 58 02:18:07,380 --> 02:18:12,280 And the way that it's getting the accuracy is the amount of good put tokens that's being generated. 59 02:18:12,500 --> 02:18:20,420 So overall, we found that from the first config, we had an improvement of 17.5% in terms of tokens generated. 60 02:18:21,720 --> 02:18:24,300 And this one, as I mentioned, is for GPT-OSS. 61 02:18:24,600 --> 02:18:29,900 We have another HTML report here, which is for the DeepSeq R1 distill model. 62 02:18:29,900 --> 02:18:36,240 In here we found that, or in here we had nine trials that was run, and out of the nine, 63 02:18:36,360 --> 02:18:37,340 the ninth one was the best. 64 02:18:37,740 --> 02:18:43,920 We found that token throughput doubled, and ITL was improved by almost half, aka it was 65 02:18:43,920 --> 02:18:45,300 reduced by almost half. 66 02:18:45,600 --> 02:18:51,120 So this is a quick intro and brief demo on the commands you run and what the output looks 67 02:18:51,120 --> 02:18:51,320 like. 68 02:18:51,440 --> 02:18:57,380 In general, you get an HTML report that tells you how each trial went, and then you also 69 02:18:57,380 --> 02:19:03,560 get a YAML file that has the optimized recipe at the end of the day. So hopefully that was helpful.