# Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean Rohan Pandey, DigitalOcean · Balaji Varadarajan, DigitalOcean [Talk page](https://ai.engineer/talks/cWrYEBlWpcY-stop-shipping-llm-settings-that-only-win) · [Video](https://www.youtube.com/watch?v=cWrYEBlWpcY) ## Stop Shipping LLM Settings That Only Win on Paper DigitalOcean’s tuning agent tests vLLM recipes on inference hardware and measures how many tokens the server can deliver within its latency targets. Two completed studies show what a winning configuration changed and why the metric matters. ### How a measured trial changes the next recipe [0:01](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=1s) Downloading a model gives an engineer a starting recipe for serving it. Finding settings that work well on a particular machine can take hours of trials. In Rohan Pandey’s demo, DigitalOcean moves that search into the background. An engineer supplies the starting configuration, and the agent takes over the work of changing settings and measuring what happens. The agent uses Optuna to run a Bayesian optimization study. Each trial pairs a configuration with a measured score. Optuna uses those observations to choose which combination of settings to try next, so the agent can spend its trial budget on promising recipes without testing every combination. When the budget runs out, the winner is the highest-scoring configuration the study tried. DigitalOcean’s description of the production setup follows a proposal onto its AMD MI355 inference servers. The agent rolls out a candidate vLLM and Kubernetes configuration, measures throughput against latency targets, then keeps the winner or restores the previous setup. Pandey calls the scoring measure *goodput*. A setting can increase total tokens per second yet do little for goodput if more requests miss their latency target. ### Four workloads define the test [2:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=128s) The demo opens on a Droplet with a **GPT-OSS** model already loaded. What traffic will exercise it? The starting recipe defines four workloads — short chat, long chat, retrieval-augmented generation (RAG) and batch classification. Each has an input sequence length (ISL) and an output sequence length (OSL), both measured in tokens. Read the pair together to see how much text the server must process for a request and how much it must generate in reply. Those lengths are adjustable. A short prompt followed by a long reply gives the server a different mix of work from a long input followed by a short classification. The starting YAML thus fixes two things for the experiment. It gives the tuner an initial serving configuration and describes the traffic under which a change will be judged. ### Launching a study with a trial budget [2:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=173s) What does the agent need to turn that YAML into a run? With the weights on the Droplet, the launch command points to their location, activates a virtual environment and runs `tuner.py`. The script takes a study name, the starting YAML file, the hardware it will run on and the number of trials to perform. The example requests 24 trials. That last argument tells the search when to stop. These runs take long enough that Pandey switches to completed studies to show the results. He recalls roughly four or five hours for GPT-OSS and a couple of hours for the other model. Each completed report comes from a run with its own trial budget. ### GPT-OSS gained 17.5% over its starting recipe [3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) The completed GPT-OSS study ran 50 trials. Its HTML report identifies the best trial in that field. The companion YAML records the serving parameters for the selected configuration, so an engineer can use the winning recipe without reconstructing it from the trial history. Pandey points out one selected value, `256`, without naming its YAML key. The comparison applies to the recipe as a whole. Measured by generated tokens that counted toward goodput, the winning configuration improved on the first configuration by 17.5%. ### DeepSeek’s ninth trial improved throughput and token timing [4:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=289s) The second report covers a **DeepSeek R1 Distill** model. The ninth of nine trials scored best. What changed for the server and for someone reading a streamed response? - **Token throughput doubled.** The server generated twice as many tokens per unit of time in the reported comparison. - **Inter-token latency fell by almost half.** Once generation was underway, the wait between successive tokens was roughly half as long. At the end of either study, the HTML report shows how each trial went and the YAML gives an engineer the settings to run again. That distinction matters for a close result. DigitalOcean’s recording description gives an example from its four studies in which an apparent 4% gain turned out to be run-to-run noise. The same description identifies the team’s work in progress: extending the agent to disaggregated serving, where prefill processes the prompt and decode generates successive tokens on separate serving resources. ## Key takeaways - Read a winning recipe alongside the workload that selected it. Input and output token lengths determine the mix of work per request, and goodput counts the throughput that meets the latency targets. - Keep the metric attached to the gain. GPT-OSS improved goodput by 17.5% over its first configuration in a 50-trial study. DeepSeek’s best trial doubled token throughput and cut the delay between successive tokens by almost half. ## Slides ### Configuration optimizer [3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) [Configuration dashboard with an objective of 1,586 weighted tokens per second and the serving settings from trial 47.](https://assets.ai.engineer/sites/aiecode2025/sha256/f58876226495195942c7f61b9d176e512ba182c079e0cf4ca5ea6e3ed6d4e005.webp) Configuration Optimizer. Best objective: 1,586 weighted tok/s under SLO. Improvement: +17.5% vs first config. Best trial #47 of 50. Four traffic types jointly optimized. ### Goodput for each traffic type [4:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=270s) [Serving parameters and goodput tables for short chat, long chat, retrieval and steady batch traffic.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a4dc4498e8e4e826367648cb7c9f7e82959cd53595ace71dd5b5f7c9986dba7.webp) Optimized serving config (deployable). Goodput per traffic type — best config. Traffic type, goodput (tok/s), concurrency, p99 TTFT, p99 TPOT, SLO (TTFT/TPOT). chat_short, chat_long, rag, batch_steady. ### Latency and throughput in a tuning report [4:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=288s) [The highlighted results compare the baseline inter-token latency with the fastest of ten trials.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ee38886bbd4cab143f1d32e733ee4f7c16dfec4a2cfc231714c989416f2dc81.webp) vLLM tuning report. Baseline ITL p50: 76 ms. Best ITL p50: 37 ms. ITL improvement: 51%. Throughput: 2.0x. Best trial #9. ### Which tuning settings changed? [4:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=296s) [Tuning table highlights settings changed from the baseline, marks reverted runs, and shows the winning row in green.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bf73678132cfd2e5199d3500971a29956a85c4cb4d1735d7c59c49b9a46d16a.webp) BEST. Highlighted cells = knob differs from baseline. Crashed trials failed to boot and were auto-reverted. ## Full transcript [0:01](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=1s) Right. Hello everyone. So my name is [0:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=3s) Right. Hello everyone. So my name is [0:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=3s) Right. Hello everyone. So my name is Rohan and this is a quick demo on the [0:06](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=6s) Rohan and this is a quick demo on the [0:06](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=6s) Rohan and this is a quick demo on the optimization agent that what has been [0:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=9s) optimization agent that what has been [0:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=9s) optimization agent that what has been recently built. So a quick intro into [0:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=12s) recently built. So a quick intro into [0:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=12s) recently built. So a quick intro into what the purpose is and what the goal is [0:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=14s) what the purpose is and what the goal is [0:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=14s) what the purpose is and what the goal is of this agent and then I'll get into a [0:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=17s) of this agent and then I'll get into a [0:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=17s) of this agent and then I'll get into a brief demo and results that uh from a [0:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=19s) brief demo and results that uh from a [0:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=19s) brief demo and results that uh from a previous run that we just did. So what [0:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=21s) previous run that we just did. So what [0:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=22s) previous run that we just did. So what the goal is here is to build an agent [0:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=24s) the goal is here is to build an agent [0:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=24s) the goal is here is to build an agent that helps um find the best VLM serving [0:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=29s) that helps um find the best VLM serving [0:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=29s) that helps um find the best VLM serving config for any given model. Uh the [0:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=32s) config for any given model. Uh the [0:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=32s) config for any given model. Uh the purpose is to reduce the time it takes [0:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=34s) purpose is to reduce the time it takes [0:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=34s) purpose is to reduce the time it takes to manually um find the best configs and [0:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=38s) to manually um find the best configs and [0:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=38s) to manually um find the best configs and the VLM serving recipe. And if we can [0:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=40s) the VLM serving recipe. And if we can [0:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=40s) the VLM serving recipe. And if we can automate this and have it run in the [0:42](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=42s) automate this and have it run in the [0:42](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=42s) automate this and have it run in the background and um have the agent find [0:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=45s) background and um have the agent find [0:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=45s) background and um have the agent find the recipes and the optimized parameters [0:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=47s) the recipes and the optimized parameters [0:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=47s) the recipes and the optimized parameters faster, then it will save a lot of time. [0:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=50s) faster, then it will save a lot of time. [0:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=50s) faster, then it will save a lot of time. So that's the goal of the agent and a [0:52](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=52s) So that's the goal of the agent and a [0:52](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=52s) So that's the goal of the agent and a little bit more into how the agent [0:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=54s) little bit more into how the agent [0:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=54s) little bit more into how the agent actually does this. So the strategy that [0:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=57s) actually does this. So the strategy that [0:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=57s) actually does this. So the strategy that the agent uses is called Beijian [0:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=59s) the agent uses is called Beijian [0:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=59s) the agent uses is called Beijian optimization and bian optimization is a [1:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=62s) optimization and bian optimization is a [1:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=62s) optimization and bian optimization is a technique um that's used to find the [1:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=65s) technique um that's used to find the [1:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=65s) technique um that's used to find the most optimal numbers um in any given [1:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=68s) most optimal numbers um in any given [1:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=68s) most optimal numbers um in any given situation. So in our situation it's to [1:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=71s) situation. So in our situation it's to [1:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=71s) situation. So in our situation it's to find the optimal numbers in this uh VLM [1:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=74s) find the optimal numbers in this uh VLM [1:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=74s) find the optimal numbers in this uh VLM recipe. Uh it uses mathematical steps [1:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=76s) recipe. Uh it uses mathematical steps [1:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=76s) recipe. Uh it uses mathematical steps that essentially minimizes the search [1:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=79s) that essentially minimizes the search [1:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=79s) that essentially minimizes the search space. So by minimizing the search space [1:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=81s) space. So by minimizing the search space [1:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=81s) space. So by minimizing the search space it uh reduces a lot of the time that it [1:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=83s) it uh reduces a lot of the time that it [1:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=83s) it uh reduces a lot of the time that it takes in finding the optimized uh [1:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=86s) takes in finding the optimized uh [1:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=86s) takes in finding the optimized uh numbers in this uh recipe during the [1:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=88s) numbers in this uh recipe during the [1:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=88s) numbers in this uh recipe during the run. So B uh basian optimization is a [1:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=91s) run. So B uh basian optimization is a [1:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=91s) run. So B uh basian optimization is a strategy that is used but optuna is the [1:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=94s) strategy that is used but optuna is the [1:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=94s) strategy that is used but optuna is the tool that uses a strategy in the search [1:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=97s) tool that uses a strategy in the search [1:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=97s) tool that uses a strategy in the search loop. So optuna uses beijan optimization [1:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=100s) loop. So optuna uses beijan optimization [1:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=100s) loop. So optuna uses beijan optimization it feeds it the current recipe the [1:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=103s) it feeds it the current recipe the [1:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=103s) it feeds it the current recipe the starting recipe. uh Beijian optimization [1:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=105s) starting recipe. uh Beijian optimization [1:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=105s) starting recipe. uh Beijian optimization runs on that and over many iterations or [1:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=108s) runs on that and over many iterations or [1:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=108s) runs on that and over many iterations or many trials, it eventually finds the [1:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=110s) many trials, it eventually finds the [1:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=110s) many trials, it eventually finds the optimized recipe and Optina basically [1:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=113s) optimized recipe and Optina basically [1:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=113s) optimized recipe and Optina basically takes each recipe after every trial, [1:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=115s) takes each recipe after every trial, [1:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=115s) takes each recipe after every trial, assesses it, gets a score, um uses a [1:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=118s) assesses it, gets a score, um uses a [1:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=118s) assesses it, gets a score, um uses a strategy to change the recipe, improve [2:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=120s) strategy to change the recipe, improve [2:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=120s) strategy to change the recipe, improve it and then it reruns it within the uh [2:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=123s) it and then it reruns it within the uh [2:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=123s) it and then it reruns it within the uh search loop um until the trials is done. [2:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=125s) search loop um until the trials is done. [2:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=125s) search loop um until the trials is done. So let's go a little bit into how uh we [2:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=128s) So let's go a little bit into how uh we [2:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=128s) So let's go a little bit into how uh we do this. So I'm currently in a droplet [2:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=131s) do this. So I'm currently in a droplet [2:11](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=131s) do this. So I'm currently in a droplet here and this droplet for example has [2:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=133s) here and this droplet for example has [2:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=133s) here and this droplet for example has GPT OSS uh 12B model loaded up. So what [2:18](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=138s) GPT OSS uh 12B model loaded up. So what [2:18](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=138s) GPT OSS uh 12B model loaded up. So what we can do is uh we can look at the [2:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=143s) we can do is uh we can look at the [2:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=143s) we can do is uh we can look at the current [2:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=144s) current [2:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=144s) current um recipe. So what we see here is four [2:27](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=147s) um recipe. So what we see here is four [2:27](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=147s) um recipe. So what we see here is four regimes right that we're testing it on. [2:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=149s) regimes right that we're testing it on. [2:29](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=149s) regimes right that we're testing it on. So there's chat short, there's chat [2:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=151s) So there's chat short, there's chat [2:31](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=151s) So there's chat short, there's chat long, there's rag and batch classify. Uh [2:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=154s) long, there's rag and batch classify. Uh [2:34](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=154s) long, there's rag and batch classify. Uh they have their own set of ISL OSL [2:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=156s) they have their own set of ISL OSL [2:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=156s) they have their own set of ISL OSL tokens that we're feeding it through. Uh [2:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=158s) tokens that we're feeding it through. Uh [2:38](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=158s) tokens that we're feeding it through. Uh all these um can be changed depending on [2:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=160s) all these um can be changed depending on [2:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=160s) all these um can be changed depending on what uh we want to see. But essentially [2:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=163s) what uh we want to see. But essentially [2:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=163s) what uh we want to see. But essentially this is the starting uh config starting [2:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=165s) this is the starting uh config starting [2:45](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=165s) this is the starting uh config starting recipe that you would get when you [2:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=167s) recipe that you would get when you [2:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=167s) recipe that you would get when you download the weights of the model. Um so [2:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=170s) download the weights of the model. Um so [2:50](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=170s) download the weights of the model. Um so moving once you have the model [2:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=173s) moving once you have the model [2:53](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=173s) moving once you have the model downloaded in your droplet and you're [2:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=175s) downloaded in your droplet and you're [2:55](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=175s) downloaded in your droplet and you're ready to run this agent you paste in [2:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=178s) ready to run this agent you paste in [2:58](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=178s) ready to run this agent you paste in this command uh which essentially tells [3:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=180s) this command uh which essentially tells [3:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=180s) this command uh which essentially tells you you know where the where the weights [3:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=182s) you you know where the where the weights [3:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=182s) you you know where the where the weights are coming from. um activate a virtual [3:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=185s) are coming from. um activate a virtual [3:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=185s) are coming from. um activate a virtual um environment and have um uh or not [3:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=189s) um environment and have um uh or not [3:09](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=189s) um environment and have um uh or not have but run the script called tuner.py. [3:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=192s) have but run the script called tuner.py. [3:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=192s) have but run the script called tuner.py. Uh this script essentially tells you or [3:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=193s) Uh this script essentially tells you or [3:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=194s) Uh this script essentially tells you or it has a couple of things you have to [3:15](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=195s) it has a couple of things you have to [3:15](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=195s) it has a couple of things you have to tell it. So you you want to give it the [3:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=197s) tell it. So you you want to give it the [3:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=197s) tell it. So you you want to give it the name of the study. You want to tell it [3:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=199s) name of the study. You want to tell it [3:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=199s) name of the study. You want to tell it the starting uh YAML file which we just [3:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=202s) the starting uh YAML file which we just [3:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=202s) the starting uh YAML file which we just saw above here. You want to tell it the [3:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=204s) saw above here. You want to tell it the [3:24](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=204s) saw above here. You want to tell it the hardware that it's running on and you [3:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=206s) hardware that it's running on and you [3:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=206s) hardware that it's running on and you want to tell it the number of trials [3:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=208s) want to tell it the number of trials [3:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=208s) want to tell it the number of trials that you want. So uh in this example we [3:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=210s) that you want. So uh in this example we [3:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=210s) that you want. So uh in this example we have 24. Um, briefly I'll show you [3:33](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=213s) have 24. Um, briefly I'll show you [3:33](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=213s) have 24. Um, briefly I'll show you another example of a past run that was [3:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=216s) another example of a past run that was [3:36](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=216s) another example of a past run that was run uh that had only nine trials. So um [3:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=220s) run uh that had only nine trials. So um [3:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=220s) run uh that had only nine trials. So um uh anyway uh this is the command that's [3:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=223s) uh anyway uh this is the command that's [3:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=223s) uh anyway uh this is the command that's that's run. Uh typically these commands [3:46](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=226s) that's run. Uh typically these commands [3:46](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=226s) that's run. Uh typically these commands take um a couple hours to run. I think [3:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=229s) take um a couple hours to run. I think [3:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=229s) take um a couple hours to run. I think for the GPTO it says it took about four [3:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=231s) for the GPTO it says it took about four [3:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=231s) for the GPTO it says it took about four or five hours and then for the other [3:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=234s) or five hours and then for the other [3:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=234s) or five hours and then for the other model it took a couple hours as well. So [3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) model it took a couple hours as well. So [3:56](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=236s) model it took a couple hours as well. So we can go into the HTML report that's [3:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=239s) we can go into the HTML report that's [3:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=239s) we can go into the HTML report that's generated after the uh agent finishes [4:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=243s) generated after the uh agent finishes [4:03](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=243s) generated after the uh agent finishes its work. Um so this one is for the [4:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=245s) its work. Um so this one is for the [4:06](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=246s) its work. Um so this one is for the GPTOSS model that I mentioned. Uh what [4:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=248s) GPTOSS model that I mentioned. Uh what [4:08](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=248s) GPTOSS model that I mentioned. Uh what what we see in the HTML report is which [4:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=250s) what we see in the HTML report is which [4:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=250s) what we see in the HTML report is which trial number was the best trial out of [4:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=252s) trial number was the best trial out of [4:12](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=252s) trial number was the best trial out of the total of how many trials. So in this [4:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=254s) the total of how many trials. So in this [4:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=254s) the total of how many trials. So in this case it was 50 trials. uh you get so at [4:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=257s) case it was 50 trials. uh you get so at [4:17](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=257s) case it was 50 trials. uh you get so at the end of the run you get an optimized [4:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=259s) the end of the run you get an optimized [4:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=259s) the end of the run you get an optimized uh YAML file which includes all the [4:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=261s) uh YAML file which includes all the [4:21](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=261s) uh YAML file which includes all the numbers that we currently look for in [4:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=263s) numbers that we currently look for in [4:23](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=263s) numbers that we currently look for in the BLM serving recipe. So in this case [4:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=266s) the BLM serving recipe. So in this case [4:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=266s) the BLM serving recipe. So in this case we found or the agent found that 256 was [4:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=268s) we found or the agent found that 256 was [4:28](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=268s) we found or the agent found that 256 was the optimized number uh so on and so [4:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=270s) the optimized number uh so on and so [4:30](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=270s) the optimized number uh so on and so forth and the way that it's getting the [4:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=272s) forth and the way that it's getting the [4:32](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=272s) forth and the way that it's getting the accuracy is the amount of goodput tokens [4:35](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=275s) accuracy is the amount of goodput tokens [4:35](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=275s) accuracy is the amount of goodput tokens that's being generated. So uh overall we [4:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=277s) that's being generated. So uh overall we [4:37](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=277s) that's being generated. So uh overall we found that from the first config we had [4:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=280s) found that from the first config we had [4:40](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=280s) found that from the first config we had an improvement of 17.5% in terms of uh [4:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=283s) an improvement of 17.5% in terms of uh [4:43](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=283s) an improvement of 17.5% in terms of uh tokens generated and this one as I [4:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=287s) tokens generated and this one as I [4:47](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=287s) tokens generated and this one as I mentioned is for GPOSS. We have another [4:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=289s) mentioned is for GPOSS. We have another [4:49](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=289s) mentioned is for GPOSS. We have another HTML report here which is for the [4:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=291s) HTML report here which is for the [4:51](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=291s) HTML report here which is for the deepseek R1 distill uh model. In here we [4:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=294s) deepseek R1 distill uh model. In here we [4:54](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=294s) deepseek R1 distill uh model. In here we found that or in here we had uh nine [4:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=297s) found that or in here we had uh nine [4:57](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=297s) found that or in here we had uh nine trials that was run and out of the nine [4:59](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=299s) trials that was run and out of the nine [5:00](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=300s) trials that was run and out of the nine the ninth one was the best. uh we found [5:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=302s) the ninth one was the best. uh we found [5:02](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=302s) the ninth one was the best. uh we found that token throughput uh doubled and ITL [5:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=305s) that token throughput uh doubled and ITL [5:05](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=305s) that token throughput uh doubled and ITL was improved by almost half aka it was [5:07](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=307s) was improved by almost half aka it was [5:07](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=307s) was improved by almost half aka it was reduced by almost half. So this is the [5:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=310s) reduced by almost half. So this is the [5:10](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=310s) reduced by almost half. So this is the quick intro and brief demo on the [5:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=313s) quick intro and brief demo on the [5:13](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=313s) quick intro and brief demo on the commands you run and what the output [5:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=314s) commands you run and what the output [5:14](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=314s) commands you run and what the output looks like in in in general you get a [5:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=316s) looks like in in in general you get a [5:16](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=316s) looks like in in in general you get a HTML report that tells you um how each [5:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=319s) HTML report that tells you um how each [5:19](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=319s) HTML report that tells you um how each trial went and then you also get um a [5:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=322s) trial went and then you also get um a [5:22](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=322s) trial went and then you also get um a YAML file that has the optimized recipe [5:25](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=325s) YAML file that has the optimized recipe [5:25](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=325s) YAML file that has the optimized recipe at the end of the day. So hopefully that [5:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=326s) at the end of the day. So hopefully that [5:26](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=326s) at the end of the day. So hopefully that was helpful. Thank you.