Introduction
Rohan Pandey demonstrates an optimization loop that tests serving configurations and returns both a report and a deployable recipe.
A model's downloaded weights do not tell an operator which serving configuration will work best for a particular machine and workload. Manually trying parameter combinations takes time, and the result can depend on whether the traffic consists of short chats, long conversations, retrieval-heavy requests, or batch classification.
Pandey's short demo starts from that practical task. He wants an optimization agent to search for a better recipe in the background, leaving the operator with a configuration they can inspect and deploy rather than a recommendation expressed only in prose.
Let measurements choose the next trial
At 02:14:07, Pandey describes the search strategy as Bayesian optimization. Instead of exhaustively trying every combination, the loop uses the results of earlier trials to guide what to try next. Optuna supplies the optimization machinery in the demonstration.
The loop starts with a recipe, tests it, assigns a score, proposes another configuration, and repeats until its trial budget is exhausted. That makes the objective important. An optimizer can search efficiently while still improving the wrong quantity if the score does not represent the intended service.
Pandey's workload includes four traffic regimes: short chat, long chat, retrieval-augmented generation, and batch classification. Each has its own input and output lengths. The mix is configurable, so the best result is conditional on the chosen traffic and hardware rather than a universal setting for the model.
Make the experiment reproducible
The demo runs on a machine with GPT-OSS 120B already available. At 02:16:26, Pandey walks through a command that selects the weights, activates an environment, and runs a tuner script.
The important inputs are the study name, starting YAML configuration, hardware specification, and number of trials. Together they identify the experiment and constrain the search. The displayed command is an example of launching a study; the report he later opens comes from a completed run with its own trial count.
That distinction prevents a misleading reading of the demo as an instantaneous optimization. Pandey says these studies take hours. The automation removes repeated manual tuning work; it does not make serving benchmarks free.
Return the evidence with the configuration
The report at 2:17:55 displays a 17.5% improvement for this study and a selected configuration. The result belongs to this workload and test setup; it is not an accuracy improvement in the language model.
The completed run produces two artifacts. The YAML file contains the selected serving parameters. The HTML report shows how trials performed and which configuration won. Keeping both lets an operator distinguish the deployable result from the measurements that justify choosing it.
Pandey describes the objective in terms of goodput: useful generated-token performance under the workload's service constraints. He also opens a second experiment on a DeepSeek R1 distilled model, reporting higher throughput and lower inter-token latency than its starting configuration. The recording does not establish that those gains transfer to a different deployment.
The useful outcome is the workflow itself. Define the traffic you care about, choose a scoring objective, let the optimizer run bounded experiments, and retain the winning recipe alongside the trial history. An agent can make the tuning process less laborious while leaving the evidence visible to the person responsible for deploying it.