Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean

Downloading a model gives an engineer a starting recipe for serving it. Finding settings that work well on a particular machine can take hours of trials. In Rohan Pandey’s demo, DigitalOcean moves that search into the background. An engineer supplies the starting configuration, and the agent takes over the work of changing settings and measuring what happens.

The agent uses Optuna to run a Bayesian optimization study. Each trial pairs a configuration with a measured score. Optuna uses those observations to choose which combination of settings to try next, so the agent can spend its trial budget on promising recipes without testing every combination. When the budget runs out, the winner is the highest-scoring configuration the study tried.

DigitalOcean’s description of the production setup follows a proposal onto its AMD MI355 inference servers. The agent rolls out a candidate vLLM and Kubernetes configuration, measures throughput against latency targets, then keeps the winner or restores the previous setup. Pandey calls the scoring measure goodput. A setting can increase total tokens per second yet do little for goodput if more requests miss their latency target.

The demo opens on a Droplet with a GPT-OSS model already loaded. What traffic will exercise it? The starting recipe defines four workloads — short chat, long chat, retrieval-augmented generation (RAG) and batch classification. Each has an input sequence length (ISL) and an output sequence length (OSL), both measured in tokens. Read the pair together to see how much text the server must process for a request and how much it must generate in reply.

Those lengths are adjustable. A short prompt followed by a long reply gives the server a different mix of work from a long input followed by a short classification. The starting YAML thus fixes two things for the experiment. It gives the tuner an initial serving configuration and describes the traffic under which a change will be judged.

What does the agent need to turn that YAML into a run? With the weights on the Droplet, the launch command points to their location, activates a virtual environment and runs tuner.py. The script takes a study name, the starting YAML file, the hardware it will run on and the number of trials to perform. The example requests 24 trials. That last argument tells the search when to stop.

These runs take long enough that Pandey switches to completed studies to show the results. He recalls roughly four or five hours for GPT-OSS and a couple of hours for the other model. Each completed report comes from a run with its own trial budget.

The completed GPT-OSS study ran 50 trials. Its HTML report identifies the best trial in that field. The companion YAML records the serving parameters for the selected configuration, so an engineer can use the winning recipe without reconstructing it from the trial history.

Pandey points out one selected value, 256, without naming its YAML key. The comparison applies to the recipe as a whole. Measured by generated tokens that counted toward goodput, the winning configuration improved on the first configuration by 17.5%.

The second report covers a DeepSeek R1 Distill model. The ninth of nine trials scored best. What changed for the server and for someone reading a streamed response?

• Token throughput doubled. The server generated twice as many tokens per unit of time in the reported comparison.
• Inter-token latency fell by almost half. Once generation was underway, the wait between successive tokens was roughly half as long.

At the end of either study, the HTML report shows how each trial went and the YAML gives an engineer the settings to run again. That distinction matters for a close result. DigitalOcean’s recording description gives an example from its four studies in which an apparent 4% gain turned out to be run-to-run noise. The same description identifies the team’s work in progress: extending the agent to disaggregated serving, where prefill processes the prompt and decode generates successive tokens on separate serving resources.