Balaji Varadarajan

Balaji Varadarajan is a staff engineer at DigitalOcean and a founding engineer of its dedicated platform for serving large language models. Based in Austin, Balaji studies how much traffic a GPU server can sustain while people using it still get a prompt first token and a steady stream of text. That limit helps determine how many servers a deployment needs and what it costs to run them.

Before DigitalOcean, Balaji held senior engineering roles at HubSpot and CCC Intelligent Solutions. At Target, he led machine-learning engineering and worked on recommendations and retrieval-augmented generation. He was also one of four inventors of Target’s system for scheduling drive-up pickups. The system could offer a shopper an incentive to choose a particular pickup window, then use the arrival registered by the shopper’s phone to determine whether they had earned it. Target filed the application in 2024 and received the patent in 2025.

Sizing a server for the workload

In his April 2026 analysis of inference throughput, latency and cost, Balaji uses Little’s Law to estimate how many requests a server must handle at once. At ten arrivals per second and an average response time of two seconds, about twenty requests are in flight. That average can hide a noticeable pause for someone using chat. When one user submits a long document, processing the entire prompt in one pass can stall replies already streaming to other users. Balaji recommends processing the prompt in chunks so the GPU can generate more tokens for those replies between chunks. For overnight batch jobs, he favors larger batches: the GPU can generate tokens for several requests each time it loads the model weights, and the whole job can finish sooner even if an individual request takes longer.

With Emilio Andere, Balaji examined a related choice in DigitalOcean’s work with Wafer on AMD GPUs. When one copy of a model spans eight GPUs, all eight repeatedly exchange partial results as they generate each token. If the model fits on four GPUs, the same server can run two independent copies. Each copy has fewer GPUs to synchronize, and the two can serve separate requests at once. Under those conditions, the server can handle more traffic and shorten the wait between tokens.

Balaji and Rohan Pandey also presented DigitalOcean’s work on automated inference tuning. Rohan demonstrated how the agent tested serving settings on workloads ranging from short chats to batch classification. It scored each configuration by how many tokens it could deliver within the latency target. In one of the team’s studies, an apparent 4% improvement fell within the ordinary variation between runs, too small to justify switching configurations on that evidence.