Lu Zhang

Lu Zhang is an infrastructure engineer whose work spans GPU clusters and large-scale inference platforms. He contributed to inference compute for GPT-4o, work also credited in the model’s 2024 system card.

Earlier in his career, Zhang built and operated GPU clusters for machine-learning workloads before focusing on model serving. His work connects the capacity of a computing fleet to individual requests: choosing which engine should handle a request, preserving computation that can be reused, and keeping services responsive under load.

In his 2026 presentation with Qianru Lao, Zhang explained how feedback-driven routing can cause traffic to oscillate between engines, disrupting KV-cache reuse across conversation turns. He also described retry budgets that constrain the extra work caused by failures, and load shedding when demand exceeds capacity. Lao explained their newer asynchronous routing architecture. Together, their work addresses the tradeoff between distributing requests across available hardware and preserving the state that makes subsequent requests cheaper to serve.