Chen Wang

Open-source model inference is becoming a compelling option for enterprises, especially those building AI-native applications. Delivering fast, efficient, and reliable inference requires engineering across the full stack, from CUDA kernels and Kubernetes orchestration to quantization, caching, and disaggregation. This talk explores the key design decisions and emerging techniques shaping production inference systems, with practical insights into balancing latency, throughput, cost, and reliability.