Yunmo Koo

Yunmo Koo is a founding engineer at FriendliAI whose work spans distributed training and the systems that generate language-model responses. He led initial development of the PeriFlow training platform and later developed inference optimizations, including speculative decoding that adapts to production traffic. His engineering work connects GPU kernels and runtime scheduling with the infrastructure developers use to train and deploy models.

From Python execution to distributed training

Yunmo studied computer science and Korean history at Seoul National University, then earned a master’s degree in computer science in its Software Platform Lab. His research included computational-graph optimization and a thesis on cost-efficient machine-learning training using preemptible cloud clusters: discounted computing resources that a provider can reclaim while a job is running.

He was one of seven co-authors of Terra, a deep-learning execution system published at NeurIPS in 2021. Terra addressed a tradeoff in training software: converting Python programs into optimized computation graphs could accelerate execution, but unsupported language features or third-party calls could prevent those programs from running. The team’s design retained ordinary Python execution while delegating deep-learning operations to a separate graph. Terra accelerated all ten programs in its evaluation; TensorFlow’s AutoGraph could not execute five of them.

Yunmo joined FriendliAI in February 2021, overlapping with his graduate studies. He led initial development of PeriFlow, a distributed-training platform for GPU resources across multiple clouds, including its fault-tolerant resource management. His work also covered the SDK, command-line tools, and web interfaces developers used to access the platform.

He helped train and release GPT-FAI 13B, a 13-billion-parameter language model that FriendliAI released in May 2022 for noncommercial research. The model was a concrete application of the company’s training platform.

Adapting inference to production workloads

Yunmo’s subsequent work covers several parts of inference, from proposing tokens to coordinating GPU execution and integrating the engine into applications:

• Adaptive speculative decoding: He led development of production speculation systems, implemented online draft-model training, and designed a hybrid speculator that routes between model-based and model-free approaches. In draft-model decoding, a smaller model proposes several tokens and the target model checks them together. Accepted proposals produce multiple tokens per expensive target-model pass. Online training lets the drafter adapt during production inference.

• Inference runtimes and kernels: He built memory management, scheduling, and KV-cache optimizations, alongside kernels for attention, sampling, and decoding. This work covers both individual GPU operations and the runtime that coordinates them.

• Application integration: He also led FriendliAI’s US solutions-architecture and forward-deployed engineering team and built integrations with LangChain and LlamaIndex, extending his responsibilities from engine implementation to customer deployments.

In “Same Model, Different Speed”, Yunmo argues that inference providers materially affect how the same model performs in an agent. He explains why long, repetitive agent inputs benefit from prefix caching: cache-aware routing sends requests toward reusable cached state, while a GPU, CPU, and NVMe hierarchy provides tiers for retaining it. He also treats failed calls as part of performance, because one failure can interrupt an entire agent run. His recommendation is to test providers on the developer’s own workload—a practical extension of his work on inference systems that adapt to the traffic they serve.