Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford

Moving gigabytes of gradients between machines gives a network a relatively forgiving job: sustain high throughput over a long transfer. Connection setup and ramp-up time contribute little to the total. John Ousterhout begins by acknowledging that TCP and RDMA perform reasonably well for these workloads; in this talk, his references to RDMA concern RoCE, or RDMA over Converged Ethernet.

Ousterhout sees a different traffic mix emerging in inference and agentic applications. Smaller pieces of computation alternate with smaller exchanges of metadata and coordination data. He gives two examples: checking whether an item is present in a distributed KV cache and synchronizing workers at a barrier. Training remains an important exception in his account because it is still dominated by massive transfers. The claim is a growing mixture of message sizes, not the disappearance of bulk traffic.

A short exchange calls for a different metric. The relevant latency includes sending a small request, doing a little remote computation, and returning a small response. Average latency can still hide the message that determines progress, so Ousterhout focuses on P99 latency: the time within which 99% of messages complete.

Consider several nodes that compute on their GPUs, exchange metadata, and then start another compute phase. The GPUs sit idle during synchronization, and the group cannot continue until every required exchange finishes. A few milliseconds of waiting barely matter after five seconds of computation. When the compute phase itself approaches the millisecond scale, millisecond-scale synchronization consumes a meaningful fraction of the cycle. One slow message can reduce application throughput even while the network continues moving bulk data quickly.

An informal audience poll produces more raised hands than Ousterhout expected from people who suspected small-message latency was already limiting their applications. The poll does not establish prevalence, but it sharpens the diagnostic question for the rest of the talk: how does a nominally tiny exchange become a tail-latency outlier?

Ousterhout’s answer begins with incast: several machines send to one destination simultaneously. If three incoming links each transmit at the rate of the destination’s single downlink, packets can arrive three times faster than that destination can receive them. The excess accumulates at the top-of-rack switch’s egress queue for the destination.

A short message sent to the same destination joins the queue behind packets from the large transfers. Its own transmission time may be tiny, but that no longer matters when waiting time is dominated by the backlog. If the switch exhausts its buffer space, it drops packets; timeouts and retransmissions then extend the delay. The problem is traffic already waiting at the final hop, not the amount of data in the short message.

Preventing the queue from growing requires the senders to slow down. TCP and the RDMA transport discussed here put that responsibility at the senders, even though the troublesome queue may be at the other end of the data-center network. Older schemes inferred congestion only after overflow caused packet loss: missing acknowledgments told the sender to reduce its rate.

Switch feedback can warn the sender before packets are lost. With Explicit Congestion Notification (ECN), a switch marks packets after an egress queue crosses a threshold. The receiver sees the marks and returns that information to the sender, for example in an acknowledgment. ECN avoids using overflow as the first warning, but the signal still appears only after a queue has begun to form and must travel through the receiver before the sender can act.

Two difficulties make the resulting control loop unstable:

• Incomplete rate information. A sender learns that congestion exists somewhere, but not precisely how far to slow down or when to accelerate again. Multiple senders adjust at the same time even though their combined rates determine the queue.
• Control lag. Ousterhout says reaching a suitable rate typically takes several round trips. During that time, old transfers finish and new ones begin, changing the available capacity.

The senders can therefore oscillate between transmitting too much and too little instead of settling. Ousterhout acknowledges improvements from more than twenty years of research, but argues that sender-side control remains poorly placed for rapidly changing data-center traffic. In the feedback scheme he describes, queue buildup supplies the evidence needed to react. By the time that evidence returns, the queue has already introduced delay.

Congestion control is only one mismatch. TCP and the RDMA transport in Ousterhout’s comparison expose a byte stream without transport-visible message boundaries. An application may write distinct messages, but TCP serializes their bytes. If two large messages precede a small one, the small message cannot pass them. This head-of-line blocking is separate from waiting behind packets in a switch queue, but it produces the same outcome: a short operation inherits the delay of long work before it.

Without message boundaries, the transport also lacks the information needed to tell how much work remains in each application message. It cannot prefer a nearly complete short message on that basis. Homa begins its clean-slate data-center design by changing this data model rather than adding another feedback algorithm to the byte stream.

Homa’s fundamental unit is a remote procedure call: a request message travels from client to server, followed by a response message from server to client. Message lengths remain visible throughout the transport. As soon as the receiver gets the first packet, it knows how much more data the sender wants to transmit. An incoming message becomes a known amount of pending work rather than an undifferentiated continuation of a stream.

That knowledge supports two related choices:

• Shortest Remaining Processing Time (SRPT). Homa favors messages with less work left to complete.
• Independent progress. Messages are not serialized into one stream, so a short message can pass a long one instead of waiting for every earlier byte.

The project began as Behnam Montazeri’s PhD dissertation. Ousterhout says the results convinced him to make production adoption his own programming project. He created a Linux kernel module, made it available on GitHub, and was working through the upstreaming process at the time of the talk.

Homa moves congestion control to the receiver, close to the final downlink where Ousterhout says congestion primarily occurs. The receiver sees the messages competing for that link and learns their remaining lengths from their initial packets. It can coordinate their progress directly instead of waiting for each remote sender to infer congestion from delayed feedback.

The packet flow has two stages. A sender breaks a message into packets and transmits only the first few immediately; Homa calls these unscheduled packets. The remaining scheduled packets wait until the receiver asks for them. The receiver sends paced grant packets back over time, with each grant authorizing another chunk.

Suppose ten messages are arriving at one receiver. Granting all ten at once would recreate congestion in the top-of-rack queue. The receiver can instead delay some grants and favor shorter messages when deciding which grants to issue. Grant timing therefore serves two purposes: it limits simultaneous incoming traffic and implements SRPT. The initial unscheduled packets remain a qualification—the receiver controls the subsequent scheduled traffic, not every packet from the outset.

Receiver grants regulate future transmissions, but a short message also needs a way around long-message packets already waiting in a switch. Homa uses the multiple hardware queues available at each egress port—typically eight in Ousterhout’s description. Packet fields select a queue, and the switch serves higher-priority queues preferentially.

Homa assigns those priorities dynamically to favor shorter messages. In the incast example, long-message packets accumulate in the lowest-priority queue. A short message enters a higher-priority queue and bypasses that backlog on its way to the same receiver. The three mechanisms now fit together: message boundaries reveal how much work remains, receiver grants regulate how much traffic proceeds, and switch queues enforce the chosen order at the congested hop.

Ousterhout closes with one benchmark used to tune and evaluate Homa. Multiple machines exchange request and response messages of equal length, ranging from about 50 bytes to one megabyte. The graph plots round-trip time against message length and includes P50 and P99 curves for TCP and Homa. P50 is the median; P99 describes the slower tail. Lower is better.

Two results stand out:

• Short-message tail latency. TCP’s P99 exceeds one millisecond, while Homa’s remains below 100 microseconds. Ousterhout reports an improvement of about 13×.
• Longest-message latency. Homa is almost a factor of two better than TCP at the largest message sizes shown. Favoring short messages did not create the expected long-message penalty in this experiment.

Ousterhout attributes the long-message result to Homa’s run-to-completion approach, which he contrasts with TCP’s fair scheduling, but he does not develop that mechanism further in this talk. The result therefore supports a narrow conclusion: this workload improved at both ends of the message-size range. It does not establish the tradeoff for every traffic mix, and the displayed figures compare Homa with TCP rather than quantifying an advantage over RoCE.

The benchmark also does not demonstrate a 13× end-to-end AI application speedup. Application throughput improves only to the extent that short-message waiting occupies its critical path. If communication is already a small part of a five-second compute phase, faster transport changes little. If synchronization consumes much of a millisecond-scale cycle, the same transport improvement can free meaningful GPU time.

Ousterhout’s closing recommendation is diagnostic before architectural: measure whether short-message tail latency is limiting application throughput. He expects these messages to become more important, while explicitly leaving that forecast to be tested over the following year or two. For teams that find the bottleneck, he recommends experimenting with Homa and offers help with setup, questions, and bug fixes. The implementation still has a material qualification for the talk’s central scenario: the kernel-module repository says the SIGCOMM paper’s incast optimization is not implemented and asks teams planning large-incast tests to contact the maintainer. Ousterhout says he moved into semi-retirement so he could spend all of his time working on the protocol.