September 16-18, 2024

Santa Clara, CA

Complementing TCP

with Homa

Stanford’s Reliable, Rapid Request-Response Protocol

Endric Schubert, Ph.D. & Ulrich Langenbach

Presentation Outline

1. The Homa Protocol from John Ousterhout @Stanford University

2. Can Homa coexist peacefully with TCP/IP?

3. RRRRP - Homa, FPGA Accelerated

4. Call for Collaboration

John Ousterhout,

Stanford University

Björn Petersen, Institute

for Micro Electronics, Ulm

University, Germany

Team MLE in Berlin and

Neu-Ulm, Germany

Acknowledgements

Stanford University:

Homa

UltraEthernet Consortium:

UET

Tesla:

TTPoE

A New Wave of Transport Layer Protocols?

Part 1

The Homa Protocol

It’s Time to Replace TCP in

the Datacenter

A Linux Kernel Implementation

of the Homa Transport Protocol

Homa - Started by John Ousterhout et al.

Homa Reduces Tail Latencies in Loaded Networks

▪ Experimental results

▪ 25 GigE Network

▪ Compares Linux kernel space

implementation of

▪ TCP/IPv4

▪ Homa/IPv4

▪ X-axis is distribution of

message mengths in workload

▪ Y-axis is Slowdown

RTT_loaded / RTT_unloaded

19x @ P99

2 ms down to 100 µs?

Nice!

Courtesy of John Ousterhout, Stanford University

HomaModule - Implemented as a Linux Kernel Module

Uses A-Priori Knowledge

▪ Link Rate between NIC and ToR switch

▪ NIC Queue Length (SRPT), i.e.

“estimated time until Tx buffer is empty”

▪ Coexistence w/ other protocols

Interaction with Tx pacer in Linux netdev

NAPI

▪ Distance between machines

Handling non-uniform RTT

▪ Priority Queues in Switches

last week!

Part 2

Can Homa coexist peacefully with TCP?

HW Setup

25 GigE network

4 or 12 nodes, each

Intel E5-2640v4

Mellanox ConnectX-4

NW Traffic

util/cp_node for the

Homa vs TCP tests

RTT and Slowdown

plus additive TCP “background

noise” via iperf (any-to-any)

SW Setup

HomaModule v2023-12-20

util/cp_vs_tcp tests in parallel

with Linux iperf

Experimental Test Setup (at Cloudlab xl170)

Experimental Test Setup

(TCP) Stream target load (using iperf)

% 10 30 40 50 60 70

Homa vs TCP RPC

target load (using

cp_node)

10 W2, W3, W4,

W2, W3, W4,

30 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5

40 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5 N/A

50 W3, W4, W5 W3, W4, W5 W3, W4, W5 W3, W4, W5 N/A N/A

60 W3, W4, W5 W3, W4, W5 W3, W4, W5 N/A N/A N/A

70 W3, W4, W5 W3, W4, W5 N/A N/A N/A N/A

How does Homa RPC behave

under increasing TCP “background

noise” vs. TCP RPC?

How much does Homa RPC

disturb the TCP iperf traffic?

How much does TCP RPC

disturb the TCP iperf traffic?

Slowdown Results Workload W4

How does Homa RPC behave

under increasing TCP “background

noise” vs. TCP RPC?

How much does Homa disturb

the TCP iperf traffic?

How much does TCP disturb

the TCP iperf traffic?

Slowdown Results Workload W4

RTT Results Workload 4

Part 3

RRRRP - Homa, FPGA Accelerated

HomaModule - A Linux Kernel Implementation of Homa

A layer just above IP,

parallel to TCP and UDP

Uses GRO (Generic

Receive Offloading)

RRRRP - Two Acceleration Approaches

Offload Engines

▪ Runs as SW on CPU

▪ Offload engines in FPGA

▪ Uses “golden” reference

implementation

(i.e. HomaModule)

▪ Low to medium eng. work

▪ Instant benefits

Full Acceleration

▪ Entire stack runs in FPGA

▪ No SW on CPU

▪ Major eng. work

(mostly in testing correctness)

▪ Integrated with MLE NPAP, a

TCP/UDP/IP FPGA Stack

▪ Maybe later

RRRRP - Offload Approach with Corundum.io

Open source FPGA NIC ported to many FPGA cards (http://www.corundum.io)

Good PCIe subsystem which supports many Rx/Tx FPGA queues.

Offload Engines

TCP (MLX 5) HomaModule RRRRP

Checksum Offload N/A N/A

Receive Side Scaling Receive Flow Steering Receive Side Scaling

Large Segmentation Offload Generic Segmentation Offload Large Segmentation Offload

Large Receive Offload Generic Receive Offload Large Receive Offload

HomaModule vs RRRRP - Slowdown for Workload W4

▪ Similar results for other

workloads

▪ Current implementation

runs on

10 GigE FPGA NIC

▪ Next: 25/50/100 GigE

Motivation for Homa Full Acceleration

Lowest RTT

BUT: lacks packet

visibility

Linux-to-Linux via 10 GigE

FPGA-to-FPGA via 10 GigE

Part 4

Call for Collaboration

⇒ Run on larger FPGA cluster

to check scalability

⇒ Try other workloads

⇒ Look at applications

Networked storage systems?

AI clusters?

⇒ Combine w/

Ultra Ethernet?

Tesla TTPoE?

Please take a moment to rate this session.

Your feedback is important to us.