Chosing the Right TCP Rx/Tx Buffer Sizes
Chosing the right TCP Rx/Tx buffer sizes is fundamental to any TCP connection: If the TCP Rx/Tx buffers are undersized, then the TCP throughput may not reach maximum line rate. Hence, in software systems we typically set those TCP Rx/Tx buffers to some large value. However, in a Full Accelerator TCP Tx buffers and TCP Rx buffers require expensive on-chip memory resources. So, we suggest to chose the “right” TCP Rx/Tx buffer sizes.
To help you pick a good tradeoff, please use the so-called Bandwidth-Delay Product (BDP) metric to determine TCP buffer sizes for TCP (keep in mind, TCP buffers are placed on both ends: Tx side and Rx side):
- Buffer size (in bits) = Bandwidth (in bits-per-second) * RTT (in seconds)
RTT is the Round-Trip Time which is the time for the sender to transmit the data plus the time-of-flight for the data, plus the time it takes the recipient to check for packet correctness (CRC), plus the time for the recipient to send out the ACK, plus the time-of-flight for the ACK, plus the time it takes the sender to process the ACK and release the buffer. Here examples:
- Assuming a direct connection with the recipient being MLE NPAP, and a 25 GBit/s link, ACK times can be assumed to be less than 1 microsecond. This translates to minimal buffer sizes of 25 Kbits, which is equivalent to a single 36 Kbit FPGA on-chip memory.
- Assuming a 10 GBit/s link and the recipient being software, then RTT can be much longer, mostly due to the longer processing times in the OS on the recipient side. For a modern Linux we can assume RTTs of 50 microseconds, or longer (you can run ‘Netperf’ on your machine to find out). Means buffer sizes shall be around 500K bits, or the 64K Bytes of on-chip memory we typically instantiate.
For resource efficiency reasons, MLE NPAP allows you to set TCP Tx buffer sizes and TCP Rx buffer sizes separately and individually for each TCP Core, to facilitate optimizations for better performance/resource trade-offs in more unidirectional dataflows.
To ensure these receive buffers are used effectively, especially when they exceed the standard 65,535-byte TCP window, MLE NPAP implements the TCP Window Scale option (RFC 7323). This is essential for achieving line-rate performance on high-latency networks (i.e., those with a large Bandwidth-Delay Product). MLE NPAP automatically calculates the correct scale factor based on the specified Rx Buffer Size for each TCP core. It then advertises this full receive capacity during the connection handshake, enabling the remote peer to send enough data to fill the pipe and maximize throughput.