Click to edit Master title style

®

PCIe Range Extension via Robust,

Long Reach Protocol Tunnels

Jim Peek

Director of Technology

Missing Link Electronics

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Disclaimer

2

Presentation Disclaimer: All opinions, judgments, recommendations,

etc. that are presented herein are the opinions of the presenter of the

material and do not necessarily reflect the opinions of the PCI-SIG

®

.

Click to edit Master title style

3

®

PCI-SIG Developers Conference 2018 3

Motivation

Example: IP based from Host complex PCI to PCI end point devices

Click to edit Master title style

4

®

PCI-SIG Developers Conference 2018 4

Introduction

PCIe to Ethernet Bandwidth Matching

Click to edit Master title style

5

®

PCI-SIG Developers Conference 2018 5

PCI Express

Click to edit Master title style

6

®

PCI-SIG Developers Conference 2018 6

Motivation

PCIe on CPUs:

standard high

performance

interface

Click to edit Master title style

7

®

PCI-SIG Developers Conference 2018 7

Motivation

PCIe on CPUs:

standard high

performance

interface

Network (TCP/IP or similar):

standard high performance

interconnect

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

PCI Express Topology

o PCIe is point-to-point. Hierarchical system topologies via

switches

o ID based routing (bus/device/function number) and address

based routing

o Transactions may require completion (posted or split-

transaction)

o Range problem: Physical line length of PCIe on PCBs is limited

to centimeter range

8

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

PCI Express Topology

o PCIe is point-to-point. Hierarchical system topologies via

switches

o ID based routing (bus/device/function number) and address

based routing

o Transactions may require completion (posted or split-

transaction)

o Range problem: Physical line length of PCIe on PCBs is limited

to centimeter range

9

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

State-of-the-Art

o PCIe external cabling

• Standard for copper cables

o FireFly PCIe over Fibre

• PCIe 3.0 x 4 over fibre by Samtec

o ExpEther

• PCIe 2.0 over 40 GigE networks by NEC

10

Source: NEC Corporation

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Alternatives

o Dolphin ICS

• PCIe based compute and

IO cluster solutions

o Storage / IO Networks

• Infiniband by Mellanox

11

Source: Mellanox

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Proposal: PCIe over TCP/IP

o Fully transparent to network equipment

• Just a bunch of TCP sessions

• No special traffic handling required

o Fully transparent to PCIe

• Reliable transport via TCP

• Congestion control via TCP

o Based on separated and distributed upstream and

downstream switch ports

• Easily scalable via TCP session count

• Support for multiple ethernet ports

• Decouples cable routing from transaction layer routing

o Independent of lower network layers, e.g. physical

layer

12

Click to edit Master title style

13

®

PCI-SIG Developers Conference 2018 13

Network Processing

Click to edit Master title style

14

®

PCI-SIG Developers Conference 2018 14

Network Processing

Click to edit Master title style

15

®

PCI-SIG Developers Conference 2018 15

Network Protocol Acceleration

Platform Architecture

Click to edit Master title style

16

®

PCI-SIG Developers Conference 2018 16

PCIe over TCP/IP Tunneling

Click to edit Master title style

17

®

PCI-SIG Developers Conference 2018 17

Why tunnel TLPs and not DLLPs?

PCIe Timeouts

Click to edit Master title style

18

®

PCI-SIG Developers Conference 2018 18

Why tunnel TLPs and not DLLPs?

Scaling to Multiple Ports

Click to edit Master title style

19

®

PCI-SIG Developers Conference 2018 19

Concept of PCIe over TCP/IP

Click to edit Master title style

20

®

PCI-SIG Developers Conference 2018 20

TLP Aggregation

• Send multiple TLPs per TCP/IP segment

• Aggregating TLPs has minor impact on latency

• TLP aggregation reduces protocol overhead

• Up to 20 % bandwidth gain with aggregation

Click to edit Master title style

21

®

PCI-SIG Developers Conference 2018 21

Implementation: FPGA Design

• Based on „XPressRICH3“ PCIe IP Core from PLDA

• Based on Fraunhofer HHI NPAP

Click to edit Master title style

22

®

PCI-SIG Developers Conference 2018 22

Implementation: Test Setup

Click to edit Master title style

23

®

PCI-SIG Developers Conference 2018 23

NVMe Performance Results

Local PCIe Switch Distributed PCIe Switch with TLP Aggregation

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Visibility for monitoring

o Using PCIe over TCP/IP also opens PCIe for simple

(performance) monitoring via network traces

24

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Visibility for monitoring

25

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Visibility for monitoring

26

• A row is a request cycle of a device DMA engine

• Request leaves PCIe device flowing

upstream, enters DN node, then UP, then

SPY, then host node

• Completions flow downstream

• Multiple completions per request

• Max read requrest size > max payload

size

• All TLPs of a transaction are captured,

timestamped and correlated

• SPY-SPY (red part) is the time a host needs to

complete a request (PCIe to DRAM to PCIe

latency)

• DN-UP, UP-DN are network transitions in

upstream or downstream direction respectively

• Network bandwidth is lower than PCIe

bandwidth, so this hop needs more time

• Observation: PCIe DMA engine requests bursts

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Conclusions

o Reliable “tunneling” of PCI Express via TCP/IP

o Fully transparent to PCIe Root Complex and PCIe

Devices

o No additional host software for compliant hosts ist

needed as PCIe switch management is natively

implemented

o The solution inter operates with all compliant software

and add-in cards out of the box

o Scalability based on FPGA processing for bandwidths

of 1, 10 , 25 GigE line rates and beyond

o Re-use existing network infrastructure

o Technology extendable with NTB and MRA concepts in

a distributed system

27

Click to edit Master title style

®

PCI-SIG Developers Conference 2018

Thank you for attending the

PCI-SIG Developers Conference 2018.

For more information please go to

www.pcisig.com

28

Click to edit Master title style