Zone-Based Automotive

Backbones Tunneling PCIe

Dr. Endric Schubert, Missing Link Electronics

• Ulrich Langenbach

Dir. Eng. Missing Link Electronics GmbH (MLE)

• Marcus Pietzsch

Group Mgr, Fraunhofer Institute for Photonic Microsystems (IPMS), Dresden, Germany

Acknowledgements

2

Apr 19 2021

• Presentation Disclaimer: All opinions, judgments, recommendations, etc.

that are presented herein are the opinions of the presenter of the material

and do not necessarily reflect the opinions of the PCI-SIG

®

.

Disclaimer

3

Apr 19 2021

WHY?

• More safe and eco-friendly vehicles drive automotive connectivity towards so-called

Zone-Based Architectures. Inside those Zone Gateways PCIe connects multiple SoC for

scalable performance. Zone Gateways connect with each other via the emerging IEEE

standards “Time Sensitive Networking” (TSN).

WHAT?

• A solution that fulfills the need for “tunneling” PCIe over TSN, supporting CPU-to-CPU

communication (PCIe NTB) and NVMe storage.

HOW?

• A digital circuit & system stack to encapsulate and to decapsulate PCIe TLPs (along

with other protocols) over real-time automotive TSN 10G/25G Ethernet

Presentation Outline

Apr 19 2021

4

Apr 19 2021

5

Automotive Network History (1967)

Apr 19 2021

6

Evolution of Automotive Networks

Courtesy: BMW

Apr 19 2021

7

Today’s Expensive Wiring Nightmare

Organized in Functional

Domains

• Powertrain, Body Control,

Comfort, Infotainment, etc

• Bus Gateways for Cross-

Domain Communication

100 ECUs, or more via many

different busses and

protocols

Automotive Networks Today (Domain-Based)

Apr 19 2021

8

More centralized, better to scale, lower cost

Next: Zone-Based Architectures

9

Apr 19 2021

Application Example: “Smart Corner”

10

Apr 19 2021

Driven by Cost/Performance, i.e.

Centralized Compute & Storage:

• PCIe (for Embedded CPUs,

GPUs, FPGAs and SoCs)

• NVMe (for SSDs)

Driven by Compliance

• FuSa ISO 26262

• Security ISO/SAE 21434

• SOTIF ISO 21448

• etc

Zone-Based Automotive Network

Needs to Transport PCIe

11

Apr 19 2021

PCIe from PCI-SIG, TSN from IEEE

Symmetric for CPU-to-CPU (e.g. PCIe NTB) or Asymmetric Sensor-to-CPU

PCIe-over-TSN is Based on Open Standards

Apr 19 2021

12

System Stack is

• Hardware (Digital Circuit)

• Software (Drivers)

3

rd

Party (IP Cores)

• Fraunhofer IPMS

• Fraunhofer HHI

• MLE

• Open Source

Apr 19 2021

13

PCIe-over-TSN System Stack Example

Labcar Setup w/ PCIe Connect to HPC Labcar Setup for Experiments

PCIe/NVMe PCIE NTB

Apr 19 2021

14

PCIe-over-TSN System Stack Implemented

Backgrounder TSN

15

Apr 19 2021

16

IEEE 802.1AS Precision Timestamping

Apr 19 2021

17

PCIe-over-TSN = “Distributed PCIe Switch”

TSN

Encapsulate and Decapsulate PCIe TLPs. PCIe demands reliability, therefore we transport

TLPs over TCP/IP over TSN over Ethernet.

PCIe-over-TSN Concept

18

Apr 19 2021

Aggregate multiple

TLPs to optimize

bandwidth dominated

communication

Do not aggregate (i.e.

send immediately) for

latency oriented

communication

PCIe-over-TSN Smart TLP Aggregation

Apr 19 2021

19

Linux lspci

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d0]

00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]

01:00.0 PCI bridge [0604]: Xilinx Corporation Device [10ee:9034] (prog-if 00 [Normal decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Bus: primary=01, secondary=02, subordinate=03, sec-latency=0

Capabilities: [70] Express (v2) Upstream Port, MSI 00

Capabilities: [1c0 v1] #19

Kernel driver in use: pcieport

Kernel modules: shpchp

02:00.0 PCI bridge [0604]: Xilinx Corporation Device [10ee:9134] (prog-if 00 [Normal decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Bus: primary=02, secondary=03, subordinate=03, sec-latency=0

Capabilities: [70] Express (v2) Downstream Port (Slot+), MSI 00

Capabilities: [1c0 v1] #19

Kernel driver in use: pcieport

Kernel modules: shpchp

03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a808] (prog-if 02 [NVM Express])

Subsystem: Samsung Electronics Co Ltd Device [144d:a801]

Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 30

NUMA node: 0

Region 0: Memory at fcf00000 (64-bit, non-prefetchable) [size=16K]

Kernel driver in use: nvme

Kernel modules: nvme

PCIe-over-TSN Results with NVMe SSD

20

Apr 19 2021

Robust transport, even if TSN “throttles” bandwidth – test results for 1GE

PCIe-over-TSN Robustness

21

Apr 19 2021

22

PCIe-over-TSN Latency Analysis

PCIe-over-TSN Latency – Results for 1 GigE

23

Apr 19 2021

> lspci -vt

-[0000:00]-+-00.0AdvancedMicroDevices,Inc.[AMD]Raven/Raven2RootComplex

+-00.2AdvancedMicroDevices,Inc.[AMD]Raven/Raven2IOMMU

+-01.0AdvancedMicroDevices,Inc.[AMD]Family17h(Models00h-1fh)PCIeDummy

Host Bridge

+-01.1-[01]----00.0MissingLinkElectronicsDevice22fb

...

> lspci -vv -s 01:00.0

01:00.0 Memory controller: Missing Link Electronics Device 22fb

Subsystem:XilinxCorporationDevice0007

Control:I/O-Mem+BusMaster+SpecCycle-MemWINV-VGASnoop-ParErr-Stepping-SERR-

FastB2B- DisINTx+

Status:Cap+66MHz-UDF-FastB2B-ParErr-DEVSEL=fast>TAbort-<TAbort-<MAbort->SERR-

<PERR- INTx-

Latency:0,CacheLineSize:64bytes

Region0:Memoryatd0000000(64-bit,prefetchable)[size=256M]

Region2:Memoryatc0000000(64-bit,prefetchable)[size=256M]

Region4:Memoryatfcf10000(32-bit,non-prefetchable)[size=64K]

Region5:Memoryatfcf00000(32-bit,non-prefetchable)[size=64K]

Capabilities:[1c0v1]#19

Kerneldriverinuse:ntb_hw_mle

Kernelmodules:ntb_hw_mle

PCIe-over-TSN for Non-Transparent Bridging (NTB)

24

Apr 19 2021

PCIe-over-TSN “Labcar”

25

Apr 19 2021

• In-vehicle connectivity is quickly evolving, driven by cost/performance and compliance

• Migration towards centralized so-called Zone-Based Architectures

• PCIe key choice for short-distance local connect between CPUs, GPUs, FPGAs, SoCs

• A layered stack of open-standard protocols extends reach of PCIe over real-time

Ethernet / Automotive Ethernet

• Implementation as a digital circuit delivers rebust behavior at deterministic, low

latencies

Conclusion

26

Apr 19 2021

Missing Link Electronics, Inc.

Zanker Road, Suite 203

San Jose, CA 95134

www.MLEcorp.com

THANK YOU!

Apr 19 2021

27