2023-03-15 Embedded World Conference
1
Composable
Edge Cloud
Systems With
NVMe over
5G URLLC
Frederik Pfautsch
Endric Schubert
2023-03-15 Embedded World Conference
2
Background
Frederik Pfautsch
frederik.pfautsch@missinglinkelectronics.com
- MSc Computer Engineering
@TU Berlin
- Master’s thesis in cooperation with
Fraunhofer HHI, MLE and Uni Ulm
- “5G Berlin” campus network (Release 15)
- MLE
R&D Lead Engineer for 5G/6G Radio
Sidelink Comm & Precision Time Synch
Endric Schubert
endric.schubert@missinglinkelectronics.com
- Dipl.-Ing. ET Univ. Karlsruhe
- PhD CS Univ. Tuebingen
- Honorary Professor Univ. Ulm
- Ambassador Startup Sued
- Background in Semiconductors, EDA,
Domain Specific Architectures w/ FPGA
- 60+ technical publications
- 20+ patents
2023-03-15 Embedded World Conference
3
MLE - Experts for Domain-Specific Compute Architectures
Our Mission:
● Deliver HW and SW for
High-Performance (Embedded) Compute
Systems & Solutions
● Offering pre-validated subsystems with FPGA IP
blocks and open-source software
● Support customer projects with deep expertise and
hands-on design services
Head-quartered in Silicon Valley with Design Offices
in Germany
● Founded 2010, employee owned
● 18+ Certified FPGA Designers
● Customers include technology leaders, US and
European government agencies, Fortune 500
companies
● Partners to:
2023-03-15 Embedded World Conference
4
Why?
⇒ 5G/6G edge cloud devices with limited storage capacity
- NVMe over 5G? It should be possible, 5G has the low latency and
bandwidth promises!
- Side effect: Measure capabilities of 5G Release 15 thoroughly
}
2023-03-15 Embedded World Conference
5
Interlude – PCIe
- De-facto standard for general purpose
peripheral connectivity within
x86 PCs and servers
- Easy extension of CotS-computers with
almost any type of peripheral
- Every new PCIe gen approx. doubles the
available bandwidth
- PCIe Gen 4: 31.5 GByte/s (x16)
- PCIe Gen 5: 63.015 GByte/s (x16)
- Packet-based, layered protocol (TLPs)
2023-03-15 Embedded World Conference
6
Interlude – NVMe
NVMe is an example of a modern, fast,
PCIe based communication protocol.
- Avoid software reads to device registers
- Hardware device implementation can issue
multiple reads in parallel, masking the round
trip time
- Also software can only transfer 64 bits per
access
- Pipeline processing for example by allowing
for lazy pointer updates of queues
- Scale with the number of CPU cores by
having independent queues/ringbuffers and
MSI-X interrupts
2023-03-15 Embedded World Conference
7
Why?
2023-03-15 Embedded World Conference
8
Why?
2023-03-15 Embedded World Conference
9
Why?
2023-03-15 Embedded World Conference
10
Why?
See: Schubert, Braun and Langenbach: “PCI Express over IP - Accelerated”
Embedded World Conference, 2016
2023-03-15 Embedded World Conference
11
Why?
2023-03-15 Embedded World Conference
12
Bringing it together, PCIe + 5G
Jim Peek
Director Of Technology
Missing Link Electronics Corp
PCI-SIG Conference 2018
2023-03-15 Embedded World Conference
13
Bringing it together, PCIe + 5G
2023-03-15 Embedded World Conference
14
5G
2023-03-15 Embedded World Conference
15
5G
2023-03-15 Embedded World Conference
16
5G – URLLC
2023-03-15 Embedded World Conference
17
Setup
2023-03-15 Embedded World Conference
18
Latency Chain
ø 16.2 ms !
PCIe over 5G is Feasible!
Default PCIe Completion Timeout: 50 µs to 50 ms
2023-03-15 Embedded World Conference
19
Latency Map
2023-03-15 Embedded World Conference
20
Latency – PCIe
ø 16.2 ms
- “PCIelat”
- Kernel module
- Ruby script
- Default PCIe
Completion Timeout:
50 µs to 50 ms
2023-03-15 Embedded World Conference
21
Latency – PL2PL
ø 20.2 ms
- VHDL counter
- Replaces PCIe traffic
- Customizable packet
size
2023-03-15 Embedded World Conference
22
Latency – PL2PL
ø 20.2 ms
- VHDL counter
- Replaces PCIe traffic
- Customizable packet
size
2023-03-15 Embedded World Conference
23
Latency – PL2PL
ø 20.2 ms
- VHDL counter
- Replaces PCIe traffic
- Customizable packet
size
2023-03-15 Embedded World Conference
24
Summary
- It works :)
- High latency, high variance (tail latency)
⇒ Attached PC does not boot, re-enumeration necessary
- Latency measured by reference setup is comparable to other published
setups
- 5G Release 15 introduces the majority (>99%) of latency in our setup!
- Latency is mostly independent of packet size (difference vanishes due to
the high latency in general)
2023-03-15 Embedded World Conference
25
Outlook URLLC
- 5G Release 15 only implements the basic requirements for URLLC, such as
“micro slots”
- 5G Release 16 and 17 will begin to support URLLC
- Does URLLC offer enough bandwidth?
- Hardware improvements during 2022 supporting Release 16
- Mediatek M80 chip platform released in Q1 2022
- Qualcomm X65 or X62
2023-03-15 Embedded World Conference
26
Backup Slides
2023-03-15 Embedded World Conference
27
Latency – PS2PS
ø 17.7 ms
- User space C-program
- Server/Client with gettimeofday
()
- Customizable packet
size
2023-03-15 Embedded World Conference
28
Latency – PL
- Xilinx Integrated Logic
Analyzer
- Count cycles
2023-03-15 Embedded World Conference
29
Latency – PS/PL
US: ø 25.6 µs
DS: ø 32.2 µs
- PL -> PS -> PL:
VHDL Counter
- PS -> PL -> PS:
User space C-program
2023-03-15 Embedded World Conference
30
Latency – iptables
US: ø 10.2 µs
DS: ø 11.0 µs
- libpcap timestamps
- tcpdump of both
interfaces
- SNAT/DNAT latency
2023-03-15 Embedded World Conference
31
Latency – Linux USB stack
- tcpdump with usbmon
- Match USB packets to
network packets