FPGA-based Highly Configurable and Low-Latency

Multi GMSL Camera 10GbE RTP Streaming

Performance and Design Choices

Agenda

Speakers

GMSL

System

Overview

Video

Subsystem

FPGA

NetStack

CSI-2

to RTP

Engine

MLE

NPAP

Hardware

Prototype

Performance

Speakers

Alin-Tudor Sferle

Analog Devices Romania

Friedrich Maximilian Sokol

(Ulrich Langenbach)

Missing Link Electronics

Germany

What is

GMSL

Gigabit Multimedia Serial Link

► High-Speed Video Link over a Single Wire

► Scalable SerDes Technology

► Camera/Display Use-Cases

► Multimarket Applications

►Up to 6/12 Gbps in GMSL2/3

►Since 2004

GMSL2

For Cameras

Serializer Deserializer

One/Two Input Ports One/Two/Four Input Ports

Parallel/CSI Input Interface Parallel/CSI Output Interface

GMSL over Coax/STP* GMSL over Coax/STP*

Virtual Channel Option Virtual Channel Option

Power/GPIOs/I2C/Video over GMSL Power/GPIOs/I2C/Video over GMSL

*STP - Shielded Twisted-Pair

GMSL2

Camera Pipeline Example

Camera

Module

Serializer

Deserializer

SoC

MIPI

C/D-PHY

MIPI

C/D-PHY

I2C

GPIO

GMSL2

Serializer

Camera

Module

MIPI

C/D-PHY

I2C

System

Overview

Motivation

High-Speed

Uncompressed

Video Streaming

from multiple

GMSL cameras

Use of Low Latency

FPGA-based UDP/IPv4 Network Accelerator

Configurable GMSL Camera Triggering

Interrupt-free

Network Accelerators Design

IEEE-1588-based Timestamping

NIC + FPGA-accelerated Video

Streaming on same 10G PHY Link

Prototype

Overview

AMD Kria K26 SoM

FPGA PL

MAX96724

Corundum

10Gb MAC&PHY + IEEE 1588

ARM PS (Linux)

GMSL Drivers

Camera Drivers

Corundum drivers

MIPI CSI

Receiver

MAX96724

MIPI CSI

Receiver

AXIS

Camera Trigger

AXIS

RTP

Engine

Instances

Host

ROS2

WebApp

GStreamer

MLE’s UDP/IPv4 NetStack

ETH1: 9.9Gbps

Linux networking stack

ADI IP

Open source

ADI Open source

Xilinx IP

AXIS

MQNIC data pipeline

ETH2: 100Mbps

AXI4

DDR

Web Server

Trigger Control

ZynqMPSoC

D-PHY 4 lanes

MLE IP

DDR

Controller

AXI4

Video

Subsystem – P1

Deserializer

MAX96724

MIPI

D-PHY

I2C

ZynqMPSoC

Xilinx’s

CSI Receiver

Subsystem

Xilinx’s

AXI I2C

Controller

FPGA Region

VC0

VC1

VC2

VC3

1/2/3/4

GMSL Cameras

AXIS

GMSL Subsystem

Video

Subsystem – P2

► HDL/FPGA Support

▪ Xilinx’s MIPI CSI Receiver Subsystem:

▪ 4 Lanes D-PHY [1500 Mbps] + YUV422 Input Support

▪ Up to 4 VCs decoding

▪ V4L2-compliant

▪ Xilinx’s AXI I2C Controller:

▪ For the GMSL Subsystem’s Components

►Linux Support

▪ V4L2-subdevices: Sensor, Serializer, Deserializer, CSI Receiver

FPGA

NetStack

Transport

Network

Ethernet

Implementation

Application

RTP

UDP

IPv4

RFC-Compliant Implementation

CSI* to RTP** Engine

MLE’s NPAP

Corundum

*CSI – Camera Serial Interface

** RTP – Real-Time Transport Protocol

CSI to RTP

Engine – P1

ZynqMPSoC

Xilinx’s

CSI Receiver

Subsystem

Xilinx’s

AXI I2C

Controller

FPGA Region

Xilinx’s AXI

Stream Switch

AXIS

CSI to RTP

Engine

AXIS

CSI to RTP

Engine

CSI to RTP

Engine

CSI to RTP

Engine

AXIS

NPAP

UDP/IP Network

Accelerator

AXIS

CSI to RTP

Engine – P2

► SW controllable through AXI4-Lite

► YUV422 format support

► RFC-compliant design - RFC4175 + RFC3550

► Configurable number of lines and pixels per line at

runtime

► 1 Line per RTP packet leveraging on Jumbo Frames

CSI to RTP

Conversion – P1

* RTP_H – RTP Header

* RTP_PH – RTP Payload Header

* PAUSE - Blanking Time [CSI-2]

CSI to RTP

Conversion – P2

► Conversion of 1 video line to 1 RTP packet

► Internal Buffers designed for Header Transmission &

Congestion on the 10 GbE link

► AXI Streaming compliant video to RTP conversion

► Designed as a packaged IP (Vivado’s Block Design)

FPGA

NetStack

Application

Transport

Network

Ethernet

Implementation

RTP

UDP

IPv4

RFC-Compliant Implementation

CSI* to RTP** Engine

MLE’s NPAP

Corundum

*CSI – Camera Serial Interface

** RTP – Real-Time Transport Protocol

NPAP

AXI4-Stream

MAC

►MLE Network Protocol Accelerator Platform (NPAP)

►TCP / UDP / IP Stack Fully in Hardware

►1 / 2.5 / 5 / 10 / 25 / 40 / 50 / 100 Gigabit Ethernet

Interface

►Full-duplex with 128 bit wide bidirectional datapath

►Designed fully in HDL

►Platform independent

NPAP

Evaluation

Reference

Design

NPAP

Evaluation

Reference

Design

NPAP

Evaluation

Reference

Design

NPAP

Evaluation

Reference

Design

NPAP

Generic Source Code with wide platform

coverage

● AMD

○ Virtex 4 to Virtex UltraScale+

○ Kintex to Kintex UltraScale+

○ Artix UltraScale+

○ Zynq-7000

○ Zynq UltraScale+ MPSoC

○ Zynq UltraScale+ RFSoC

○ Versal ACAP Series

● Altera

○ Cyclone IV series

○ Cyclone 10 GX series

○ Stratix V

○ Stratix 10 GX series

○ Agilex 5 D, E Series

○ Agilex 7 F, I, M Series

● Microchip

○ Polarfire and PolarFire SoC

● Lattice

○ Avant-G

FPGA

NetStack

RTP

UDP

IPv4

RFC-Compliant Implementation

CSI* to RTP** Engine

MLE’s NPAP

Corundum

*CSI – Camera Serial Interface

** RTP – Real-Time Transport Protocol

Application

Transport

Network

Ethernet

Implementation

Prototype

Overview – Corundum Subsystem

AMD Kria K26 SoM

FPGA PL

MAX96724

Corundum

10Gb MAC&PHY + IEEE 1588

ARM PS (Linux)

GMSL Drivers

Camera Drivers

Corundum drivers

MIPI CSI

Receiver

MAX96724

MIPI CSI

Receiver

AXIS

Camera Trigger

AXIS

RTP

Host

ROS2

WebApp

GStreamer

UDP/IPv4 NetStack

ETH1: 9.9Gbps

Linux networking stack

ADI IP

Open source

ADI Open source

Xilinx IP

AXIS

MQNIC data pipeline

ETH2: 100Mbps

AXI4

DDR

Web Server

Trigger Control

ZynqMPSoC

D-PHY 4 lanes

MLE IP

DDR

Controller

AXI4

Corundum

Subsystem – P1

► Open-source, high-performance, FPGA-based NIC

○ PCIe gen 3 x16, multiple 10G/25G/100G Ethernet ports

○ Fully custom, high-performance DMA engine; Linux driver

► Application block for custom logic

► Access to network traffic, DMA engine, on-card RAM, PTP

time

► Fine-grained traffic control

► 10,000+ hardware queues, customizable schedulers

Corundum

Subsystem – P2

Corundum

Subsystem – P3

MLE NPAP

Prototype

Overview – Corundum and Camera Interface

AMD Kria K26 SoM

FPGA PL

MAX96724

Corundum

10Gb MAC&PHY + IEEE 1588

ARM PS (Linux)

GMSL Drivers

Camera Drivers

Corundum drivers

MIPI CSI

Receiver

MAX96724

MIPI CSI

Receiver

AXIS

Camera Trigger

AXIS

RTP

Host

ROS2

WebApp

GStreamer

UDP/IPv4 NetStack

ETH1: 9.9Gbps

Linux networking stack

ADI IP

Open source

ADI Open source

Xilinx IP

AXIS

MQNIC data pipeline

ETH2: 100Mbps

AXI4

DDR

Web Server

Trigger Control

ZynqMPSoC

D-PHY 4 lanes

MLE IP

DDR

Controller

AXI4

Corundum

Corundum and Camera Interface - P1

Corundum

APP block / wrapper

MPSoC PL

AXI4-S

Data

path

split

Video Source

MIPI RX

Subsystem

Encap RTP

MAC

Ethernet

NPAP

Subsystem

MPSoC PS

AXI4-MM

AXI4-L

CSI-2 Data

Corundum

Corundum and Camera Interface - P2

APP block / wrapper

tcp_udp_netperf_npap_wrapper

Corundum

MAC

TX Path

UDP/ TCP

Demo

Application

AXI4-S

Data path

split

Higher

Prio

PHC

Video Source

MIPI RX

Subsystem

TX TS

Corundum

Interface

TX TS

Adaptation

Logic

AXI4-L Status &

Control

Encap RTP

Rate Limiter

(2)

NPAP

CSI-2 Data

Hardware

Prototype

TierIV GMSL to 10G Ethernet Converter, enabled by ADI

Two Quad MAX96724 Deserializers

One 10 GbE-capable SFP+ Connector

Built on top of Xilinx’s K26

Industrial-Grade SoM

External Camera Triggering

through GPIO header’s pins

Source: TierIV

Prototyped

Cameras

TierIV C1

- 2.5 MP – FullHD+ resolution

- HDR

- Image Sensor + ISP + GMSL2 Serializer

- Output: YUV422

TierIV C2

- 5.4 MP – 2.5K+ resolution

- HDR

- Image Sensor + ISP + GMSL2 Serializer

- Output: YUV422

Prototype

System

RX: ~ 9495 Mbps

RX: 135 Mbps

RX: - Mbps

TX: 135 Mbps

135 Mbps

(1x UDP)

TX: 95 Mbps

95 Mbps

1x UDP

- -

TX: 9400 Mbps

4x 2350 Mbps

4x UDP

- -

Performance

Traffic Separation

Up to 7 x 30 FPS

8 x 20 FPS

Up to 4 x 20 FPS

8 x 10 FPS

Performance

Overview

Performance

Overview – in Gbps

~8.2 Gbps

~6.3 Gpbs

~6.85 Gbps

~ 6.85 Gbps

Streaming

Demo – C1

Streaming

Demo – C2

Thank You!

Questions?