NVMe Fast FPGA RAID Recorder System
Unlike 3rd-party Network-Attached-Storage (NAS) systems which have limited read/write bandwidth, MLE NVMe Fast FPGA RAID (FFRAID) Recorder System, based on MLE’s NVMe FFRAID Accelerator, can scale to 400 Gbps or more, delivering loss-less and gapless data recording from multiple data sources onto a RAID of NVMeTM SSDs.
MLE NVMe FFRAID Recorder is a turnkey system – delivered as a ready-to-run appliance – to support data-in-motion pre- and post-processing and is highly scalable with regards to bandwidth and recording capacity.
Key Features
- Scalable from 100 to 400 Gbps, or more
- Cascading of multiple systems with time-synchronization
- Start-Pause-Stop Data Recording
- Pre-trigger Data Recording in circular buffers
- Adaptable signal front-ends
- Read/write compatible with Linux Software-RAID
Applications
- Autonomous Vehicle Path Record & Replay
- Automotive / Medical / Industrial Test Equipment
- Broadcast Recording
- High-speed Radar / Lidar / Camera Data Acquisition & Storage
- Network Telemetry and Analytics
- Very Deep Network Packet Capture of Ethernet or IPv4 or TCP/UDP Data
NVMe FFRAID Recorder Turnkey System Availability
MLE NVMe FFRAID Recorder is a ready-to-run turnkey system which
- integrates multiple MLE NVMe FFRAID Accelerator subsystems
- on off-the-shelf FPGA cards
- along with a standard Linux server
- which has been optimized for PCIe/NVMe cost/performance and
- with a choice of pre-validated NVMe U.2/U.3 SSDs
Formfactor choices for a turnkey NVMe FFRAID recording system include bench-top appliance, 19”-rack mount systems, or embedded recording systems which have been optimized for Size, Weight, Cost and Power.
Bench-Top NVMe FFRAID Recorder System
MLE NVMe FFRAID Bench-Top Recorder System can be highly customized based on the “Mayflower” system from Inonet GmbH. The Inonet Quicktray makes it easy to swap a RAID-0 unit of 4x NVMe SSDs – a nice feature for data recording in the field, or the recording of many different data sets.
19” Rackmount NVMe FFRAID Recorder System
MLE NVMe FFRAID 19” Server is intended for rack-level integration.
Embedded Recording Systems
Embedded recording systems can come in various form factors and use different SoC-FPGAs, all optimized for Size, Weight and Power, and/or Cost (SWaPC).
On the right is an example of NVMe FFRAID Embedded Recorder system based on AMD Versal AI Edge System-on-Chip.
Exemplary Remote User Interface
From a user’s perspective, MLE NVMe Fast FPGA RAID Recorder is implemented via a Remote Procedure Call (RPC) API. This enables users to quickly implement their own look&feel GUI via a webpage running on a separate machine, connected to the MLE NVMe Fast FPGA RAID Recorder via LAN.
To facilitate integration and testing, MLE provides a complete example set of Python and CURL commands for running the Recorder via this RPC API.
MLE NVMe FFRAID vs Network-Attached Storage (NAS)
Unlike 3rd-party Network-Attached-Storage (NAS) systems which have limited read/write bandwidth, NVMe FFRAID can scale to 400 Gbps, or more.
MLE NVMe FFRAID | Typical NAS system | |
Data Rate | 100 to 400 Gbps | limited to < 80 Gbps |
High-accuracy IEEE time synchronization | yes | no |
Recording Capability | highly optimized to store high-speed data to NVMe storage | file system based |
Add-on Capability |
| None |
Storage Location | RAID of NVMe SSDs | RAID of NVMe SSDs |
Remote Management | yes, via host Linux | yes |
Front-End | adaptable, channel-based, high-speed I/O | network only |
SSD Security | Self-encryption TCG OPAL | Self-encryption TCG OPAL |
RAID compatibility | open-source Linux | proprietary or open-source Linux |
NVMe FFRAID Accelerator Subsystem
MLE NVMe FFRAID Recorder is a customizable turnkey solution which is based on the MLE NVMe FFRAID Accelerator Subsystem, featuring:
- Scalable channel-based architecture
- Compatibility and interoperability with Linux MDRAID
- Support for many data acquisition use cases including decimation or metadata indexing
- Simplex record, simplex replay, half-duplex and full-duplex modes
Channel-Based Architecture
MLE’s NVMe FFRAID Recorder implements a channel-based architecture where each data source/sink can be associated with a dedicated RAID engine and a dedicated storage space. Each channel can have 10/25/50/100 Gbps, or combinations thereof.
Adaptable signal front-ends support many different I/O standards in a “mix & match” fashion.
This channel-based architecture along with the combination of FPGA NVMe Recording Stack plus a well-tuned PCIe setup, delivers a best-in-class price/performance ratio for high-speed data acquisition, recording and replay. MLE’s multi-core NVMe Host Controller Subsystem supports dedicated NVMe queues per SSD in a PCIe Peer-to-Peer communication.
The NVMe FFRAID Recorder also supports high-performance and high-endurance NVMe U.2/U.3 SSDs with self-encryption TCG OPAL security function!
Recording Capacity and Scalability
MLE’s NVMe FFRAID Recorder supports a wide range of NVMe SSDs and can be scaled from M.2 SSDs for small and light-weight embedded systems up to large 19” racks using high-performance U.2 or U.3 SSDs. Scalability also includes selecting from different SSD capacities and Drive-Writes-per-Day (DWPD) models. Here a table of possible recording times in minutes:
| Recording Speed (Gbps) | ||||||||
| Storage (TiB) | 100 | 150 | 200 | 250 | 300 | 350 | 400 | |
| 5 | 7.2 | 4.8 | 3.6 | 2.9 | 2.4 | 2 | 1.8 | |
| 10 | 14.3 | 9.5 | 7.2 | 5.7 | 4.8 | 4.1 | 3.6 | |
| 15 | 21.5 | 14.3 | 10.7 | 8.6 | 7.2 | 6.1 | 5.4 | |
| 20 | 28.6 | 19.1 | 14.3 | 11.5 | 9.5 | 8.2 | 7.2 | |
| 25 | 35.8 | 23.9 | 17.9 | 14.3 | 11.9 | 10.2 | 8.9 | |
| 30 | 42.9 | 28.6 | 21.5 | 17.2 | 14.3 | 12.3 | 10.7 | |
| 35 | 50.1 | 33.4 | 25.1 | 20.0 | 16.7 | 14.3 | 12.5 | |
| 40 | 57.3 | 38.2 | 28.6 | 22.9 | 19.1 | 16.4 | 14.3 | |
| 45 | 64.4 | 42.9 | 32.2 | 25.8 | 21.5 | 18.4 | 16.1 | |
| 50 | 71.6 | 47.7 | 35.8 | 28.6 | 23.9 | 20.5 | 17.9 | |
| Recording Time in Minute(s) | ||||||||
Data Acquisition Pre- and Post-Processing
Besides record/replay of raw data NVMe FFRAID supports data-in-motion pre- and post-processing that enables you to add your custom algorithms for indexing and metadata generation, on-the-fly data decimation, or running in “spy-mode” as a transparent data proxy.
Plain Recording, Loss-Less and Gapless
Ingress data from the high-speed sensors are transferred and recorded at-speed and as-is onto the NVMe FFRAID.
Data Proxy & Record
Communication from a high-speed data source can be transported to a data sink while this data is also recorded at-speed.
Data Decimation & Record
Unwanted pieces of the ingress data is removed on-the-fly prior to storage, certain regions-of-interest (ROI), for example.
Adding Meta-Data & Record
Ingress data can be analyzed on-the-fly to generate indexing information for later search, for example. This metadata is then recorded along with the ingress data. Metadata can, for example, be: Hardware timestamps, regions-of-interest, search indexes.
NVMe FFRAID is Linux Compatible
NVMe FFRAID is fully compatible with Linux Software-RAID (via the Linux MD driver). This allows recording at high data rates and replaying at slower speeds, or vice versa. For performance reasons, NVMe FFRAID stores your data as so-called Linux block storage, i.e. no filesystems are used which slow down data acquisition and/or retrieval. Hence, you can record via NVMe FFRAID and replay that same data from a Linux MDRAID, and vice versa:
“Simplex Record”
Ingress data (1) is recorded at high-speed using NVMe FFRAID (2). Once recording is done the NVMe FFRAID releases the SSD RAID and Linux opens this as an MDRAID. Then data can be replayed via Linux (3), typically at lower speeds, and, for example sent out via a Linux network connection (4).
“Simplex Replay”
Ingress data (1) comes in via a Linux NIC, or any other Linux userspace software, for example, and is recorded onto a Linux MDRAID (2). Once recording is done, Linux releases the SSD RAID and NVMe FFRAID then opens it. Then data can be replayed via NVMe FFRAID (3) and be streamed-out at high data rates (4).
“Half-Duplex Record & Replay”
Ingress data (1) is recorded at high-speed using NVMe FFRAID (2). Once recording is done, then data can be replayed via NVMe FFRAID (3) and be streamed-out at high data rates (4). Because you operate the NVMe SSDs purely in sequential read (ex-or write), this features best performance.
“Full-Duplex Record & Replay”
Ingress data is recorded at high-speed using NVMe FFRAID. At the same time, while recording, data is replayed from the NVMe FFRAID and be streamed-out at high data rates. Because typical NVMe SSDs deliver less performance when writes happen parallel to rads, you will experience less performance in this mode.
Availability & Pricing
NVMe FFRAID is available as IP cores (for select FPGA devices), as NVMe FFRAID Cards (for select off-the-shelf FPGA cards), and as NVMe FFRAID Recorder (a variety of turnkey system appliances from MLE or MLE partners).
| VistProduct Name | Deliverables | Example Pricing |
|---|---|---|
| Evaluation Reference Design (ERD) | For evaluation puposes, and upon request, we can provide ready-to-run loaner cards or systems based on our PCIe 4.0 or PCIe 5.0 implementation. | Upon Request |
| NVMe FFRAID Recorder System | A ready-to-use yet customizable turnkey system built by MLE or MLE partners where hardware, software and “gateware” is fully integrated and tested. Different formfactor choices from medium-sized embedded PC to 19″ rack mount, lab-use or ruggedized. You can bring your own SSDs, or choose SSDs from our many options depending on the storage capacity required. | Starting at $24,800.- (depends on SSD performance and capacity) |
| NVMe FFRAID Card | Our NVMe FFRAID Cards are based on off-the-shelf 3rd party FPGA cards (such as AMD Alveo or Altera). Includes a Single-Project-Use netlist FPGA design license so you can alter the FPGA-based signal frontend. | Starting at $9,800.- |
| Intellectual Property (IP) Cores | Single-Project or Multi-Project Use for select FPGA devices; Modular and application-specific IP cores, and example design projects; delivered as encrypted netlist or RTL. Learn more about MLE NVMe FFRAID Accelerator offerings. |
Documentation
- NVMe Fast FPGA RAID Recorder Datasheet
- System Architecture
- Exemplary Evaluation Reference Design
- SSD options
- Linux operating system choices
- NVMe Fast FPGA RAID Accelerator Datasheet
- FPGA Based 400GBit/s Data Recorder – Insight into Different Pitfalls and Design Choices
- Linux ZynqMP PS-PCIe Root Port Driver (A software-only, non-accelerated alternative described by the Xilinx Wiki)
- Example designs on GitHub from Opsero (for PS-based NVMe supporting various FPGA and MPSoC evaluation boards)
Frequently Asked Questions
Does the NVMe FFRAID Recorder support file systems?
No, the NVMe FFRAID Recorder uses so-called Block Storage. So, no file systems are not supported. For each data transfer the user application logic selects a start and maximum end address, and then data is written to flash in a linear fashion. This achieves best performance and avoids write amplifications.
Does the NVMe FFRAID Recorder support drive partitions?
Partitions are not explicitly supported. However, the user application logic can use the NVMe FFRAID Recorder to read the SSD’s partition table and then set up transfers with start and maximum end address to be aligned to partitions.
Does the NVMe FFRAID Recorder support NVMe namespaces?
Only one single namespace is supported per SSD.
How many SSDs can be connected to the NVMe FFRAID Recorder?
The standard for the NVMe FFRAID Recorder is 4/8/16 SSDs. The number of SSDs can be adjusted to your application within certain limits, for example: the accumulated sustained write speed should be faster than the incoming data stream, or too many SSDs can cause latency issues. However, we can customize the NVMe FFRAID Recorder for your application to support more complex PCIe topologies. Please ask us for more details.
How many NVMe IO Queues does the NVMe FFRAID Recorder support and what is the depth of the NVMe IO Queue?
NVMe FFRAID Recorder currently supports one single IO Queue per SSD. This IO Queue can have up to 128 entries, each with up to 128 KiB data. I.e. you can have up to 16 MiB of “data in flight” per SSD. If needed, we can change the depth and size of this IO Queue. However, given the needs of streaming applications increasing the number of IO Queues may not be advantageous.
Does the NVMe FFRAID Recorder support PCIe Peer-to-Peer?
Yes, this is supported. Peer-to-Peer transfers can be very attractive as it frees up the host CPU. Team MLE can customize the NVMe FFRAID Recorder for your application to support many more complex PCIe topologies, including multiple direct-attached SSDs, multiple SSDs connected via a 3rd party PCIe switch chip, including PCIe Peer-to-Peer. Please ask us for more details.
How many parallel streams can be processed?
Currently, the NVMe FFRAID Recorder handles 16 independent data streams. To save resources, the number of streams can be reduced without losing the overall performance by widening the data paths.
Does the NVMe FFRAID Recorder support M.2 PCIe connectivity?
Yes. Because the NVMe FFRAID Recorder is agnostic to the formfactor of your SSD M.2, U.2, U.3, EDSFF and so on are supported, as long as your SSD “speaks the NVMe protocol” and not SATA nor SAS.
What are the best SSDs to use and from which vendor?
While, again, the NVMe FFRAID Recorder is compatible to work with any NVMe SSD, there are a couple of other aspects to keep in mind when selecting an NVMe SSD: Noise, vibration, harshness, temperature throttling, local RAM buffers, SLC, MLC, TLC, QLC, 3D-XPoint, etc. To enable our customers to deliver dependable performance solutions, we have worked with a set of 3rd party SSD vendors and would be happy to give you technical guidance in your project. Please inquire.