Enterprise Data Storage System Development

Project in Nutshell: We designed an enterprise NAS (network attached storage) system with DPDK / SPDK support to enhance data transfer speed and network efficiency for a European customer. The scalable and portable design features video broadcasting in NDI streaming networks, modifiable storage unit, a dual-power system for reliability, and high-performing I/O units.

Client & Challenge

A European company turned to us to design an enterprise data storage system (NAS device) with DPDK / SPDK support and a possibility of video broadcasting in NDI streaming networks. The device should ensure optimal performance in high-quality video intake, storage, encoding/decoding, and streaming from multiple pro cameras and can be used with standard and customized storage clusters based on SAS, SSD NVMe, and CFexpress.

Solution

1. Concept Development

Our work included two layers:

Hardware design. Schematic and PCB design to ensure optimal capacity using the customizable storage clusters, high performance, scalability, and modularity of the solution.
Software development for data storage to facilitate efficient video management, encoding/decoding, and seamless streaming capabilities.

2. Hardware Design

There are four components to the hardware solution which contribute to scalability, reliability, and performance:

Modifiable storage unit:

A smooth software GUI controls a hot-plug capability using SPDK or ZFS as RAID 0.
A custom hardware solution: with various options to scale storage array, providing various PCIe lanes and SSD NVMe / CF-express Type C combinations.

Reliable power unit:

Compact OEM power supply units and innovative schematics for V-mount battery sourcing.
A dual-power system ensures continuous, reliable power supply.

High-performing I/O unit:

Maximised connectivity. It features a PCIe 1x16 extension and high-speed OEM NIC PCIe 4.0 x16 / dual-port 100G.
Local administration via video monitor and USB interfaces.

The robust processing unit is equipped to host two PCIe cards RTX A4500 for unrivalled processing power for video decoding (up to 8K / 60 fps).

_{The high-level hardware architecture of the NAS device and its communication with external devices}

In terms of performance, the system achieves peak efficiency by utilising the RTX A6000 ADA and Mellanox NIC. This configuration guarantees high-bandwidth capabilities and robust performance, facilitating the smooth management of 8K video streams from multiple sources.

3. Software Development

The system performance was improved by using DPDK / SPDK, which helped us solve the following issues:

Packet frame management. DPDK facilitates access to packetised frames, compiles them in physical memory and sends them to the GPU for decoding while simultaneously storing the frame on the disk using data storage software.
Improved performance due to Zero-copy. By receiving packets bypassing the kernel using zero-copy, the designed system provides a two to three times performance improvement over standard technologies such as Linux sockets.
SPDK integration. Leveraging DPDK for data storages, the project ensures zero-copy packet retrieval and significantly augments performance as opposed to traditional technologies like Linux sockets. Furthermore, the foresight of SPDK integration hints at the potential to double data-saving speeds in the future.
Advanced hot-plugging and storage management. The project safeguards data and simplifies storage management by employing SPDK or ZFS as RAID 0 for SSD NVMe / CFexpress and supporting software GUI-controlled hot plugging.

Business Value

The new NAS storage system with DPDK / SPDK enhances data transfer speed and network efficiency. The project boasts several competitive advantages that enhance its performance and usability. The design is both scalable and portable. With the modifiable storage unit and various storage card options, the system allows for scaling while maintaining the flexibility to transport and set up easily in diverse locations.

Including a dual-power system is a key feature contributing to power optimisation. This system uses compact OEM power units and leverages a V-mount battery sourcing method, ensuring reliable and uninterrupted operation in various settings.

Our expertise has enabled the integration of robust NDI streaming, elevating media streaming to new levels of efficiency and reliability. This feature offers seamless and high-quality streaming over Ethernet.

More of What We Do with DPDK / SDPK
SPDK и DPDK Solutions: explore our engineering expertise in high-speed data transfer — up to 10 times faster than the Linux kernel network stack.
TCP PEP and QoS Software Modules: we used DPDK for an HTS satellite communications system for a leading supplier of satellite telecoms equipment.

Tell us about your project!

All submitted information will be kept confidential.