Cisco TRex Traffic Generator: Running Load Tests on a Network


 

As we were designing yet another router, we used a very handy tool to test the network's performance – the Cisco TRex traffic generator. So what kind of tool is it? How does one use it? And how can it help network developers and engineers? Let's tackle these questions one at a time.

 

1. What is Cisco TRex?

It is an open-source traffic generator, which works on standard Intel DPDK-based processors and supports both stateful and stateless modes. It is a relatively simple to use piece of software and is fully scalable.
 
The documentation for it can be found here.
 
TRex allows developers to generate different types of traffic and analyze the data that is received as a result. The data analysis is performed on the MAC and IP levels. TRex users can set packet size and quantity, as well as control the speed of the data transfer.
 
The traffic generator works in a Linux environment.
 
A key aspect that sets the TRex generator apart is its utilization of DPDK technology, which allows for the bypass of performance bottlenecks in the Linux network stack. DPDK, or Data Plane Development Kit, is a whole set of libraries and drivers that are used for fast packet processing by allowing the system to exclude the Linux network stack from the packet processing algorithms so that the tool can interact directly with the network device that's being tested.
 
DPDK transforms a general-purpose processor into a packet-forwarding server. This transformation eliminates the need for expensive switches and routers. However, DPDK does have some hardware limitations and the list of supported network adapters can be found here. The most most popular platform here is from Intel, i.e. it supports hardware that works with Linux drivers e1000, ixgbe, i40e, ice, fm10k, ipn3ke, ifc, and igc.
 
Another important nuance to consider is that in order for a TRex server to work with speeds of 10Gb/s and above, it requires the use of a multi-core processor, meaning at least 4 cores, and preferably an Intel CPU with hyper-threading support.
 

2. How to get and try out TRex

1) Download the archive from the trex-tgn.cisco.com server: trex-tgn.cisco.com/trex/release/

Unpack the archive in the user's home directory "/home/user", where "user" is replaced by the user name.

[bash]>wget --no-cache https://trex-tgn.cisco.com/trex/release/latest

[bash]>tar -xzvf latest

Unpack the archive in the user's home directory "/home/user", where "user" is replaced by the user name.

 

2) Configure the interfaces to send and receive data

Configure the interfaces by using the utility "dpdk_setup_ports.py", which is included in the TRex package. The network interfaces used by TRex can be configured at the MAC or IP level. To launch it, run the utility with the interactive configuration key "sudo ./dpdk_setup_ports.py -i".
 
The first step is to drop the MAC-based configuration (Do you want to use MAC-based config? (y/n) n).
 
The second step is to select a pair of network interfaces. In our example, the Intel X710 network card supports 4 network interfaces, and we will use the 1st and 4th network card ports.
 


In the third step, the system will give the option to automatically create a closed configuration, which is when data leaves port 1 and goes to port 2 (and back), all on the same PC. We chose to abandon this scheme and configure a routing scheme for 2 PCs.
 
The fourth and fifth step are save the configuration to the file "/etc/trex_cfg.yaml" and confirm.
 
As an example, let's take a look at the configuration at the IP level for the following connection scheme:



 

The configuration file is located here: "/etc/trex_cfg.yaml". An example of a simple configuration file is shown below, using a network card with 2 ports and a CPU supporting 8 threads:

### Config file generated by dpdk_setup_ports.py ###
- version: 2 interfaces: ['01:00.0', '01:00.3']
port_info:
- ip: 192.168.253.106
default_gw: 192.168.253.107
- ip: 192.168.254.106
default_gw: 192.168.254.107
platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 0
threads: [2,3,4,5,6,7]

In the configuration:

  • '01:00.0', '01:00.3' are the names of the Ethernet interfaces in the Linux system used.
  • ip: 192.168.253.106 is the address of the Server TRex PC port from which the traffic is generated.
  • default_gw: 192.168.253.107 is the address of port 1 of the PC DUT (Device under test).
  • ip: 192.168.254.106 is the address of Server TRex PC port from which traffic is returned after going through QOS rules.
  • default_gw: 192.168.253.107 is the address of 2nd port of the DUT.

Warning! The TRex system prohibits using the same subnet when generating traffic flows. For this purpose the subnets 16.0.0.0 and 48.0.0.0 should be used when generating packets.
 

3) Configuring the interfaces on a remote machine

Now we have to set up the forwarding and routes so the DUT that we are routing traffic through knows where to send and receive the packets.

Let's configure routing rules on the DUT:

sudo echo 1 > /proc/sys/net/ipv4/ip_forward

sudo route add -net 16.0.0.0 netmask 255.0.0.0 gw 192.168.253.106

sudo route add -net 48.0.0.0 netmask 255.0.0.0 gw 192.168.254.106

 

4) Launch the TRex server in astf mode:

cd v2.XX

sudo ./t-rex-64 -i --astf

If the TRex server is launched successfully, the system will display that the Ethernet ports that will be used for testing are bound:

The ports are bound/configured.

port : 0

------------
link : link : Link Up - speed 10000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 10000 Mbps - full-duplex>
promiscuous : 0
number of ports : 2
max cores for 2 ports : 1
tx queues per port : 3

 

5) Launch the TRex console

With the help of the console, which should be open in a separate window, start the generation of the traffic flow using the pre-made examples (the folder with the examples can also be found in the TRex archive):

cd v2.XX
./trex-console
start -f astf/http_simple.py -m 1

start (options):
-a (all ports)
-port 1 2 3 (ports 1 2 3)>
-d duration (-d 100 -d 10m -d 1h)
-m stream strength (-m 1 -m 1gb -m 40%)
-f load from disk the streams file

After a successful launch, the traffic statistics will be displayed in the TRex server console:

Global stats enabled
Cpu Utilization : 0.3 % 0.6 Gb/core
Platform_factor : 1.0
Total-Tx : 759.81 Kbps
Total-Rx : 759.81 Kbps
Total-PPS : 82.81 pps
Total-CPS : 2.69 cps

Expected-PPS : 0.00 pps
Expected-CPS : 0.00 cps
Expected-L7-BPS : 0.00 bps

Active-flows : 2 Clients : 0 Socket-util : 0.0000 %
Open-flows : 641

 

3. Development and testing automation using TRex

We wrote a lot of tests for TRex while we were developing the network router. Naturally, we considered whether we could run them automatically using python. This is what we came up with as a result:
 
We started the TRex server in stl mode:

cd v2.XX
sudo ./t-rex-64 -i --stl

We set an environment variable for python, since TRex works in conjunction with python.

export PYTHONPATH=/home/!!!user!!!/v2.XX/automation/trex_control_plane/interactive, where "!!!user!!" is the user name and home directory, and v2.XX is version of TRex software that was downloaded and unpacked into this folder.

The we launched the traffic generator with python. The example configuration listing is shown below.

python example_test_2bidirectstream.py

Expected result:

Transmit: 10000.24576MByte/s Receive: 10000.272384MByte/s
Stream 1 TX: 4487179200 Bit/s RX: 4487179200 Bit/s
Stream 2 TX: 2492873600 Bit/s RX: 2492873600 Bit/s
Stream 3 TX: 1994294400 Bit/s RX: 1994294400 Bit/s
Stream 4 TX: 997147200 Bit/s RX: 997147200 Bit/s

Let's break this example down:

c = STLClient(server = '127.0.0.1')

Create a connection to the TRex server. In our example, the connection was created on the same machine as the server.

  • «base_pkt_dir_a, base_pkt_dir_b, base_pkt_dir_c, base_pkt_dir_d» are packet templates with source and destination addresses and source and destination ports. In this example, 4 streams are created: 2 in one direction and 2 in the opposite direction.
  • «s1, s2, s3, s4» – we queried the STLStream class for the parameters of the generated streams, such as the stream ID and bitrate. In our case, the parameters we: ID1=4.5 Gbps, ID2=2.5 Gbps, ID3=2 Gbps, ID4=1 Gbps.

Stream configuration file listing example_test_2bidirectstream.py

# get TRex APIs
from trex_stl_lib.api import *
 
c = STLClient(server = '127.0.0.1')
c.connect()
 
try:
    # create a base packet with scapy
    base_pkt_dir_a = Ether()/IP(src="16.0.0.1",dst="48.0.0.1")/UDP(dport=5001,sport=50001)
    base_pkt_dir_b = Ether()/IP(src="48.0.0.1",dst="16.0.0.1")/UDP(dport=50001,sport=5001)
 
    base_pkt_dir_c = Ether()/IP(src="16.0.0.2",dst="48.0.0.2")/UDP(dport=5002,sport=50002)
    base_pkt_dir_d = Ether()/IP(src="48.0.0.2",dst="16.0.0.2")/UDP(dport=50002,sport=5002)
 
    # pps : float
    # Packets per second
    #
    # bps_L1 : float
    # Bits per second L1 (with IPG)
    #
    # bps_L2 : float
    # Bits per second L2 (Ethernet-FCS)
    packet_size = 1400
 
    def pad(base_pkt):
        pad = (packet_size - len(base_pkt)) * 'x'
        return pad
 
    s1 = STLStream(packet=STLPktBuilder(base_pkt_dir_a/pad(base_pkt_dir_a)), mode=STLTXCont(bps_L2=4500000000), flow_stats=STLFlowStats(pg_id=1))
    s2 = STLStream(packet=STLPktBuilder(base_pkt_dir_b/pad(base_pkt_dir_b)), mode=STLTXCont(bps_L2=2500000000), flow_stats=STLFlowStats(pg_id=2))
    s3 = STLStream(packet=STLPktBuilder(base_pkt_dir_c/pad(base_pkt_dir_c)), mode=STLTXCont(bps_L2=2000000000), flow_stats=STLFlowStats(pg_id=3))
    s4 = STLStream(packet=STLPktBuilder(base_pkt_dir_d/pad(base_pkt_dir_d)), mode=STLTXCont(bps_L2=1000000000), flow_stats=STLFlowStats(pg_id=4))
 
    my_ports = [0, 1]
 
    c.reset(ports = [my_ports[0], my_ports[1]])
 
    # add the streams
    c.add_streams(s1, ports = my_ports[0])
    c.add_streams(s2, ports = my_ports[1])
    c.add_streams(s3, ports = my_ports[0])
    c.add_streams(s4, ports = my_ports[1])
 
    # start traffic with limit of 10 seconds (otherwise it will continue forever)
    # bi direction
    testduration = 10
    c.start(ports=[my_ports[0], my_ports[1]], duration=testduration)
    # hold until traffic ends
    c.wait_on_traffic()
 
    # check out the stats
    stats = c.get_stats()
 
    # get global stats
    totalstats = stats['global']
    totaltx = round(totalstats.get('tx_bps'))
    totalrx = round(totalstats.get('rx_bps'))
    print('Transmit: {}MByte/s Receive: {}MByte/s'.format((totaltx / 1000000), (totalrx / 1000000)))
    c.clear_stats(ports = [my_ports[0], my_ports[1]])
 
    # get flow stats
    totalstats = stats['flow_stats']
    stream1 = totalstats[1]
 
    stream2 = totalstats[2]
    stream3 = totalstats[3]
    stream4 = totalstats[4]
    totaltx_1 = stream1.get('tx_pkts')
    totalrx_1 = stream1.get('rx_pkts')
    print('Stream 1 TX: {} Bit/s RX: {} Bit/s'.format((totaltx_1['total'] / testduration * packet_size * 8),
                                                               (totalrx_1['total'] / testduration * packet_size * 8)))
    totaltx_2 = stream2.get('tx_pkts')
    totalrx_2 = stream2.get('rx_pkts')
    print('Stream 2 TX: {} Bit/s RX: {} Bit/s'.format((totaltx_2['total'] / testduration * packet_size * 8),
                                                               (totalrx_2['total'] / testduration * packet_size * 8)))
    totaltx_3 = stream3.get('tx_pkts')
    totalrx_3 = stream3.get('rx_pkts')
    print('Stream 3 TX: {} Bit/s RX: {} Bit/s'.format((totaltx_3['total'] / testduration * packet_size * 8),
                                                               (totalrx_3['total'] / testduration * packet_size * 8)))
    totaltx_4 = stream4.get('tx_pkts')
    totalrx_4 = stream4.get('rx_pkts')
    print('Stream 4 TX: {} Bit/s RX: {} Bit/s'.format((totaltx_4['total'] / testduration * packet_size * 8),
                                                               (totalrx_4['total'] / testduration * packet_size * 8)))
except STLError as e:
    print(e)
 
finally:
    c.disconnect()

 

Conclusion

While preparing this tutorial, we ran and tested the DUT system with 4 streams, and then collected information on the streams and global statistics.

The operation described above was run using python, which means that TRex can be used to automate testing and debugging of network devices and software products. This can be done in a loop or by running sequential tests in python.

So is TRex by Cisco better or worse than other similar traffic generators and why? Let's take the popular client-server program iperf as an example. In the TRex use case, we described the stream setup and process. Both of the aforementioned testing and debugging tools are good in their own ways – iperf is great for running quick functionality checks on the fly, while TRex does a good job managing the automation of testing and designing network devices and complex systems for the telecom industry. In the latter case, in is important to be able to configure multithreaded streams, so that each stream can be configured for a particular task and the output then analyzed.

TRex supports the creation of templates for almost any kind of traffic and can amplify them in order to generate large-scale DDoS attacks, including TCP-SYN, UDP and ICMP streams. The ability to generate massive traffic flows allows the user to simulate attacks from different clients and on multiple target servers.

So if you haven't tried this tool out yet, keep this mind. But if you have, feel free to share your examples and feedback in the comments. It would be interesting to find out what our fellow engineers think about TRex and how they've used it.

 

Our projects

 

Industrial TSN routerWe designed a hardware platform, created 3D models of PCB to fit into a new enclosure, installed Linux OS, and ensured our DFM principle.

IEEE C37.94 Multiplexer for teleprotection system We've done mechanical design, PCB, and FPGA design for the IEEE C37.94 interface converter used in Smart Grid teleprotection system

Microcomputer 88F6282 to design network devices We designed a solution that decreases the time, costs and risks when designing new products with WiFi and Gigabit Ethernet interfaces

TI DM373-based SoM for video surveillance projects We've designed the CPU module based on the TI DM3730, which helps cut the time, costs, and risks of designing new products

Broadband SDR satellite modem on Zynq Ultrascale+ We designed a hardware & software platform for a broadband software-defined radio satellite modem

Industrial switches with 10GbE and 48 ports We have designed a new product line of 1Gb/10Gbps Ethernet switches operating at an industrial temperature range (−20 to +70°C) 

AI application for set-top box: searching and buying products from video streaming

Shoppable video: AI-based app for STB & Smart TV We've developed and installed on STBs the first AI application for searching and buying clothes directly from the video stream

digital signal processing module for wireless communication systems

DSP + FPGA: module for wireless communication We have designed a platform with an open architecture for wireless communication systems based on the SDR technology

A model range of industrial managed gigabit switch equipment with 8 and 16 ports
Design of industrial managed L2 switch We created reference designs of industrial managed switches with 10Gbase-X and 10/100/1000Base-T ports
Network traffic analyzer with Bypass function
BYPASS: design of network traffic analyzer We created prototypes of a network traffic analyzer for setting in a network connection gap (BYPASS function)
a series of managed 16- and 24-port Gigabit switches based on Realtek 83xx chipset
Design of L2 Gigabit managed switches, 24 ports Promwad engineering team designed a custom managed L2 switches based on the Realtek 83xx chipset and Linux kernel
DECT base station, VolP gateway, WiFi router, Office PBX, Server
7-in-1 modem router design with ​​VoIP, PBX & DECT We have designed and prepared for mass production an advanced VoIP modem router that provides all types of office communications
we developed a specialized Embedded Linux distribution for routers and other network devices
OpenWRT for Realtek: Embedded Linux for routersWe developed a specialized Embedded Linux distribution for routers and adaptable web interface to manage LuCi based
VoIP broadband router design
VoIP broadband router: HW & SW development Our engineers designed a subscriber router for providing voice over IP services in broadband networks
we created a reference design of a set-top box DVB-T/C + IPTV, which is intended for rapid platform development and launch of a series of new digital devices
IPTV+T/C hybrid set-top box: reference design We created a reference design of a set-top box with rapid platform development and launch of a series of new digital devices
promwad have designed a digital IPTV STB with DVB-T and DVB-S/S2 support, based on the SMP8654 processor by SigmaDesigns

Digital IPTV STB with DVB-T/S/S2 supportWe have designed a digital IPTV STB with DVB-T and DVB-S/S2 support, based on the SMP8654 processor by SigmaDesigns

we have developed a computer for networks with a client-server architecture that transfers basic processes to a remote server

AK1100 thin client: turnkey designWe have developed a new turn-key product, the AK1100 thin client based on the processor Marvell Sheeva 88F6282

IPTV set-top box design for SmartLabs We have developed a range of IPTV set-top boxes, ensured license coverage for all the proprietary coding standards for audio and video
we developed a video registration device for digitalization, storage and distribution of audio and video streams on request from 4 analogue sources

JPEG2000 4-channel recorder: HW, SW developmentWe developed a video registration device for storage and distribution of audio and video streams on request from 4 analogue sources