0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ====================================
0004 HOWTO for the linux packet generator
0005 ====================================
0006
0007 Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
0008 or as a module. A module is preferred; modprobe pktgen if needed. Once
0009 running, pktgen creates a thread for each CPU with affinity to that CPU.
0010 Monitoring and controlling is done via /proc. It is easiest to select a
0011 suitable sample script and configure that.
0012
0013 On a dual CPU::
0014
0015 ps aux | grep pkt
0016 root 129 0.3 0.0 0 0 ? SW 2003 523:20 [kpktgend_0]
0017 root 130 0.3 0.0 0 0 ? SW 2003 509:50 [kpktgend_1]
0018
0019
0020 For monitoring and control pktgen creates::
0021
0022 /proc/net/pktgen/pgctrl
0023 /proc/net/pktgen/kpktgend_X
0024 /proc/net/pktgen/ethX
0025
0026
0027 Tuning NIC for max performance
0028 ==============================
0029
0030 The default NIC settings are (likely) not tuned for pktgen's artificial
0031 overload type of benchmarking, as this could hurt the normal use-case.
0032
0033 Specifically increasing the TX ring buffer in the NIC::
0034
0035 # ethtool -G ethX tx 1024
0036
0037 A larger TX ring can improve pktgen's performance, while it can hurt
0038 in the general case, 1) because the TX ring buffer might get larger
0039 than the CPU's L1/L2 cache, 2) because it allows more queueing in the
0040 NIC HW layer (which is bad for bufferbloat).
0041
0042 One should hesitate to conclude that packets/descriptors in the HW
0043 TX ring cause delay. Drivers usually delay cleaning up the
0044 ring-buffers for various performance reasons, and packets stalling
0045 the TX ring might just be waiting for cleanup.
0046
0047 This cleanup issue is specifically the case for the driver ixgbe
0048 (Intel 82599 chip). This driver (ixgbe) combines TX+RX ring cleanups,
0049 and the cleanup interval is affected by the ethtool --coalesce setting
0050 of parameter "rx-usecs".
0051
0052 For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6)::
0053
0054 # ethtool -C ethX rx-usecs 30
0055
0056
0057 Kernel threads
0058 ==============
0059 Pktgen creates a thread for each CPU with affinity to that CPU.
0060 Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
0061
0062 Example: /proc/net/pktgen/kpktgend_0::
0063
0064 Running:
0065 Stopped: eth4@0
0066 Result: OK: add_device=eth4@0
0067
0068 Most important are the devices assigned to the thread.
0069
0070 The two basic thread commands are:
0071
0072 * add_device DEVICE@NAME -- adds a single device
0073 * rem_device_all -- remove all associated devices
0074
0075 When adding a device to a thread, a corresponding procfile is created
0076 which is used for configuring this device. Thus, device names need to
0077 be unique.
0078
0079 To support adding the same device to multiple threads, which is useful
0080 with multi queue NICs, the device naming scheme is extended with "@":
0081 device@something
0082
0083 The part after "@" can be anything, but it is custom to use the thread
0084 number.
0085
0086 Viewing devices
0087 ===============
0088
0089 The Params section holds configured information. The Current section
0090 holds running statistics. The Result is printed after a run or after
0091 interruption. Example::
0092
0093 /proc/net/pktgen/eth4@0
0094
0095 Params: count 100000 min_pkt_size: 60 max_pkt_size: 60
0096 frags: 0 delay: 0 clone_skb: 64 ifname: eth4@0
0097 flows: 0 flowlen: 0
0098 queue_map_min: 0 queue_map_max: 0
0099 dst_min: 192.168.81.2 dst_max:
0100 src_min: src_max:
0101 src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
0102 udp_src_min: 9 udp_src_max: 109 udp_dst_min: 9 udp_dst_max: 9
0103 src_mac_count: 0 dst_mac_count: 0
0104 Flags: UDPSRC_RND NO_TIMESTAMP QUEUE_MAP_CPU
0105 Current:
0106 pkts-sofar: 100000 errors: 0
0107 started: 623913381008us stopped: 623913396439us idle: 25us
0108 seq_num: 100001 cur_dst_mac_offset: 0 cur_src_mac_offset: 0
0109 cur_saddr: 192.168.8.3 cur_daddr: 192.168.81.2
0110 cur_udp_dst: 9 cur_udp_src: 42
0111 cur_queue_map: 0
0112 flows: 0
0113 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
0114 6480562pps 3110Mb/sec (3110669760bps) errors: 0
0115
0116
0117 Configuring devices
0118 ===================
0119 This is done via the /proc interface, and most easily done via pgset
0120 as defined in the sample scripts.
0121 You need to specify PGDEV environment variable to use functions from sample
0122 scripts, i.e.::
0123
0124 export PGDEV=/proc/net/pktgen/eth4@0
0125 source samples/pktgen/functions.sh
0126
0127 Examples::
0128
0129 pg_ctrl start starts injection.
0130 pg_ctrl stop aborts injection. Also, ^C aborts generator.
0131
0132 pgset "clone_skb 1" sets the number of copies of the same packet
0133 pgset "clone_skb 0" use single SKB for all transmits
0134 pgset "burst 8" uses xmit_more API to queue 8 copies of the same
0135 packet and update HW tx queue tail pointer once.
0136 "burst 1" is the default
0137 pgset "pkt_size 9014" sets packet size to 9014
0138 pgset "frags 5" packet will consist of 5 fragments
0139 pgset "count 200000" sets number of packets to send, set to zero
0140 for continuous sends until explicitly stopped.
0141
0142 pgset "delay 5000" adds delay to hard_start_xmit(). nanoseconds
0143
0144 pgset "dst 10.0.0.1" sets IP destination address
0145 (BEWARE! This generator is very aggressive!)
0146
0147 pgset "dst_min 10.0.0.1" Same as dst
0148 pgset "dst_max 10.0.0.254" Set the maximum destination IP.
0149 pgset "src_min 10.0.0.1" Set the minimum (or only) source IP.
0150 pgset "src_max 10.0.0.254" Set the maximum source IP.
0151 pgset "dst6 fec0::1" IPV6 destination address
0152 pgset "src6 fec0::2" IPV6 source address
0153 pgset "dstmac 00:00:00:00:00:00" sets MAC destination address
0154 pgset "srcmac 00:00:00:00:00:00" sets MAC source address
0155
0156 pgset "queue_map_min 0" Sets the min value of tx queue interval
0157 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
0158 To select queue 1 of a given device,
0159 use queue_map_min=1 and queue_map_max=1
0160
0161 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
0162 The 'minimum' MAC is what you set with srcmac.
0163
0164 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
0165 The 'minimum' MAC is what you set with dstmac.
0166
0167 pgset "flag [name]" Set a flag to determine behaviour. Current flags
0168 are: IPSRC_RND # IP source is random (between min/max)
0169 IPDST_RND # IP destination is random
0170 UDPSRC_RND, UDPDST_RND,
0171 MACSRC_RND, MACDST_RND
0172 TXSIZE_RND, IPV6,
0173 MPLS_RND, VID_RND, SVID_RND
0174 FLOW_SEQ,
0175 QUEUE_MAP_RND # queue map random
0176 QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
0177 UDPCSUM,
0178 IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
0179 NODE_ALLOC # node specific memory allocation
0180 NO_TIMESTAMP # disable timestamping
0181 pgset 'flag ![name]' Clear a flag to determine behaviour.
0182 Note that you might need to use single quote in
0183 interactive mode, so that your shell wouldn't expand
0184 the specified flag as a history command.
0185
0186 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
0187
0188 pgset "udp_src_min 9" set UDP source port min, If < udp_src_max, then
0189 cycle through the port range.
0190
0191 pgset "udp_src_max 9" set UDP source port max.
0192 pgset "udp_dst_min 9" set UDP destination port min, If < udp_dst_max, then
0193 cycle through the port range.
0194 pgset "udp_dst_max 9" set UDP destination port max.
0195
0196 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
0197 outer label=16,middle label=32,
0198 inner label=0 (IPv4 NULL)) Note that
0199 there must be no spaces between the
0200 arguments. Leading zeros are required.
0201 Do not set the bottom of stack bit,
0202 that's done automatically. If you do
0203 set the bottom of stack bit, that
0204 indicates that you want to randomly
0205 generate that address and the flag
0206 MPLS_RND will be turned on. You
0207 can have any mix of random and fixed
0208 labels in the label stack.
0209
0210 pgset "mpls 0" turn off mpls (or any invalid argument works too!)
0211
0212 pgset "vlan_id 77" set VLAN ID 0-4095
0213 pgset "vlan_p 3" set priority bit 0-7 (default 0)
0214 pgset "vlan_cfi 0" set canonical format identifier 0-1 (default 0)
0215
0216 pgset "svlan_id 22" set SVLAN ID 0-4095
0217 pgset "svlan_p 3" set priority bit 0-7 (default 0)
0218 pgset "svlan_cfi 0" set canonical format identifier 0-1 (default 0)
0219
0220 pgset "vlan_id 9999" > 4095 remove vlan and svlan tags
0221 pgset "svlan 9999" > 4095 remove svlan tag
0222
0223
0224 pgset "tos XX" set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
0225 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
0226
0227 pgset "rate 300M" set rate to 300 Mb/s
0228 pgset "ratep 1000000" set rate to 1Mpps
0229
0230 pgset "xmit_mode netif_receive" RX inject into stack netif_receive_skb()
0231 Works with "burst" but not with "clone_skb".
0232 Default xmit_mode is "start_xmit".
0233
0234 Sample scripts
0235 ==============
0236
0237 A collection of tutorial scripts and helpers for pktgen is in the
0238 samples/pktgen directory. The helper parameters.sh file support easy
0239 and consistent parameter parsing across the sample scripts.
0240
0241 Usage example and help::
0242
0243 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
0244
0245 Usage:::
0246
0247 ./pktgen_sample01_simple.sh [-vx] -i ethX
0248
0249 -i : ($DEV) output interface/device (required)
0250 -s : ($PKT_SIZE) packet size
0251 -d : ($DEST_IP) destination IP. CIDR (e.g. 198.18.0.0/15) is also allowed
0252 -m : ($DST_MAC) destination MAC-addr
0253 -p : ($DST_PORT) destination PORT range (e.g. 433-444) is also allowed
0254 -t : ($THREADS) threads to start
0255 -f : ($F_THREAD) index of first thread (zero indexed CPU number)
0256 -c : ($SKB_CLONE) SKB clones send before alloc new SKB
0257 -n : ($COUNT) num messages to send per thread, 0 means indefinitely
0258 -b : ($BURST) HW level bursting of SKBs
0259 -v : ($VERBOSE) verbose
0260 -x : ($DEBUG) debug
0261 -6 : ($IP6) IPv6
0262 -w : ($DELAY) Tx Delay value (ns)
0263 -a : ($APPEND) Script will not reset generator's state, but will append its config
0264
0265 The global variables being set are also listed. E.g. the required
0266 interface/device parameter "-i" sets variable $DEV. Copy the
0267 pktgen_sampleXX scripts and modify them to fit your own needs.
0268
0269
0270 Interrupt affinity
0271 ===================
0272 Note that when adding devices to a specific CPU it is a good idea to
0273 also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
0274 to the same CPU. This reduces cache bouncing when freeing skbs.
0275
0276 Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
0277 to the running threads CPU (directly from smp_processor_id()).
0278
0279 Enable IPsec
0280 ============
0281 Default IPsec transformation with ESP encapsulation plus transport mode
0282 can be enabled by simply setting::
0283
0284 pgset "flag IPSEC"
0285 pgset "flows 1"
0286
0287 To avoid breaking existing testbed scripts for using AH type and tunnel mode,
0288 you can use "pgset spi SPI_VALUE" to specify which transformation mode
0289 to employ.
0290
0291
0292 Current commands and configuration options
0293 ==========================================
0294
0295 **Pgcontrol commands**::
0296
0297 start
0298 stop
0299 reset
0300
0301 **Thread commands**::
0302
0303 add_device
0304 rem_device_all
0305
0306
0307 **Device commands**::
0308
0309 count
0310 clone_skb
0311 burst
0312 debug
0313
0314 frags
0315 delay
0316
0317 src_mac_count
0318 dst_mac_count
0319
0320 pkt_size
0321 min_pkt_size
0322 max_pkt_size
0323
0324 queue_map_min
0325 queue_map_max
0326 skb_priority
0327
0328 tos (ipv4)
0329 traffic_class (ipv6)
0330
0331 mpls
0332
0333 udp_src_min
0334 udp_src_max
0335
0336 udp_dst_min
0337 udp_dst_max
0338
0339 node
0340
0341 flag
0342 IPSRC_RND
0343 IPDST_RND
0344 UDPSRC_RND
0345 UDPDST_RND
0346 MACSRC_RND
0347 MACDST_RND
0348 TXSIZE_RND
0349 IPV6
0350 MPLS_RND
0351 VID_RND
0352 SVID_RND
0353 FLOW_SEQ
0354 QUEUE_MAP_RND
0355 QUEUE_MAP_CPU
0356 UDPCSUM
0357 IPSEC
0358 NODE_ALLOC
0359 NO_TIMESTAMP
0360
0361 spi (ipsec)
0362
0363 dst_min
0364 dst_max
0365
0366 src_min
0367 src_max
0368
0369 dst_mac
0370 src_mac
0371
0372 clear_counters
0373
0374 src6
0375 dst6
0376 dst6_max
0377 dst6_min
0378
0379 flows
0380 flowlen
0381
0382 rate
0383 ratep
0384
0385 xmit_mode <start_xmit|netif_receive>
0386
0387 vlan_cfi
0388 vlan_id
0389 vlan_p
0390
0391 svlan_cfi
0392 svlan_id
0393 svlan_p
0394
0395
0396 References:
0397
0398 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
0399 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
0400
0401 Paper from Linux-Kongress in Erlangen 2004.
0402 - ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
0403
0404 Thanks to:
0405
0406 Grant Grundler for testing on IA-64 and parisc, Harald Welte, Lennert Buytenhek
0407 Stephen Hemminger, Andi Kleen, Dave Miller and many others.
0408
0409
0410 Good luck with the linux net-development.
