0001 tdc - Linux Traffic Control (tc) unit testing suite
0002
0003 Author: Lucas Bates - lucasb@mojatatu.com
0004
0005 tdc is a Python script to load tc unit tests from a separate JSON file and
0006 execute them inside a network namespace dedicated to the task.
0007
0008
0009 REQUIREMENTS
0010 ------------
0011
0012 * Minimum Python version of 3.4. Earlier 3.X versions may work but are not
0013 guaranteed.
0014
0015 * The kernel must have network namespace support if using nsPlugin
0016
0017 * The kernel must have veth support available, as a veth pair is created
0018 prior to running the tests when using nsPlugin.
0019
0020 * The kernel must have the appropriate infrastructure enabled to run all tdc
0021 unit tests. See the config file in this directory for minimum required
0022 features. As new tests will be added, config options list will be updated.
0023
0024 * All tc-related features being tested must be built in or available as
0025 modules. To check what is required in current setup run:
0026 ./tdc.py -c
0027
0028 Note:
0029 In the current release, tdc run will abort due to a failure in setup or
0030 teardown commands - which includes not being able to run a test simply
0031 because the kernel did not support a specific feature. (This will be
0032 handled in a future version - the current workaround is to run the tests
0033 on specific test categories that your kernel supports)
0034
0035
0036 BEFORE YOU RUN
0037 --------------
0038
0039 The path to the tc executable that will be most commonly tested can be defined
0040 in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
0041 define the path.
0042
0043 If you need to test a different tc executable on the fly, you can do so by
0044 using the -p option when running tdc:
0045 ./tdc.py -p /path/to/tc
0046
0047
0048 RUNNING TDC
0049 -----------
0050
0051 To use tdc, root privileges are required. This is because the
0052 commands being tested must be run as root. The code that enforces
0053 execution by root uid has been moved into a plugin (see PLUGIN
0054 ARCHITECTURE, below).
0055
0056 Tests that use a network device should have nsPlugin.py listed as a
0057 requirement for that test. nsPlugin executes all commands within a
0058 network namespace and creates a veth pair which may be used in those test
0059 cases. To disable execution within the namespace, pass the -N option
0060 to tdc when starting a test run; the veth pair will still be created
0061 by the plugin.
0062
0063 Running tdc without any arguments will run all tests. Refer to the section
0064 on command line arguments for more information, or run:
0065 ./tdc.py -h
0066
0067 tdc will list the test names as they are being run, and print a summary in
0068 TAP (Test Anything Protocol) format when they are done. If tests fail,
0069 output captured from the failing test will be printed immediately following
0070 the failed test in the TAP output.
0071
0072
0073 OVERVIEW OF TDC EXECUTION
0074 -------------------------
0075
0076 One run of tests is considered a "test suite" (this will be refined in the
0077 future). A test suite has one or more test cases in it.
0078
0079 A test case has four stages:
0080
0081 - setup
0082 - execute
0083 - verify
0084 - teardown
0085
0086 The setup and teardown stages can run zero or more commands. The setup
0087 stage does some setup if the test needs it. The teardown stage undoes
0088 the setup and returns the system to a "neutral" state so any other test
0089 can be run next. These two stages require any commands run to return
0090 success, but do not otherwise verify the results.
0091
0092 The execute and verify stages each run one command. The execute stage
0093 tests the return code against one or more acceptable values. The
0094 verify stage checks the return code for success, and also compares
0095 the stdout with a regular expression.
0096
0097 Each of the commands in any stage will run in a shell instance.
0098
0099
0100 USER-DEFINED CONSTANTS
0101 ----------------------
0102
0103 The tdc_config.py file contains multiple values that can be altered to suit
0104 your needs. Any value in the NAMES dictionary can be altered without affecting
0105 the tests to be run. These values are used in the tc commands that will be
0106 executed as part of the test. More will be added as test cases require.
0107
0108 Example:
0109 $TC qdisc add dev $DEV1 ingress
0110
0111 The NAMES values are used to substitute into the commands in the test cases.
0112
0113
0114 COMMAND LINE ARGUMENTS
0115 ----------------------
0116
0117 Run tdc.py -h to see the full list of available arguments.
0118
0119 usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
0120 [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v] [-N]
0121 [-d DEVICE] [-P] [-n] [-V]
0122
0123 Linux TC unit tests
0124
0125 optional arguments:
0126 -h, --help show this help message and exit
0127 -p PATH, --path PATH The full path to the tc executable to use
0128 -v, --verbose Show the commands that are being run
0129 -N, --notap Suppress tap results for command under test
0130 -d DEVICE, --device DEVICE
0131 Execute test cases that use a physical device, where
0132 DEVICE is its name. (If not defined, tests that require
0133 a physical device will be skipped)
0134 -P, --pause Pause execution just before post-suite stage
0135
0136 selection:
0137 select which test cases: files plus directories; filtered by categories
0138 plus testids
0139
0140 -D DIR [DIR ...], --directory DIR [DIR ...]
0141 Collect tests from the specified directory(ies)
0142 (default [tc-tests])
0143 -f FILE [FILE ...], --file FILE [FILE ...]
0144 Run tests from the specified file(s)
0145 -c [CATG [CATG ...]], --category [CATG [CATG ...]]
0146 Run tests only from the specified category/ies, or if
0147 no category/ies is/are specified, list known
0148 categories.
0149 -e ID [ID ...], --execute ID [ID ...]
0150 Execute the specified test cases with specified IDs
0151
0152 action:
0153 select action to perform on selected test cases
0154
0155 -l, --list List all test cases, or those only within the
0156 specified category
0157 -s, --show Display the selected test cases
0158 -i, --id Generate ID numbers for new test cases
0159
0160 netns:
0161 options for nsPlugin (run commands in net namespace)
0162
0163 -N, --no-namespace
0164 Do not run commands in a network namespace.
0165
0166 valgrind:
0167 options for valgrindPlugin (run command under test under Valgrind)
0168
0169 -V, --valgrind Run commands under valgrind
0170
0171
0172 PLUGIN ARCHITECTURE
0173 -------------------
0174
0175 There is now a plugin architecture, and some of the functionality that
0176 was in the tdc.py script has been moved into the plugins.
0177
0178 The plugins are in the directory plugin-lib. The are executed from
0179 directory plugins. Put symbolic links from plugins to plugin-lib,
0180 and name them according to the order you want them to run. This is not
0181 necessary if a test case being run requires a specific plugin to work.
0182
0183 Example:
0184
0185 bjb@bee:~/work/tc-testing$ ls -l plugins
0186 total 4
0187 lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
0188 lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
0189 -rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
0190
0191 The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
0192 must be called "SubPlugin" so tdc can find them. They are
0193 distinguished from each other in the python program by their module
0194 name.
0195
0196 This base class supplies "hooks" to run extra functions. These hooks are as follows:
0197
0198 pre- and post-suite
0199 pre- and post-case
0200 pre- and post-execute stage
0201 adjust-command (runs in all stages and receives the stage name)
0202
0203 The pre-suite hook receives the number of tests and an array of test ids.
0204 This allows you to dump out the list of skipped tests in the event of a
0205 failure during setup or teardown stage.
0206
0207 The pre-case hook receives the ordinal number and test id of the current test.
0208
0209 The adjust-command hook receives the stage id (see list below) and the
0210 full command to be executed. This allows for last-minute adjustment
0211 of the command.
0212
0213 The stages are identified by the following strings:
0214
0215 - pre (pre-suite)
0216 - setup
0217 - command
0218 - verify
0219 - teardown
0220 - post (post-suite)
0221
0222
0223 To write a plugin, you need to inherit from TdcPlugin in
0224 TdcPlugin.py. To use the plugin, you have to put the
0225 implementation file in plugin-lib, and add a symbolic link to it from
0226 plugins. It will be detected at run time and invoked at the
0227 appropriate times. There are a few examples in the plugin-lib
0228 directory:
0229
0230 - rootPlugin.py:
0231 implements the enforcement of running as root
0232 - nsPlugin.py:
0233 sets up a network namespace and runs all commands in that namespace,
0234 while also setting up dummy devices to be used in testing.
0235 - valgrindPlugin.py
0236 runs each command in the execute stage under valgrind,
0237 and checks for leaks.
0238 This plugin will output an extra test for each test in the test file,
0239 one is the existing output as to whether the test passed or failed,
0240 and the other is a test whether the command leaked memory or not.
0241 (This one is a preliminary version, it may not work quite right yet,
0242 but the overall template is there and it should only need tweaks.)
0243 - buildebpfPlugin.py:
0244 builds all programs in $EBPFDIR.
0245
0246
0247 ACKNOWLEDGEMENTS
0248 ----------------
0249
0250 Thanks to:
0251
0252 Jamal Hadi Salim, for providing valuable test cases
0253 Keara Leibovitz, who wrote the CLI test driver that I used as a base for the
0254 first version of the tc testing suite. This work was presented at
0255 Netdev 1.2 Tokyo in October 2016.
0256 Samir Hussain, for providing help while I dove into Python for the first time
0257 and being a second eye for this code.
