0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ===================================================
0004 The Kernel Test Anything Protocol (KTAP), version 1
0005 ===================================================
0006
0007 TAP, or the Test Anything Protocol is a format for specifying test results used
0008 by a number of projects. It's website and specification are found at this `link
0009 <https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test
0010 results. However, Kernel testing frameworks have special needs for test results
0011 which don't align with the original TAP specification. Thus, a "Kernel TAP"
0012 (KTAP) format is specified to extend and alter TAP to support these use-cases.
0013 This specification describes the generally accepted format of KTAP as it is
0014 currently used in the kernel.
0015
0016 KTAP test results describe a series of tests (which may be nested: i.e., test
0017 can have subtests), each of which can contain both diagnostic data -- e.g., log
0018 lines -- and a final result. The test structure and results are
0019 machine-readable, whereas the diagnostic data is unstructured and is there to
0020 aid human debugging.
0021
0022 KTAP output is built from four different types of lines:
0023 - Version lines
0024 - Plan lines
0025 - Test case result lines
0026 - Diagnostic lines
0027
0028 In general, valid KTAP output should also form valid TAP output, but some
0029 information, in particular nested test results, may be lost. Also note that
0030 there is a stagnant draft specification for TAP14, KTAP diverges from this in
0031 a couple of places (notably the "Subtest" header), which are described where
0032 relevant later in this document.
0033
0034 Version lines
0035 -------------
0036
0037 All KTAP-formatted results begin with a "version line" which specifies which
0038 version of the (K)TAP standard the result is compliant with.
0039
0040 For example:
0041 - "KTAP version 1"
0042 - "TAP version 13"
0043 - "TAP version 14"
0044
0045 Note that, in KTAP, subtests also begin with a version line, which denotes the
0046 start of the nested test results. This differs from TAP14, which uses a
0047 separate "Subtest" line.
0048
0049 While, going forward, "KTAP version 1" should be used by compliant tests, it
0050 is expected that most parsers and other tooling will accept the other versions
0051 listed here for compatibility with existing tests and frameworks.
0052
0053 Plan lines
0054 ----------
0055
0056 A test plan provides the number of tests (or subtests) in the KTAP output.
0057
0058 Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
0059 Plan lines follow version lines to indicate the number of nested tests.
0060
0061 While there are cases where the number of tests is not known in advance -- in
0062 which case the test plan may be omitted -- it is strongly recommended one is
0063 present where possible.
0064
0065 Test case result lines
0066 ----------------------
0067
0068 Test case result lines indicate the final status of a test.
0069 They are required and must have the format:
0070
0071 .. code-block:: none
0072
0073 <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
0074
0075 The result can be either "ok", which indicates the test case passed,
0076 or "not ok", which indicates that the test case failed.
0077
0078 <number> represents the number of the test being performed. The first test must
0079 have the number 1 and the number then must increase by 1 for each additional
0080 subtest within the same test at the same nesting level.
0081
0082 The description is a description of the test, generally the name of
0083 the test, and can be any string of words (can't include #). The
0084 description is optional, but recommended.
0085
0086 The directive and any diagnostic data is optional. If either are present, they
0087 must follow a hash sign, "#".
0088
0089 A directive is a keyword that indicates a different outcome for a test other
0090 than passed and failed. The directive is optional, and consists of a single
0091 keyword preceding the diagnostic data. In the event that a parser encounters
0092 a directive it doesn't support, it should fall back to the "ok" / "not ok"
0093 result.
0094
0095 Currently accepted directives are:
0096
0097 - "SKIP", which indicates a test was skipped (note the result of the test case
0098 result line can be either "ok" or "not ok" if the SKIP directive is used)
0099 - "TODO", which indicates that a test is not expected to pass at the moment,
0100 e.g. because the feature it is testing is known to be broken. While this
0101 directive is inherited from TAP, its use in the kernel is discouraged.
0102 - "XFAIL", which indicates that a test is expected to fail. This is similar
0103 to "TODO", above, and is used by some kselftest tests.
0104 - “TIMEOUT”, which indicates a test has timed out (note the result of the test
0105 case result line should be “not ok” if the TIMEOUT directive is used)
0106 - “ERROR”, which indicates that the execution of a test has failed due to a
0107 specific error that is included in the diagnostic data. (note the result of
0108 the test case result line should be “not ok” if the ERROR directive is used)
0109
0110 The diagnostic data is a plain-text field which contains any additional details
0111 about why this result was produced. This is typically an error message for ERROR
0112 or failed tests, or a description of missing dependencies for a SKIP result.
0113
0114 The diagnostic data field is optional, and results which have neither a
0115 directive nor any diagnostic data do not need to include the "#" field
0116 separator.
0117
0118 Example result lines include::
0119
0120 ok 1 test_case_name
0121
0122 The test "test_case_name" passed.
0123
0124 ::
0125
0126 not ok 1 test_case_name
0127
0128 The test "test_case_name" failed.
0129
0130 ::
0131
0132 ok 1 test # SKIP necessary dependency unavailable
0133
0134 The test "test" was SKIPPED with the diagnostic message "necessary dependency
0135 unavailable".
0136
0137 ::
0138
0139 not ok 1 test # TIMEOUT 30 seconds
0140
0141 The test "test" timed out, with diagnostic data "30 seconds".
0142
0143 ::
0144
0145 ok 5 check return code # rcode=0
0146
0147 The test "check return code" passed, with additional diagnostic data “rcode=0”
0148
0149
0150 Diagnostic lines
0151 ----------------
0152
0153 If tests wish to output any further information, they should do so using
0154 "diagnostic lines". Diagnostic lines are optional, freeform text, and are
0155 often used to describe what is being tested and any intermediate results in
0156 more detail than the final result and diagnostic data line provides.
0157
0158 Diagnostic lines are formatted as "# <diagnostic_description>", where the
0159 description can be any string. Diagnostic lines can be anywhere in the test
0160 output. As a rule, diagnostic lines regarding a test are directly before the
0161 test result line for that test.
0162
0163 Note that most tools will treat unknown lines (see below) as diagnostic lines,
0164 even if they do not start with a "#": this is to capture any other useful
0165 kernel output which may help debug the test. It is nevertheless recommended
0166 that tests always prefix any diagnostic output they have with a "#" character.
0167
0168 Unknown lines
0169 -------------
0170
0171 There may be lines within KTAP output that do not follow the format of one of
0172 the four formats for lines described above. This is allowed, however, they will
0173 not influence the status of the tests.
0174
0175 This is an important difference from TAP. Kernel tests may print messages
0176 to the system console or a log file. Both of these destinations may contain
0177 messages either from unrelated kernel or userspace activity, or kernel
0178 messages from non-test code that is invoked by the test. The kernel code
0179 invoked by the test likely is not aware that a test is in progress and
0180 thus can not print the message as a diagnostic message.
0181
0182 Nested tests
0183 ------------
0184
0185 In KTAP, tests can be nested. This is done by having a test include within its
0186 output an entire set of KTAP-formatted results. This can be used to categorize
0187 and group related tests, or to split out different results from the same test.
0188
0189 The "parent" test's result should consist of all of its subtests' results,
0190 starting with another KTAP version line and test plan, and end with the overall
0191 result. If one of the subtests fail, for example, the parent test should also
0192 fail.
0193
0194 Additionally, all lines in a subtest should be indented. One level of
0195 indentation is two spaces: " ". The indentation should begin at the version
0196 line and should end before the parent test's result line.
0197
0198 "Unknown lines" are not considered to be lines in a subtest and thus are
0199 allowed to be either indented or not indented.
0200
0201 An example of a test with two nested subtests:
0202
0203 ::
0204
0205 KTAP version 1
0206 1..1
0207 KTAP version 1
0208 1..2
0209 ok 1 test_1
0210 not ok 2 test_2
0211 # example failed
0212 not ok 1 example
0213
0214 An example format with multiple levels of nested testing:
0215
0216 ::
0217
0218 KTAP version 1
0219 1..2
0220 KTAP version 1
0221 1..2
0222 KTAP version 1
0223 1..2
0224 not ok 1 test_1
0225 ok 2 test_2
0226 not ok 1 test_3
0227 ok 2 test_4 # SKIP
0228 not ok 1 example_test_1
0229 ok 2 example_test_2
0230
0231
0232 Major differences between TAP and KTAP
0233 --------------------------------------
0234
0235 ================================================== ========= ===============
0236 Feature TAP KTAP
0237 ================================================== ========= ===============
0238 yaml and json in diagnosic message ok not recommended
0239 TODO directive ok not recognized
0240 allows an arbitrary number of tests to be nested no yes
0241 "Unknown lines" are in category of "Anything else" yes no
0242 "Unknown lines" are incorrect allowed
0243 ================================================== ========= ===============
0244
0245 The TAP14 specification does permit nested tests, but instead of using another
0246 nested version line, uses a line of the form
0247 "Subtest: <name>" where <name> is the name of the parent test.
0248
0249 Example KTAP output
0250 --------------------
0251 ::
0252
0253 KTAP version 1
0254 1..1
0255 KTAP version 1
0256 1..3
0257 KTAP version 1
0258 1..1
0259 # test_1: initializing test_1
0260 ok 1 test_1
0261 ok 1 example_test_1
0262 KTAP version 1
0263 1..2
0264 ok 1 test_1 # SKIP test_1 skipped
0265 ok 2 test_2
0266 ok 2 example_test_2
0267 KTAP version 1
0268 1..3
0269 ok 1 test_1
0270 # test_2: FAIL
0271 not ok 2 test_2
0272 ok 3 test_3 # SKIP test_3 skipped
0273 not ok 3 example_test_3
0274 not ok 1 main_test
0275
0276 This output defines the following hierarchy:
0277
0278 A single test called "main_test", which fails, and has three subtests:
0279 - "example_test_1", which passes, and has one subtest:
0280
0281 - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1"
0282
0283 - "example_test_2", which passes, and has two subtests:
0284
0285 - "test_1", which is skipped, with the explanation "test_1 skipped"
0286 - "test_2", which passes
0287
0288 - "example_test_3", which fails, and has three subtests
0289
0290 - "test_1", which passes
0291 - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.
0292 - "test_3", which is skipped with the explanation "test_3 skipped"
0293
0294 Note that the individual subtests with the same names do not conflict, as they
0295 are found in different parent tests. This output also exhibits some sensible
0296 rules for "bubbling up" test results: a test fails if any of its subtests fail.
0297 Skipped tests do not affect the result of the parent test (though it often
0298 makes sense for a test to be marked skipped if _all_ of its subtests have been
0299 skipped).
0300
0301 See also:
0302 ---------
0303
0304 - The TAP specification:
0305 https://testanything.org/tap-version-13-specification.html
0306 - The (stagnant) TAP version 14 specification:
0307 https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
0308 - The kselftest documentation:
0309 Documentation/dev-tools/kselftest.rst
0310 - The KUnit documentation:
0311 Documentation/dev-tools/kunit/index.rst
