0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 =================
0004 Boot-time tracing
0005 =================
0006
0007 :Author: Masami Hiramatsu <mhiramat@kernel.org>
0008
0009 Overview
0010 ========
0011
0012 Boot-time tracing allows users to trace boot-time process including
0013 device initialization with full features of ftrace including per-event
0014 filter and actions, histograms, kprobe-events and synthetic-events,
0015 and trace instances.
0016 Since kernel command line is not enough to control these complex features,
0017 this uses bootconfig file to describe tracing feature programming.
0018
0019 Options in the Boot Config
0020 ==========================
0021
0022 Here is the list of available options list for boot time tracing in
0023 boot config file [1]_. All options are under "ftrace." or "kernel."
0024 prefix. See kernel parameters for the options which starts
0025 with "kernel." prefix [2]_.
0026
0027 .. [1] See :ref:`Documentation/admin-guide/bootconfig.rst <bootconfig>`
0028 .. [2] See :ref:`Documentation/admin-guide/kernel-parameters.rst <kernelparameters>`
0029
0030 Ftrace Global Options
0031 ---------------------
0032
0033 Ftrace global options have "kernel." prefix in boot config, which means
0034 these options are passed as a part of kernel legacy command line.
0035
0036 kernel.tp_printk
0037 Output trace-event data on printk buffer too.
0038
0039 kernel.dump_on_oops [= MODE]
0040 Dump ftrace on Oops. If MODE = 1 or omitted, dump trace buffer
0041 on all CPUs. If MODE = 2, dump a buffer on a CPU which kicks Oops.
0042
0043 kernel.traceoff_on_warning
0044 Stop tracing if WARN_ON() occurs.
0045
0046 kernel.fgraph_max_depth = MAX_DEPTH
0047 Set MAX_DEPTH to maximum depth of fgraph tracer.
0048
0049 kernel.fgraph_filters = FILTER[, FILTER2...]
0050 Add fgraph tracing function filters.
0051
0052 kernel.fgraph_notraces = FILTER[, FILTER2...]
0053 Add fgraph non-tracing function filters.
0054
0055
0056 Ftrace Per-instance Options
0057 ---------------------------
0058
0059 These options can be used for each instance including global ftrace node.
0060
0061 ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
0062 Enable given ftrace options.
0063
0064 ftrace.[instance.INSTANCE.]tracing_on = 0|1
0065 Enable/Disable tracing on this instance when starting boot-time tracing.
0066 (you can enable it by the "traceon" event trigger action)
0067
0068 ftrace.[instance.INSTANCE.]trace_clock = CLOCK
0069 Set given CLOCK to ftrace's trace_clock.
0070
0071 ftrace.[instance.INSTANCE.]buffer_size = SIZE
0072 Configure ftrace buffer size to SIZE. You can use "KB" or "MB"
0073 for that SIZE.
0074
0075 ftrace.[instance.INSTANCE.]alloc_snapshot
0076 Allocate snapshot buffer.
0077
0078 ftrace.[instance.INSTANCE.]cpumask = CPUMASK
0079 Set CPUMASK as trace cpu-mask.
0080
0081 ftrace.[instance.INSTANCE.]events = EVENT[, EVENT2[...]]
0082 Enable given events on boot. You can use a wild card in EVENT.
0083
0084 ftrace.[instance.INSTANCE.]tracer = TRACER
0085 Set TRACER to current tracer on boot. (e.g. function)
0086
0087 ftrace.[instance.INSTANCE.]ftrace.filters
0088 This will take an array of tracing function filter rules.
0089
0090 ftrace.[instance.INSTANCE.]ftrace.notraces
0091 This will take an array of NON-tracing function filter rules.
0092
0093
0094 Ftrace Per-Event Options
0095 ------------------------
0096
0097 These options are setting per-event options.
0098
0099 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.enable
0100 Enable GROUP:EVENT tracing.
0101
0102 ftrace.[instance.INSTANCE.]event.GROUP.enable
0103 Enable all event tracing within GROUP.
0104
0105 ftrace.[instance.INSTANCE.]event.enable
0106 Enable all event tracing.
0107
0108 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.filter = FILTER
0109 Set FILTER rule to the GROUP:EVENT.
0110
0111 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.actions = ACTION[, ACTION2[...]]
0112 Set ACTIONs to the GROUP:EVENT.
0113
0114 ftrace.[instance.INSTANCE.]event.kprobes.EVENT.probes = PROBE[, PROBE2[...]]
0115 Defines new kprobe event based on PROBEs. It is able to define
0116 multiple probes on one event, but those must have same type of
0117 arguments. This option is available only for the event which
0118 group name is "kprobes".
0119
0120 ftrace.[instance.INSTANCE.]event.synthetic.EVENT.fields = FIELD[, FIELD2[...]]
0121 Defines new synthetic event with FIELDs. Each field should be
0122 "type varname".
0123
0124 Note that kprobe and synthetic event definitions can be written under
0125 instance node, but those are also visible from other instances. So please
0126 take care for event name conflict.
0127
0128 Ftrace Histogram Options
0129 ------------------------
0130
0131 Since it is too long to write a histogram action as a string for per-event
0132 action option, there are tree-style options under per-event 'hist' subkey
0133 for the histogram actions. For the detail of the each parameter,
0134 please read the event histogram document (Documentation/trace/histogram.rst)
0135
0136 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]keys = KEY1[, KEY2[...]]
0137 Set histogram key parameters. (Mandatory)
0138 The 'N' is a digit string for the multiple histogram. You can omit it
0139 if there is one histogram on the event.
0140
0141 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]values = VAL1[, VAL2[...]]
0142 Set histogram value parameters.
0143
0144 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]sort = SORT1[, SORT2[...]]
0145 Set histogram sort parameter options.
0146
0147 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]size = NR_ENTRIES
0148 Set histogram size (number of entries).
0149
0150 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]name = NAME
0151 Set histogram name.
0152
0153 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]var.VARIABLE = EXPR
0154 Define a new VARIABLE by EXPR expression.
0155
0156 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<pause|continue|clear>
0157 Set histogram control parameter. You can set one of them.
0158
0159 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]event = GROUP.EVENT
0160 Set histogram 'onmatch' handler matching event parameter.
0161 The 'M' is a digit string for the multiple 'onmatch' handler. You can omit it
0162 if there is one 'onmatch' handler on this histogram.
0163
0164 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmatch.[M.]trace = EVENT[, ARG1[...]]
0165 Set histogram 'trace' action for 'onmatch'.
0166 EVENT must be a synthetic event name, and ARG1... are parameters
0167 for that event. Mandatory if 'onmatch.event' option is set.
0168
0169 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onmax.[M.]var = VAR
0170 Set histogram 'onmax' handler variable parameter.
0171
0172 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]onchange.[M.]var = VAR
0173 Set histogram 'onchange' handler variable parameter.
0174
0175 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]save = ARG1[, ARG2[...]]
0176 Set histogram 'save' action parameters for 'onmax' or 'onchange' handler.
0177 This option or below 'snapshot' option is mandatory if 'onmax.var' or
0178 'onchange.var' option is set.
0179
0180 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.[N.]<onmax|onchange>.[M.]snapshot
0181 Set histogram 'snapshot' action for 'onmax' or 'onchange' handler.
0182 This option or above 'save' option is mandatory if 'onmax.var' or
0183 'onchange.var' option is set.
0184
0185 ftrace.[instance.INSTANCE.]event.GROUP.EVENT.hist.filter = FILTER_EXPR
0186 Set histogram filter expression. You don't need 'if' in the FILTER_EXPR.
0187
0188 Note that this 'hist' option can conflict with the per-event 'actions'
0189 option if the 'actions' option has a histogram action.
0190
0191
0192 When to Start
0193 =============
0194
0195 All boot-time tracing options starting with ``ftrace`` will be enabled at the
0196 end of core_initcall. This means you can trace the events from postcore_initcall.
0197 Most of the subsystems and architecture dependent drivers will be initialized
0198 after that (arch_initcall or subsys_initcall). Thus, you can trace those with
0199 boot-time tracing.
0200 If you want to trace events before core_initcall, you can use the options
0201 starting with ``kernel``. Some of them will be enabled eariler than the initcall
0202 processing (for example,. ``kernel.ftrace=function`` and ``kernel.trace_event``
0203 will start before the initcall.)
0204
0205
0206 Examples
0207 ========
0208
0209 For example, to add filter and actions for each event, define kprobe
0210 events, and synthetic events with histogram, write a boot config like
0211 below::
0212
0213 ftrace.event {
0214 task.task_newtask {
0215 filter = "pid < 128"
0216 enable
0217 }
0218 kprobes.vfs_read {
0219 probes = "vfs_read $arg1 $arg2"
0220 filter = "common_pid < 200"
0221 enable
0222 }
0223 synthetic.initcall_latency {
0224 fields = "unsigned long func", "u64 lat"
0225 hist {
0226 keys = func.sym, lat
0227 values = lat
0228 sort = lat
0229 }
0230 }
0231 initcall.initcall_start.hist {
0232 keys = func
0233 var.ts0 = common_timestamp.usecs
0234 }
0235 initcall.initcall_finish.hist {
0236 keys = func
0237 var.lat = common_timestamp.usecs - $ts0
0238 onmatch {
0239 event = initcall.initcall_start
0240 trace = initcall_latency, func, $lat
0241 }
0242 }
0243 }
0244
0245 Also, boot-time tracing supports "instance" node, which allows us to run
0246 several tracers for different purpose at once. For example, one tracer
0247 is for tracing functions starting with "user\_", and others tracing
0248 "kernel\_" functions, you can write boot config as below::
0249
0250 ftrace.instance {
0251 foo {
0252 tracer = "function"
0253 ftrace.filters = "user_*"
0254 }
0255 bar {
0256 tracer = "function"
0257 ftrace.filters = "kernel_*"
0258 }
0259 }
0260
0261 The instance node also accepts event nodes so that each instance
0262 can customize its event tracing.
0263
0264 With the trigger action and kprobes, you can trace function-graph while
0265 a function is called. For example, this will trace all function calls in
0266 the pci_proc_init()::
0267
0268 ftrace {
0269 tracing_on = 0
0270 tracer = function_graph
0271 event.kprobes {
0272 start_event {
0273 probes = "pci_proc_init"
0274 actions = "traceon"
0275 }
0276 end_event {
0277 probes = "pci_proc_init%return"
0278 actions = "traceoff"
0279 }
0280 }
0281 }
0282
0283
0284 This boot-time tracing also supports ftrace kernel parameters via boot
0285 config.
0286 For example, following kernel parameters::
0287
0288 trace_options=sym-addr trace_event=initcall:* tp_printk trace_buf_size=1M ftrace=function ftrace_filter="vfs*"
0289
0290 This can be written in boot config like below::
0291
0292 kernel {
0293 trace_options = sym-addr
0294 trace_event = "initcall:*"
0295 tp_printk
0296 trace_buf_size = 1M
0297 ftrace = function
0298 ftrace_filter = "vfs*"
0299 }
0300
0301 Note that parameters start with "kernel" prefix instead of "ftrace".
