0001 ======================================
0002 Coresight - HW Assisted Tracing on ARM
0003 ======================================
0004
0005 :Author: Mathieu Poirier <mathieu.poirier@linaro.org>
0006 :Date: September 11th, 2014
0007
0008 Introduction
0009 ------------
0010
0011 Coresight is an umbrella of technologies allowing for the debugging of ARM
0012 based SoC. It includes solutions for JTAG and HW assisted tracing. This
0013 document is concerned with the latter.
0014
0015 HW assisted tracing is becoming increasingly useful when dealing with systems
0016 that have many SoCs and other components like GPU and DMA engines. ARM has
0017 developed a HW assisted tracing solution by means of different components, each
0018 being added to a design at synthesis time to cater to specific tracing needs.
0019 Components are generally categorised as source, link and sinks and are
0020 (usually) discovered using the AMBA bus.
0021
0022 "Sources" generate a compressed stream representing the processor instruction
0023 path based on tracing scenarios as configured by users. From there the stream
0024 flows through the coresight system (via ATB bus) using links that are connecting
0025 the emanating source to a sink(s). Sinks serve as endpoints to the coresight
0026 implementation, either storing the compressed stream in a memory buffer or
0027 creating an interface to the outside world where data can be transferred to a
0028 host without fear of filling up the onboard coresight memory buffer.
0029
0030 At typical coresight system would look like this::
0031
0032 *****************************************************************
0033 **************************** AMBA AXI ****************************===||
0034 ***************************************************************** ||
0035 ^ ^ | ||
0036 | | * **
0037 0000000 ::::: 0000000 ::::: ::::: @@@@@@@ ||||||||||||
0038 0 CPU 0<-->: C : 0 CPU 0<-->: C : : C : @ STM @ || System ||
0039 |->0000000 : T : |->0000000 : T : : T :<--->@@@@@ || Memory ||
0040 | #######<-->: I : | #######<-->: I : : I : @@@<-| ||||||||||||
0041 | # ETM # ::::: | # PTM # ::::: ::::: @ |
0042 | ##### ^ ^ | ##### ^ ! ^ ! . | |||||||||
0043 | |->### | ! | |->### | ! | ! . | || DAP ||
0044 | | # | ! | | # | ! | ! . | |||||||||
0045 | | . | ! | | . | ! | ! . | | |
0046 | | . | ! | | . | ! | ! . | | *
0047 | | . | ! | | . | ! | ! . | | SWD/
0048 | | . | ! | | . | ! | ! . | | JTAG
0049 *****************************************************************<-|
0050 *************************** AMBA Debug APB ************************
0051 *****************************************************************
0052 | . ! . ! ! . |
0053 | . * . * * . |
0054 *****************************************************************
0055 ******************** Cross Trigger Matrix (CTM) *******************
0056 *****************************************************************
0057 | . ^ . . |
0058 | * ! * * |
0059 *****************************************************************
0060 ****************** AMBA Advanced Trace Bus (ATB) ******************
0061 *****************************************************************
0062 | ! =============== |
0063 | * ===== F =====<---------|
0064 | ::::::::: ==== U ====
0065 |-->:: CTI ::<!! === N ===
0066 | ::::::::: ! == N ==
0067 | ^ * == E ==
0068 | ! &&&&&&&&& IIIIIII == L ==
0069 |------>&& ETB &&<......II I =======
0070 | ! &&&&&&&&& II I .
0071 | ! I I .
0072 | ! I REP I<..........
0073 | ! I I
0074 | !!>&&&&&&&&& II I *Source: ARM ltd.
0075 |------>& TPIU &<......II I DAP = Debug Access Port
0076 &&&&&&&&& IIIIIII ETM = Embedded Trace Macrocell
0077 ; PTM = Program Trace Macrocell
0078 ; CTI = Cross Trigger Interface
0079 * ETB = Embedded Trace Buffer
0080 To trace port TPIU= Trace Port Interface Unit
0081 SWD = Serial Wire Debug
0082
0083 While on target configuration of the components is done via the APB bus,
0084 all trace data are carried out-of-band on the ATB bus. The CTM provides
0085 a way to aggregate and distribute signals between CoreSight components.
0086
0087 The coresight framework provides a central point to represent, configure and
0088 manage coresight devices on a platform. This first implementation centers on
0089 the basic tracing functionality, enabling components such ETM/PTM, funnel,
0090 replicator, TMC, TPIU and ETB. Future work will enable more
0091 intricate IP blocks such as STM and CTI.
0092
0093
0094 Acronyms and Classification
0095 ---------------------------
0096
0097 Acronyms:
0098
0099 PTM:
0100 Program Trace Macrocell
0101 ETM:
0102 Embedded Trace Macrocell
0103 STM:
0104 System trace Macrocell
0105 ETB:
0106 Embedded Trace Buffer
0107 ITM:
0108 Instrumentation Trace Macrocell
0109 TPIU:
0110 Trace Port Interface Unit
0111 TMC-ETR:
0112 Trace Memory Controller, configured as Embedded Trace Router
0113 TMC-ETF:
0114 Trace Memory Controller, configured as Embedded Trace FIFO
0115 CTI:
0116 Cross Trigger Interface
0117
0118 Classification:
0119
0120 Source:
0121 ETMv3.x ETMv4, PTMv1.0, PTMv1.1, STM, STM500, ITM
0122 Link:
0123 Funnel, replicator (intelligent or not), TMC-ETR
0124 Sinks:
0125 ETBv1.0, ETB1.1, TPIU, TMC-ETF
0126 Misc:
0127 CTI
0128
0129
0130 Device Tree Bindings
0131 --------------------
0132
0133 See Documentation/devicetree/bindings/arm/arm,coresight-\*.yaml for details.
0134
0135 As of this writing drivers for ITM, STMs and CTIs are not provided but are
0136 expected to be added as the solution matures.
0137
0138
0139 Framework and implementation
0140 ----------------------------
0141
0142 The coresight framework provides a central point to represent, configure and
0143 manage coresight devices on a platform. Any coresight compliant device can
0144 register with the framework for as long as they use the right APIs:
0145
0146 .. c:function:: struct coresight_device *coresight_register(struct coresight_desc *desc);
0147 .. c:function:: void coresight_unregister(struct coresight_device *csdev);
0148
0149 The registering function is taking a ``struct coresight_desc *desc`` and
0150 register the device with the core framework. The unregister function takes
0151 a reference to a ``struct coresight_device *csdev`` obtained at registration time.
0152
0153 If everything goes well during the registration process the new devices will
0154 show up under /sys/bus/coresight/devices, as showns here for a TC2 platform::
0155
0156 root:~# ls /sys/bus/coresight/devices/
0157 replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
0158 20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
0159 root:~#
0160
0161 The functions take a ``struct coresight_device``, which looks like this::
0162
0163 struct coresight_desc {
0164 enum coresight_dev_type type;
0165 struct coresight_dev_subtype subtype;
0166 const struct coresight_ops *ops;
0167 struct coresight_platform_data *pdata;
0168 struct device *dev;
0169 const struct attribute_group **groups;
0170 };
0171
0172
0173 The "coresight_dev_type" identifies what the device is, i.e, source link or
0174 sink while the "coresight_dev_subtype" will characterise that type further.
0175
0176 The ``struct coresight_ops`` is mandatory and will tell the framework how to
0177 perform base operations related to the components, each component having
0178 a different set of requirement. For that ``struct coresight_ops_sink``,
0179 ``struct coresight_ops_link`` and ``struct coresight_ops_source`` have been
0180 provided.
0181
0182 The next field ``struct coresight_platform_data *pdata`` is acquired by calling
0183 ``of_get_coresight_platform_data()``, as part of the driver's _probe routine and
0184 ``struct device *dev`` gets the device reference embedded in the ``amba_device``::
0185
0186 static int etm_probe(struct amba_device *adev, const struct amba_id *id)
0187 {
0188 ...
0189 ...
0190 drvdata->dev = &adev->dev;
0191 ...
0192 }
0193
0194 Specific class of device (source, link, or sink) have generic operations
0195 that can be performed on them (see ``struct coresight_ops``). The ``**groups``
0196 is a list of sysfs entries pertaining to operations
0197 specific to that component only. "Implementation defined" customisations are
0198 expected to be accessed and controlled using those entries.
0199
0200 Device Naming scheme
0201 --------------------
0202
0203 The devices that appear on the "coresight" bus were named the same as their
0204 parent devices, i.e, the real devices that appears on AMBA bus or the platform bus.
0205 Thus the names were based on the Linux Open Firmware layer naming convention,
0206 which follows the base physical address of the device followed by the device
0207 type. e.g::
0208
0209 root:~# ls /sys/bus/coresight/devices/
0210 20010000.etf 20040000.funnel 20100000.stm 22040000.etm
0211 22140000.etm 230c0000.funnel 23240000.etm 20030000.tpiu
0212 20070000.etr 20120000.replicator 220c0000.funnel
0213 23040000.etm 23140000.etm 23340000.etm
0214
0215 However, with the introduction of ACPI support, the names of the real
0216 devices are a bit cryptic and non-obvious. Thus, a new naming scheme was
0217 introduced to use more generic names based on the type of the device. The
0218 following rules apply::
0219
0220 1) Devices that are bound to CPUs, are named based on the CPU logical
0221 number.
0222
0223 e.g, ETM bound to CPU0 is named "etm0"
0224
0225 2) All other devices follow a pattern, "<device_type_prefix>N", where :
0226
0227 <device_type_prefix> - A prefix specific to the type of the device
0228 N - a sequential number assigned based on the order
0229 of probing.
0230
0231 e.g, tmc_etf0, tmc_etr0, funnel0, funnel1
0232
0233 Thus, with the new scheme the devices could appear as ::
0234
0235 root:~# ls /sys/bus/coresight/devices/
0236 etm0 etm1 etm2 etm3 etm4 etm5 funnel0
0237 funnel1 funnel2 replicator0 stm0 tmc_etf0 tmc_etr0 tpiu0
0238
0239 Some of the examples below might refer to old naming scheme and some
0240 to the newer scheme, to give a confirmation that what you see on your
0241 system is not unexpected. One must use the "names" as they appear on
0242 the system under specified locations.
0243
0244 Topology Representation
0245 -----------------------
0246
0247 Each CoreSight component has a ``connections`` directory which will contain
0248 links to other CoreSight components. This allows the user to explore the trace
0249 topology and for larger systems, determine the most appropriate sink for a
0250 given source. The connection information can also be used to establish
0251 which CTI devices are connected to a given component. This directory contains a
0252 ``nr_links`` attribute detailing the number of links in the directory.
0253
0254 For an ETM source, in this case ``etm0`` on a Juno platform, a typical
0255 arrangement will be::
0256
0257 linaro-developer:~# ls - l /sys/bus/coresight/devices/etm0/connections
0258 <file details> cti_cpu0 -> ../../../23020000.cti/cti_cpu0
0259 <file details> nr_links
0260 <file details> out:0 -> ../../../230c0000.funnel/funnel2
0261
0262 Following the out port to ``funnel2``::
0263
0264 linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel2/connections
0265 <file details> in:0 -> ../../../23040000.etm/etm0
0266 <file details> in:1 -> ../../../23140000.etm/etm3
0267 <file details> in:2 -> ../../../23240000.etm/etm4
0268 <file details> in:3 -> ../../../23340000.etm/etm5
0269 <file details> nr_links
0270 <file details> out:0 -> ../../../20040000.funnel/funnel0
0271
0272 And again to ``funnel0``::
0273
0274 linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel0/connections
0275 <file details> in:0 -> ../../../220c0000.funnel/funnel1
0276 <file details> in:1 -> ../../../230c0000.funnel/funnel2
0277 <file details> nr_links
0278 <file details> out:0 -> ../../../20010000.etf/tmc_etf0
0279
0280 Finding the first sink ``tmc_etf0``. This can be used to collect data
0281 as a sink, or as a link to propagate further along the chain::
0282
0283 linaro-developer:~# ls -l /sys/bus/coresight/devices/tmc_etf0/connections
0284 <file details> cti_sys0 -> ../../../20020000.cti/cti_sys0
0285 <file details> in:0 -> ../../../20040000.funnel/funnel0
0286 <file details> nr_links
0287 <file details> out:0 -> ../../../20150000.funnel/funnel4
0288
0289 via ``funnel4``::
0290
0291 linaro-developer:~# ls -l /sys/bus/coresight/devices/funnel4/connections
0292 <file details> in:0 -> ../../../20010000.etf/tmc_etf0
0293 <file details> in:1 -> ../../../20140000.etf/tmc_etf1
0294 <file details> nr_links
0295 <file details> out:0 -> ../../../20120000.replicator/replicator0
0296
0297 and a ``replicator0``::
0298
0299 linaro-developer:~# ls -l /sys/bus/coresight/devices/replicator0/connections
0300 <file details> in:0 -> ../../../20150000.funnel/funnel4
0301 <file details> nr_links
0302 <file details> out:0 -> ../../../20030000.tpiu/tpiu0
0303 <file details> out:1 -> ../../../20070000.etr/tmc_etr0
0304
0305 Arriving at the final sink in the chain, ``tmc_etr0``::
0306
0307 linaro-developer:~# ls -l /sys/bus/coresight/devices/tmc_etr0/connections
0308 <file details> cti_sys0 -> ../../../20020000.cti/cti_sys0
0309 <file details> in:0 -> ../../../20120000.replicator/replicator0
0310 <file details> nr_links
0311
0312 As described below, when using sysfs it is sufficient to enable a sink and
0313 a source for successful trace. The framework will correctly enable all
0314 intermediate links as required.
0315
0316 Note: ``cti_sys0`` appears in two of the connections lists above.
0317 CTIs can connect to multiple devices and are arranged in a star topology
0318 via the CTM. See (Documentation/trace/coresight/coresight-ect.rst)
0319 [#fourth]_ for further details.
0320 Looking at this device we see 4 connections::
0321
0322 linaro-developer:~# ls -l /sys/bus/coresight/devices/cti_sys0/connections
0323 <file details> nr_links
0324 <file details> stm0 -> ../../../20100000.stm/stm0
0325 <file details> tmc_etf0 -> ../../../20010000.etf/tmc_etf0
0326 <file details> tmc_etr0 -> ../../../20070000.etr/tmc_etr0
0327 <file details> tpiu0 -> ../../../20030000.tpiu/tpiu0
0328
0329
0330 How to use the tracer modules
0331 -----------------------------
0332
0333 There are two ways to use the Coresight framework:
0334
0335 1. using the perf cmd line tools.
0336 2. interacting directly with the Coresight devices using the sysFS interface.
0337
0338 Preference is given to the former as using the sysFS interface
0339 requires a deep understanding of the Coresight HW. The following sections
0340 provide details on using both methods.
0341
0342 Using the sysFS interface
0343 ~~~~~~~~~~~~~~~~~~~~~~~~~
0344
0345 Before trace collection can start, a coresight sink needs to be identified.
0346 There is no limit on the amount of sinks (nor sources) that can be enabled at
0347 any given moment. As a generic operation, all device pertaining to the sink
0348 class will have an "active" entry in sysfs::
0349
0350 root:/sys/bus/coresight/devices# ls
0351 replicator 20030000.tpiu 2201c000.ptm 2203c000.etm 2203e000.etm
0352 20010000.etb 20040000.funnel 2201d000.ptm 2203d000.etm
0353 root:/sys/bus/coresight/devices# ls 20010000.etb
0354 enable_sink status trigger_cntr
0355 root:/sys/bus/coresight/devices# echo 1 > 20010000.etb/enable_sink
0356 root:/sys/bus/coresight/devices# cat 20010000.etb/enable_sink
0357 1
0358 root:/sys/bus/coresight/devices#
0359
0360 At boot time the current etm3x driver will configure the first address
0361 comparator with "_stext" and "_etext", essentially tracing any instruction
0362 that falls within that range. As such "enabling" a source will immediately
0363 trigger a trace capture::
0364
0365 root:/sys/bus/coresight/devices# echo 1 > 2201c000.ptm/enable_source
0366 root:/sys/bus/coresight/devices# cat 2201c000.ptm/enable_source
0367 1
0368 root:/sys/bus/coresight/devices# cat 20010000.etb/status
0369 Depth: 0x2000
0370 Status: 0x1
0371 RAM read ptr: 0x0
0372 RAM wrt ptr: 0x19d3 <----- The write pointer is moving
0373 Trigger cnt: 0x0
0374 Control: 0x1
0375 Flush status: 0x0
0376 Flush ctrl: 0x2001
0377 root:/sys/bus/coresight/devices#
0378
0379 Trace collection is stopped the same way::
0380
0381 root:/sys/bus/coresight/devices# echo 0 > 2201c000.ptm/enable_source
0382 root:/sys/bus/coresight/devices#
0383
0384 The content of the ETB buffer can be harvested directly from /dev::
0385
0386 root:/sys/bus/coresight/devices# dd if=/dev/20010000.etb \
0387 of=~/cstrace.bin
0388 64+0 records in
0389 64+0 records out
0390 32768 bytes (33 kB) copied, 0.00125258 s, 26.2 MB/s
0391 root:/sys/bus/coresight/devices#
0392
0393 The file cstrace.bin can be decompressed using "ptm2human", DS-5 or Trace32.
0394
0395 Following is a DS-5 output of an experimental loop that increments a variable up
0396 to a certain value. The example is simple and yet provides a glimpse of the
0397 wealth of possibilities that coresight provides.
0398 ::
0399
0400 Info Tracing enabled
0401 Instruction 106378866 0x8026B53C E52DE004 false PUSH {lr}
0402 Instruction 0 0x8026B540 E24DD00C false SUB sp,sp,#0xc
0403 Instruction 0 0x8026B544 E3A03000 false MOV r3,#0
0404 Instruction 0 0x8026B548 E58D3004 false STR r3,[sp,#4]
0405 Instruction 0 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0406 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0407 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0408 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0409 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0410 Timestamp Timestamp: 17106715833
0411 Instruction 319 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0412 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0413 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0414 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0415 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0416 Instruction 9 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0417 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0418 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0419 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0420 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0421 Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0422 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0423 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0424 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0425 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0426 Instruction 7 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0427 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0428 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0429 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0430 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0431 Instruction 10 0x8026B54C E59D3004 false LDR r3,[sp,#4]
0432 Instruction 0 0x8026B550 E3530004 false CMP r3,#4
0433 Instruction 0 0x8026B554 E2833001 false ADD r3,r3,#1
0434 Instruction 0 0x8026B558 E58D3004 false STR r3,[sp,#4]
0435 Instruction 0 0x8026B55C DAFFFFFA true BLE {pc}-0x10 ; 0x8026b54c
0436 Instruction 6 0x8026B560 EE1D3F30 false MRC p15,#0x0,r3,c13,c0,#1
0437 Instruction 0 0x8026B564 E1A0100D false MOV r1,sp
0438 Instruction 0 0x8026B568 E3C12D7F false BIC r2,r1,#0x1fc0
0439 Instruction 0 0x8026B56C E3C2203F false BIC r2,r2,#0x3f
0440 Instruction 0 0x8026B570 E59D1004 false LDR r1,[sp,#4]
0441 Instruction 0 0x8026B574 E59F0010 false LDR r0,[pc,#16] ; [0x8026B58C] = 0x80550368
0442 Instruction 0 0x8026B578 E592200C false LDR r2,[r2,#0xc]
0443 Instruction 0 0x8026B57C E59221D0 false LDR r2,[r2,#0x1d0]
0444 Instruction 0 0x8026B580 EB07A4CF true BL {pc}+0x1e9344 ; 0x804548c4
0445 Info Tracing enabled
0446 Instruction 13570831 0x8026B584 E28DD00C false ADD sp,sp,#0xc
0447 Instruction 0 0x8026B588 E8BD8000 true LDM sp!,{pc}
0448 Timestamp Timestamp: 17107041535
0449
0450 Using perf framework
0451 ~~~~~~~~~~~~~~~~~~~~
0452
0453 Coresight tracers are represented using the Perf framework's Performance
0454 Monitoring Unit (PMU) abstraction. As such the perf framework takes charge of
0455 controlling when tracing gets enabled based on when the process of interest is
0456 scheduled. When configured in a system, Coresight PMUs will be listed when
0457 queried by the perf command line tool:
0458
0459 linaro@linaro-nano:~$ ./perf list pmu
0460
0461 List of pre-defined events (to be used in -e):
0462
0463 cs_etm// [Kernel PMU event]
0464
0465 linaro@linaro-nano:~$
0466
0467 Regardless of the number of tracers available in a system (usually equal to the
0468 amount of processor cores), the "cs_etm" PMU will be listed only once.
0469
0470 A Coresight PMU works the same way as any other PMU, i.e the name of the PMU is
0471 listed along with configuration options within forward slashes '/'. Since a
0472 Coresight system will typically have more than one sink, the name of the sink to
0473 work with needs to be specified as an event option.
0474 On newer kernels the available sinks are listed in sysFS under
0475 ($SYSFS)/bus/event_source/devices/cs_etm/sinks/::
0476
0477 root@localhost:/sys/bus/event_source/devices/cs_etm/sinks# ls
0478 tmc_etf0 tmc_etr0 tpiu0
0479
0480 On older kernels, this may need to be found from the list of coresight devices,
0481 available under ($SYSFS)/bus/coresight/devices/::
0482
0483 root:~# ls /sys/bus/coresight/devices/
0484 etm0 etm1 etm2 etm3 etm4 etm5 funnel0
0485 funnel1 funnel2 replicator0 stm0 tmc_etf0 tmc_etr0 tpiu0
0486 root@linaro-nano:~# perf record -e cs_etm/@tmc_etr0/u --per-thread program
0487
0488 As mentioned above in section "Device Naming scheme", the names of the devices could
0489 look different from what is used in the example above. One must use the device names
0490 as it appears under the sysFS.
0491
0492 The syntax within the forward slashes '/' is important. The '@' character
0493 tells the parser that a sink is about to be specified and that this is the sink
0494 to use for the trace session.
0495
0496 More information on the above and other example on how to use Coresight with
0497 the perf tools can be found in the "HOWTO.md" file of the openCSD gitHub
0498 repository [#third]_.
0499
0500 Advanced perf framework usage
0501 -----------------------------
0502
0503 AutoFDO analysis using the perf tools
0504 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0505
0506 perf can be used to record and analyze trace of programs.
0507
0508 Execution can be recorded using 'perf record' with the cs_etm event,
0509 specifying the name of the sink to record to, e.g::
0510
0511 perf record -e cs_etm/@tmc_etr0/u --per-thread
0512
0513 The 'perf report' and 'perf script' commands can be used to analyze execution,
0514 synthesizing instruction and branch events from the instruction trace.
0515 'perf inject' can be used to replace the trace data with the synthesized events.
0516 The --itrace option controls the type and frequency of synthesized events
0517 (see perf documentation).
0518
0519 Note that only 64-bit programs are currently supported - further work is
0520 required to support instruction decode of 32-bit Arm programs.
0521
0522 Tracing PID
0523 ~~~~~~~~~~~
0524
0525 The kernel can be built to write the PID value into the PE ContextID registers.
0526 For a kernel running at EL1, the PID is stored in CONTEXTIDR_EL1. A PE may
0527 implement Arm Virtualization Host Extensions (VHE), which the kernel can
0528 run at EL2 as a virtualisation host; in this case, the PID value is stored in
0529 CONTEXTIDR_EL2.
0530
0531 perf provides PMU formats that program the ETM to insert these values into the
0532 trace data; the PMU formats are defined as below:
0533
0534 "contextid1": Available on both EL1 kernel and EL2 kernel. When the
0535 kernel is running at EL1, "contextid1" enables the PID
0536 tracing; when the kernel is running at EL2, this enables
0537 tracing the PID of guest applications.
0538
0539 "contextid2": Only usable when the kernel is running at EL2. When
0540 selected, enables PID tracing on EL2 kernel.
0541
0542 "contextid": Will be an alias for the option that enables PID
0543 tracing. I.e,
0544 contextid == contextid1, on EL1 kernel.
0545 contextid == contextid2, on EL2 kernel.
0546
0547 perf will always enable PID tracing at the relevant EL, this is accomplished by
0548 automatically enable the "contextid" config - but for EL2 it is possible to make
0549 specific adjustments using configs "contextid1" and "contextid2", E.g. if a user
0550 wants to trace PIDs for both host and guest, the two configs "contextid1" and
0551 "contextid2" can be set at the same time:
0552
0553 perf record -e cs_etm/contextid1,contextid2/u -- vm
0554
0555
0556 Generating coverage files for Feedback Directed Optimization: AutoFDO
0557 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0558
0559 'perf inject' accepts the --itrace option in which case tracing data is
0560 removed and replaced with the synthesized events. e.g.
0561 ::
0562
0563 perf inject --itrace --strip -i perf.data -o perf.data.new
0564
0565 Below is an example of using ARM ETM for autoFDO. It requires autofdo
0566 (https://github.com/google/autofdo) and gcc version 5. The bubble
0567 sort example is from the AutoFDO tutorial (https://gcc.gnu.org/wiki/AutoFDO/Tutorial).
0568 ::
0569
0570 $ gcc-5 -O3 sort.c -o sort
0571 $ taskset -c 2 ./sort
0572 Bubble sorting array of 30000 elements
0573 5910 ms
0574
0575 $ perf record -e cs_etm/@tmc_etr0/u --per-thread taskset -c 2 ./sort
0576 Bubble sorting array of 30000 elements
0577 12543 ms
0578 [ perf record: Woken up 35 times to write data ]
0579 [ perf record: Captured and wrote 69.640 MB perf.data ]
0580
0581 $ perf inject -i perf.data -o inj.data --itrace=il64 --strip
0582 $ create_gcov --binary=./sort --profile=inj.data --gcov=sort.gcov -gcov_version=1
0583 $ gcc-5 -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo
0584 $ taskset -c 2 ./sort_autofdo
0585 Bubble sorting array of 30000 elements
0586 5806 ms
0587
0588 Config option formats
0589 ~~~~~~~~~~~~~~~~~~~~~
0590
0591 The following strings can be provided between // on the perf command line to enable various options.
0592 They are also listed in the folder /sys/bus/event_source/devices/cs_etm/format/
0593
0594 .. list-table::
0595 :header-rows: 1
0596
0597 * - Option
0598 - Description
0599 * - branch_broadcast
0600 - Session local version of the system wide setting:
0601 :ref:`ETM_MODE_BB <coresight-branch-broadcast>`
0602 * - contextid
0603 - See `Tracing PID`_
0604 * - contextid1
0605 - See `Tracing PID`_
0606 * - contextid2
0607 - See `Tracing PID`_
0608 * - configid
0609 - Selection for a custom configuration. This is an implementation detail and not used directly,
0610 see :ref:`trace/coresight/coresight-config:Using Configurations in perf`
0611 * - preset
0612 - Override for parameters in a custom configuration, see
0613 :ref:`trace/coresight/coresight-config:Using Configurations in perf`
0614 * - sinkid
0615 - Hashed version of the string to select a sink, automatically set when using the @ notation.
0616 This is an internal implementation detail and is not used directly, see `Using perf
0617 framework`_.
0618 * - cycacc
0619 - Session local version of the system wide setting: :ref:`ETMv4_MODE_CYCACC
0620 <coresight-cycle-accurate>`
0621 * - retstack
0622 - Session local version of the system wide setting: :ref:`ETM_MODE_RETURNSTACK
0623 <coresight-return-stack>`
0624 * - timestamp
0625 - Session local version of the system wide setting: :ref:`ETMv4_MODE_TIMESTAMP
0626 <coresight-timestamp>`
0627
0628 How to use the STM module
0629 -------------------------
0630
0631 Using the System Trace Macrocell module is the same as the tracers - the only
0632 difference is that clients are driving the trace capture rather
0633 than the program flow through the code.
0634
0635 As with any other CoreSight component, specifics about the STM tracer can be
0636 found in sysfs with more information on each entry being found in [#first]_::
0637
0638 root@genericarmv8:~# ls /sys/bus/coresight/devices/stm0
0639 enable_source hwevent_select port_enable subsystem uevent
0640 hwevent_enable mgmt port_select traceid
0641 root@genericarmv8:~#
0642
0643 Like any other source a sink needs to be identified and the STM enabled before
0644 being used::
0645
0646 root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/tmc_etf0/enable_sink
0647 root@genericarmv8:~# echo 1 > /sys/bus/coresight/devices/stm0/enable_source
0648
0649 From there user space applications can request and use channels using the devfs
0650 interface provided for that purpose by the generic STM API::
0651
0652 root@genericarmv8:~# ls -l /dev/stm0
0653 crw------- 1 root root 10, 61 Jan 3 18:11 /dev/stm0
0654 root@genericarmv8:~#
0655
0656 Details on how to use the generic STM API can be found here:
0657 - Documentation/trace/stm.rst [#second]_.
0658
0659 The CTI & CTM Modules
0660 ---------------------
0661
0662 The CTI (Cross Trigger Interface) provides a set of trigger signals between
0663 individual CTIs and components, and can propagate these between all CTIs via
0664 channels on the CTM (Cross Trigger Matrix).
0665
0666 A separate documentation file is provided to explain the use of these devices.
0667 (Documentation/trace/coresight/coresight-ect.rst) [#fourth]_.
0668
0669 CoreSight System Configuration
0670 ------------------------------
0671
0672 CoreSight components can be complex devices with many programming options.
0673 Furthermore, components can be programmed to interact with each other across the
0674 complete system.
0675
0676 A CoreSight System Configuration manager is provided to allow these complex programming
0677 configurations to be selected and used easily from perf and sysfs.
0678
0679 See the separate document for further information.
0680 (Documentation/trace/coresight/coresight-config.rst) [#fifth]_.
0681
0682
0683 .. [#first] Documentation/ABI/testing/sysfs-bus-coresight-devices-stm
0684
0685 .. [#second] Documentation/trace/stm.rst
0686
0687 .. [#third] https://github.com/Linaro/perf-opencsd
0688
0689 .. [#fourth] Documentation/trace/coresight/coresight-ect.rst
0690
0691 .. [#fifth] Documentation/trace/coresight/coresight-config.rst
