0001 _
0002 _ __ _ __ ___ __ _ _ __ __ _ _ __ | |__
0003 | '_ \| '_ ` _ \ _____ / _` | '__/ _` | '_ \| '_ \
0004 | |_) | | | | | |_____| (_| | | | (_| | |_) | | | |
0005 | .__/|_| |_| |_| \__, |_| \__,_| .__/|_| |_|
0006 |_| |___/ |_|
0007
0008 pm-graph: suspend/resume/boot timing analysis tools
0009 Version: 5.9
0010 Author: Todd Brandt <todd.e.brandt@intel.com>
0011 Home Page: https://01.org/pm-graph
0012
0013 Report bugs/issues at bugzilla.kernel.org Tools/pm-graph
0014 - https://bugzilla.kernel.org/buglist.cgi?component=pm-graph&product=Tools
0015
0016 Full documentation available online & in man pages
0017 - Getting Started:
0018 https://01.org/pm-graph/documentation/getting-started
0019
0020 - Config File Format:
0021 https://01.org/pm-graph/documentation/3-config-file-format
0022
0023 - upstream version in git:
0024 https://github.com/intel/pm-graph/
0025
0026 Table of Contents
0027 - Overview
0028 - Setup
0029 - Usage
0030 - Basic Usage
0031 - Dev Mode Usage
0032 - Proc Mode Usage
0033 - Endurance Testing
0034 - Usage Examples
0035 - Configuration Files
0036 - Usage Examples
0037 - Config File Options
0038 - Custom Timeline Entries
0039 - Adding/Editing Timeline Functions
0040 - Adding/Editing Dev Timeline Source Functions
0041 - Verifying your Custom Functions
0042 - Testing on consumer linux Operating Systems
0043 - Android
0044
0045 ------------------------------------------------------------------
0046 | OVERVIEW |
0047 ------------------------------------------------------------------
0048
0049 This tool suite is designed to assist kernel and OS developers in optimizing
0050 their linux stack's suspend/resume & boot time. Using a kernel image built
0051 with a few extra options enabled, the tools will execute a suspend or boot,
0052 and will capture dmesg and ftrace data. This data is transformed into a set of
0053 timelines and a callgraph to give a quick and detailed view of which devices
0054 and kernel processes are taking the most time in suspend/resume & boot.
0055
0056 ------------------------------------------------------------------
0057 | SETUP |
0058 ------------------------------------------------------------------
0059
0060 Package Requirements
0061 - runs with python2 or python3, choice is made by /usr/bin/python link
0062 - python
0063 - python-configparser (for python2 sleepgraph)
0064 - python-requests (for stresstester.py)
0065 - linux-tools-common (for turbostat usage in sleepgraph)
0066
0067 Ubuntu:
0068 sudo apt-get install python python-configparser python-requests linux-tools-common
0069
0070 Fedora:
0071 sudo dnf install python python-configparser python-requests linux-tools-common
0072
0073 The tools can most easily be installed via git clone and make install
0074
0075 $> git clone http://github.com/intel/pm-graph.git
0076 $> cd pm-graph
0077 $> sudo make install
0078 $> man sleepgraph ; man bootgraph
0079
0080 Setup involves some minor kernel configuration
0081
0082 The following kernel build options are required for all kernels:
0083 CONFIG_DEVMEM=y
0084 CONFIG_PM_DEBUG=y
0085 CONFIG_PM_SLEEP_DEBUG=y
0086 CONFIG_FTRACE=y
0087 CONFIG_FUNCTION_TRACER=y
0088 CONFIG_FUNCTION_GRAPH_TRACER=y
0089 CONFIG_KPROBES=y
0090 CONFIG_KPROBES_ON_FTRACE=y
0091
0092 In kernel 3.15.0, two patches were upstreamed which enable the
0093 v3.0 behavior. These patches allow the tool to read all the
0094 data from trace events instead of from dmesg. You can enable
0095 this behavior on earlier kernels with these patches:
0096
0097 (kernel/pre-3.15/enable_trace_events_suspend_resume.patch)
0098 (kernel/pre-3.15/enable_trace_events_device_pm_callback.patch)
0099
0100 If you're using bootgraph, or sleepgraph with a kernel older than 3.15.0,
0101 the following additional kernel parameters are required:
0102 (e.g. in file /etc/default/grub)
0103 GRUB_CMDLINE_LINUX_DEFAULT="... initcall_debug log_buf_len=32M ..."
0104
0105 If you're using a kernel older than 3.11-rc2, the following simple
0106 patch must be applied to enable ftrace data:
0107 in file: kernel/power/suspend.c
0108 in function: int suspend_devices_and_enter(suspend_state_t state)
0109 remove call to "ftrace_stop();"
0110 remove call to "ftrace_start();"
0111
0112 There is a patch which does this for kernel v3.8.0:
0113 (kernel/pre-3.11-rc2/enable_ftrace_in_suspendresume.patch)
0114
0115
0116
0117 ------------------------------------------------------------------
0118 | USAGE |
0119 ------------------------------------------------------------------
0120
0121 Basic Usage
0122 ___________
0123
0124 1) First configure a kernel using the instructions from the previous sections.
0125 Then build, install, and boot with it.
0126 2) Open up a terminal window and execute the mode list command:
0127
0128 %> sudo ./sleepgraph.py -modes
0129 ['freeze', 'mem', 'disk']
0130
0131 Execute a test using one of the available power modes, e.g. mem (S3):
0132
0133 %> sudo ./sleepgraph.py -m mem -rtcwake 15
0134
0135 or with a config file
0136
0137 %> sudo ./sleepgraph.py -config config/suspend.cfg
0138
0139 When the system comes back you'll see the script finishing up and
0140 creating the output files in the test subdir. It generates output
0141 files in subdirectory: suspend-mmddyy-HHMMSS. The ftrace file can
0142 be used to regenerate the html timeline with different options
0143
0144 HTML output: <hostname>_<mode>.html
0145 raw dmesg output: <hostname>_<mode>_dmesg.txt
0146 raw ftrace output: <hostname>_<mode>_ftrace.txt
0147
0148 View the html in firefox or chrome.
0149
0150
0151 Dev Mode Usage
0152 ______________
0153
0154 Developer mode adds information on low level source calls to the timeline.
0155 The tool sets kprobes on all delay and mutex calls to see which devices
0156 are waiting for something and when. It also sets a suite of kprobes on
0157 subsystem dependent calls to better fill out the timeline.
0158
0159 The tool will also expose kernel threads that don't normally show up in the
0160 timeline. This is useful in discovering dependent threads to get a better
0161 idea of what each device is waiting for. For instance, the scsi_eh thread,
0162 a.k.a. scsi resume error handler, is what each SATA disk device waits for
0163 before it can continue resume.
0164
0165 The timeline will be much larger if run with dev mode, so it can be useful
0166 to set the -mindev option to clip out any device blocks that are too small
0167 to see easily. The following command will give a nice dev mode run:
0168
0169 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -mindev 1 -dev
0170
0171 or with a config file
0172
0173 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
0174
0175
0176 Proc Mode Usage
0177 _______________
0178
0179 Proc mode adds user process info to the timeline. This is done in a manner
0180 similar to the bootchart utility, which graphs init processes and their
0181 execution as the system boots. This tool option does the same thing but for
0182 the period before and after suspend/resume.
0183
0184 In order to see any process info, there needs to be some delay before or
0185 after resume since processes are frozen in suspend_prepare and thawed in
0186 resume_complete. The predelay and postdelay args allow you to do this. It
0187 can also be useful to run in x2 mode with an x2 delay, this way you can
0188 see process activity before and after resume, and in between two
0189 successive suspend/resumes.
0190
0191 The command can be run like this:
0192
0193 %> sudo ./sleepgraph.py -m mem -rtcwake 15 -x2 -x2delay 1000 -predelay 1000 -postdelay 1000 -proc
0194
0195 or with a config file
0196
0197 %> sudo ./sleepgraph.py -config config/suspend-proc.cfg
0198
0199 ------------------------------------------------------------------
0200 | ENDURANCE TESTING |
0201 ------------------------------------------------------------------
0202
0203 The best way to gauge the health of a system is to run a series of
0204 suspend/resumes over an extended period and analyze the behavior. This can be
0205 accomplished with sleepgraph's -multi argument. You specify two numbers: the
0206 number of tests to run OR the duration in days, hours, or minutes, and the
0207 delay in seconds between them. For instance, -multi 20 5: execute 20 tests with
0208 a 5 second delay between each, or -multi 24h 0: execute tests over a 24 hour
0209 period with no delay between tests. You can include any other options you like
0210 to generate the data you want. It's most useful to collect dev mode timelines
0211 as the kprobes don't alter the performance much and you get more insight.
0212
0213 On completion, the output folder contains a series of folders for the
0214 individual test data and a set of summary pages in the root. The summary.html
0215 file is a tabular list of the tests with relevant info and links. The
0216 summary-issue.html and summary-devices.html files include data taken from
0217 all tests on kernel issues and device performance. The folder looks like this:
0218
0219 suspend-xN-{date}-{time}:
0220 summary.html
0221 summary-issues.html
0222 summary-devices.html
0223 suspend-{date}-{time} (1)
0224 suspend-{date}-{time} (2)
0225 ...
0226
0227 These are the relevant arguments to use for testing:
0228
0229 -m mode
0230 Mode to initiate for suspend e.g. mem, freeze, standby (default: mem).
0231
0232 -rtcwake t
0233 Use rtcwake to autoresume after t seconds (default: 15).
0234
0235 -gzip (optional)
0236 Gzip the trace and dmesg logs to save space. The tool can also read in
0237 gzipped logs for processing. This reduces the multitest folder size.
0238
0239 -dev (optional)
0240 Add kernel source calls and threads to the timeline (default: disabled).
0241
0242 -multi n d
0243 Execute n consecutive tests at d seconds intervals. The outputs will be
0244 created in a new subdirectory: suspend-xN-{date}-{time}. When the multitest
0245 run is done, the -summary command is called automatically to create summary
0246 html files for all the data (unless you use -skiphtml). -skiphtml will
0247 speed up the testing by not creating timelines or summary html files. You
0248 can then run the tool again at a later time with -summary and -genhtml to
0249 create the timelines.
0250
0251 -skiphtml (optional)
0252 Run the test and capture the trace logs, but skip the timeline and summary
0253 html generation. This can greatly speed up overall testing. You can then
0254 copy the data to a faster host machine and run -summary -genhtml to
0255 generate the timelines and summary.
0256
0257 These are the relevant commands to use after testing is complete:
0258
0259 -summary indir
0260 Generate or regenerate the summary for a -multi test run. Creates three
0261 files: summary.html, summary-issues.html, and summary-devices.html in the
0262 current folder. summary.html is a table of tests with relevant info sorted
0263 by kernel/host/mode, and links to the test html files. summary-issues.html
0264 is a list of kernel issues found in dmesg from all the tests.
0265 summary-devices.html is a list of devices and times from all the tests.
0266
0267 -genhtml
0268 Used with -summary to regenerate any missing html timelines from their
0269 dmesg and ftrace logs. This will require a significant amount of time if
0270 there are thousands of tests.
0271
0272 Usage Examples
0273 _______________
0274
0275 A multitest is initiated like this:
0276
0277 %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0
0278
0279 or you can skip timeline generation in order to speed things up
0280
0281 %> sudo ./sleepgraph.py -m mem -rtcwake 10 -dev -gzip -multi 2000 0 -skiphtml
0282
0283 The tool will produce an output folder with all the test subfolders inside.
0284 Each test subfolder contains the dmesg/ftrace logs and/or the html timeline
0285 depending on whether you used the -skiphtml option. The root folder contains
0286 the summary.html files.
0287
0288 The summary for an existing multitest is generated like this:
0289
0290 %> cd suspend-x2000-{date}-{time}
0291 %> sleepgraph.py -summary .
0292
0293 or if you need to generate the html timelines you can use -genhtml
0294
0295 %> cd suspend-xN-{date}-{time}
0296 %> sleepgraph.py -summary . -genhtml
0297
0298 ------------------------------------------------------------------
0299 | CONFIGURATION FILES |
0300 ------------------------------------------------------------------
0301
0302 Since 4.0 we've moved to using config files in lieu of command line options.
0303 The config folder contains a collection of typical use cases.
0304 There are corresponding configs for other power modes:
0305
0306 Simple suspend/resume with basic timeline (mem/freeze/standby)
0307 config/suspend.cfg
0308 config/freeze.cfg
0309 config/standby.cfg
0310
0311 Dev mode suspend/resume with dev timeline (mem/freeze/standby)
0312 config/suspend-dev.cfg
0313 config/freeze-dev.cfg
0314 config/standby-dev.cfg
0315
0316 Simple suspend/resume with timeline and callgraph (mem/freeze/standby)
0317 config/suspend-callgraph.cfg
0318 config/freeze-callgraph.cfg
0319 config/standby-callgraph.cfg
0320
0321 Sample proc mode x2 run using mem suspend
0322 config/suspend-x2-proc.cfg
0323
0324 Sample for editing timeline funcs (moves internal functions into config)
0325 config/custom-timeline-functions.cfg
0326
0327 Sample debug config for serio subsystem
0328 config/debug-serio-suspend.cfg
0329
0330
0331 Usage Examples
0332 ______________
0333
0334 Run a simple mem suspend:
0335 %> sudo ./sleepgraph.py -config config/suspend.cfg
0336
0337 Run a mem suspend with callgraph data:
0338 %> sudo ./sleepgraph.py -config config/suspend-callgraph.cfg
0339
0340 Run a mem suspend with dev mode detail:
0341 %> sudo ./sleepgraph.py -config config/suspend-dev.cfg
0342
0343
0344 Config File Options
0345 ___________________
0346
0347 [Settings]
0348
0349 # Verbosity: print verbose messages (def: false)
0350 verbose: false
0351
0352 # Suspend Mode: e.g. standby, mem, freeze, disk (def: mem)
0353 mode: mem
0354
0355 # Output Directory Format: {hostname}, {date}, {time} give current values
0356 output-dir: suspend-{hostname}-{date}-{time}
0357
0358 # Automatic Wakeup: use rtcwake to wakeup after X seconds (def: infinity)
0359 rtcwake: 15
0360
0361 # Add Logs: add the dmesg and ftrace log to the html output (def: false)
0362 addlogs: false
0363
0364 # Sus/Res Gap: insert a gap between sus & res in the timeline (def: false)
0365 srgap: false
0366
0367 # Custom Command: Command to execute in lieu of suspend (def: "")
0368 command: echo mem > /sys/power/state
0369
0370 # Proc mode: graph user processes and cpu usage in the timeline (def: false)
0371 proc: false
0372
0373 # Dev mode: graph source functions in the timeline (def: false)
0374 dev: false
0375
0376 # Suspend/Resume x2: run 2 suspend/resumes back to back (def: false)
0377 x2: false
0378
0379 # x2 Suspend Delay: time delay between the two test runs in ms (def: 0 ms)
0380 x2delay: 0
0381
0382 # Pre Suspend Delay: nclude an N ms delay before (1st) suspend (def: 0 ms)
0383 predelay: 0
0384
0385 # Post Resume Delay: include an N ms delay after (last) resume (def: 0 ms)
0386 postdelay: 0
0387
0388 # Min Device Length: graph only dev callbacks longer than min (def: 0.001 ms)
0389 mindev: 0.001
0390
0391 # Callgraph: gather ftrace callgraph data on all timeline events (def: false)
0392 callgraph: false
0393
0394 # Expand Callgraph: pre-expand the callgraph treeviews in html (def: false)
0395 expandcg: false
0396
0397 # Min Callgraph Length: show callgraphs only if longer than min (def: 1 ms)
0398 mincg: 1
0399
0400 # Timestamp Precision: number of sig digits in timestamps (0:S, [3:ms], 6:us)
0401 timeprec: 3
0402
0403 # Device Filter: show only devs whose name/driver includes one of these strings
0404 devicefilter: _cpu_up,_cpu_down,i915,usb
0405
0406 # Override default timeline entries:
0407 # Do not use the internal default functions for timeline entries (def: false)
0408 # Set this to true if you intend to only use the ones defined in the config
0409 override-timeline-functions: true
0410
0411 # Override default dev timeline entries:
0412 # Do not use the internal default functions for dev timeline entries (def: false)
0413 # Set this to true if you intend to only use the ones defined in the config
0414 override-dev-timeline-functions: true
0415
0416 # Call Loop Max Gap (dev mode only)
0417 # merge loops of the same call if each is less than maxgap apart (def: 100us)
0418 callloop-maxgap: 0.0001
0419
0420 # Call Loop Max Length (dev mode only)
0421 # merge loops of the same call if each is less than maxlen in length (def: 5ms)
0422 callloop-maxlen: 0.005
0423
0424 ------------------------------------------------------------------
0425 | CUSTOM TIMELINE ENTRIES |
0426 ------------------------------------------------------------------
0427
0428 Adding or Editing Timeline Functions
0429 ____________________________________
0430
0431 The tool uses an array of function names to fill out empty spaces in the
0432 timeline where device callbacks don't appear. For instance, in suspend_prepare
0433 the tool adds the sys_sync and freeze_processes calls as virtual device blocks
0434 in the timeline to show you where the time is going. These calls should fill
0435 the timeline with contiguous data so that most kernel execution is covered.
0436
0437 It is possible to add new function calls to the timeline by adding them to
0438 the config. It's also possible to copy the internal timeline functions into
0439 the config so that you can override and edit them. Place them in the
0440 timeline_functions_ARCH section with the name of your architecture appended.
0441 i.e. for x86_64: [timeline_functions_x86_64]
0442
0443 Use the override-timeline-functions option if you only want to use your
0444 custom calls, or leave it false to append them to the internal ones.
0445
0446 This section includes a list of functions (set using kprobes) which use both
0447 symbol data and function arg data. The args are pulled directly from the
0448 stack using this architecture's registers and stack formatting. Each entry
0449 can include up to four pieces of info: The function name, a format string,
0450 an argument list, and a color. But only a function name is required.
0451
0452 For a full example config, see config/custom-timeline-functions.cfg. It pulls
0453 all the internal timeline functions into the config and allows you to edit
0454 them.
0455
0456 Entry format:
0457
0458 function: format{fn_arg1}_{fn_arg2} fn_arg1 fn_arg2 ... [color=purple]
0459
0460 Required Arguments:
0461
0462 function: The symbol name for the function you want probed, this is the
0463 minimum required for an entry, it will show up as the function
0464 name with no arguments.
0465
0466 example: _cpu_up:
0467
0468 Optional Arguments:
0469
0470 format: The format to display the data on the timeline in. Use braces to
0471 enclose the arg names.
0472
0473 example: CPU_ON[{cpu}]
0474
0475 color: The color of the entry block in the timeline. The default color is
0476 transparent, so the entry shares the phase color. The color is an
0477 html color string, either a word, or an RGB.
0478
0479 example: [color=#CC00CC]
0480
0481 arglist: A list of arguments from registers/stack addresses. See URL:
0482 https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt
0483
0484 example: cpu=%di:s32
0485
0486 Here is a full example entry. It displays cpu resume calls in the timeline
0487 in orange. They will appear as CPU_ON[0], CPU_ON[1], etc.
0488
0489 [timeline_functions_x86_64]
0490 _cpu_up: CPU_ON[{cpu}] cpu=%di:s32 [color=orange]
0491
0492
0493 Adding or Editing Dev Mode Timeline Source Functions
0494 ____________________________________________________
0495
0496 In dev mode, the tool uses an array of function names to monitor source
0497 execution within the timeline entries.
0498
0499 The function calls are displayed inside the main device/call blocks in the
0500 timeline. However, if a function call is not within a main timeline event,
0501 it will spawn an entirely new event named after the caller's kernel thread.
0502 These asynchronous kernel threads will populate in a separate section
0503 beneath the main device/call section.
0504
0505 The tool has a set of hard coded calls which focus on the most common use
0506 cases: msleep, udelay, schedule_timeout, mutex_lock_slowpath, etc. These are
0507 the functions that add a hardcoded time delay to the suspend/resume path.
0508 The tool also includes some common functions native to important
0509 subsystems: ata, i915, and ACPI, etc.
0510
0511 It is possible to add new function calls to the dev timeline by adding them
0512 to the config. It's also possible to copy the internal dev timeline
0513 functions into the config so that you can override and edit them. Place them
0514 in the dev_timeline_functions_ARCH section with the name of your architecture
0515 appended. i.e. for x86_64: [dev_timeline_functions_x86_64]
0516
0517 Use the override-dev-timeline-functions option if you only want to use your
0518 custom calls, or leave it false to append them to the internal ones.
0519
0520 The format is the same as the timeline_functions_x86_64 section. It's a
0521 list of functions (set using kprobes) which use both symbol data and function
0522 arg data. The args are pulled directly from the stack using this
0523 architecture's registers and stack formatting. Each entry can include up
0524 to four pieces of info: The function name, a format string, an argument list,
0525 and a color. But only the function name is required.
0526
0527 For a full example config, see config/custom-timeline-functions.cfg. It pulls
0528 all the internal dev timeline functions into the config and allows you to edit
0529 them.
0530
0531 Here is a full example entry. It displays the ATA port reset calls as
0532 ataN_port_reset in the timeline. This is where most of the SATA disk resume
0533 time goes, so it can be helpful to see the low level call.
0534
0535 [dev_timeline_functions_x86_64]
0536 ata_eh_recover: ata{port}_port_reset port=+36(%di):s32 [color=#CC00CC]
0537
0538
0539 Verifying your custom functions
0540 _______________________________
0541
0542 Once you have a set of functions (kprobes) defined, it can be useful to
0543 perform a quick check to see if you formatted them correctly and if the system
0544 actually supports them. To do this, run the tool with your config file
0545 and the -status option. The tool will go through all the kprobes (both
0546 custom and internal if you haven't overridden them) and actually attempts
0547 to set them in ftrace. It will then print out success or fail for you.
0548
0549 Note that kprobes which don't actually exist in the kernel won't stop the
0550 tool, they just wont show up.
0551
0552 For example:
0553
0554 sudo ./sleepgraph.py -config config/custom-timeline-functions.cfg -status
0555 Checking this system (myhostname)...
0556 have root access: YES
0557 is sysfs mounted: YES
0558 is "mem" a valid power mode: YES
0559 is ftrace supported: YES
0560 are kprobes supported: YES
0561 timeline data source: FTRACE (all trace events found)
0562 is rtcwake supported: YES
0563 verifying timeline kprobes work:
0564 _cpu_down: YES
0565 _cpu_up: YES
0566 acpi_pm_finish: YES
0567 acpi_pm_prepare: YES
0568 freeze_kernel_threads: YES
0569 freeze_processes: YES
0570 sys_sync: YES
0571 thaw_processes: YES
0572 verifying dev kprobes work:
0573 __const_udelay: YES
0574 __mutex_lock_slowpath: YES
0575 acpi_os_stall: YES
0576 acpi_ps_parse_aml: YES
0577 intel_opregion_init: NO
0578 intel_opregion_register: NO
0579 intel_opregion_setup: NO
0580 msleep: YES
0581 schedule_timeout: YES
0582 schedule_timeout_uninterruptible: YES
0583 usleep_range: YES
0584
0585
0586 ------------------------------------------------------------------
0587 | TESTING ON CONSUMER LINUX OPERATING SYSTEMS |
0588 ------------------------------------------------------------------
0589
0590 Android
0591 _______
0592
0593 The easiest way to execute on an android device is to run the android.sh
0594 script on the device, then pull the ftrace log back to the host and run
0595 sleepgraph.py on it.
0596
0597 Here are the steps:
0598
0599 [download and install the tool on the device]
0600
0601 host%> wget https://raw.githubusercontent.com/intel/pm-graph/master/tools/android.sh
0602 host%> adb connect 192.168.1.6
0603 host%> adb root
0604 # push the script to a writeable location
0605 host%> adb push android.sh /sdcard/
0606
0607 [check whether the tool will run on your device]
0608
0609 host%> adb shell
0610 dev%> cd /sdcard
0611 dev%> sh android.sh status
0612 host : asus_t100
0613 kernel : 3.14.0-i386-dirty
0614 modes : freeze mem
0615 rtcwake : supported
0616 ftrace : supported
0617 trace events {
0618 suspend_resume: found
0619 device_pm_callback_end: found
0620 device_pm_callback_start: found
0621 }
0622 # the above is what you see on a system that's properly patched
0623
0624 [execute the suspend]
0625
0626 # NOTE: The suspend will only work if the screen isn't timed out,
0627 # so you have to press some keys first to wake it up b4 suspend)
0628 dev%> sh android.sh suspend mem
0629 ------------------------------------
0630 Suspend/Resume timing test initiated
0631 ------------------------------------
0632 hostname : asus_t100
0633 kernel : 3.14.0-i386-dirty
0634 mode : mem
0635 ftrace out : /mnt/shell/emulated/0/ftrace.txt
0636 dmesg out : /mnt/shell/emulated/0/dmesg.txt
0637 log file : /mnt/shell/emulated/0/log.txt
0638 ------------------------------------
0639 INITIALIZING FTRACE........DONE
0640 STARTING FTRACE
0641 SUSPEND START @ 21:24:02 (rtcwake in 10 seconds)
0642 <adb connection will now terminate>
0643
0644 [retrieve the data from the device]
0645
0646 # I find that you have to actually kill the adb process and
0647 # reconnect sometimes in order for the connection to work post-suspend
0648 host%> adb connect 192.168.1.6
0649 # (required) get the ftrace data, this is the most important piece
0650 host%> adb pull /sdcard/ftrace.txt
0651 # (optional) get the dmesg data, this is for debugging
0652 host%> adb pull /sdcard/dmesg.txt
0653 # (optional) get the log, which just lists some test times for comparison
0654 host%> adb pull /sdcard/log.txt
0655
0656 [create an output html file using sleepgraph.py]
0657
0658 host%> sleepgraph.py -ftrace ftrace.txt
0659
0660 You should now have an output.html with the android data, enjoy!
