0001 .. highlight:: none
0002
0003 Debugging kernel and modules via gdb
0004 ====================================
0005
0006 The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
0007 interfaces allow to debug the Linux kernel and its modules during runtime
0008 using gdb. Gdb comes with a powerful scripting interface for python. The
0009 kernel provides a collection of helper scripts that can simplify typical
0010 kernel debugging steps. This is a short tutorial about how to enable and use
0011 them. It focuses on QEMU/KVM virtual machines as target, but the examples can
0012 be transferred to the other gdb stubs as well.
0013
0014
0015 Requirements
0016 ------------
0017
0018 - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
0019 for distributions)
0020
0021
0022 Setup
0023 -----
0024
0025 - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
0026 www.qemu.org for more details). For cross-development,
0027 https://landley.net/aboriginal/bin keeps a pool of machine images and
0028 toolchains that can be helpful to start from.
0029
0030 - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
0031 CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
0032 CONFIG_FRAME_POINTER, keep it enabled.
0033
0034 - Install that kernel on the guest, turn off KASLR if necessary by adding
0035 "nokaslr" to the kernel command line.
0036 Alternatively, QEMU allows to boot the kernel directly using -kernel,
0037 -append, -initrd command line switches. This is generally only useful if
0038 you do not depend on modules. See QEMU documentation for more details on
0039 this mode. In this case, you should build the kernel with
0040 CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
0041
0042 - Enable the gdb stub of QEMU/KVM, either
0043
0044 - at VM startup time by appending "-s" to the QEMU command line
0045
0046 or
0047
0048 - during runtime by issuing "gdbserver" from the QEMU monitor
0049 console
0050
0051 - cd /path/to/linux-build
0052
0053 - Start gdb: gdb vmlinux
0054
0055 Note: Some distros may restrict auto-loading of gdb scripts to known safe
0056 directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
0057
0058 add-auto-load-safe-path /path/to/linux-build
0059
0060 to ~/.gdbinit. See gdb help for more details.
0061
0062 - Attach to the booted guest::
0063
0064 (gdb) target remote :1234
0065
0066
0067 Examples of using the Linux-provided gdb helpers
0068 ------------------------------------------------
0069
0070 - Load module (and main kernel) symbols::
0071
0072 (gdb) lx-symbols
0073 loading vmlinux
0074 scanning for modules in /home/user/linux/build
0075 loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
0076 loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
0077 loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
0078 loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
0079 loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
0080 ...
0081 loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
0082
0083 - Set a breakpoint on some not yet loaded module function, e.g.::
0084
0085 (gdb) b btrfs_init_sysfs
0086 Function "btrfs_init_sysfs" not defined.
0087 Make breakpoint pending on future shared library load? (y or [n]) y
0088 Breakpoint 1 (btrfs_init_sysfs) pending.
0089
0090 - Continue the target::
0091
0092 (gdb) c
0093
0094 - Load the module on the target and watch the symbols being loaded as well as
0095 the breakpoint hit::
0096
0097 loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
0098 loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
0099 loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
0100 loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
0101
0102 Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
0103 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
0104
0105 - Dump the log buffer of the target kernel::
0106
0107 (gdb) lx-dmesg
0108 [ 0.000000] Initializing cgroup subsys cpuset
0109 [ 0.000000] Initializing cgroup subsys cpu
0110 [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
0111 [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
0112 [ 0.000000] e820: BIOS-provided physical RAM map:
0113 [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
0114 [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
0115 ....
0116
0117 - Examine fields of the current task struct(supported by x86 and arm64 only)::
0118
0119 (gdb) p $lx_current().pid
0120 $1 = 4998
0121 (gdb) p $lx_current().comm
0122 $2 = "modprobe\000\000\000\000\000\000\000"
0123
0124 - Make use of the per-cpu function for the current or a specified CPU::
0125
0126 (gdb) p $lx_per_cpu("runqueues").nr_running
0127 $3 = 1
0128 (gdb) p $lx_per_cpu("runqueues", 2).nr_running
0129 $4 = 0
0130
0131 - Dig into hrtimers using the container_of helper::
0132
0133 (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
0134 (gdb) p *$container_of($next, "struct hrtimer", "node")
0135 $5 = {
0136 node = {
0137 node = {
0138 __rb_parent_color = 18446612133355256072,
0139 rb_right = 0x0 <irq_stack_union>,
0140 rb_left = 0x0 <irq_stack_union>
0141 },
0142 expires = {
0143 tv64 = 1835268000000
0144 }
0145 },
0146 _softexpires = {
0147 tv64 = 1835268000000
0148 },
0149 function = 0xffffffff81078232 <tick_sched_timer>,
0150 base = 0xffff88003fd0d6f0,
0151 state = 1,
0152 start_pid = 0,
0153 start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
0154 start_comm = "swapper/2\000\000\000\000\000\000"
0155 }
0156
0157
0158 List of commands and functions
0159 ------------------------------
0160
0161 The number of commands and convenience functions may evolve over the time,
0162 this is just a snapshot of the initial version::
0163
0164 (gdb) apropos lx
0165 function lx_current -- Return current task
0166 function lx_module -- Find module by name and return the module variable
0167 function lx_per_cpu -- Return per-cpu variable
0168 function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
0169 function lx_thread_info -- Calculate Linux thread_info from task variable
0170 lx-dmesg -- Print Linux kernel log buffer
0171 lx-lsmod -- List currently loaded modules
0172 lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
0173
0174 Detailed help can be obtained via "help <command-name>" for commands and "help
0175 function <function-name>" for convenience functions.
