0001 ======================
0002 Firmware-Assisted Dump
0003 ======================
0004
0005 July 2011
0006
0007 The goal of firmware-assisted dump is to enable the dump of
0008 a crashed system, and to do so from a fully-reset system, and
0009 to minimize the total elapsed time until the system is back
0010 in production use.
0011
0012 - Firmware-Assisted Dump (FADump) infrastructure is intended to replace
0013 the existing phyp assisted dump.
0014 - Fadump uses the same firmware interfaces and memory reservation model
0015 as phyp assisted dump.
0016 - Unlike phyp dump, FADump exports the memory dump through /proc/vmcore
0017 in the ELF format in the same way as kdump. This helps us reuse the
0018 kdump infrastructure for dump capture and filtering.
0019 - Unlike phyp dump, userspace tool does not need to refer any sysfs
0020 interface while reading /proc/vmcore.
0021 - Unlike phyp dump, FADump allows user to release all the memory reserved
0022 for dump, with a single operation of echo 1 > /sys/kernel/fadump_release_mem.
0023 - Once enabled through kernel boot parameter, FADump can be
0024 started/stopped through /sys/kernel/fadump_registered interface (see
0025 sysfs files section below) and can be easily integrated with kdump
0026 service start/stop init scripts.
0027
0028 Comparing with kdump or other strategies, firmware-assisted
0029 dump offers several strong, practical advantages:
0030
0031 - Unlike kdump, the system has been reset, and loaded
0032 with a fresh copy of the kernel. In particular,
0033 PCI and I/O devices have been reinitialized and are
0034 in a clean, consistent state.
0035 - Once the dump is copied out, the memory that held the dump
0036 is immediately available to the running kernel. And therefore,
0037 unlike kdump, FADump doesn't need a 2nd reboot to get back
0038 the system to the production configuration.
0039
0040 The above can only be accomplished by coordination with,
0041 and assistance from the Power firmware. The procedure is
0042 as follows:
0043
0044 - The first kernel registers the sections of memory with the
0045 Power firmware for dump preservation during OS initialization.
0046 These registered sections of memory are reserved by the first
0047 kernel during early boot.
0048
0049 - When system crashes, the Power firmware will copy the registered
0050 low memory regions (boot memory) from source to destination area.
0051 It will also save hardware PTE's.
0052
0053 NOTE:
0054 The term 'boot memory' means size of the low memory chunk
0055 that is required for a kernel to boot successfully when
0056 booted with restricted memory. By default, the boot memory
0057 size will be the larger of 5% of system RAM or 256MB.
0058 Alternatively, user can also specify boot memory size
0059 through boot parameter 'crashkernel=' which will override
0060 the default calculated size. Use this option if default
0061 boot memory size is not sufficient for second kernel to
0062 boot successfully. For syntax of crashkernel= parameter,
0063 refer to Documentation/admin-guide/kdump/kdump.rst. If any
0064 offset is provided in crashkernel= parameter, it will be
0065 ignored as FADump uses a predefined offset to reserve memory
0066 for boot memory dump preservation in case of a crash.
0067
0068 - After the low memory (boot memory) area has been saved, the
0069 firmware will reset PCI and other hardware state. It will
0070 *not* clear the RAM. It will then launch the bootloader, as
0071 normal.
0072
0073 - The freshly booted kernel will notice that there is a new node
0074 (rtas/ibm,kernel-dump on pSeries or ibm,opal/dump/mpipl-boot
0075 on OPAL platform) in the device tree, indicating that
0076 there is crash data available from a previous boot. During
0077 the early boot OS will reserve rest of the memory above
0078 boot memory size effectively booting with restricted memory
0079 size. This will make sure that this kernel (also, referred
0080 to as second kernel or capture kernel) will not touch any
0081 of the dump memory area.
0082
0083 - User-space tools will read /proc/vmcore to obtain the contents
0084 of memory, which holds the previous crashed kernel dump in ELF
0085 format. The userspace tools may copy this info to disk, or
0086 network, nas, san, iscsi, etc. as desired.
0087
0088 - Once the userspace tool is done saving dump, it will echo
0089 '1' to /sys/kernel/fadump_release_mem to release the reserved
0090 memory back to general use, except the memory required for
0091 next firmware-assisted dump registration.
0092
0093 e.g.::
0094
0095 # echo 1 > /sys/kernel/fadump_release_mem
0096
0097 Please note that the firmware-assisted dump feature
0098 is only available on POWER6 and above systems on pSeries
0099 (PowerVM) platform and POWER9 and above systems with OP940
0100 or later firmware versions on PowerNV (OPAL) platform.
0101 Note that, OPAL firmware exports ibm,opal/dump node when
0102 FADump is supported on PowerNV platform.
0103
0104 On OPAL based machines, system first boots into an intermittent
0105 kernel (referred to as petitboot kernel) before booting into the
0106 capture kernel. This kernel would have minimal kernel and/or
0107 userspace support to process crash data. Such kernel needs to
0108 preserve previously crash'ed kernel's memory for the subsequent
0109 capture kernel boot to process this crash data. Kernel config
0110 option CONFIG_PRESERVE_FA_DUMP has to be enabled on such kernel
0111 to ensure that crash data is preserved to process later.
0112
0113 -- On OPAL based machines (PowerNV), if the kernel is build with
0114 CONFIG_OPAL_CORE=y, OPAL memory at the time of crash is also
0115 exported as /sys/firmware/opal/mpipl/core file. This procfs file is
0116 helpful in debugging OPAL crashes with GDB. The kernel memory
0117 used for exporting this procfs file can be released by echo'ing
0118 '1' to /sys/firmware/opal/mpipl/release_core node.
0119
0120 e.g.
0121 # echo 1 > /sys/firmware/opal/mpipl/release_core
0122
0123 Implementation details:
0124 -----------------------
0125
0126 During boot, a check is made to see if firmware supports
0127 this feature on that particular machine. If it does, then
0128 we check to see if an active dump is waiting for us. If yes
0129 then everything but boot memory size of RAM is reserved during
0130 early boot (See Fig. 2). This area is released once we finish
0131 collecting the dump from user land scripts (e.g. kdump scripts)
0132 that are run. If there is dump data, then the
0133 /sys/kernel/fadump_release_mem file is created, and the reserved
0134 memory is held.
0135
0136 If there is no waiting dump data, then only the memory required to
0137 hold CPU state, HPTE region, boot memory dump, FADump header and
0138 elfcore header, is usually reserved at an offset greater than boot
0139 memory size (see Fig. 1). This area is *not* released: this region
0140 will be kept permanently reserved, so that it can act as a receptacle
0141 for a copy of the boot memory content in addition to CPU state and
0142 HPTE region, in the case a crash does occur.
0143
0144 Since this reserved memory area is used only after the system crash,
0145 there is no point in blocking this significant chunk of memory from
0146 production kernel. Hence, the implementation uses the Linux kernel's
0147 Contiguous Memory Allocator (CMA) for memory reservation if CMA is
0148 configured for kernel. With CMA reservation this memory will be
0149 available for applications to use it, while kernel is prevented from
0150 using it. With this FADump will still be able to capture all of the
0151 kernel memory and most of the user space memory except the user pages
0152 that were present in CMA region::
0153
0154 o Memory Reservation during first kernel
0155
0156 Low memory Top of memory
0157 0 boot memory size |<--- Reserved dump area --->| |
0158 | | | Permanent Reservation | |
0159 V V | | V
0160 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
0161 | | |///|////| DUMP | HDR | ELF |////| |
0162 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
0163 | ^ ^ ^ ^ ^
0164 | | | | | |
0165 \ CPU HPTE / | |
0166 ------------------------------ | |
0167 Boot memory content gets transferred | |
0168 to reserved area by firmware at the | |
0169 time of crash. | |
0170 FADump Header |
0171 (meta area) |
0172 |
0173 |
0174 Metadata: This area holds a metadata structure whose
0175 address is registered with f/w and retrieved in the
0176 second kernel after crash, on platforms that support
0177 tags (OPAL). Having such structure with info needed
0178 to process the crashdump eases dump capture process.
0179
0180 Fig. 1
0181
0182
0183 o Memory Reservation during second kernel after crash
0184
0185 Low memory Top of memory
0186 0 boot memory size |
0187 | |<------------ Crash preserved area ------------>|
0188 V V |<--- Reserved dump area --->| |
0189 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
0190 | | |///|////| DUMP | HDR | ELF |////| |
0191 +-----------+-----/ /---+---+----+-------+-----+-----+----+--+
0192 | |
0193 V V
0194 Used by second /proc/vmcore
0195 kernel to boot
0196
0197 +---+
0198 |///| -> Regions (CPU, HPTE & Metadata) marked like this in the above
0199 +---+ figures are not always present. For example, OPAL platform
0200 does not have CPU & HPTE regions while Metadata region is
0201 not supported on pSeries currently.
0202
0203 Fig. 2
0204
0205
0206 Currently the dump will be copied from /proc/vmcore to a new file upon
0207 user intervention. The dump data available through /proc/vmcore will be
0208 in ELF format. Hence the existing kdump infrastructure (kdump scripts)
0209 to save the dump works fine with minor modifications. KDump scripts on
0210 major Distro releases have already been modified to work seamlessly (no
0211 user intervention in saving the dump) when FADump is used, instead of
0212 KDump, as dump mechanism.
0213
0214 The tools to examine the dump will be same as the ones
0215 used for kdump.
0216
0217 How to enable firmware-assisted dump (FADump):
0218 ----------------------------------------------
0219
0220 1. Set config option CONFIG_FA_DUMP=y and build kernel.
0221 2. Boot into linux kernel with 'fadump=on' kernel cmdline option.
0222 By default, FADump reserved memory will be initialized as CMA area.
0223 Alternatively, user can boot linux kernel with 'fadump=nocma' to
0224 prevent FADump to use CMA.
0225 3. Optionally, user can also set 'crashkernel=' kernel cmdline
0226 to specify size of the memory to reserve for boot memory dump
0227 preservation.
0228
0229 NOTE:
0230 1. 'fadump_reserve_mem=' parameter has been deprecated. Instead
0231 use 'crashkernel=' to specify size of the memory to reserve
0232 for boot memory dump preservation.
0233 2. If firmware-assisted dump fails to reserve memory then it
0234 will fallback to existing kdump mechanism if 'crashkernel='
0235 option is set at kernel cmdline.
0236 3. if user wants to capture all of user space memory and ok with
0237 reserved memory not available to production system, then
0238 'fadump=nocma' kernel parameter can be used to fallback to
0239 old behaviour.
0240
0241 Sysfs/debugfs files:
0242 --------------------
0243
0244 Firmware-assisted dump feature uses sysfs file system to hold
0245 the control files and debugfs file to display memory reserved region.
0246
0247 Here is the list of files under kernel sysfs:
0248
0249 /sys/kernel/fadump_enabled
0250 This is used to display the FADump status.
0251
0252 - 0 = FADump is disabled
0253 - 1 = FADump is enabled
0254
0255 This interface can be used by kdump init scripts to identify if
0256 FADump is enabled in the kernel and act accordingly.
0257
0258 /sys/kernel/fadump_registered
0259 This is used to display the FADump registration status as well
0260 as to control (start/stop) the FADump registration.
0261
0262 - 0 = FADump is not registered.
0263 - 1 = FADump is registered and ready to handle system crash.
0264
0265 To register FADump echo 1 > /sys/kernel/fadump_registered and
0266 echo 0 > /sys/kernel/fadump_registered for un-register and stop the
0267 FADump. Once the FADump is un-registered, the system crash will not
0268 be handled and vmcore will not be captured. This interface can be
0269 easily integrated with kdump service start/stop.
0270
0271 /sys/kernel/fadump/mem_reserved
0272
0273 This is used to display the memory reserved by FADump for saving the
0274 crash dump.
0275
0276 /sys/kernel/fadump_release_mem
0277 This file is available only when FADump is active during
0278 second kernel. This is used to release the reserved memory
0279 region that are held for saving crash dump. To release the
0280 reserved memory echo 1 to it::
0281
0282 echo 1 > /sys/kernel/fadump_release_mem
0283
0284 After echo 1, the content of the /sys/kernel/debug/powerpc/fadump_region
0285 file will change to reflect the new memory reservations.
0286
0287 The existing userspace tools (kdump infrastructure) can be easily
0288 enhanced to use this interface to release the memory reserved for
0289 dump and continue without 2nd reboot.
0290
0291 Note: /sys/kernel/fadump_release_opalcore sysfs has moved to
0292 /sys/firmware/opal/mpipl/release_core
0293
0294 /sys/firmware/opal/mpipl/release_core
0295
0296 This file is available only on OPAL based machines when FADump is
0297 active during capture kernel. This is used to release the memory
0298 used by the kernel to export /sys/firmware/opal/mpipl/core file. To
0299 release this memory, echo '1' to it:
0300
0301 echo 1 > /sys/firmware/opal/mpipl/release_core
0302
0303 Note: The following FADump sysfs files are deprecated.
0304
0305 +----------------------------------+--------------------------------+
0306 | Deprecated | Alternative |
0307 +----------------------------------+--------------------------------+
0308 | /sys/kernel/fadump_enabled | /sys/kernel/fadump/enabled |
0309 +----------------------------------+--------------------------------+
0310 | /sys/kernel/fadump_registered | /sys/kernel/fadump/registered |
0311 +----------------------------------+--------------------------------+
0312 | /sys/kernel/fadump_release_mem | /sys/kernel/fadump/release_mem |
0313 +----------------------------------+--------------------------------+
0314
0315 Here is the list of files under powerpc debugfs:
0316 (Assuming debugfs is mounted on /sys/kernel/debug directory.)
0317
0318 /sys/kernel/debug/powerpc/fadump_region
0319 This file shows the reserved memory regions if FADump is
0320 enabled otherwise this file is empty. The output format
0321 is::
0322
0323 <region>: [<start>-<end>] <reserved-size> bytes, Dumped: <dump-size>
0324
0325 and for kernel DUMP region is:
0326
0327 DUMP: Src: <src-addr>, Dest: <dest-addr>, Size: <size>, Dumped: # bytes
0328
0329 e.g.
0330 Contents when FADump is registered during first kernel::
0331
0332 # cat /sys/kernel/debug/powerpc/fadump_region
0333 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x0
0334 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x0
0335 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x0
0336
0337 Contents when FADump is active during second kernel::
0338
0339 # cat /sys/kernel/debug/powerpc/fadump_region
0340 CPU : [0x0000006ffb0000-0x0000006fff001f] 0x40020 bytes, Dumped: 0x40020
0341 HPTE: [0x0000006fff0020-0x0000006fff101f] 0x1000 bytes, Dumped: 0x1000
0342 DUMP: [0x0000006fff1020-0x0000007fff101f] 0x10000000 bytes, Dumped: 0x10000000
0343 : [0x00000010000000-0x0000006ffaffff] 0x5ffb0000 bytes, Dumped: 0x5ffb0000
0344
0345
0346 NOTE:
0347 Please refer to Documentation/filesystems/debugfs.rst on
0348 how to mount the debugfs filesystem.
0349
0350
0351 TODO:
0352 -----
0353 - Need to come up with the better approach to find out more
0354 accurate boot memory size that is required for a kernel to
0355 boot successfully when booted with restricted memory.
0356 - The FADump implementation introduces a FADump crash info structure
0357 in the scratch area before the ELF core header. The idea of introducing
0358 this structure is to pass some important crash info data to the second
0359 kernel which will help second kernel to populate ELF core header with
0360 correct data before it gets exported through /proc/vmcore. The current
0361 design implementation does not address a possibility of introducing
0362 additional fields (in future) to this structure without affecting
0363 compatibility. Need to come up with the better approach to address this.
0364
0365 The possible approaches are:
0366
0367 1. Introduce version field for version tracking, bump up the version
0368 whenever a new field is added to the structure in future. The version
0369 field can be used to find out what fields are valid for the current
0370 version of the structure.
0371 2. Reserve the area of predefined size (say PAGE_SIZE) for this
0372 structure and have unused area as reserved (initialized to zero)
0373 for future field additions.
0374
0375 The advantage of approach 1 over 2 is we don't need to reserve extra space.
0376
0377 Author: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
0378
0379 This document is based on the original documentation written for phyp
0380
0381 assisted dump by Linas Vepstas and Manish Ahuja.
