0001 .. _slub:
0002
0003 ==========================
0004 Short users guide for SLUB
0005 ==========================
0006
0007 The basic philosophy of SLUB is very different from SLAB. SLAB
0008 requires rebuilding the kernel to activate debug options for all
0009 slab caches. SLUB always includes full debugging but it is off by default.
0010 SLUB can enable debugging only for selected slabs in order to avoid
0011 an impact on overall system performance which may make a bug more
0012 difficult to find.
0013
0014 In order to switch debugging on one can add an option ``slub_debug``
0015 to the kernel command line. That will enable full debugging for
0016 all slabs.
0017
0018 Typically one would then use the ``slabinfo`` command to get statistical
0019 data and perform operation on the slabs. By default ``slabinfo`` only lists
0020 slabs that have data in them. See "slabinfo -h" for more options when
0021 running the command. ``slabinfo`` can be compiled with
0022 ::
0023
0024 gcc -o slabinfo tools/vm/slabinfo.c
0025
0026 Some of the modes of operation of ``slabinfo`` require that slub debugging
0027 be enabled on the command line. F.e. no tracking information will be
0028 available without debugging on and validation can only partially
0029 be performed if debugging was not switched on.
0030
0031 Some more sophisticated uses of slub_debug:
0032 -------------------------------------------
0033
0034 Parameters may be given to ``slub_debug``. If none is specified then full
0035 debugging is enabled. Format:
0036
0037 slub_debug=<Debug-Options>
0038 Enable options for all slabs
0039
0040 slub_debug=<Debug-Options>,<slab name1>,<slab name2>,...
0041 Enable options only for select slabs (no spaces
0042 after a comma)
0043
0044 Multiple blocks of options for all slabs or selected slabs can be given, with
0045 blocks of options delimited by ';'. The last of "all slabs" blocks is applied
0046 to all slabs except those that match one of the "select slabs" block. Options
0047 of the first "select slabs" blocks that matches the slab's name are applied.
0048
0049 Possible debug options are::
0050
0051 F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS
0052 Sorry SLAB legacy issues)
0053 Z Red zoning
0054 P Poisoning (object and padding)
0055 U User tracking (free and alloc)
0056 T Trace (please only use on single slabs)
0057 A Enable failslab filter mark for the cache
0058 O Switch debugging off for caches that would have
0059 caused higher minimum slab orders
0060 - Switch all debugging off (useful if the kernel is
0061 configured with CONFIG_SLUB_DEBUG_ON)
0062
0063 F.e. in order to boot just with sanity checks and red zoning one would specify::
0064
0065 slub_debug=FZ
0066
0067 Trying to find an issue in the dentry cache? Try::
0068
0069 slub_debug=,dentry
0070
0071 to only enable debugging on the dentry cache. You may use an asterisk at the
0072 end of the slab name, in order to cover all slabs with the same prefix. For
0073 example, here's how you can poison the dentry cache as well as all kmalloc
0074 slabs::
0075
0076 slub_debug=P,kmalloc-*,dentry
0077
0078 Red zoning and tracking may realign the slab. We can just apply sanity checks
0079 to the dentry cache with::
0080
0081 slub_debug=F,dentry
0082
0083 Debugging options may require the minimum possible slab order to increase as
0084 a result of storing the metadata (for example, caches with PAGE_SIZE object
0085 sizes). This has a higher liklihood of resulting in slab allocation errors
0086 in low memory situations or if there's high fragmentation of memory. To
0087 switch off debugging for such caches by default, use::
0088
0089 slub_debug=O
0090
0091 You can apply different options to different list of slab names, using blocks
0092 of options. This will enable red zoning for dentry and user tracking for
0093 kmalloc. All other slabs will not get any debugging enabled::
0094
0095 slub_debug=Z,dentry;U,kmalloc-*
0096
0097 You can also enable options (e.g. sanity checks and poisoning) for all caches
0098 except some that are deemed too performance critical and don't need to be
0099 debugged by specifying global debug options followed by a list of slab names
0100 with "-" as options::
0101
0102 slub_debug=FZ;-,zs_handle,zspage
0103
0104 The state of each debug option for a slab can be found in the respective files
0105 under::
0106
0107 /sys/kernel/slab/<slab name>/
0108
0109 If the file contains 1, the option is enabled, 0 means disabled. The debug
0110 options from the ``slub_debug`` parameter translate to the following files::
0111
0112 F sanity_checks
0113 Z red_zone
0114 P poison
0115 U store_user
0116 T trace
0117 A failslab
0118
0119 Careful with tracing: It may spew out lots of information and never stop if
0120 used on the wrong slab.
0121
0122 Slab merging
0123 ============
0124
0125 If no debug options are specified then SLUB may merge similar slabs together
0126 in order to reduce overhead and increase cache hotness of objects.
0127 ``slabinfo -a`` displays which slabs were merged together.
0128
0129 Slab validation
0130 ===============
0131
0132 SLUB can validate all object if the kernel was booted with slub_debug. In
0133 order to do so you must have the ``slabinfo`` tool. Then you can do
0134 ::
0135
0136 slabinfo -v
0137
0138 which will test all objects. Output will be generated to the syslog.
0139
0140 This also works in a more limited way if boot was without slab debug.
0141 In that case ``slabinfo -v`` simply tests all reachable objects. Usually
0142 these are in the cpu slabs and the partial slabs. Full slabs are not
0143 tracked by SLUB in a non debug situation.
0144
0145 Getting more performance
0146 ========================
0147
0148 To some degree SLUB's performance is limited by the need to take the
0149 list_lock once in a while to deal with partial slabs. That overhead is
0150 governed by the order of the allocation for each slab. The allocations
0151 can be influenced by kernel parameters:
0152
0153 .. slub_min_objects=x (default 4)
0154 .. slub_min_order=x (default 0)
0155 .. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER))
0156
0157 ``slub_min_objects``
0158 allows to specify how many objects must at least fit into one
0159 slab in order for the allocation order to be acceptable. In
0160 general slub will be able to perform this number of
0161 allocations on a slab without consulting centralized resources
0162 (list_lock) where contention may occur.
0163
0164 ``slub_min_order``
0165 specifies a minimum order of slabs. A similar effect like
0166 ``slub_min_objects``.
0167
0168 ``slub_max_order``
0169 specified the order at which ``slub_min_objects`` should no
0170 longer be checked. This is useful to avoid SLUB trying to
0171 generate super large order pages to fit ``slub_min_objects``
0172 of a slab cache with large object sizes into one high order
0173 page. Setting command line parameter
0174 ``debug_guardpage_minorder=N`` (N > 0), forces setting
0175 ``slub_max_order`` to 0, what cause minimum possible order of
0176 slabs allocation.
0177
0178 SLUB Debug output
0179 =================
0180
0181 Here is a sample of slub debug output::
0182
0183 ====================================================================
0184 BUG kmalloc-8: Right Redzone overwritten
0185 --------------------------------------------------------------------
0186
0187 INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc
0188 INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58
0189 INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58
0190 INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554
0191
0192 Bytes b4 (0xc90f6d10): 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
0193 Object (0xc90f6d20): 31 30 31 39 2e 30 30 35 1019.005
0194 Redzone (0xc90f6d28): 00 cc cc cc .
0195 Padding (0xc90f6d50): 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
0196
0197 [<c010523d>] dump_trace+0x63/0x1eb
0198 [<c01053df>] show_trace_log_lvl+0x1a/0x2f
0199 [<c010601d>] show_trace+0x12/0x14
0200 [<c0106035>] dump_stack+0x16/0x18
0201 [<c017e0fa>] object_err+0x143/0x14b
0202 [<c017e2cc>] check_object+0x66/0x234
0203 [<c017eb43>] __slab_free+0x239/0x384
0204 [<c017f446>] kfree+0xa6/0xc6
0205 [<c02e2335>] get_modalias+0xb9/0xf5
0206 [<c02e23b7>] dmi_dev_uevent+0x27/0x3c
0207 [<c027866a>] dev_uevent+0x1ad/0x1da
0208 [<c0205024>] kobject_uevent_env+0x20a/0x45b
0209 [<c020527f>] kobject_uevent+0xa/0xf
0210 [<c02779f1>] store_uevent+0x4f/0x58
0211 [<c027758e>] dev_attr_store+0x29/0x2f
0212 [<c01bec4f>] sysfs_write_file+0x16e/0x19c
0213 [<c0183ba7>] vfs_write+0xd1/0x15a
0214 [<c01841d7>] sys_write+0x3d/0x72
0215 [<c0104112>] sysenter_past_esp+0x5f/0x99
0216 [<b7f7b410>] 0xb7f7b410
0217 =======================
0218
0219 FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc
0220
0221 If SLUB encounters a corrupted object (full detection requires the kernel
0222 to be booted with slub_debug) then the following output will be dumped
0223 into the syslog:
0224
0225 1. Description of the problem encountered
0226
0227 This will be a message in the system log starting with::
0228
0229 ===============================================
0230 BUG <slab cache affected>: <What went wrong>
0231 -----------------------------------------------
0232
0233 INFO: <corruption start>-<corruption_end> <more info>
0234 INFO: Slab <address> <slab information>
0235 INFO: Object <address> <object information>
0236 INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by
0237 cpu> pid=<pid of the process>
0238 INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu>
0239 pid=<pid of the process>
0240
0241 (Object allocation / free information is only available if SLAB_STORE_USER is
0242 set for the slab. slub_debug sets that option)
0243
0244 2. The object contents if an object was involved.
0245
0246 Various types of lines can follow the BUG SLUB line:
0247
0248 Bytes b4 <address> : <bytes>
0249 Shows a few bytes before the object where the problem was detected.
0250 Can be useful if the corruption does not stop with the start of the
0251 object.
0252
0253 Object <address> : <bytes>
0254 The bytes of the object. If the object is inactive then the bytes
0255 typically contain poison values. Any non-poison value shows a
0256 corruption by a write after free.
0257
0258 Redzone <address> : <bytes>
0259 The Redzone following the object. The Redzone is used to detect
0260 writes after the object. All bytes should always have the same
0261 value. If there is any deviation then it is due to a write after
0262 the object boundary.
0263
0264 (Redzone information is only available if SLAB_RED_ZONE is set.
0265 slub_debug sets that option)
0266
0267 Padding <address> : <bytes>
0268 Unused data to fill up the space in order to get the next object
0269 properly aligned. In the debug case we make sure that there are
0270 at least 4 bytes of padding. This allows the detection of writes
0271 before the object.
0272
0273 3. A stackdump
0274
0275 The stackdump describes the location where the error was detected. The cause
0276 of the corruption is may be more likely found by looking at the function that
0277 allocated or freed the object.
0278
0279 4. Report on how the problem was dealt with in order to ensure the continued
0280 operation of the system.
0281
0282 These are messages in the system log beginning with::
0283
0284 FIX <slab cache affected>: <corrective action taken>
0285
0286 In the above sample SLUB found that the Redzone of an active object has
0287 been overwritten. Here a string of 8 characters was written into a slab that
0288 has the length of 8 characters. However, a 8 character string needs a
0289 terminating 0. That zero has overwritten the first byte of the Redzone field.
0290 After reporting the details of the issue encountered the FIX SLUB message
0291 tells us that SLUB has restored the Redzone to its proper value and then
0292 system operations continue.
0293
0294 Emergency operations
0295 ====================
0296
0297 Minimal debugging (sanity checks alone) can be enabled by booting with::
0298
0299 slub_debug=F
0300
0301 This will be generally be enough to enable the resiliency features of slub
0302 which will keep the system running even if a bad kernel component will
0303 keep corrupting objects. This may be important for production systems.
0304 Performance will be impacted by the sanity checks and there will be a
0305 continual stream of error messages to the syslog but no additional memory
0306 will be used (unlike full debugging).
0307
0308 No guarantees. The kernel component still needs to be fixed. Performance
0309 may be optimized further by locating the slab that experiences corruption
0310 and enabling debugging only for that cache
0311
0312 I.e.::
0313
0314 slub_debug=F,dentry
0315
0316 If the corruption occurs by writing after the end of the object then it
0317 may be advisable to enable a Redzone to avoid corrupting the beginning
0318 of other objects::
0319
0320 slub_debug=FZ,dentry
0321
0322 Extended slabinfo mode and plotting
0323 ===================================
0324
0325 The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes:
0326 - Slabcache Totals
0327 - Slabs sorted by size (up to -N <num> slabs, default 1)
0328 - Slabs sorted by loss (up to -N <num> slabs, default 1)
0329
0330 Additionally, in this mode ``slabinfo`` does not dynamically scale
0331 sizes (G/M/K) and reports everything in bytes (this functionality is
0332 also available to other slabinfo modes via '-B' option) which makes
0333 reporting more precise and accurate. Moreover, in some sense the `-X'
0334 mode also simplifies the analysis of slabs' behaviour, because its
0335 output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it
0336 pushes the analysis from looking through the numbers (tons of numbers)
0337 to something easier -- visual analysis.
0338
0339 To generate plots:
0340
0341 a) collect slabinfo extended records, for example::
0342
0343 while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done
0344
0345 b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script::
0346
0347 slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN]
0348
0349 The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records
0350 and generates 3 png files (and 3 pre-processing cache files) per STATS
0351 file:
0352 - Slabcache Totals: FOO_STATS-totals.png
0353 - Slabs sorted by size: FOO_STATS-slabs-by-size.png
0354 - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png
0355
0356 Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you
0357 need to compare slabs' behaviour "prior to" and "after" some code
0358 modification. To help you out there, ``slabinfo-gnuplot.sh`` script
0359 can 'merge' the `Slabcache Totals` sections from different
0360 measurements. To visually compare N plots:
0361
0362 a) Collect as many STATS1, STATS2, .. STATSN files as you need::
0363
0364 while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done
0365
0366 b) Pre-process those STATS files::
0367
0368 slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN
0369
0370 c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the
0371 generated pre-processed \*-totals::
0372
0373 slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals
0374
0375 This will produce a single plot (png file).
0376
0377 Plots, expectedly, can be large so some fluctuations or small spikes
0378 can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two
0379 options to 'zoom-in'/'zoom-out':
0380
0381 a) ``-s %d,%d`` -- overwrites the default image width and height
0382 b) ``-r %d,%d`` -- specifies a range of samples to use (for example,
0383 in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r
0384 40,60`` range will plot only samples collected between 40th and
0385 60th seconds).
0386
0387
0388 DebugFS files for SLUB
0389 ======================
0390
0391 For more information about current state of SLUB caches with the user tracking
0392 debug option enabled, debugfs files are available, typically under
0393 /sys/kernel/debug/slab/<cache>/ (created only for caches with enabled user
0394 tracking). There are 2 types of these files with the following debug
0395 information:
0396
0397 1. alloc_traces::
0398
0399 Prints information about unique allocation traces of the currently
0400 allocated objects. The output is sorted by frequency of each trace.
0401
0402 Information in the output:
0403 Number of objects, allocating function, minimal/average/maximal jiffies since alloc,
0404 pid range of the allocating processes, cpu mask of allocating cpus, and stack trace.
0405
0406 Example:::
0407
0408 1085 populate_error_injection_list+0x97/0x110 age=166678/166680/166682 pid=1 cpus=1::
0409 __slab_alloc+0x6d/0x90
0410 kmem_cache_alloc_trace+0x2eb/0x300
0411 populate_error_injection_list+0x97/0x110
0412 init_error_injection+0x1b/0x71
0413 do_one_initcall+0x5f/0x2d0
0414 kernel_init_freeable+0x26f/0x2d7
0415 kernel_init+0xe/0x118
0416 ret_from_fork+0x22/0x30
0417
0418
0419 2. free_traces::
0420
0421 Prints information about unique freeing traces of the currently allocated
0422 objects. The freeing traces thus come from the previous life-cycle of the
0423 objects and are reported as not available for objects allocated for the first
0424 time. The output is sorted by frequency of each trace.
0425
0426 Information in the output:
0427 Number of objects, freeing function, minimal/average/maximal jiffies since free,
0428 pid range of the freeing processes, cpu mask of freeing cpus, and stack trace.
0429
0430 Example:::
0431
0432 1980 <not-available> age=4294912290 pid=0 cpus=0
0433 51 acpi_ut_update_ref_count+0x6a6/0x782 age=236886/237027/237772 pid=1 cpus=1
0434 kfree+0x2db/0x420
0435 acpi_ut_update_ref_count+0x6a6/0x782
0436 acpi_ut_update_object_reference+0x1ad/0x234
0437 acpi_ut_remove_reference+0x7d/0x84
0438 acpi_rs_get_prt_method_data+0x97/0xd6
0439 acpi_get_irq_routing_table+0x82/0xc4
0440 acpi_pci_irq_find_prt_entry+0x8e/0x2e0
0441 acpi_pci_irq_lookup+0x3a/0x1e0
0442 acpi_pci_irq_enable+0x77/0x240
0443 pcibios_enable_device+0x39/0x40
0444 do_pci_enable_device.part.0+0x5d/0xe0
0445 pci_enable_device_flags+0xfc/0x120
0446 pci_enable_device+0x13/0x20
0447 virtio_pci_probe+0x9e/0x170
0448 local_pci_probe+0x48/0x80
0449 pci_device_probe+0x105/0x1c0
0450
0451 Christoph Lameter, May 30, 2007
0452 Sergey Senozhatsky, October 23, 2015
