0001 .. include:: <isonum.txt>
0002
0003 ============================================
0004 Reliability, Availability and Serviceability
0005 ============================================
0006
0007 RAS concepts
0008 ************
0009
0010 Reliability, Availability and Serviceability (RAS) is a concept used on
0011 servers meant to measure their robustness.
0012
0013 Reliability
0014 is the probability that a system will produce correct outputs.
0015
0016 * Generally measured as Mean Time Between Failures (MTBF)
0017 * Enhanced by features that help to avoid, detect and repair hardware faults
0018
0019 Availability
0020 is the probability that a system is operational at a given time
0021
0022 * Generally measured as a percentage of downtime per a period of time
0023 * Often uses mechanisms to detect and correct hardware faults in
0024 runtime;
0025
0026 Serviceability (or maintainability)
0027 is the simplicity and speed with which a system can be repaired or
0028 maintained
0029
0030 * Generally measured on Mean Time Between Repair (MTBR)
0031
0032 Improving RAS
0033 -------------
0034
0035 In order to reduce systems downtime, a system should be capable of detecting
0036 hardware errors, and, when possible correcting them in runtime. It should
0037 also provide mechanisms to detect hardware degradation, in order to warn
0038 the system administrator to take the action of replacing a component before
0039 it causes data loss or system downtime.
0040
0041 Among the monitoring measures, the most usual ones include:
0042
0043 * CPU – detect errors at instruction execution and at L1/L2/L3 caches;
0044 * Memory – add error correction logic (ECC) to detect and correct errors;
0045 * I/O – add CRC checksums for transferred data;
0046 * Storage – RAID, journal file systems, checksums,
0047 Self-Monitoring, Analysis and Reporting Technology (SMART).
0048
0049 By monitoring the number of occurrences of error detections, it is possible
0050 to identify if the probability of hardware errors is increasing, and, on such
0051 case, do a preventive maintenance to replace a degraded component while
0052 those errors are correctable.
0053
0054 Types of errors
0055 ---------------
0056
0057 Most mechanisms used on modern systems use technologies like Hamming
0058 Codes that allow error correction when the number of errors on a bit packet
0059 is below a threshold. If the number of errors is above, those mechanisms
0060 can indicate with a high degree of confidence that an error happened, but
0061 they can't correct.
0062
0063 Also, sometimes an error occur on a component that it is not used. For
0064 example, a part of the memory that it is not currently allocated.
0065
0066 That defines some categories of errors:
0067
0068 * **Correctable Error (CE)** - the error detection mechanism detected and
0069 corrected the error. Such errors are usually not fatal, although some
0070 Kernel mechanisms allow the system administrator to consider them as fatal.
0071
0072 * **Uncorrected Error (UE)** - the amount of errors happened above the error
0073 correction threshold, and the system was unable to auto-correct.
0074
0075 * **Fatal Error** - when an UE error happens on a critical component of the
0076 system (for example, a piece of the Kernel got corrupted by an UE), the
0077 only reliable way to avoid data corruption is to hang or reboot the machine.
0078
0079 * **Non-fatal Error** - when an UE error happens on an unused component,
0080 like a CPU in power down state or an unused memory bank, the system may
0081 still run, eventually replacing the affected hardware by a hot spare,
0082 if available.
0083
0084 Also, when an error happens on a userspace process, it is also possible to
0085 kill such process and let userspace restart it.
0086
0087 The mechanism for handling non-fatal errors is usually complex and may
0088 require the help of some userspace application, in order to apply the
0089 policy desired by the system administrator.
0090
0091 Identifying a bad hardware component
0092 ------------------------------------
0093
0094 Just detecting a hardware flaw is usually not enough, as the system needs
0095 to pinpoint to the minimal replaceable unit (MRU) that should be exchanged
0096 to make the hardware reliable again.
0097
0098 So, it requires not only error logging facilities, but also mechanisms that
0099 will translate the error message to the silkscreen or component label for
0100 the MRU.
0101
0102 Typically, it is very complex for memory, as modern CPUs interlace memory
0103 from different memory modules, in order to provide a better performance. The
0104 DMI BIOS usually have a list of memory module labels, with can be obtained
0105 using the ``dmidecode`` tool. For example, on a desktop machine, it shows::
0106
0107 Memory Device
0108 Total Width: 64 bits
0109 Data Width: 64 bits
0110 Size: 16384 MB
0111 Form Factor: SODIMM
0112 Set: None
0113 Locator: ChannelA-DIMM0
0114 Bank Locator: BANK 0
0115 Type: DDR4
0116 Type Detail: Synchronous
0117 Speed: 2133 MHz
0118 Rank: 2
0119 Configured Clock Speed: 2133 MHz
0120
0121 On the above example, a DDR4 SO-DIMM memory module is located at the
0122 system's memory labeled as "BANK 0", as given by the *bank locator* field.
0123 Please notice that, on such system, the *total width* is equal to the
0124 *data width*. It means that such memory module doesn't have error
0125 detection/correction mechanisms.
0126
0127 Unfortunately, not all systems use the same field to specify the memory
0128 bank. On this example, from an older server, ``dmidecode`` shows::
0129
0130 Memory Device
0131 Array Handle: 0x1000
0132 Error Information Handle: Not Provided
0133 Total Width: 72 bits
0134 Data Width: 64 bits
0135 Size: 8192 MB
0136 Form Factor: DIMM
0137 Set: 1
0138 Locator: DIMM_A1
0139 Bank Locator: Not Specified
0140 Type: DDR3
0141 Type Detail: Synchronous Registered (Buffered)
0142 Speed: 1600 MHz
0143 Rank: 2
0144 Configured Clock Speed: 1600 MHz
0145
0146 There, the DDR3 RDIMM memory module is located at the system's memory labeled
0147 as "DIMM_A1", as given by the *locator* field. Please notice that this
0148 memory module has 64 bits of *data width* and 72 bits of *total width*. So,
0149 it has 8 extra bits to be used by error detection and correction mechanisms.
0150 Such kind of memory is called Error-correcting code memory (ECC memory).
0151
0152 To make things even worse, it is not uncommon that systems with different
0153 labels on their system's board to use exactly the same BIOS, meaning that
0154 the labels provided by the BIOS won't match the real ones.
0155
0156 ECC memory
0157 ----------
0158
0159 As mentioned in the previous section, ECC memory has extra bits to be
0160 used for error correction. In the above example, a memory module has
0161 64 bits of *data width*, and 72 bits of *total width*. The extra 8
0162 bits which are used for the error detection and correction mechanisms
0163 are referred to as the *syndrome*\ [#f1]_\ [#f2]_.
0164
0165 So, when the cpu requests the memory controller to write a word with
0166 *data width*, the memory controller calculates the *syndrome* in real time,
0167 using Hamming code, or some other error correction code, like SECDED+,
0168 producing a code with *total width* size. Such code is then written
0169 on the memory modules.
0170
0171 At read, the *total width* bits code is converted back, using the same
0172 ECC code used on write, producing a word with *data width* and a *syndrome*.
0173 The word with *data width* is sent to the CPU, even when errors happen.
0174
0175 The memory controller also looks at the *syndrome* in order to check if
0176 there was an error, and if the ECC code was able to fix such error.
0177 If the error was corrected, a Corrected Error (CE) happened. If not, an
0178 Uncorrected Error (UE) happened.
0179
0180 The information about the CE/UE errors is stored on some special registers
0181 at the memory controller and can be accessed by reading such registers,
0182 either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64
0183 bit CPUs, such errors can also be retrieved via the Machine Check
0184 Architecture (MCA)\ [#f3]_.
0185
0186 .. [#f1] Please notice that several memory controllers allow operation on a
0187 mode called "Lock-Step", where it groups two memory modules together,
0188 doing 128-bit reads/writes. That gives 16 bits for error correction, with
0189 significantly improves the error correction mechanism, at the expense
0190 that, when an error happens, there's no way to know what memory module is
0191 to blame. So, it has to blame both memory modules.
0192
0193 .. [#f2] Some memory controllers also allow using memory in mirror mode.
0194 On such mode, the same data is written to two memory modules. At read,
0195 the system checks both memory modules, in order to check if both provide
0196 identical data. On such configuration, when an error happens, there's no
0197 way to know what memory module is to blame. So, it has to blame both
0198 memory modules (or 4 memory modules, if the system is also on Lock-step
0199 mode).
0200
0201 .. [#f3] For more details about the Machine Check Architecture (MCA),
0202 please read Documentation/x86/x86_64/machinecheck.rst at the Kernel tree.
0203
0204 EDAC - Error Detection And Correction
0205 *************************************
0206
0207 .. note::
0208
0209 "bluesmoke" was the name for this device driver subsystem when it
0210 was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net.
0211 That site is mostly archaic now and can be used only for historical
0212 purposes.
0213
0214 When the subsystem was pushed upstream for the first time, on
0215 Kernel 2.6.16, it was renamed to ``EDAC``.
0216
0217 Purpose
0218 -------
0219
0220 The ``edac`` kernel module's goal is to detect and report hardware errors
0221 that occur within the computer system running under linux.
0222
0223 Memory
0224 ------
0225
0226 Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
0227 primary errors being harvested. These types of errors are harvested by
0228 the ``edac_mc`` device.
0229
0230 Detecting CE events, then harvesting those events and reporting them,
0231 **can** but must not necessarily be a predictor of future UE events. With
0232 CE events only, the system can and will continue to operate as no data
0233 has been damaged yet.
0234
0235 However, preventive maintenance and proactive part replacement of memory
0236 modules exhibiting CEs can reduce the likelihood of the dreaded UE events
0237 and system panics.
0238
0239 Other hardware elements
0240 -----------------------
0241
0242 A new feature for EDAC, the ``edac_device`` class of device, was added in
0243 the 2.6.23 version of the kernel.
0244
0245 This new device type allows for non-memory type of ECC hardware detectors
0246 to have their states harvested and presented to userspace via the sysfs
0247 interface.
0248
0249 Some architectures have ECC detectors for L1, L2 and L3 caches,
0250 along with DMA engines, fabric switches, main data path switches,
0251 interconnections, and various other hardware data paths. If the hardware
0252 reports it, then a edac_device device probably can be constructed to
0253 harvest and present that to userspace.
0254
0255
0256 PCI bus scanning
0257 ----------------
0258
0259 In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
0260 in order to determine if errors are occurring during data transfers.
0261
0262 The presence of PCI Parity errors must be examined with a grain of salt.
0263 There are several add-in adapters that do **not** follow the PCI specification
0264 with regards to Parity generation and reporting. The specification says
0265 the vendor should tie the parity status bits to 0 if they do not intend
0266 to generate parity. Some vendors do not do this, and thus the parity bit
0267 can "float" giving false positives.
0268
0269 There is a PCI device attribute located in sysfs that is checked by
0270 the EDAC PCI scanning code. If that attribute is set, PCI parity/error
0271 scanning is skipped for that device. The attribute is::
0272
0273 broken_parity_status
0274
0275 and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for
0276 PCI devices.
0277
0278
0279 Versioning
0280 ----------
0281
0282 EDAC is composed of a "core" module (``edac_core.ko``) and several Memory
0283 Controller (MC) driver modules. On a given system, the CORE is loaded
0284 and one MC driver will be loaded. Both the CORE and the MC driver (or
0285 ``edac_device`` driver) have individual versions that reflect current
0286 release level of their respective modules.
0287
0288 Thus, to "report" on what version a system is running, one must report
0289 both the CORE's and the MC driver's versions.
0290
0291
0292 Loading
0293 -------
0294
0295 If ``edac`` was statically linked with the kernel then no loading
0296 is necessary. If ``edac`` was built as modules then simply modprobe
0297 the ``edac`` pieces that you need. You should be able to modprobe
0298 hardware-specific modules and have the dependencies load the necessary
0299 core modules.
0300
0301 Example::
0302
0303 $ modprobe amd76x_edac
0304
0305 loads both the ``amd76x_edac.ko`` memory controller module and the
0306 ``edac_mc.ko`` core module.
0307
0308
0309 Sysfs interface
0310 ---------------
0311
0312 EDAC presents a ``sysfs`` interface for control and reporting purposes. It
0313 lives in the /sys/devices/system/edac directory.
0314
0315 Within this directory there currently reside 2 components:
0316
0317 ======= ==============================
0318 mc memory controller(s) system
0319 pci PCI control and status system
0320 ======= ==============================
0321
0322
0323
0324 Memory Controller (mc) Model
0325 ----------------------------
0326
0327 Each ``mc`` device controls a set of memory modules [#f4]_. These modules
0328 are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``).
0329 There can be multiple csrows and multiple channels.
0330
0331 .. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely
0332 used to refer to a memory module, although there are other memory
0333 packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI
0334 specification (Version 2.7) defines a memory module in the Common
0335 Platform Error Record (CPER) section to be an SMBIOS Memory Device
0336 (Type 17). Along this document, and inside the EDAC subsystem, the term
0337 "dimm" is used for all memory modules, even when they use a
0338 different kind of packaging.
0339
0340 Memory controllers allow for several csrows, with 8 csrows being a
0341 typical value. Yet, the actual number of csrows depends on the layout of
0342 a given motherboard, memory controller and memory module characteristics.
0343
0344 Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems)
0345 data transfers to/from the CPU from/to memory. Some newer chipsets allow
0346 for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory
0347 controllers. The following example will assume 2 channels:
0348
0349 +------------+-----------------------+
0350 | CS Rows | Channels |
0351 +------------+-----------+-----------+
0352 | | ``ch0`` | ``ch1`` |
0353 +============+===========+===========+
0354 | |**DIMM_A0**|**DIMM_B0**|
0355 +------------+-----------+-----------+
0356 | ``csrow0`` | rank0 | rank0 |
0357 +------------+-----------+-----------+
0358 | ``csrow1`` | rank1 | rank1 |
0359 +------------+-----------+-----------+
0360 | |**DIMM_A1**|**DIMM_B1**|
0361 +------------+-----------+-----------+
0362 | ``csrow2`` | rank0 | rank0 |
0363 +------------+-----------+-----------+
0364 | ``csrow3`` | rank1 | rank1 |
0365 +------------+-----------+-----------+
0366
0367 In the above example, there are 4 physical slots on the motherboard
0368 for memory DIMMs:
0369
0370 +---------+---------+
0371 | DIMM_A0 | DIMM_B0 |
0372 +---------+---------+
0373 | DIMM_A1 | DIMM_B1 |
0374 +---------+---------+
0375
0376 Labels for these slots are usually silk-screened on the motherboard.
0377 Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are
0378 channel 1. Notice that there are two csrows possible on a physical DIMM.
0379 These csrows are allocated their csrow assignment based on the slot into
0380 which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
0381 Channel, the csrows cross both DIMMs.
0382
0383 Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
0384 In the example above 2 dual ranked DIMMs are similarly placed. Thus,
0385 both csrow0 and csrow1 are populated. On the other hand, when 2 single
0386 ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will
0387 have just one csrow (csrow0) and csrow1 will be empty. The pattern
0388 repeats itself for csrow2 and csrow3. Also note that some memory
0389 controllers don't have any logic to identify the memory module, see
0390 ``rankX`` directories below.
0391
0392 The representation of the above is reflected in the directory
0393 tree in EDAC's sysfs interface. Starting in directory
0394 ``/sys/devices/system/edac/mc``, each memory controller will be
0395 represented by its own ``mcX`` directory, where ``X`` is the
0396 index of the MC::
0397
0398 ..../edac/mc/
0399 |
0400 |->mc0
0401 |->mc1
0402 |->mc2
0403 ....
0404
0405 Under each ``mcX`` directory each ``csrowX`` is again represented by a
0406 ``csrowX``, where ``X`` is the csrow index::
0407
0408 .../mc/mc0/
0409 |
0410 |->csrow0
0411 |->csrow2
0412 |->csrow3
0413 ....
0414
0415 Notice that there is no csrow1, which indicates that csrow0 is composed
0416 of a single ranked DIMMs. This should also apply in both Channels, in
0417 order to have dual-channel mode be operational. Since both csrow2 and
0418 csrow3 are populated, this indicates a dual ranked set of DIMMs for
0419 channels 0 and 1.
0420
0421 Within each of the ``mcX`` and ``csrowX`` directories are several EDAC
0422 control and attribute files.
0423
0424 ``mcX`` directories
0425 -------------------
0426
0427 In ``mcX`` directories are EDAC control and attribute files for
0428 this ``X`` instance of the memory controllers.
0429
0430 For a description of the sysfs API, please see:
0431
0432 Documentation/ABI/testing/sysfs-devices-edac
0433
0434
0435 ``dimmX`` or ``rankX`` directories
0436 ----------------------------------
0437
0438 The recommended way to use the EDAC subsystem is to look at the information
0439 provided by the ``dimmX`` or ``rankX`` directories [#f5]_.
0440
0441 A typical EDAC system has the following structure under
0442 ``/sys/devices/system/edac/``\ [#f6]_::
0443
0444 /sys/devices/system/edac/
0445 ├── mc
0446 │ ├── mc0
0447 │ │ ├── ce_count
0448 │ │ ├── ce_noinfo_count
0449 │ │ ├── dimm0
0450 │ │ │ ├── dimm_ce_count
0451 │ │ │ ├── dimm_dev_type
0452 │ │ │ ├── dimm_edac_mode
0453 │ │ │ ├── dimm_label
0454 │ │ │ ├── dimm_location
0455 │ │ │ ├── dimm_mem_type
0456 │ │ │ ├── dimm_ue_count
0457 │ │ │ ├── size
0458 │ │ │ └── uevent
0459 │ │ ├── max_location
0460 │ │ ├── mc_name
0461 │ │ ├── reset_counters
0462 │ │ ├── seconds_since_reset
0463 │ │ ├── size_mb
0464 │ │ ├── ue_count
0465 │ │ ├── ue_noinfo_count
0466 │ │ └── uevent
0467 │ ├── mc1
0468 │ │ ├── ce_count
0469 │ │ ├── ce_noinfo_count
0470 │ │ ├── dimm0
0471 │ │ │ ├── dimm_ce_count
0472 │ │ │ ├── dimm_dev_type
0473 │ │ │ ├── dimm_edac_mode
0474 │ │ │ ├── dimm_label
0475 │ │ │ ├── dimm_location
0476 │ │ │ ├── dimm_mem_type
0477 │ │ │ ├── dimm_ue_count
0478 │ │ │ ├── size
0479 │ │ │ └── uevent
0480 │ │ ├── max_location
0481 │ │ ├── mc_name
0482 │ │ ├── reset_counters
0483 │ │ ├── seconds_since_reset
0484 │ │ ├── size_mb
0485 │ │ ├── ue_count
0486 │ │ ├── ue_noinfo_count
0487 │ │ └── uevent
0488 │ └── uevent
0489 └── uevent
0490
0491 In the ``dimmX`` directories are EDAC control and attribute files for
0492 this ``X`` memory module:
0493
0494 - ``size`` - Total memory managed by this csrow attribute file
0495
0496 This attribute file displays, in count of megabytes, the memory
0497 that this csrow contains.
0498
0499 - ``dimm_ue_count`` - Uncorrectable Errors count attribute file
0500
0501 This attribute file displays the total count of uncorrectable
0502 errors that have occurred on this DIMM. If panic_on_ue is set
0503 this counter will not have a chance to increment, since EDAC
0504 will panic the system.
0505
0506 - ``dimm_ce_count`` - Correctable Errors count attribute file
0507
0508 This attribute file displays the total count of correctable
0509 errors that have occurred on this DIMM. This count is very
0510 important to examine. CEs provide early indications that a
0511 DIMM is beginning to fail. This count field should be
0512 monitored for non-zero values and report such information
0513 to the system administrator.
0514
0515 - ``dimm_dev_type`` - Device type attribute file
0516
0517 This attribute file will display what type of DRAM device is
0518 being utilized on this DIMM.
0519 Examples:
0520
0521 - x1
0522 - x2
0523 - x4
0524 - x8
0525
0526 - ``dimm_edac_mode`` - EDAC Mode of operation attribute file
0527
0528 This attribute file will display what type of Error detection
0529 and correction is being utilized.
0530
0531 - ``dimm_label`` - memory module label control file
0532
0533 This control file allows this DIMM to have a label assigned
0534 to it. With this label in the module, when errors occur
0535 the output can provide the DIMM label in the system log.
0536 This becomes vital for panic events to isolate the
0537 cause of the UE event.
0538
0539 DIMM Labels must be assigned after booting, with information
0540 that correctly identifies the physical slot with its
0541 silk screen label. This information is currently very
0542 motherboard specific and determination of this information
0543 must occur in userland at this time.
0544
0545 - ``dimm_location`` - location of the memory module
0546
0547 The location can have up to 3 levels, and describe how the
0548 memory controller identifies the location of a memory module.
0549 Depending on the type of memory and memory controller, it
0550 can be:
0551
0552 - *csrow* and *channel* - used when the memory controller
0553 doesn't identify a single DIMM - e. g. in ``rankX`` dir;
0554 - *branch*, *channel*, *slot* - typically used on FB-DIMM memory
0555 controllers;
0556 - *channel*, *slot* - used on Nehalem and newer Intel drivers.
0557
0558 - ``dimm_mem_type`` - Memory Type attribute file
0559
0560 This attribute file will display what type of memory is currently
0561 on this csrow. Normally, either buffered or unbuffered memory.
0562 Examples:
0563
0564 - Registered-DDR
0565 - Unbuffered-DDR
0566
0567 .. [#f5] On some systems, the memory controller doesn't have any logic
0568 to identify the memory module. On such systems, the directory is called ``rankX`` and works on a similar way as the ``csrowX`` directories.
0569 On modern Intel memory controllers, the memory controller identifies the
0570 memory modules directly. On such systems, the directory is called ``dimmX``.
0571
0572 .. [#f6] There are also some ``power`` directories and ``subsystem``
0573 symlinks inside the sysfs mapping that are automatically created by
0574 the sysfs subsystem. Currently, they serve no purpose.
0575
0576 ``csrowX`` directories
0577 ----------------------
0578
0579 When CONFIG_EDAC_LEGACY_SYSFS is enabled, sysfs will contain the ``csrowX``
0580 directories. As this API doesn't work properly for Rambus, FB-DIMMs and
0581 modern Intel Memory Controllers, this is being deprecated in favor of
0582 ``dimmX`` directories.
0583
0584 In the ``csrowX`` directories are EDAC control and attribute files for
0585 this ``X`` instance of csrow:
0586
0587
0588 - ``ue_count`` - Total Uncorrectable Errors count attribute file
0589
0590 This attribute file displays the total count of uncorrectable
0591 errors that have occurred on this csrow. If panic_on_ue is set
0592 this counter will not have a chance to increment, since EDAC
0593 will panic the system.
0594
0595
0596 - ``ce_count`` - Total Correctable Errors count attribute file
0597
0598 This attribute file displays the total count of correctable
0599 errors that have occurred on this csrow. This count is very
0600 important to examine. CEs provide early indications that a
0601 DIMM is beginning to fail. This count field should be
0602 monitored for non-zero values and report such information
0603 to the system administrator.
0604
0605
0606 - ``size_mb`` - Total memory managed by this csrow attribute file
0607
0608 This attribute file displays, in count of megabytes, the memory
0609 that this csrow contains.
0610
0611
0612 - ``mem_type`` - Memory Type attribute file
0613
0614 This attribute file will display what type of memory is currently
0615 on this csrow. Normally, either buffered or unbuffered memory.
0616 Examples:
0617
0618 - Registered-DDR
0619 - Unbuffered-DDR
0620
0621
0622 - ``edac_mode`` - EDAC Mode of operation attribute file
0623
0624 This attribute file will display what type of Error detection
0625 and correction is being utilized.
0626
0627
0628 - ``dev_type`` - Device type attribute file
0629
0630 This attribute file will display what type of DRAM device is
0631 being utilized on this DIMM.
0632 Examples:
0633
0634 - x1
0635 - x2
0636 - x4
0637 - x8
0638
0639
0640 - ``ch0_ce_count`` - Channel 0 CE Count attribute file
0641
0642 This attribute file will display the count of CEs on this
0643 DIMM located in channel 0.
0644
0645
0646 - ``ch0_ue_count`` - Channel 0 UE Count attribute file
0647
0648 This attribute file will display the count of UEs on this
0649 DIMM located in channel 0.
0650
0651
0652 - ``ch0_dimm_label`` - Channel 0 DIMM Label control file
0653
0654
0655 This control file allows this DIMM to have a label assigned
0656 to it. With this label in the module, when errors occur
0657 the output can provide the DIMM label in the system log.
0658 This becomes vital for panic events to isolate the
0659 cause of the UE event.
0660
0661 DIMM Labels must be assigned after booting, with information
0662 that correctly identifies the physical slot with its
0663 silk screen label. This information is currently very
0664 motherboard specific and determination of this information
0665 must occur in userland at this time.
0666
0667
0668 - ``ch1_ce_count`` - Channel 1 CE Count attribute file
0669
0670
0671 This attribute file will display the count of CEs on this
0672 DIMM located in channel 1.
0673
0674
0675 - ``ch1_ue_count`` - Channel 1 UE Count attribute file
0676
0677
0678 This attribute file will display the count of UEs on this
0679 DIMM located in channel 0.
0680
0681
0682 - ``ch1_dimm_label`` - Channel 1 DIMM Label control file
0683
0684 This control file allows this DIMM to have a label assigned
0685 to it. With this label in the module, when errors occur
0686 the output can provide the DIMM label in the system log.
0687 This becomes vital for panic events to isolate the
0688 cause of the UE event.
0689
0690 DIMM Labels must be assigned after booting, with information
0691 that correctly identifies the physical slot with its
0692 silk screen label. This information is currently very
0693 motherboard specific and determination of this information
0694 must occur in userland at this time.
0695
0696
0697 System Logging
0698 --------------
0699
0700 If logging for UEs and CEs is enabled, then system logs will contain
0701 information indicating that errors have been detected::
0702
0703 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac
0704 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac
0705
0706
0707 The structure of the message is:
0708
0709 +---------------------------------------+-------------+
0710 | Content | Example |
0711 +=======================================+=============+
0712 | The memory controller | MC0 |
0713 +---------------------------------------+-------------+
0714 | Error type | CE |
0715 +---------------------------------------+-------------+
0716 | Memory page | 0x283 |
0717 +---------------------------------------+-------------+
0718 | Offset in the page | 0xce0 |
0719 +---------------------------------------+-------------+
0720 | The byte granularity | grain 8 |
0721 | or resolution of the error | |
0722 +---------------------------------------+-------------+
0723 | The error syndrome | 0xb741 |
0724 +---------------------------------------+-------------+
0725 | Memory row | row 0 |
0726 +---------------------------------------+-------------+
0727 | Memory channel | channel 1 |
0728 +---------------------------------------+-------------+
0729 | DIMM label, if set prior | DIMM B1 |
0730 +---------------------------------------+-------------+
0731 | And then an optional, driver-specific | |
0732 | message that may have additional | |
0733 | information. | |
0734 +---------------------------------------+-------------+
0735
0736 Both UEs and CEs with no info will lack all but memory controller, error
0737 type, a notice of "no info" and then an optional, driver-specific error
0738 message.
0739
0740
0741 PCI Bus Parity Detection
0742 ------------------------
0743
0744 On Header Type 00 devices, the primary status is looked at for any
0745 parity error regardless of whether parity is enabled on the device or
0746 not. (The spec indicates parity is generated in some cases). On Header
0747 Type 01 bridges, the secondary status register is also looked at to see
0748 if parity occurred on the bus on the other side of the bridge.
0749
0750
0751 Sysfs configuration
0752 -------------------
0753
0754 Under ``/sys/devices/system/edac/pci`` are control and attribute files as
0755 follows:
0756
0757
0758 - ``check_pci_parity`` - Enable/Disable PCI Parity checking control file
0759
0760 This control file enables or disables the PCI Bus Parity scanning
0761 operation. Writing a 1 to this file enables the scanning. Writing
0762 a 0 to this file disables the scanning.
0763
0764 Enable::
0765
0766 echo "1" >/sys/devices/system/edac/pci/check_pci_parity
0767
0768 Disable::
0769
0770 echo "0" >/sys/devices/system/edac/pci/check_pci_parity
0771
0772
0773 - ``pci_parity_count`` - Parity Count
0774
0775 This attribute file will display the number of parity errors that
0776 have been detected.
0777
0778
0779 Module parameters
0780 -----------------
0781
0782 - ``edac_mc_panic_on_ue`` - Panic on UE control file
0783
0784 An uncorrectable error will cause a machine panic. This is usually
0785 desirable. It is a bad idea to continue when an uncorrectable error
0786 occurs - it is indeterminate what was uncorrected and the operating
0787 system context might be so mangled that continuing will lead to further
0788 corruption. If the kernel has MCE configured, then EDAC will never
0789 notice the UE.
0790
0791 LOAD TIME::
0792
0793 module/kernel parameter: edac_mc_panic_on_ue=[0|1]
0794
0795 RUN TIME::
0796
0797 echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
0798
0799
0800 - ``edac_mc_log_ue`` - Log UE control file
0801
0802
0803 Generate kernel messages describing uncorrectable errors. These errors
0804 are reported through the system message log system. UE statistics
0805 will be accumulated even when UE logging is disabled.
0806
0807 LOAD TIME::
0808
0809 module/kernel parameter: edac_mc_log_ue=[0|1]
0810
0811 RUN TIME::
0812
0813 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
0814
0815
0816 - ``edac_mc_log_ce`` - Log CE control file
0817
0818
0819 Generate kernel messages describing correctable errors. These
0820 errors are reported through the system message log system.
0821 CE statistics will be accumulated even when CE logging is disabled.
0822
0823 LOAD TIME::
0824
0825 module/kernel parameter: edac_mc_log_ce=[0|1]
0826
0827 RUN TIME::
0828
0829 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
0830
0831
0832 - ``edac_mc_poll_msec`` - Polling period control file
0833
0834
0835 The time period, in milliseconds, for polling for error information.
0836 Too small a value wastes resources. Too large a value might delay
0837 necessary handling of errors and might loose valuable information for
0838 locating the error. 1000 milliseconds (once each second) is the current
0839 default. Systems which require all the bandwidth they can get, may
0840 increase this.
0841
0842 LOAD TIME::
0843
0844 module/kernel parameter: edac_mc_poll_msec=[0|1]
0845
0846 RUN TIME::
0847
0848 echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
0849
0850
0851 - ``panic_on_pci_parity`` - Panic on PCI PARITY Error
0852
0853
0854 This control file enables or disables panicking when a parity
0855 error has been detected.
0856
0857
0858 module/kernel parameter::
0859
0860 edac_panic_on_pci_pe=[0|1]
0861
0862 Enable::
0863
0864 echo "1" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
0865
0866 Disable::
0867
0868 echo "0" > /sys/module/edac_core/parameters/edac_panic_on_pci_pe
0869
0870
0871
0872 EDAC device type
0873 ----------------
0874
0875 In the header file, edac_pci.h, there is a series of edac_device structures
0876 and APIs for the EDAC_DEVICE.
0877
0878 User space access to an edac_device is through the sysfs interface.
0879
0880 At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices
0881 will appear.
0882
0883 There is a three level tree beneath the above ``edac`` directory. For example,
0884 the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net
0885 website) installs itself as::
0886
0887 /sys/devices/system/edac/test-instance
0888
0889 in this directory are various controls, a symlink and one or more ``instance``
0890 directories.
0891
0892 The standard default controls are:
0893
0894 ============== =======================================================
0895 log_ce boolean to log CE events
0896 log_ue boolean to log UE events
0897 panic_on_ue boolean to ``panic`` the system if an UE is encountered
0898 (default off, can be set true via startup script)
0899 poll_msec time period between POLL cycles for events
0900 ============== =======================================================
0901
0902 The test_device_edac device adds at least one of its own custom control:
0903
0904 ============== ==================================================
0905 test_bits which in the current test driver does nothing but
0906 show how it is installed. A ported driver can
0907 add one or more such controls and/or attributes
0908 for specific uses.
0909 One out-of-tree driver uses controls here to allow
0910 for ERROR INJECTION operations to hardware
0911 injection registers
0912 ============== ==================================================
0913
0914 The symlink points to the 'struct dev' that is registered for this edac_device.
0915
0916 Instances
0917 ---------
0918
0919 One or more instance directories are present. For the ``test_device_edac``
0920 case:
0921
0922 +----------------+
0923 | test-instance0 |
0924 +----------------+
0925
0926
0927 In this directory there are two default counter attributes, which are totals of
0928 counter in deeper subdirectories.
0929
0930 ============== ====================================
0931 ce_count total of CE events of subdirectories
0932 ue_count total of UE events of subdirectories
0933 ============== ====================================
0934
0935 Blocks
0936 ------
0937
0938 At the lowest directory level is the ``block`` directory. There can be 0, 1
0939 or more blocks specified in each instance:
0940
0941 +-------------+
0942 | test-block0 |
0943 +-------------+
0944
0945 In this directory the default attributes are:
0946
0947 ============== ================================================
0948 ce_count which is counter of CE events for this ``block``
0949 of hardware being monitored
0950 ue_count which is counter of UE events for this ``block``
0951 of hardware being monitored
0952 ============== ================================================
0953
0954
0955 The ``test_device_edac`` device adds 4 attributes and 1 control:
0956
0957 ================== ====================================================
0958 test-block-bits-0 for every POLL cycle this counter
0959 is incremented
0960 test-block-bits-1 every 10 cycles, this counter is bumped once,
0961 and test-block-bits-0 is set to 0
0962 test-block-bits-2 every 100 cycles, this counter is bumped once,
0963 and test-block-bits-1 is set to 0
0964 test-block-bits-3 every 1000 cycles, this counter is bumped once,
0965 and test-block-bits-2 is set to 0
0966 ================== ====================================================
0967
0968
0969 ================== ====================================================
0970 reset-counters writing ANY thing to this control will
0971 reset all the above counters.
0972 ================== ====================================================
0973
0974
0975 Use of the ``test_device_edac`` driver should enable any others to create their own
0976 unique drivers for their hardware systems.
0977
0978 The ``test_device_edac`` sample driver is located at the
0979 http://bluesmoke.sourceforge.net project site for EDAC.
0980
0981
0982 Usage of EDAC APIs on Nehalem and newer Intel CPUs
0983 --------------------------------------------------
0984
0985 On older Intel architectures, the memory controller was part of the North
0986 Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and
0987 newer Intel architectures integrated an enhanced version of the memory
0988 controller (MC) inside the CPUs.
0989
0990 This chapter will cover the differences of the enhanced memory controllers
0991 found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and
0992 ``sbx_edac`` drivers.
0993
0994 .. note::
0995
0996 The Xeon E7 processor families use a separate chip for the memory
0997 controller, called Intel Scalable Memory Buffer. This section doesn't
0998 apply for such families.
0999
1000 1) There is one Memory Controller per Quick Patch Interconnect
1001 (QPI). At the driver, the term "socket" means one QPI. This is
1002 associated with a physical CPU socket.
1003
1004 Each MC have 3 physical read channels, 3 physical write channels and
1005 3 logic channels. The driver currently sees it as just 3 channels.
1006 Each channel can have up to 3 DIMMs.
1007
1008 The minimum known unity is DIMMs. There are no information about csrows.
1009 As EDAC API maps the minimum unity is csrows, the driver sequentially
1010 maps channel/DIMM into different csrows.
1011
1012 For example, supposing the following layout::
1013
1014 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
1015 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
1016 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
1017 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
1018 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
1019 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
1020 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
1021
1022 The driver will map it as::
1023
1024 csrow0: channel 0, dimm0
1025 csrow1: channel 0, dimm1
1026 csrow2: channel 1, dimm0
1027 csrow3: channel 2, dimm0
1028
1029 exports one DIMM per csrow.
1030
1031 Each QPI is exported as a different memory controller.
1032
1033 2) The MC has the ability to inject errors to test drivers. The drivers
1034 implement this functionality via some error injection nodes:
1035
1036 For injecting a memory error, there are some sysfs nodes, under
1037 ``/sys/devices/system/edac/mc/mc?/``:
1038
1039 - ``inject_addrmatch/*``:
1040 Controls the error injection mask register. It is possible to specify
1041 several characteristics of the address to match an error code::
1042
1043 dimm = the affected dimm. Numbers are relative to a channel;
1044 rank = the memory rank;
1045 channel = the channel that will generate an error;
1046 bank = the affected bank;
1047 page = the page address;
1048 column (or col) = the address column.
1049
1050 each of the above values can be set to "any" to match any valid value.
1051
1052 At driver init, all values are set to any.
1053
1054 For example, to generate an error at rank 1 of dimm 2, for any channel,
1055 any bank, any page, any column::
1056
1057 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
1058 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
1059
1060 To return to the default behaviour of matching any, you can do::
1061
1062 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
1063 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
1064
1065 - ``inject_eccmask``:
1066 specifies what bits will have troubles,
1067
1068 - ``inject_section``:
1069 specifies what ECC cache section will get the error::
1070
1071 3 for both
1072 2 for the highest
1073 1 for the lowest
1074
1075 - ``inject_type``:
1076 specifies the type of error, being a combination of the following bits::
1077
1078 bit 0 - repeat
1079 bit 1 - ecc
1080 bit 2 - parity
1081
1082 - ``inject_enable``:
1083 starts the error generation when something different than 0 is written.
1084
1085 All inject vars can be read. root permission is needed for write.
1086
1087 Datasheet states that the error will only be generated after a write on an
1088 address that matches inject_addrmatch. It seems, however, that reading will
1089 also produce an error.
1090
1091 For example, the following code will generate an error for any write access
1092 at socket 0, on any DIMM/address on channel 2::
1093
1094 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
1095 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
1096 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
1097 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
1098 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
1099 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
1100
1101 For socket 1, it is needed to replace "mc0" by "mc1" at the above
1102 commands.
1103
1104 The generated error message will look like::
1105
1106 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
1107
1108 3) Corrected Error memory register counters
1109
1110 Those newer MCs have some registers to count memory errors. The driver
1111 uses those registers to report Corrected Errors on devices with Registered
1112 DIMMs.
1113
1114 However, those counters don't work with Unregistered DIMM. As the chipset
1115 offers some counters that also work with UDIMMs (but with a worse level of
1116 granularity than the default ones), the driver exposes those registers for
1117 UDIMM memories.
1118
1119 They can be read by looking at the contents of ``all_channel_counts/``::
1120
1121 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
1122 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
1123 0
1124 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
1125 0
1126 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
1127 0
1128
1129 What happens here is that errors on different csrows, but at the same
1130 dimm number will increment the same counter.
1131 So, in this memory mapping::
1132
1133 csrow0: channel 0, dimm0
1134 csrow1: channel 0, dimm1
1135 csrow2: channel 1, dimm0
1136 csrow3: channel 2, dimm0
1137
1138 The hardware will increment udimm0 for an error at the first dimm at either
1139 csrow0, csrow2 or csrow3;
1140
1141 The hardware will increment udimm1 for an error at the second dimm at either
1142 csrow0, csrow2 or csrow3;
1143
1144 The hardware will increment udimm2 for an error at the third dimm at either
1145 csrow0, csrow2 or csrow3;
1146
1147 4) Standard error counters
1148
1149 The standard error counters are generated when an mcelog error is received
1150 by the driver. Since, with UDIMM, this is counted by software, it is
1151 possible that some errors could be lost. With RDIMM's, they display the
1152 contents of the registers
1153
1154 Reference documents used on ``amd64_edac``
1155 ------------------------------------------
1156
1157 ``amd64_edac`` module is based on the following documents
1158 (available from http://support.amd.com/en-us/search/tech-docs):
1159
1160 1. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
1161 Opteron Processors
1162 :AMD publication #: 26094
1163 :Revision: 3.26
1164 :Link: http://support.amd.com/TechDocs/26094.PDF
1165
1166 2. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
1167 Processors
1168 :AMD publication #: 32559
1169 :Revision: 3.00
1170 :Issue Date: May 2006
1171 :Link: http://support.amd.com/TechDocs/32559.pdf
1172
1173 3. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
1174 Processors
1175 :AMD publication #: 31116
1176 :Revision: 3.00
1177 :Issue Date: September 07, 2007
1178 :Link: http://support.amd.com/TechDocs/31116.pdf
1179
1180 4. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1181 Models 30h-3Fh Processors
1182 :AMD publication #: 49125
1183 :Revision: 3.06
1184 :Issue Date: 2/12/2015 (latest release)
1185 :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
1186
1187 5. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1188 Models 60h-6Fh Processors
1189 :AMD publication #: 50742
1190 :Revision: 3.01
1191 :Issue Date: 7/23/2015 (latest release)
1192 :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
1193
1194 6. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
1195 Models 00h-0Fh Processors
1196 :AMD publication #: 48751
1197 :Revision: 3.03
1198 :Issue Date: 2/23/2015 (latest release)
1199 :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
1200
1201 Credits
1202 =======
1203
1204 * Written by Doug Thompson <dougthompson@xmission.com>
1205
1206 - 7 Dec 2005
1207 - 17 Jul 2007 Updated
1208
1209 * |copy| Mauro Carvalho Chehab
1210
1211 - 05 Aug 2009 Nehalem interface
1212 - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section
1213
1214 * EDAC authors/maintainers:
1215
1216 - Doug Thompson, Dave Jiang, Dave Peterson et al,
1217 - Mauro Carvalho Chehab
1218 - Borislav Petkov
1219 - original author: Thayne Harbaugh
