0001 ====================================================================
0002 Interaction of Suspend code (S3) with the CPU hotplug infrastructure
0003 ====================================================================
0004
0005 (C) 2011 - 2014 Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
0006
0007
0008 I. Differences between CPU hotplug and Suspend-to-RAM
0009 ======================================================
0010
0011 How does the regular CPU hotplug code differ from how the Suspend-to-RAM
0012 infrastructure uses it internally? And where do they share common code?
0013
0014 Well, a picture is worth a thousand words... So ASCII art follows :-)
0015
0016 [This depicts the current design in the kernel, and focusses only on the
0017 interactions involving the freezer and CPU hotplug and also tries to explain
0018 the locking involved. It outlines the notifications involved as well.
0019 But please note that here, only the call paths are illustrated, with the aim
0020 of describing where they take different paths and where they share code.
0021 What happens when regular CPU hotplug and Suspend-to-RAM race with each other
0022 is not depicted here.]
0023
0024 On a high level, the suspend-resume cycle goes like this::
0025
0026 |Freeze| -> |Disable nonboot| -> |Do suspend| -> |Enable nonboot| -> |Thaw |
0027 |tasks | | cpus | | | | cpus | |tasks|
0028
0029
0030 More details follow::
0031
0032 Suspend call path
0033 -----------------
0034
0035 Write 'mem' to
0036 /sys/power/state
0037 sysfs file
0038 |
0039 v
0040 Acquire system_transition_mutex lock
0041 |
0042 v
0043 Send PM_SUSPEND_PREPARE
0044 notifications
0045 |
0046 v
0047 Freeze tasks
0048 |
0049 |
0050 v
0051 freeze_secondary_cpus()
0052 /* start */
0053 |
0054 v
0055 Acquire cpu_add_remove_lock
0056 |
0057 v
0058 Iterate over CURRENTLY
0059 online CPUs
0060 |
0061 |
0062 | ----------
0063 v | L
0064 ======> _cpu_down() |
0065 | [This takes cpuhotplug.lock |
0066 Common | before taking down the CPU |
0067 code | and releases it when done] | O
0068 | While it is at it, notifications |
0069 | are sent when notable events occur, |
0070 ======> by running all registered callbacks. |
0071 | | O
0072 | |
0073 | |
0074 v |
0075 Note down these cpus in | P
0076 frozen_cpus mask ----------
0077 |
0078 v
0079 Disable regular cpu hotplug
0080 by increasing cpu_hotplug_disabled
0081 |
0082 v
0083 Release cpu_add_remove_lock
0084 |
0085 v
0086 /* freeze_secondary_cpus() complete */
0087 |
0088 v
0089 Do suspend
0090
0091
0092
0093 Resuming back is likewise, with the counterparts being (in the order of
0094 execution during resume):
0095
0096 * thaw_secondary_cpus() which involves::
0097
0098 | Acquire cpu_add_remove_lock
0099 | Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
0100 | Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
0101 | Release cpu_add_remove_lock
0102 v
0103
0104 * thaw tasks
0105 * send PM_POST_SUSPEND notifications
0106 * Release system_transition_mutex lock.
0107
0108
0109 It is to be noted here that the system_transition_mutex lock is acquired at the
0110 very beginning, when we are just starting out to suspend, and then released only
0111 after the entire cycle is complete (i.e., suspend + resume).
0112
0113 ::
0114
0115
0116
0117 Regular CPU hotplug call path
0118 -----------------------------
0119
0120 Write 0 (or 1) to
0121 /sys/devices/system/cpu/cpu*/online
0122 sysfs file
0123 |
0124 |
0125 v
0126 cpu_down()
0127 |
0128 v
0129 Acquire cpu_add_remove_lock
0130 |
0131 v
0132 If cpu_hotplug_disabled > 0
0133 return gracefully
0134 |
0135 |
0136 v
0137 ======> _cpu_down()
0138 | [This takes cpuhotplug.lock
0139 Common | before taking down the CPU
0140 code | and releases it when done]
0141 | While it is at it, notifications
0142 | are sent when notable events occur,
0143 ======> by running all registered callbacks.
0144 |
0145 |
0146 v
0147 Release cpu_add_remove_lock
0148 [That's it!, for
0149 regular CPU hotplug]
0150
0151
0152
0153 So, as can be seen from the two diagrams (the parts marked as "Common code"),
0154 regular CPU hotplug and the suspend code path converge at the _cpu_down() and
0155 _cpu_up() functions. They differ in the arguments passed to these functions,
0156 in that during regular CPU hotplug, 0 is passed for the 'tasks_frozen'
0157 argument. But during suspend, since the tasks are already frozen by the time
0158 the non-boot CPUs are offlined or onlined, the _cpu_*() functions are called
0159 with the 'tasks_frozen' argument set to 1.
0160 [See below for some known issues regarding this.]
0161
0162
0163 Important files and functions/entry points:
0164 -------------------------------------------
0165
0166 - kernel/power/process.c : freeze_processes(), thaw_processes()
0167 - kernel/power/suspend.c : suspend_prepare(), suspend_enter(), suspend_finish()
0168 - kernel/cpu.c: cpu_[up|down](), _cpu_[up|down](),
0169 [disable|enable]_nonboot_cpus()
0170
0171
0172
0173 II. What are the issues involved in CPU hotplug?
0174 ------------------------------------------------
0175
0176 There are some interesting situations involving CPU hotplug and microcode
0177 update on the CPUs, as discussed below:
0178
0179 [Please bear in mind that the kernel requests the microcode images from
0180 userspace, using the request_firmware() function defined in
0181 drivers/base/firmware_loader/main.c]
0182
0183
0184 a. When all the CPUs are identical:
0185
0186 This is the most common situation and it is quite straightforward: we want
0187 to apply the same microcode revision to each of the CPUs.
0188 To give an example of x86, the collect_cpu_info() function defined in
0189 arch/x86/kernel/microcode_core.c helps in discovering the type of the CPU
0190 and thereby in applying the correct microcode revision to it.
0191 But note that the kernel does not maintain a common microcode image for the
0192 all CPUs, in order to handle case 'b' described below.
0193
0194
0195 b. When some of the CPUs are different than the rest:
0196
0197 In this case since we probably need to apply different microcode revisions
0198 to different CPUs, the kernel maintains a copy of the correct microcode
0199 image for each CPU (after appropriate CPU type/model discovery using
0200 functions such as collect_cpu_info()).
0201
0202
0203 c. When a CPU is physically hot-unplugged and a new (and possibly different
0204 type of) CPU is hot-plugged into the system:
0205
0206 In the current design of the kernel, whenever a CPU is taken offline during
0207 a regular CPU hotplug operation, upon receiving the CPU_DEAD notification
0208 (which is sent by the CPU hotplug code), the microcode update driver's
0209 callback for that event reacts by freeing the kernel's copy of the
0210 microcode image for that CPU.
0211
0212 Hence, when a new CPU is brought online, since the kernel finds that it
0213 doesn't have the microcode image, it does the CPU type/model discovery
0214 afresh and then requests the userspace for the appropriate microcode image
0215 for that CPU, which is subsequently applied.
0216
0217 For example, in x86, the mc_cpu_callback() function (which is the microcode
0218 update driver's callback registered for CPU hotplug events) calls
0219 microcode_update_cpu() which would call microcode_init_cpu() in this case,
0220 instead of microcode_resume_cpu() when it finds that the kernel doesn't
0221 have a valid microcode image. This ensures that the CPU type/model
0222 discovery is performed and the right microcode is applied to the CPU after
0223 getting it from userspace.
0224
0225
0226 d. Handling microcode update during suspend/hibernate:
0227
0228 Strictly speaking, during a CPU hotplug operation which does not involve
0229 physically removing or inserting CPUs, the CPUs are not actually powered
0230 off during a CPU offline. They are just put to the lowest C-states possible.
0231 Hence, in such a case, it is not really necessary to re-apply microcode
0232 when the CPUs are brought back online, since they wouldn't have lost the
0233 image during the CPU offline operation.
0234
0235 This is the usual scenario encountered during a resume after a suspend.
0236 However, in the case of hibernation, since all the CPUs are completely
0237 powered off, during restore it becomes necessary to apply the microcode
0238 images to all the CPUs.
0239
0240 [Note that we don't expect someone to physically pull out nodes and insert
0241 nodes with a different type of CPUs in-between a suspend-resume or a
0242 hibernate/restore cycle.]
0243
0244 In the current design of the kernel however, during a CPU offline operation
0245 as part of the suspend/hibernate cycle (cpuhp_tasks_frozen is set),
0246 the existing copy of microcode image in the kernel is not freed up.
0247 And during the CPU online operations (during resume/restore), since the
0248 kernel finds that it already has copies of the microcode images for all the
0249 CPUs, it just applies them to the CPUs, avoiding any re-discovery of CPU
0250 type/model and the need for validating whether the microcode revisions are
0251 right for the CPUs or not (due to the above assumption that physical CPU
0252 hotplug will not be done in-between suspend/resume or hibernate/restore
0253 cycles).
0254
0255
0256 III. Known problems
0257 ===================
0258
0259 Are there any known problems when regular CPU hotplug and suspend race
0260 with each other?
0261
0262 Yes, they are listed below:
0263
0264 1. When invoking regular CPU hotplug, the 'tasks_frozen' argument passed to
0265 the _cpu_down() and _cpu_up() functions is *always* 0.
0266 This might not reflect the true current state of the system, since the
0267 tasks could have been frozen by an out-of-band event such as a suspend
0268 operation in progress. Hence, the cpuhp_tasks_frozen variable will not
0269 reflect the frozen state and the CPU hotplug callbacks which evaluate
0270 that variable might execute the wrong code path.
0271
0272 2. If a regular CPU hotplug stress test happens to race with the freezer due
0273 to a suspend operation in progress at the same time, then we could hit the
0274 situation described below:
0275
0276 * A regular cpu online operation continues its journey from userspace
0277 into the kernel, since the freezing has not yet begun.
0278 * Then freezer gets to work and freezes userspace.
0279 * If cpu online has not yet completed the microcode update stuff by now,
0280 it will now start waiting on the frozen userspace in the
0281 TASK_UNINTERRUPTIBLE state, in order to get the microcode image.
0282 * Now the freezer continues and tries to freeze the remaining tasks. But
0283 due to this wait mentioned above, the freezer won't be able to freeze
0284 the cpu online hotplug task and hence freezing of tasks fails.
0285
0286 As a result of this task freezing failure, the suspend operation gets
0287 aborted.
