Back to home page

LXR

 
 

    


0001 
0002 Control Group v2
0003 
0004 October, 2015           Tejun Heo <tj@kernel.org>
0005 
0006 This is the authoritative documentation on the design, interface and
0007 conventions of cgroup v2.  It describes all userland-visible aspects
0008 of cgroup including core and specific controller behaviors.  All
0009 future changes must be reflected in this document.  Documentation for
0010 v1 is available under Documentation/cgroup-v1/.
0011 
0012 CONTENTS
0013 
0014 1. Introduction
0015   1-1. Terminology
0016   1-2. What is cgroup?
0017 2. Basic Operations
0018   2-1. Mounting
0019   2-2. Organizing Processes
0020   2-3. [Un]populated Notification
0021   2-4. Controlling Controllers
0022     2-4-1. Enabling and Disabling
0023     2-4-2. Top-down Constraint
0024     2-4-3. No Internal Process Constraint
0025   2-5. Delegation
0026     2-5-1. Model of Delegation
0027     2-5-2. Delegation Containment
0028   2-6. Guidelines
0029     2-6-1. Organize Once and Control
0030     2-6-2. Avoid Name Collisions
0031 3. Resource Distribution Models
0032   3-1. Weights
0033   3-2. Limits
0034   3-3. Protections
0035   3-4. Allocations
0036 4. Interface Files
0037   4-1. Format
0038   4-2. Conventions
0039   4-3. Core Interface Files
0040 5. Controllers
0041   5-1. CPU
0042     5-1-1. CPU Interface Files
0043   5-2. Memory
0044     5-2-1. Memory Interface Files
0045     5-2-2. Usage Guidelines
0046     5-2-3. Memory Ownership
0047   5-3. IO
0048     5-3-1. IO Interface Files
0049     5-3-2. Writeback
0050 6. Namespace
0051   6-1. Basics
0052   6-2. The Root and Views
0053   6-3. Migration and setns(2)
0054   6-4. Interaction with Other Namespaces
0055 P. Information on Kernel Programming
0056   P-1. Filesystem Support for Writeback
0057 D. Deprecated v1 Core Features
0058 R. Issues with v1 and Rationales for v2
0059   R-1. Multiple Hierarchies
0060   R-2. Thread Granularity
0061   R-3. Competition Between Inner Nodes and Threads
0062   R-4. Other Interface Issues
0063   R-5. Controller Issues and Remedies
0064     R-5-1. Memory
0065 
0066 
0067 1. Introduction
0068 
0069 1-1. Terminology
0070 
0071 "cgroup" stands for "control group" and is never capitalized.  The
0072 singular form is used to designate the whole feature and also as a
0073 qualifier as in "cgroup controllers".  When explicitly referring to
0074 multiple individual control groups, the plural form "cgroups" is used.
0075 
0076 
0077 1-2. What is cgroup?
0078 
0079 cgroup is a mechanism to organize processes hierarchically and
0080 distribute system resources along the hierarchy in a controlled and
0081 configurable manner.
0082 
0083 cgroup is largely composed of two parts - the core and controllers.
0084 cgroup core is primarily responsible for hierarchically organizing
0085 processes.  A cgroup controller is usually responsible for
0086 distributing a specific type of system resource along the hierarchy
0087 although there are utility controllers which serve purposes other than
0088 resource distribution.
0089 
0090 cgroups form a tree structure and every process in the system belongs
0091 to one and only one cgroup.  All threads of a process belong to the
0092 same cgroup.  On creation, all processes are put in the cgroup that
0093 the parent process belongs to at the time.  A process can be migrated
0094 to another cgroup.  Migration of a process doesn't affect already
0095 existing descendant processes.
0096 
0097 Following certain structural constraints, controllers may be enabled or
0098 disabled selectively on a cgroup.  All controller behaviors are
0099 hierarchical - if a controller is enabled on a cgroup, it affects all
0100 processes which belong to the cgroups consisting the inclusive
0101 sub-hierarchy of the cgroup.  When a controller is enabled on a nested
0102 cgroup, it always restricts the resource distribution further.  The
0103 restrictions set closer to the root in the hierarchy can not be
0104 overridden from further away.
0105 
0106 
0107 2. Basic Operations
0108 
0109 2-1. Mounting
0110 
0111 Unlike v1, cgroup v2 has only single hierarchy.  The cgroup v2
0112 hierarchy can be mounted with the following mount command.
0113 
0114   # mount -t cgroup2 none $MOUNT_POINT
0115 
0116 cgroup2 filesystem has the magic number 0x63677270 ("cgrp").  All
0117 controllers which support v2 and are not bound to a v1 hierarchy are
0118 automatically bound to the v2 hierarchy and show up at the root.
0119 Controllers which are not in active use in the v2 hierarchy can be
0120 bound to other hierarchies.  This allows mixing v2 hierarchy with the
0121 legacy v1 multiple hierarchies in a fully backward compatible way.
0122 
0123 A controller can be moved across hierarchies only after the controller
0124 is no longer referenced in its current hierarchy.  Because per-cgroup
0125 controller states are destroyed asynchronously and controllers may
0126 have lingering references, a controller may not show up immediately on
0127 the v2 hierarchy after the final umount of the previous hierarchy.
0128 Similarly, a controller should be fully disabled to be moved out of
0129 the unified hierarchy and it may take some time for the disabled
0130 controller to become available for other hierarchies; furthermore, due
0131 to inter-controller dependencies, other controllers may need to be
0132 disabled too.
0133 
0134 While useful for development and manual configurations, moving
0135 controllers dynamically between the v2 and other hierarchies is
0136 strongly discouraged for production use.  It is recommended to decide
0137 the hierarchies and controller associations before starting using the
0138 controllers after system boot.
0139 
0140 During transition to v2, system management software might still
0141 automount the v1 cgroup filesystem and so hijack all controllers
0142 during boot, before manual intervention is possible. To make testing
0143 and experimenting easier, the kernel parameter cgroup_no_v1= allows
0144 disabling controllers in v1 and make them always available in v2.
0145 
0146 
0147 2-2. Organizing Processes
0148 
0149 Initially, only the root cgroup exists to which all processes belong.
0150 A child cgroup can be created by creating a sub-directory.
0151 
0152   # mkdir $CGROUP_NAME
0153 
0154 A given cgroup may have multiple child cgroups forming a tree
0155 structure.  Each cgroup has a read-writable interface file
0156 "cgroup.procs".  When read, it lists the PIDs of all processes which
0157 belong to the cgroup one-per-line.  The PIDs are not ordered and the
0158 same PID may show up more than once if the process got moved to
0159 another cgroup and then back or the PID got recycled while reading.
0160 
0161 A process can be migrated into a cgroup by writing its PID to the
0162 target cgroup's "cgroup.procs" file.  Only one process can be migrated
0163 on a single write(2) call.  If a process is composed of multiple
0164 threads, writing the PID of any thread migrates all threads of the
0165 process.
0166 
0167 When a process forks a child process, the new process is born into the
0168 cgroup that the forking process belongs to at the time of the
0169 operation.  After exit, a process stays associated with the cgroup
0170 that it belonged to at the time of exit until it's reaped; however, a
0171 zombie process does not appear in "cgroup.procs" and thus can't be
0172 moved to another cgroup.
0173 
0174 A cgroup which doesn't have any children or live processes can be
0175 destroyed by removing the directory.  Note that a cgroup which doesn't
0176 have any children and is associated only with zombie processes is
0177 considered empty and can be removed.
0178 
0179   # rmdir $CGROUP_NAME
0180 
0181 "/proc/$PID/cgroup" lists a process's cgroup membership.  If legacy
0182 cgroup is in use in the system, this file may contain multiple lines,
0183 one for each hierarchy.  The entry for cgroup v2 is always in the
0184 format "0::$PATH".
0185 
0186   # cat /proc/842/cgroup
0187   ...
0188   0::/test-cgroup/test-cgroup-nested
0189 
0190 If the process becomes a zombie and the cgroup it was associated with
0191 is removed subsequently, " (deleted)" is appended to the path.
0192 
0193   # cat /proc/842/cgroup
0194   ...
0195   0::/test-cgroup/test-cgroup-nested (deleted)
0196 
0197 
0198 2-3. [Un]populated Notification
0199 
0200 Each non-root cgroup has a "cgroup.events" file which contains
0201 "populated" field indicating whether the cgroup's sub-hierarchy has
0202 live processes in it.  Its value is 0 if there is no live process in
0203 the cgroup and its descendants; otherwise, 1.  poll and [id]notify
0204 events are triggered when the value changes.  This can be used, for
0205 example, to start a clean-up operation after all processes of a given
0206 sub-hierarchy have exited.  The populated state updates and
0207 notifications are recursive.  Consider the following sub-hierarchy
0208 where the numbers in the parentheses represent the numbers of processes
0209 in each cgroup.
0210 
0211   A(4) - B(0) - C(1)
0212               \ D(0)
0213 
0214 A, B and C's "populated" fields would be 1 while D's 0.  After the one
0215 process in C exits, B and C's "populated" fields would flip to "0" and
0216 file modified events will be generated on the "cgroup.events" files of
0217 both cgroups.
0218 
0219 
0220 2-4. Controlling Controllers
0221 
0222 2-4-1. Enabling and Disabling
0223 
0224 Each cgroup has a "cgroup.controllers" file which lists all
0225 controllers available for the cgroup to enable.
0226 
0227   # cat cgroup.controllers
0228   cpu io memory
0229 
0230 No controller is enabled by default.  Controllers can be enabled and
0231 disabled by writing to the "cgroup.subtree_control" file.
0232 
0233   # echo "+cpu +memory -io" > cgroup.subtree_control
0234 
0235 Only controllers which are listed in "cgroup.controllers" can be
0236 enabled.  When multiple operations are specified as above, either they
0237 all succeed or fail.  If multiple operations on the same controller
0238 are specified, the last one is effective.
0239 
0240 Enabling a controller in a cgroup indicates that the distribution of
0241 the target resource across its immediate children will be controlled.
0242 Consider the following sub-hierarchy.  The enabled controllers are
0243 listed in parentheses.
0244 
0245   A(cpu,memory) - B(memory) - C()
0246                             \ D()
0247 
0248 As A has "cpu" and "memory" enabled, A will control the distribution
0249 of CPU cycles and memory to its children, in this case, B.  As B has
0250 "memory" enabled but not "CPU", C and D will compete freely on CPU
0251 cycles but their division of memory available to B will be controlled.
0252 
0253 As a controller regulates the distribution of the target resource to
0254 the cgroup's children, enabling it creates the controller's interface
0255 files in the child cgroups.  In the above example, enabling "cpu" on B
0256 would create the "cpu." prefixed controller interface files in C and
0257 D.  Likewise, disabling "memory" from B would remove the "memory."
0258 prefixed controller interface files from C and D.  This means that the
0259 controller interface files - anything which doesn't start with
0260 "cgroup." are owned by the parent rather than the cgroup itself.
0261 
0262 
0263 2-4-2. Top-down Constraint
0264 
0265 Resources are distributed top-down and a cgroup can further distribute
0266 a resource only if the resource has been distributed to it from the
0267 parent.  This means that all non-root "cgroup.subtree_control" files
0268 can only contain controllers which are enabled in the parent's
0269 "cgroup.subtree_control" file.  A controller can be enabled only if
0270 the parent has the controller enabled and a controller can't be
0271 disabled if one or more children have it enabled.
0272 
0273 
0274 2-4-3. No Internal Process Constraint
0275 
0276 Non-root cgroups can only distribute resources to their children when
0277 they don't have any processes of their own.  In other words, only
0278 cgroups which don't contain any processes can have controllers enabled
0279 in their "cgroup.subtree_control" files.
0280 
0281 This guarantees that, when a controller is looking at the part of the
0282 hierarchy which has it enabled, processes are always only on the
0283 leaves.  This rules out situations where child cgroups compete against
0284 internal processes of the parent.
0285 
0286 The root cgroup is exempt from this restriction.  Root contains
0287 processes and anonymous resource consumption which can't be associated
0288 with any other cgroups and requires special treatment from most
0289 controllers.  How resource consumption in the root cgroup is governed
0290 is up to each controller.
0291 
0292 Note that the restriction doesn't get in the way if there is no
0293 enabled controller in the cgroup's "cgroup.subtree_control".  This is
0294 important as otherwise it wouldn't be possible to create children of a
0295 populated cgroup.  To control resource distribution of a cgroup, the
0296 cgroup must create children and transfer all its processes to the
0297 children before enabling controllers in its "cgroup.subtree_control"
0298 file.
0299 
0300 
0301 2-5. Delegation
0302 
0303 2-5-1. Model of Delegation
0304 
0305 A cgroup can be delegated to a less privileged user by granting write
0306 access of the directory and its "cgroup.procs" file to the user.  Note
0307 that resource control interface files in a given directory control the
0308 distribution of the parent's resources and thus must not be delegated
0309 along with the directory.
0310 
0311 Once delegated, the user can build sub-hierarchy under the directory,
0312 organize processes as it sees fit and further distribute the resources
0313 it received from the parent.  The limits and other settings of all
0314 resource controllers are hierarchical and regardless of what happens
0315 in the delegated sub-hierarchy, nothing can escape the resource
0316 restrictions imposed by the parent.
0317 
0318 Currently, cgroup doesn't impose any restrictions on the number of
0319 cgroups in or nesting depth of a delegated sub-hierarchy; however,
0320 this may be limited explicitly in the future.
0321 
0322 
0323 2-5-2. Delegation Containment
0324 
0325 A delegated sub-hierarchy is contained in the sense that processes
0326 can't be moved into or out of the sub-hierarchy by the delegatee.  For
0327 a process with a non-root euid to migrate a target process into a
0328 cgroup by writing its PID to the "cgroup.procs" file, the following
0329 conditions must be met.
0330 
0331 - The writer's euid must match either uid or suid of the target process.
0332 
0333 - The writer must have write access to the "cgroup.procs" file.
0334 
0335 - The writer must have write access to the "cgroup.procs" file of the
0336   common ancestor of the source and destination cgroups.
0337 
0338 The above three constraints ensure that while a delegatee may migrate
0339 processes around freely in the delegated sub-hierarchy it can't pull
0340 in from or push out to outside the sub-hierarchy.
0341 
0342 For an example, let's assume cgroups C0 and C1 have been delegated to
0343 user U0 who created C00, C01 under C0 and C10 under C1 as follows and
0344 all processes under C0 and C1 belong to U0.
0345 
0346   ~~~~~~~~~~~~~ - C0 - C00
0347   ~ cgroup    ~      \ C01
0348   ~ hierarchy ~
0349   ~~~~~~~~~~~~~ - C1 - C10
0350 
0351 Let's also say U0 wants to write the PID of a process which is
0352 currently in C10 into "C00/cgroup.procs".  U0 has write access to the
0353 file and uid match on the process; however, the common ancestor of the
0354 source cgroup C10 and the destination cgroup C00 is above the points
0355 of delegation and U0 would not have write access to its "cgroup.procs"
0356 files and thus the write will be denied with -EACCES.
0357 
0358 
0359 2-6. Guidelines
0360 
0361 2-6-1. Organize Once and Control
0362 
0363 Migrating a process across cgroups is a relatively expensive operation
0364 and stateful resources such as memory are not moved together with the
0365 process.  This is an explicit design decision as there often exist
0366 inherent trade-offs between migration and various hot paths in terms
0367 of synchronization cost.
0368 
0369 As such, migrating processes across cgroups frequently as a means to
0370 apply different resource restrictions is discouraged.  A workload
0371 should be assigned to a cgroup according to the system's logical and
0372 resource structure once on start-up.  Dynamic adjustments to resource
0373 distribution can be made by changing controller configuration through
0374 the interface files.
0375 
0376 
0377 2-6-2. Avoid Name Collisions
0378 
0379 Interface files for a cgroup and its children cgroups occupy the same
0380 directory and it is possible to create children cgroups which collide
0381 with interface files.
0382 
0383 All cgroup core interface files are prefixed with "cgroup." and each
0384 controller's interface files are prefixed with the controller name and
0385 a dot.  A controller's name is composed of lower case alphabets and
0386 '_'s but never begins with an '_' so it can be used as the prefix
0387 character for collision avoidance.  Also, interface file names won't
0388 start or end with terms which are often used in categorizing workloads
0389 such as job, service, slice, unit or workload.
0390 
0391 cgroup doesn't do anything to prevent name collisions and it's the
0392 user's responsibility to avoid them.
0393 
0394 
0395 3. Resource Distribution Models
0396 
0397 cgroup controllers implement several resource distribution schemes
0398 depending on the resource type and expected use cases.  This section
0399 describes major schemes in use along with their expected behaviors.
0400 
0401 
0402 3-1. Weights
0403 
0404 A parent's resource is distributed by adding up the weights of all
0405 active children and giving each the fraction matching the ratio of its
0406 weight against the sum.  As only children which can make use of the
0407 resource at the moment participate in the distribution, this is
0408 work-conserving.  Due to the dynamic nature, this model is usually
0409 used for stateless resources.
0410 
0411 All weights are in the range [1, 10000] with the default at 100.  This
0412 allows symmetric multiplicative biases in both directions at fine
0413 enough granularity while staying in the intuitive range.
0414 
0415 As long as the weight is in range, all configuration combinations are
0416 valid and there is no reason to reject configuration changes or
0417 process migrations.
0418 
0419 "cpu.weight" proportionally distributes CPU cycles to active children
0420 and is an example of this type.
0421 
0422 
0423 3-2. Limits
0424 
0425 A child can only consume upto the configured amount of the resource.
0426 Limits can be over-committed - the sum of the limits of children can
0427 exceed the amount of resource available to the parent.
0428 
0429 Limits are in the range [0, max] and defaults to "max", which is noop.
0430 
0431 As limits can be over-committed, all configuration combinations are
0432 valid and there is no reason to reject configuration changes or
0433 process migrations.
0434 
0435 "io.max" limits the maximum BPS and/or IOPS that a cgroup can consume
0436 on an IO device and is an example of this type.
0437 
0438 
0439 3-3. Protections
0440 
0441 A cgroup is protected to be allocated upto the configured amount of
0442 the resource if the usages of all its ancestors are under their
0443 protected levels.  Protections can be hard guarantees or best effort
0444 soft boundaries.  Protections can also be over-committed in which case
0445 only upto the amount available to the parent is protected among
0446 children.
0447 
0448 Protections are in the range [0, max] and defaults to 0, which is
0449 noop.
0450 
0451 As protections can be over-committed, all configuration combinations
0452 are valid and there is no reason to reject configuration changes or
0453 process migrations.
0454 
0455 "memory.low" implements best-effort memory protection and is an
0456 example of this type.
0457 
0458 
0459 3-4. Allocations
0460 
0461 A cgroup is exclusively allocated a certain amount of a finite
0462 resource.  Allocations can't be over-committed - the sum of the
0463 allocations of children can not exceed the amount of resource
0464 available to the parent.
0465 
0466 Allocations are in the range [0, max] and defaults to 0, which is no
0467 resource.
0468 
0469 As allocations can't be over-committed, some configuration
0470 combinations are invalid and should be rejected.  Also, if the
0471 resource is mandatory for execution of processes, process migrations
0472 may be rejected.
0473 
0474 "cpu.rt.max" hard-allocates realtime slices and is an example of this
0475 type.
0476 
0477 
0478 4. Interface Files
0479 
0480 4-1. Format
0481 
0482 All interface files should be in one of the following formats whenever
0483 possible.
0484 
0485   New-line separated values
0486   (when only one value can be written at once)
0487 
0488         VAL0\n
0489         VAL1\n
0490         ...
0491 
0492   Space separated values
0493   (when read-only or multiple values can be written at once)
0494 
0495         VAL0 VAL1 ...\n
0496 
0497   Flat keyed
0498 
0499         KEY0 VAL0\n
0500         KEY1 VAL1\n
0501         ...
0502 
0503   Nested keyed
0504 
0505         KEY0 SUB_KEY0=VAL00 SUB_KEY1=VAL01...
0506         KEY1 SUB_KEY0=VAL10 SUB_KEY1=VAL11...
0507         ...
0508 
0509 For a writable file, the format for writing should generally match
0510 reading; however, controllers may allow omitting later fields or
0511 implement restricted shortcuts for most common use cases.
0512 
0513 For both flat and nested keyed files, only the values for a single key
0514 can be written at a time.  For nested keyed files, the sub key pairs
0515 may be specified in any order and not all pairs have to be specified.
0516 
0517 
0518 4-2. Conventions
0519 
0520 - Settings for a single feature should be contained in a single file.
0521 
0522 - The root cgroup should be exempt from resource control and thus
0523   shouldn't have resource control interface files.  Also,
0524   informational files on the root cgroup which end up showing global
0525   information available elsewhere shouldn't exist.
0526 
0527 - If a controller implements weight based resource distribution, its
0528   interface file should be named "weight" and have the range [1,
0529   10000] with 100 as the default.  The values are chosen to allow
0530   enough and symmetric bias in both directions while keeping it
0531   intuitive (the default is 100%).
0532 
0533 - If a controller implements an absolute resource guarantee and/or
0534   limit, the interface files should be named "min" and "max"
0535   respectively.  If a controller implements best effort resource
0536   guarantee and/or limit, the interface files should be named "low"
0537   and "high" respectively.
0538 
0539   In the above four control files, the special token "max" should be
0540   used to represent upward infinity for both reading and writing.
0541 
0542 - If a setting has a configurable default value and keyed specific
0543   overrides, the default entry should be keyed with "default" and
0544   appear as the first entry in the file.
0545 
0546   The default value can be updated by writing either "default $VAL" or
0547   "$VAL".
0548 
0549   When writing to update a specific override, "default" can be used as
0550   the value to indicate removal of the override.  Override entries
0551   with "default" as the value must not appear when read.
0552 
0553   For example, a setting which is keyed by major:minor device numbers
0554   with integer values may look like the following.
0555 
0556     # cat cgroup-example-interface-file
0557     default 150
0558     8:0 300
0559 
0560   The default value can be updated by
0561 
0562     # echo 125 > cgroup-example-interface-file
0563 
0564   or
0565 
0566     # echo "default 125" > cgroup-example-interface-file
0567 
0568   An override can be set by
0569 
0570     # echo "8:16 170" > cgroup-example-interface-file
0571 
0572   and cleared by
0573 
0574     # echo "8:0 default" > cgroup-example-interface-file
0575     # cat cgroup-example-interface-file
0576     default 125
0577     8:16 170
0578 
0579 - For events which are not very high frequency, an interface file
0580   "events" should be created which lists event key value pairs.
0581   Whenever a notifiable event happens, file modified event should be
0582   generated on the file.
0583 
0584 
0585 4-3. Core Interface Files
0586 
0587 All cgroup core files are prefixed with "cgroup."
0588 
0589   cgroup.procs
0590 
0591         A read-write new-line separated values file which exists on
0592         all cgroups.
0593 
0594         When read, it lists the PIDs of all processes which belong to
0595         the cgroup one-per-line.  The PIDs are not ordered and the
0596         same PID may show up more than once if the process got moved
0597         to another cgroup and then back or the PID got recycled while
0598         reading.
0599 
0600         A PID can be written to migrate the process associated with
0601         the PID to the cgroup.  The writer should match all of the
0602         following conditions.
0603 
0604         - Its euid is either root or must match either uid or suid of
0605           the target process.
0606 
0607         - It must have write access to the "cgroup.procs" file.
0608 
0609         - It must have write access to the "cgroup.procs" file of the
0610           common ancestor of the source and destination cgroups.
0611 
0612         When delegating a sub-hierarchy, write access to this file
0613         should be granted along with the containing directory.
0614 
0615   cgroup.controllers
0616 
0617         A read-only space separated values file which exists on all
0618         cgroups.
0619 
0620         It shows space separated list of all controllers available to
0621         the cgroup.  The controllers are not ordered.
0622 
0623   cgroup.subtree_control
0624 
0625         A read-write space separated values file which exists on all
0626         cgroups.  Starts out empty.
0627 
0628         When read, it shows space separated list of the controllers
0629         which are enabled to control resource distribution from the
0630         cgroup to its children.
0631 
0632         Space separated list of controllers prefixed with '+' or '-'
0633         can be written to enable or disable controllers.  A controller
0634         name prefixed with '+' enables the controller and '-'
0635         disables.  If a controller appears more than once on the list,
0636         the last one is effective.  When multiple enable and disable
0637         operations are specified, either all succeed or all fail.
0638 
0639   cgroup.events
0640 
0641         A read-only flat-keyed file which exists on non-root cgroups.
0642         The following entries are defined.  Unless specified
0643         otherwise, a value change in this file generates a file
0644         modified event.
0645 
0646           populated
0647 
0648                 1 if the cgroup or its descendants contains any live
0649                 processes; otherwise, 0.
0650 
0651 
0652 5. Controllers
0653 
0654 5-1. CPU
0655 
0656 [NOTE: The interface for the cpu controller hasn't been merged yet]
0657 
0658 The "cpu" controllers regulates distribution of CPU cycles.  This
0659 controller implements weight and absolute bandwidth limit models for
0660 normal scheduling policy and absolute bandwidth allocation model for
0661 realtime scheduling policy.
0662 
0663 
0664 5-1-1. CPU Interface Files
0665 
0666 All time durations are in microseconds.
0667 
0668   cpu.stat
0669 
0670         A read-only flat-keyed file which exists on non-root cgroups.
0671 
0672         It reports the following six stats.
0673 
0674           usage_usec
0675           user_usec
0676           system_usec
0677           nr_periods
0678           nr_throttled
0679           throttled_usec
0680 
0681   cpu.weight
0682 
0683         A read-write single value file which exists on non-root
0684         cgroups.  The default is "100".
0685 
0686         The weight in the range [1, 10000].
0687 
0688   cpu.max
0689 
0690         A read-write two value file which exists on non-root cgroups.
0691         The default is "max 100000".
0692 
0693         The maximum bandwidth limit.  It's in the following format.
0694 
0695           $MAX $PERIOD
0696 
0697         which indicates that the group may consume upto $MAX in each
0698         $PERIOD duration.  "max" for $MAX indicates no limit.  If only
0699         one number is written, $MAX is updated.
0700 
0701   cpu.rt.max
0702 
0703   [NOTE: The semantics of this file is still under discussion and the
0704    interface hasn't been merged yet]
0705 
0706         A read-write two value file which exists on all cgroups.
0707         The default is "0 100000".
0708 
0709         The maximum realtime runtime allocation.  Over-committing
0710         configurations are disallowed and process migrations are
0711         rejected if not enough bandwidth is available.  It's in the
0712         following format.
0713 
0714           $MAX $PERIOD
0715 
0716         which indicates that the group may consume upto $MAX in each
0717         $PERIOD duration.  If only one number is written, $MAX is
0718         updated.
0719 
0720 
0721 5-2. Memory
0722 
0723 The "memory" controller regulates distribution of memory.  Memory is
0724 stateful and implements both limit and protection models.  Due to the
0725 intertwining between memory usage and reclaim pressure and the
0726 stateful nature of memory, the distribution model is relatively
0727 complex.
0728 
0729 While not completely water-tight, all major memory usages by a given
0730 cgroup are tracked so that the total memory consumption can be
0731 accounted and controlled to a reasonable extent.  Currently, the
0732 following types of memory usages are tracked.
0733 
0734 - Userland memory - page cache and anonymous memory.
0735 
0736 - Kernel data structures such as dentries and inodes.
0737 
0738 - TCP socket buffers.
0739 
0740 The above list may expand in the future for better coverage.
0741 
0742 
0743 5-2-1. Memory Interface Files
0744 
0745 All memory amounts are in bytes.  If a value which is not aligned to
0746 PAGE_SIZE is written, the value may be rounded up to the closest
0747 PAGE_SIZE multiple when read back.
0748 
0749   memory.current
0750 
0751         A read-only single value file which exists on non-root
0752         cgroups.
0753 
0754         The total amount of memory currently being used by the cgroup
0755         and its descendants.
0756 
0757   memory.low
0758 
0759         A read-write single value file which exists on non-root
0760         cgroups.  The default is "0".
0761 
0762         Best-effort memory protection.  If the memory usages of a
0763         cgroup and all its ancestors are below their low boundaries,
0764         the cgroup's memory won't be reclaimed unless memory can be
0765         reclaimed from unprotected cgroups.
0766 
0767         Putting more memory than generally available under this
0768         protection is discouraged.
0769 
0770   memory.high
0771 
0772         A read-write single value file which exists on non-root
0773         cgroups.  The default is "max".
0774 
0775         Memory usage throttle limit.  This is the main mechanism to
0776         control memory usage of a cgroup.  If a cgroup's usage goes
0777         over the high boundary, the processes of the cgroup are
0778         throttled and put under heavy reclaim pressure.
0779 
0780         Going over the high limit never invokes the OOM killer and
0781         under extreme conditions the limit may be breached.
0782 
0783   memory.max
0784 
0785         A read-write single value file which exists on non-root
0786         cgroups.  The default is "max".
0787 
0788         Memory usage hard limit.  This is the final protection
0789         mechanism.  If a cgroup's memory usage reaches this limit and
0790         can't be reduced, the OOM killer is invoked in the cgroup.
0791         Under certain circumstances, the usage may go over the limit
0792         temporarily.
0793 
0794         This is the ultimate protection mechanism.  As long as the
0795         high limit is used and monitored properly, this limit's
0796         utility is limited to providing the final safety net.
0797 
0798   memory.events
0799 
0800         A read-only flat-keyed file which exists on non-root cgroups.
0801         The following entries are defined.  Unless specified
0802         otherwise, a value change in this file generates a file
0803         modified event.
0804 
0805           low
0806 
0807                 The number of times the cgroup is reclaimed due to
0808                 high memory pressure even though its usage is under
0809                 the low boundary.  This usually indicates that the low
0810                 boundary is over-committed.
0811 
0812           high
0813 
0814                 The number of times processes of the cgroup are
0815                 throttled and routed to perform direct memory reclaim
0816                 because the high memory boundary was exceeded.  For a
0817                 cgroup whose memory usage is capped by the high limit
0818                 rather than global memory pressure, this event's
0819                 occurrences are expected.
0820 
0821           max
0822 
0823                 The number of times the cgroup's memory usage was
0824                 about to go over the max boundary.  If direct reclaim
0825                 fails to bring it down, the OOM killer is invoked.
0826 
0827           oom
0828 
0829                 The number of times the OOM killer has been invoked in
0830                 the cgroup.  This may not exactly match the number of
0831                 processes killed but should generally be close.
0832 
0833   memory.stat
0834 
0835         A read-only flat-keyed file which exists on non-root cgroups.
0836 
0837         This breaks down the cgroup's memory footprint into different
0838         types of memory, type-specific details, and other information
0839         on the state and past events of the memory management system.
0840 
0841         All memory amounts are in bytes.
0842 
0843         The entries are ordered to be human readable, and new entries
0844         can show up in the middle. Don't rely on items remaining in a
0845         fixed position; use the keys to look up specific values!
0846 
0847           anon
0848 
0849                 Amount of memory used in anonymous mappings such as
0850                 brk(), sbrk(), and mmap(MAP_ANONYMOUS)
0851 
0852           file
0853 
0854                 Amount of memory used to cache filesystem data,
0855                 including tmpfs and shared memory.
0856 
0857           kernel_stack
0858 
0859                 Amount of memory allocated to kernel stacks.
0860 
0861           slab
0862 
0863                 Amount of memory used for storing in-kernel data
0864                 structures.
0865 
0866           sock
0867 
0868                 Amount of memory used in network transmission buffers
0869 
0870           file_mapped
0871 
0872                 Amount of cached filesystem data mapped with mmap()
0873 
0874           file_dirty
0875 
0876                 Amount of cached filesystem data that was modified but
0877                 not yet written back to disk
0878 
0879           file_writeback
0880 
0881                 Amount of cached filesystem data that was modified and
0882                 is currently being written back to disk
0883 
0884           inactive_anon
0885           active_anon
0886           inactive_file
0887           active_file
0888           unevictable
0889 
0890                 Amount of memory, swap-backed and filesystem-backed,
0891                 on the internal memory management lists used by the
0892                 page reclaim algorithm
0893 
0894           slab_reclaimable
0895 
0896                 Part of "slab" that might be reclaimed, such as
0897                 dentries and inodes.
0898 
0899           slab_unreclaimable
0900 
0901                 Part of "slab" that cannot be reclaimed on memory
0902                 pressure.
0903 
0904           pgfault
0905 
0906                 Total number of page faults incurred
0907 
0908           pgmajfault
0909 
0910                 Number of major page faults incurred
0911 
0912   memory.swap.current
0913 
0914         A read-only single value file which exists on non-root
0915         cgroups.
0916 
0917         The total amount of swap currently being used by the cgroup
0918         and its descendants.
0919 
0920   memory.swap.max
0921 
0922         A read-write single value file which exists on non-root
0923         cgroups.  The default is "max".
0924 
0925         Swap usage hard limit.  If a cgroup's swap usage reaches this
0926         limit, anonymous meomry of the cgroup will not be swapped out.
0927 
0928 
0929 5-2-2. Usage Guidelines
0930 
0931 "memory.high" is the main mechanism to control memory usage.
0932 Over-committing on high limit (sum of high limits > available memory)
0933 and letting global memory pressure to distribute memory according to
0934 usage is a viable strategy.
0935 
0936 Because breach of the high limit doesn't trigger the OOM killer but
0937 throttles the offending cgroup, a management agent has ample
0938 opportunities to monitor and take appropriate actions such as granting
0939 more memory or terminating the workload.
0940 
0941 Determining whether a cgroup has enough memory is not trivial as
0942 memory usage doesn't indicate whether the workload can benefit from
0943 more memory.  For example, a workload which writes data received from
0944 network to a file can use all available memory but can also operate as
0945 performant with a small amount of memory.  A measure of memory
0946 pressure - how much the workload is being impacted due to lack of
0947 memory - is necessary to determine whether a workload needs more
0948 memory; unfortunately, memory pressure monitoring mechanism isn't
0949 implemented yet.
0950 
0951 
0952 5-2-3. Memory Ownership
0953 
0954 A memory area is charged to the cgroup which instantiated it and stays
0955 charged to the cgroup until the area is released.  Migrating a process
0956 to a different cgroup doesn't move the memory usages that it
0957 instantiated while in the previous cgroup to the new cgroup.
0958 
0959 A memory area may be used by processes belonging to different cgroups.
0960 To which cgroup the area will be charged is in-deterministic; however,
0961 over time, the memory area is likely to end up in a cgroup which has
0962 enough memory allowance to avoid high reclaim pressure.
0963 
0964 If a cgroup sweeps a considerable amount of memory which is expected
0965 to be accessed repeatedly by other cgroups, it may make sense to use
0966 POSIX_FADV_DONTNEED to relinquish the ownership of memory areas
0967 belonging to the affected files to ensure correct memory ownership.
0968 
0969 
0970 5-3. IO
0971 
0972 The "io" controller regulates the distribution of IO resources.  This
0973 controller implements both weight based and absolute bandwidth or IOPS
0974 limit distribution; however, weight based distribution is available
0975 only if cfq-iosched is in use and neither scheme is available for
0976 blk-mq devices.
0977 
0978 
0979 5-3-1. IO Interface Files
0980 
0981   io.stat
0982 
0983         A read-only nested-keyed file which exists on non-root
0984         cgroups.
0985 
0986         Lines are keyed by $MAJ:$MIN device numbers and not ordered.
0987         The following nested keys are defined.
0988 
0989           rbytes        Bytes read
0990           wbytes        Bytes written
0991           rios          Number of read IOs
0992           wios          Number of write IOs
0993 
0994         An example read output follows.
0995 
0996           8:16 rbytes=1459200 wbytes=314773504 rios=192 wios=353
0997           8:0 rbytes=90430464 wbytes=299008000 rios=8950 wios=1252
0998 
0999   io.weight
1000 
1001         A read-write flat-keyed file which exists on non-root cgroups.
1002         The default is "default 100".
1003 
1004         The first line is the default weight applied to devices
1005         without specific override.  The rest are overrides keyed by
1006         $MAJ:$MIN device numbers and not ordered.  The weights are in
1007         the range [1, 10000] and specifies the relative amount IO time
1008         the cgroup can use in relation to its siblings.
1009 
1010         The default weight can be updated by writing either "default
1011         $WEIGHT" or simply "$WEIGHT".  Overrides can be set by writing
1012         "$MAJ:$MIN $WEIGHT" and unset by writing "$MAJ:$MIN default".
1013 
1014         An example read output follows.
1015 
1016           default 100
1017           8:16 200
1018           8:0 50
1019 
1020   io.max
1021 
1022         A read-write nested-keyed file which exists on non-root
1023         cgroups.
1024 
1025         BPS and IOPS based IO limit.  Lines are keyed by $MAJ:$MIN
1026         device numbers and not ordered.  The following nested keys are
1027         defined.
1028 
1029           rbps          Max read bytes per second
1030           wbps          Max write bytes per second
1031           riops         Max read IO operations per second
1032           wiops         Max write IO operations per second
1033 
1034         When writing, any number of nested key-value pairs can be
1035         specified in any order.  "max" can be specified as the value
1036         to remove a specific limit.  If the same key is specified
1037         multiple times, the outcome is undefined.
1038 
1039         BPS and IOPS are measured in each IO direction and IOs are
1040         delayed if limit is reached.  Temporary bursts are allowed.
1041 
1042         Setting read limit at 2M BPS and write at 120 IOPS for 8:16.
1043 
1044           echo "8:16 rbps=2097152 wiops=120" > io.max
1045 
1046         Reading returns the following.
1047 
1048           8:16 rbps=2097152 wbps=max riops=max wiops=120
1049 
1050         Write IOPS limit can be removed by writing the following.
1051 
1052           echo "8:16 wiops=max" > io.max
1053 
1054         Reading now returns the following.
1055 
1056           8:16 rbps=2097152 wbps=max riops=max wiops=max
1057 
1058 
1059 5-3-2. Writeback
1060 
1061 Page cache is dirtied through buffered writes and shared mmaps and
1062 written asynchronously to the backing filesystem by the writeback
1063 mechanism.  Writeback sits between the memory and IO domains and
1064 regulates the proportion of dirty memory by balancing dirtying and
1065 write IOs.
1066 
1067 The io controller, in conjunction with the memory controller,
1068 implements control of page cache writeback IOs.  The memory controller
1069 defines the memory domain that dirty memory ratio is calculated and
1070 maintained for and the io controller defines the io domain which
1071 writes out dirty pages for the memory domain.  Both system-wide and
1072 per-cgroup dirty memory states are examined and the more restrictive
1073 of the two is enforced.
1074 
1075 cgroup writeback requires explicit support from the underlying
1076 filesystem.  Currently, cgroup writeback is implemented on ext2, ext4
1077 and btrfs.  On other filesystems, all writeback IOs are attributed to
1078 the root cgroup.
1079 
1080 There are inherent differences in memory and writeback management
1081 which affects how cgroup ownership is tracked.  Memory is tracked per
1082 page while writeback per inode.  For the purpose of writeback, an
1083 inode is assigned to a cgroup and all IO requests to write dirty pages
1084 from the inode are attributed to that cgroup.
1085 
1086 As cgroup ownership for memory is tracked per page, there can be pages
1087 which are associated with different cgroups than the one the inode is
1088 associated with.  These are called foreign pages.  The writeback
1089 constantly keeps track of foreign pages and, if a particular foreign
1090 cgroup becomes the majority over a certain period of time, switches
1091 the ownership of the inode to that cgroup.
1092 
1093 While this model is enough for most use cases where a given inode is
1094 mostly dirtied by a single cgroup even when the main writing cgroup
1095 changes over time, use cases where multiple cgroups write to a single
1096 inode simultaneously are not supported well.  In such circumstances, a
1097 significant portion of IOs are likely to be attributed incorrectly.
1098 As memory controller assigns page ownership on the first use and
1099 doesn't update it until the page is released, even if writeback
1100 strictly follows page ownership, multiple cgroups dirtying overlapping
1101 areas wouldn't work as expected.  It's recommended to avoid such usage
1102 patterns.
1103 
1104 The sysctl knobs which affect writeback behavior are applied to cgroup
1105 writeback as follows.
1106 
1107   vm.dirty_background_ratio
1108   vm.dirty_ratio
1109 
1110         These ratios apply the same to cgroup writeback with the
1111         amount of available memory capped by limits imposed by the
1112         memory controller and system-wide clean memory.
1113 
1114   vm.dirty_background_bytes
1115   vm.dirty_bytes
1116 
1117         For cgroup writeback, this is calculated into ratio against
1118         total available memory and applied the same way as
1119         vm.dirty[_background]_ratio.
1120 
1121 
1122 6. Namespace
1123 
1124 6-1. Basics
1125 
1126 cgroup namespace provides a mechanism to virtualize the view of the
1127 "/proc/$PID/cgroup" file and cgroup mounts.  The CLONE_NEWCGROUP clone
1128 flag can be used with clone(2) and unshare(2) to create a new cgroup
1129 namespace.  The process running inside the cgroup namespace will have
1130 its "/proc/$PID/cgroup" output restricted to cgroupns root.  The
1131 cgroupns root is the cgroup of the process at the time of creation of
1132 the cgroup namespace.
1133 
1134 Without cgroup namespace, the "/proc/$PID/cgroup" file shows the
1135 complete path of the cgroup of a process.  In a container setup where
1136 a set of cgroups and namespaces are intended to isolate processes the
1137 "/proc/$PID/cgroup" file may leak potential system level information
1138 to the isolated processes.  For Example:
1139 
1140   # cat /proc/self/cgroup
1141   0::/batchjobs/container_id1
1142 
1143 The path '/batchjobs/container_id1' can be considered as system-data
1144 and undesirable to expose to the isolated processes.  cgroup namespace
1145 can be used to restrict visibility of this path.  For example, before
1146 creating a cgroup namespace, one would see:
1147 
1148   # ls -l /proc/self/ns/cgroup
1149   lrwxrwxrwx 1 root root 0 2014-07-15 10:37 /proc/self/ns/cgroup -> cgroup:[4026531835]
1150   # cat /proc/self/cgroup
1151   0::/batchjobs/container_id1
1152 
1153 After unsharing a new namespace, the view changes.
1154 
1155   # ls -l /proc/self/ns/cgroup
1156   lrwxrwxrwx 1 root root 0 2014-07-15 10:35 /proc/self/ns/cgroup -> cgroup:[4026532183]
1157   # cat /proc/self/cgroup
1158   0::/
1159 
1160 When some thread from a multi-threaded process unshares its cgroup
1161 namespace, the new cgroupns gets applied to the entire process (all
1162 the threads).  This is natural for the v2 hierarchy; however, for the
1163 legacy hierarchies, this may be unexpected.
1164 
1165 A cgroup namespace is alive as long as there are processes inside or
1166 mounts pinning it.  When the last usage goes away, the cgroup
1167 namespace is destroyed.  The cgroupns root and the actual cgroups
1168 remain.
1169 
1170 
1171 6-2. The Root and Views
1172 
1173 The 'cgroupns root' for a cgroup namespace is the cgroup in which the
1174 process calling unshare(2) is running.  For example, if a process in
1175 /batchjobs/container_id1 cgroup calls unshare, cgroup
1176 /batchjobs/container_id1 becomes the cgroupns root.  For the
1177 init_cgroup_ns, this is the real root ('/') cgroup.
1178 
1179 The cgroupns root cgroup does not change even if the namespace creator
1180 process later moves to a different cgroup.
1181 
1182   # ~/unshare -c # unshare cgroupns in some cgroup
1183   # cat /proc/self/cgroup
1184   0::/
1185   # mkdir sub_cgrp_1
1186   # echo 0 > sub_cgrp_1/cgroup.procs
1187   # cat /proc/self/cgroup
1188   0::/sub_cgrp_1
1189 
1190 Each process gets its namespace-specific view of "/proc/$PID/cgroup"
1191 
1192 Processes running inside the cgroup namespace will be able to see
1193 cgroup paths (in /proc/self/cgroup) only inside their root cgroup.
1194 From within an unshared cgroupns:
1195 
1196   # sleep 100000 &
1197   [1] 7353
1198   # echo 7353 > sub_cgrp_1/cgroup.procs
1199   # cat /proc/7353/cgroup
1200   0::/sub_cgrp_1
1201 
1202 From the initial cgroup namespace, the real cgroup path will be
1203 visible:
1204 
1205   $ cat /proc/7353/cgroup
1206   0::/batchjobs/container_id1/sub_cgrp_1
1207 
1208 From a sibling cgroup namespace (that is, a namespace rooted at a
1209 different cgroup), the cgroup path relative to its own cgroup
1210 namespace root will be shown.  For instance, if PID 7353's cgroup
1211 namespace root is at '/batchjobs/container_id2', then it will see
1212 
1213   # cat /proc/7353/cgroup
1214   0::/../container_id2/sub_cgrp_1
1215 
1216 Note that the relative path always starts with '/' to indicate that
1217 its relative to the cgroup namespace root of the caller.
1218 
1219 
1220 6-3. Migration and setns(2)
1221 
1222 Processes inside a cgroup namespace can move into and out of the
1223 namespace root if they have proper access to external cgroups.  For
1224 example, from inside a namespace with cgroupns root at
1225 /batchjobs/container_id1, and assuming that the global hierarchy is
1226 still accessible inside cgroupns:
1227 
1228   # cat /proc/7353/cgroup
1229   0::/sub_cgrp_1
1230   # echo 7353 > batchjobs/container_id2/cgroup.procs
1231   # cat /proc/7353/cgroup
1232   0::/../container_id2
1233 
1234 Note that this kind of setup is not encouraged.  A task inside cgroup
1235 namespace should only be exposed to its own cgroupns hierarchy.
1236 
1237 setns(2) to another cgroup namespace is allowed when:
1238 
1239 (a) the process has CAP_SYS_ADMIN against its current user namespace
1240 (b) the process has CAP_SYS_ADMIN against the target cgroup
1241     namespace's userns
1242 
1243 No implicit cgroup changes happen with attaching to another cgroup
1244 namespace.  It is expected that the someone moves the attaching
1245 process under the target cgroup namespace root.
1246 
1247 
1248 6-4. Interaction with Other Namespaces
1249 
1250 Namespace specific cgroup hierarchy can be mounted by a process
1251 running inside a non-init cgroup namespace.
1252 
1253   # mount -t cgroup2 none $MOUNT_POINT
1254 
1255 This will mount the unified cgroup hierarchy with cgroupns root as the
1256 filesystem root.  The process needs CAP_SYS_ADMIN against its user and
1257 mount namespaces.
1258 
1259 The virtualization of /proc/self/cgroup file combined with restricting
1260 the view of cgroup hierarchy by namespace-private cgroupfs mount
1261 provides a properly isolated cgroup view inside the container.
1262 
1263 
1264 P. Information on Kernel Programming
1265 
1266 This section contains kernel programming information in the areas
1267 where interacting with cgroup is necessary.  cgroup core and
1268 controllers are not covered.
1269 
1270 
1271 P-1. Filesystem Support for Writeback
1272 
1273 A filesystem can support cgroup writeback by updating
1274 address_space_operations->writepage[s]() to annotate bio's using the
1275 following two functions.
1276 
1277   wbc_init_bio(@wbc, @bio)
1278 
1279         Should be called for each bio carrying writeback data and
1280         associates the bio with the inode's owner cgroup.  Can be
1281         called anytime between bio allocation and submission.
1282 
1283   wbc_account_io(@wbc, @page, @bytes)
1284 
1285         Should be called for each data segment being written out.
1286         While this function doesn't care exactly when it's called
1287         during the writeback session, it's the easiest and most
1288         natural to call it as data segments are added to a bio.
1289 
1290 With writeback bio's annotated, cgroup support can be enabled per
1291 super_block by setting SB_I_CGROUPWB in ->s_iflags.  This allows for
1292 selective disabling of cgroup writeback support which is helpful when
1293 certain filesystem features, e.g. journaled data mode, are
1294 incompatible.
1295 
1296 wbc_init_bio() binds the specified bio to its cgroup.  Depending on
1297 the configuration, the bio may be executed at a lower priority and if
1298 the writeback session is holding shared resources, e.g. a journal
1299 entry, may lead to priority inversion.  There is no one easy solution
1300 for the problem.  Filesystems can try to work around specific problem
1301 cases by skipping wbc_init_bio() or using bio_associate_blkcg()
1302 directly.
1303 
1304 
1305 D. Deprecated v1 Core Features
1306 
1307 - Multiple hierarchies including named ones are not supported.
1308 
1309 - All mount options and remounting are not supported.
1310 
1311 - The "tasks" file is removed and "cgroup.procs" is not sorted.
1312 
1313 - "cgroup.clone_children" is removed.
1314 
1315 - /proc/cgroups is meaningless for v2.  Use "cgroup.controllers" file
1316   at the root instead.
1317 
1318 
1319 R. Issues with v1 and Rationales for v2
1320 
1321 R-1. Multiple Hierarchies
1322 
1323 cgroup v1 allowed an arbitrary number of hierarchies and each
1324 hierarchy could host any number of controllers.  While this seemed to
1325 provide a high level of flexibility, it wasn't useful in practice.
1326 
1327 For example, as there is only one instance of each controller, utility
1328 type controllers such as freezer which can be useful in all
1329 hierarchies could only be used in one.  The issue is exacerbated by
1330 the fact that controllers couldn't be moved to another hierarchy once
1331 hierarchies were populated.  Another issue was that all controllers
1332 bound to a hierarchy were forced to have exactly the same view of the
1333 hierarchy.  It wasn't possible to vary the granularity depending on
1334 the specific controller.
1335 
1336 In practice, these issues heavily limited which controllers could be
1337 put on the same hierarchy and most configurations resorted to putting
1338 each controller on its own hierarchy.  Only closely related ones, such
1339 as the cpu and cpuacct controllers, made sense to be put on the same
1340 hierarchy.  This often meant that userland ended up managing multiple
1341 similar hierarchies repeating the same steps on each hierarchy
1342 whenever a hierarchy management operation was necessary.
1343 
1344 Furthermore, support for multiple hierarchies came at a steep cost.
1345 It greatly complicated cgroup core implementation but more importantly
1346 the support for multiple hierarchies restricted how cgroup could be
1347 used in general and what controllers was able to do.
1348 
1349 There was no limit on how many hierarchies there might be, which meant
1350 that a thread's cgroup membership couldn't be described in finite
1351 length.  The key might contain any number of entries and was unlimited
1352 in length, which made it highly awkward to manipulate and led to
1353 addition of controllers which existed only to identify membership,
1354 which in turn exacerbated the original problem of proliferating number
1355 of hierarchies.
1356 
1357 Also, as a controller couldn't have any expectation regarding the
1358 topologies of hierarchies other controllers might be on, each
1359 controller had to assume that all other controllers were attached to
1360 completely orthogonal hierarchies.  This made it impossible, or at
1361 least very cumbersome, for controllers to cooperate with each other.
1362 
1363 In most use cases, putting controllers on hierarchies which are
1364 completely orthogonal to each other isn't necessary.  What usually is
1365 called for is the ability to have differing levels of granularity
1366 depending on the specific controller.  In other words, hierarchy may
1367 be collapsed from leaf towards root when viewed from specific
1368 controllers.  For example, a given configuration might not care about
1369 how memory is distributed beyond a certain level while still wanting
1370 to control how CPU cycles are distributed.
1371 
1372 
1373 R-2. Thread Granularity
1374 
1375 cgroup v1 allowed threads of a process to belong to different cgroups.
1376 This didn't make sense for some controllers and those controllers
1377 ended up implementing different ways to ignore such situations but
1378 much more importantly it blurred the line between API exposed to
1379 individual applications and system management interface.
1380 
1381 Generally, in-process knowledge is available only to the process
1382 itself; thus, unlike service-level organization of processes,
1383 categorizing threads of a process requires active participation from
1384 the application which owns the target process.
1385 
1386 cgroup v1 had an ambiguously defined delegation model which got abused
1387 in combination with thread granularity.  cgroups were delegated to
1388 individual applications so that they can create and manage their own
1389 sub-hierarchies and control resource distributions along them.  This
1390 effectively raised cgroup to the status of a syscall-like API exposed
1391 to lay programs.
1392 
1393 First of all, cgroup has a fundamentally inadequate interface to be
1394 exposed this way.  For a process to access its own knobs, it has to
1395 extract the path on the target hierarchy from /proc/self/cgroup,
1396 construct the path by appending the name of the knob to the path, open
1397 and then read and/or write to it.  This is not only extremely clunky
1398 and unusual but also inherently racy.  There is no conventional way to
1399 define transaction across the required steps and nothing can guarantee
1400 that the process would actually be operating on its own sub-hierarchy.
1401 
1402 cgroup controllers implemented a number of knobs which would never be
1403 accepted as public APIs because they were just adding control knobs to
1404 system-management pseudo filesystem.  cgroup ended up with interface
1405 knobs which were not properly abstracted or refined and directly
1406 revealed kernel internal details.  These knobs got exposed to
1407 individual applications through the ill-defined delegation mechanism
1408 effectively abusing cgroup as a shortcut to implementing public APIs
1409 without going through the required scrutiny.
1410 
1411 This was painful for both userland and kernel.  Userland ended up with
1412 misbehaving and poorly abstracted interfaces and kernel exposing and
1413 locked into constructs inadvertently.
1414 
1415 
1416 R-3. Competition Between Inner Nodes and Threads
1417 
1418 cgroup v1 allowed threads to be in any cgroups which created an
1419 interesting problem where threads belonging to a parent cgroup and its
1420 children cgroups competed for resources.  This was nasty as two
1421 different types of entities competed and there was no obvious way to
1422 settle it.  Different controllers did different things.
1423 
1424 The cpu controller considered threads and cgroups as equivalents and
1425 mapped nice levels to cgroup weights.  This worked for some cases but
1426 fell flat when children wanted to be allocated specific ratios of CPU
1427 cycles and the number of internal threads fluctuated - the ratios
1428 constantly changed as the number of competing entities fluctuated.
1429 There also were other issues.  The mapping from nice level to weight
1430 wasn't obvious or universal, and there were various other knobs which
1431 simply weren't available for threads.
1432 
1433 The io controller implicitly created a hidden leaf node for each
1434 cgroup to host the threads.  The hidden leaf had its own copies of all
1435 the knobs with "leaf_" prefixed.  While this allowed equivalent
1436 control over internal threads, it was with serious drawbacks.  It
1437 always added an extra layer of nesting which wouldn't be necessary
1438 otherwise, made the interface messy and significantly complicated the
1439 implementation.
1440 
1441 The memory controller didn't have a way to control what happened
1442 between internal tasks and child cgroups and the behavior was not
1443 clearly defined.  There were attempts to add ad-hoc behaviors and
1444 knobs to tailor the behavior to specific workloads which would have
1445 led to problems extremely difficult to resolve in the long term.
1446 
1447 Multiple controllers struggled with internal tasks and came up with
1448 different ways to deal with it; unfortunately, all the approaches were
1449 severely flawed and, furthermore, the widely different behaviors
1450 made cgroup as a whole highly inconsistent.
1451 
1452 This clearly is a problem which needs to be addressed from cgroup core
1453 in a uniform way.
1454 
1455 
1456 R-4. Other Interface Issues
1457 
1458 cgroup v1 grew without oversight and developed a large number of
1459 idiosyncrasies and inconsistencies.  One issue on the cgroup core side
1460 was how an empty cgroup was notified - a userland helper binary was
1461 forked and executed for each event.  The event delivery wasn't
1462 recursive or delegatable.  The limitations of the mechanism also led
1463 to in-kernel event delivery filtering mechanism further complicating
1464 the interface.
1465 
1466 Controller interfaces were problematic too.  An extreme example is
1467 controllers completely ignoring hierarchical organization and treating
1468 all cgroups as if they were all located directly under the root
1469 cgroup.  Some controllers exposed a large amount of inconsistent
1470 implementation details to userland.
1471 
1472 There also was no consistency across controllers.  When a new cgroup
1473 was created, some controllers defaulted to not imposing extra
1474 restrictions while others disallowed any resource usage until
1475 explicitly configured.  Configuration knobs for the same type of
1476 control used widely differing naming schemes and formats.  Statistics
1477 and information knobs were named arbitrarily and used different
1478 formats and units even in the same controller.
1479 
1480 cgroup v2 establishes common conventions where appropriate and updates
1481 controllers so that they expose minimal and consistent interfaces.
1482 
1483 
1484 R-5. Controller Issues and Remedies
1485 
1486 R-5-1. Memory
1487 
1488 The original lower boundary, the soft limit, is defined as a limit
1489 that is per default unset.  As a result, the set of cgroups that
1490 global reclaim prefers is opt-in, rather than opt-out.  The costs for
1491 optimizing these mostly negative lookups are so high that the
1492 implementation, despite its enormous size, does not even provide the
1493 basic desirable behavior.  First off, the soft limit has no
1494 hierarchical meaning.  All configured groups are organized in a global
1495 rbtree and treated like equal peers, regardless where they are located
1496 in the hierarchy.  This makes subtree delegation impossible.  Second,
1497 the soft limit reclaim pass is so aggressive that it not just
1498 introduces high allocation latencies into the system, but also impacts
1499 system performance due to overreclaim, to the point where the feature
1500 becomes self-defeating.
1501 
1502 The memory.low boundary on the other hand is a top-down allocated
1503 reserve.  A cgroup enjoys reclaim protection when it and all its
1504 ancestors are below their low boundaries, which makes delegation of
1505 subtrees possible.  Secondly, new cgroups have no reserve per default
1506 and in the common case most cgroups are eligible for the preferred
1507 reclaim pass.  This allows the new low boundary to be efficiently
1508 implemented with just a minor addition to the generic reclaim code,
1509 without the need for out-of-band data structures and reclaim passes.
1510 Because the generic reclaim code considers all cgroups except for the
1511 ones running low in the preferred first reclaim pass, overreclaim of
1512 individual groups is eliminated as well, resulting in much better
1513 overall workload performance.
1514 
1515 The original high boundary, the hard limit, is defined as a strict
1516 limit that can not budge, even if the OOM killer has to be called.
1517 But this generally goes against the goal of making the most out of the
1518 available memory.  The memory consumption of workloads varies during
1519 runtime, and that requires users to overcommit.  But doing that with a
1520 strict upper limit requires either a fairly accurate prediction of the
1521 working set size or adding slack to the limit.  Since working set size
1522 estimation is hard and error prone, and getting it wrong results in
1523 OOM kills, most users tend to err on the side of a looser limit and
1524 end up wasting precious resources.
1525 
1526 The memory.high boundary on the other hand can be set much more
1527 conservatively.  When hit, it throttles allocations by forcing them
1528 into direct reclaim to work off the excess, but it never invokes the
1529 OOM killer.  As a result, a high boundary that is chosen too
1530 aggressively will not terminate the processes, but instead it will
1531 lead to gradual performance degradation.  The user can monitor this
1532 and make corrections until the minimal memory footprint that still
1533 gives acceptable performance is found.
1534 
1535 In extreme cases, with many concurrent allocations and a complete
1536 breakdown of reclaim progress within the group, the high boundary can
1537 be exceeded.  But even then it's mostly better to satisfy the
1538 allocation from the slack available in other groups or the rest of the
1539 system than killing the group.  Otherwise, memory.max is there to
1540 limit this type of spillover and ultimately contain buggy or even
1541 malicious applications.
1542 
1543 Setting the original memory.limit_in_bytes below the current usage was
1544 subject to a race condition, where concurrent charges could cause the
1545 limit setting to fail. memory.max on the other hand will first set the
1546 limit to prevent new charges, and then reclaim and OOM kill until the
1547 new limit is met - or the task writing to memory.max is killed.
1548 
1549 The combined memory+swap accounting and limiting is replaced by real
1550 control over swap space.
1551 
1552 The main argument for a combined memory+swap facility in the original
1553 cgroup design was that global or parental pressure would always be
1554 able to swap all anonymous memory of a child group, regardless of the
1555 child's own (possibly untrusted) configuration.  However, untrusted
1556 groups can sabotage swapping by other means - such as referencing its
1557 anonymous memory in a tight loop - and an admin can not assume full
1558 swappability when overcommitting untrusted jobs.
1559 
1560 For trusted jobs, on the other hand, a combined counter is not an
1561 intuitive userspace interface, and it flies in the face of the idea
1562 that cgroup controllers should account and limit specific physical
1563 resources.  Swap space is a resource like all others in the system,
1564 and that's why unified hierarchy allows distributing it separately.