0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ============
0004 x86 Topology
0005 ============
0006
0007 This documents and clarifies the main aspects of x86 topology modelling and
0008 representation in the kernel. Update/change when doing changes to the
0009 respective code.
0010
0011 The architecture-agnostic topology definitions are in
0012 Documentation/admin-guide/cputopology.rst. This file holds x86-specific
0013 differences/specialities which must not necessarily apply to the generic
0014 definitions. Thus, the way to read up on Linux topology on x86 is to start
0015 with the generic one and look at this one in parallel for the x86 specifics.
0016
0017 Needless to say, code should use the generic functions - this file is *only*
0018 here to *document* the inner workings of x86 topology.
0019
0020 Started by Thomas Gleixner <tglx@linutronix.de> and Borislav Petkov <bp@alien8.de>.
0021
0022 The main aim of the topology facilities is to present adequate interfaces to
0023 code which needs to know/query/use the structure of the running system wrt
0024 threads, cores, packages, etc.
0025
0026 The kernel does not care about the concept of physical sockets because a
0027 socket has no relevance to software. It's an electromechanical component. In
0028 the past a socket always contained a single package (see below), but with the
0029 advent of Multi Chip Modules (MCM) a socket can hold more than one package. So
0030 there might be still references to sockets in the code, but they are of
0031 historical nature and should be cleaned up.
0032
0033 The topology of a system is described in the units of:
0034
0035 - packages
0036 - cores
0037 - threads
0038
0039 Package
0040 =======
0041 Packages contain a number of cores plus shared resources, e.g. DRAM
0042 controller, shared caches etc.
0043
0044 Modern systems may also use the term 'Die' for package.
0045
0046 AMD nomenclature for package is 'Node'.
0047
0048 Package-related topology information in the kernel:
0049
0050 - cpuinfo_x86.x86_max_cores:
0051
0052 The number of cores in a package. This information is retrieved via CPUID.
0053
0054 - cpuinfo_x86.x86_max_dies:
0055
0056 The number of dies in a package. This information is retrieved via CPUID.
0057
0058 - cpuinfo_x86.cpu_die_id:
0059
0060 The physical ID of the die. This information is retrieved via CPUID.
0061
0062 - cpuinfo_x86.phys_proc_id:
0063
0064 The physical ID of the package. This information is retrieved via CPUID
0065 and deduced from the APIC IDs of the cores in the package.
0066
0067 Modern systems use this value for the socket. There may be multiple
0068 packages within a socket. This value may differ from cpu_die_id.
0069
0070 - cpuinfo_x86.logical_proc_id:
0071
0072 The logical ID of the package. As we do not trust BIOSes to enumerate the
0073 packages in a consistent way, we introduced the concept of logical package
0074 ID so we can sanely calculate the number of maximum possible packages in
0075 the system and have the packages enumerated linearly.
0076
0077 - topology_max_packages():
0078
0079 The maximum possible number of packages in the system. Helpful for per
0080 package facilities to preallocate per package information.
0081
0082 - cpu_llc_id:
0083
0084 A per-CPU variable containing:
0085
0086 - On Intel, the first APIC ID of the list of CPUs sharing the Last Level
0087 Cache
0088
0089 - On AMD, the Node ID or Core Complex ID containing the Last Level
0090 Cache. In general, it is a number identifying an LLC uniquely on the
0091 system.
0092
0093 Cores
0094 =====
0095 A core consists of 1 or more threads. It does not matter whether the threads
0096 are SMT- or CMT-type threads.
0097
0098 AMDs nomenclature for a CMT core is "Compute Unit". The kernel always uses
0099 "core".
0100
0101 Core-related topology information in the kernel:
0102
0103 - smp_num_siblings:
0104
0105 The number of threads in a core. The number of threads in a package can be
0106 calculated by::
0107
0108 threads_per_package = cpuinfo_x86.x86_max_cores * smp_num_siblings
0109
0110
0111 Threads
0112 =======
0113 A thread is a single scheduling unit. It's the equivalent to a logical Linux
0114 CPU.
0115
0116 AMDs nomenclature for CMT threads is "Compute Unit Core". The kernel always
0117 uses "thread".
0118
0119 Thread-related topology information in the kernel:
0120
0121 - topology_core_cpumask():
0122
0123 The cpumask contains all online threads in the package to which a thread
0124 belongs.
0125
0126 The number of online threads is also printed in /proc/cpuinfo "siblings."
0127
0128 - topology_sibling_cpumask():
0129
0130 The cpumask contains all online threads in the core to which a thread
0131 belongs.
0132
0133 - topology_logical_package_id():
0134
0135 The logical package ID to which a thread belongs.
0136
0137 - topology_physical_package_id():
0138
0139 The physical package ID to which a thread belongs.
0140
0141 - topology_core_id();
0142
0143 The ID of the core to which a thread belongs. It is also printed in /proc/cpuinfo
0144 "core_id."
0145
0146
0147
0148 System topology examples
0149 ========================
0150
0151 .. note::
0152 The alternative Linux CPU enumeration depends on how the BIOS enumerates the
0153 threads. Many BIOSes enumerate all threads 0 first and then all threads 1.
0154 That has the "advantage" that the logical Linux CPU numbers of threads 0 stay
0155 the same whether threads are enabled or not. That's merely an implementation
0156 detail and has no practical impact.
0157
0158 1) Single Package, Single Core::
0159
0160 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0161
0162 2) Single Package, Dual Core
0163
0164 a) One thread per core::
0165
0166 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0167 -> [core 1] -> [thread 0] -> Linux CPU 1
0168
0169 b) Two threads per core::
0170
0171 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0172 -> [thread 1] -> Linux CPU 1
0173 -> [core 1] -> [thread 0] -> Linux CPU 2
0174 -> [thread 1] -> Linux CPU 3
0175
0176 Alternative enumeration::
0177
0178 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0179 -> [thread 1] -> Linux CPU 2
0180 -> [core 1] -> [thread 0] -> Linux CPU 1
0181 -> [thread 1] -> Linux CPU 3
0182
0183 AMD nomenclature for CMT systems::
0184
0185 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
0186 -> [Compute Unit Core 1] -> Linux CPU 1
0187 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
0188 -> [Compute Unit Core 1] -> Linux CPU 3
0189
0190 4) Dual Package, Dual Core
0191
0192 a) One thread per core::
0193
0194 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0195 -> [core 1] -> [thread 0] -> Linux CPU 1
0196
0197 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
0198 -> [core 1] -> [thread 0] -> Linux CPU 3
0199
0200 b) Two threads per core::
0201
0202 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0203 -> [thread 1] -> Linux CPU 1
0204 -> [core 1] -> [thread 0] -> Linux CPU 2
0205 -> [thread 1] -> Linux CPU 3
0206
0207 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 4
0208 -> [thread 1] -> Linux CPU 5
0209 -> [core 1] -> [thread 0] -> Linux CPU 6
0210 -> [thread 1] -> Linux CPU 7
0211
0212 Alternative enumeration::
0213
0214 [package 0] -> [core 0] -> [thread 0] -> Linux CPU 0
0215 -> [thread 1] -> Linux CPU 4
0216 -> [core 1] -> [thread 0] -> Linux CPU 1
0217 -> [thread 1] -> Linux CPU 5
0218
0219 [package 1] -> [core 0] -> [thread 0] -> Linux CPU 2
0220 -> [thread 1] -> Linux CPU 6
0221 -> [core 1] -> [thread 0] -> Linux CPU 3
0222 -> [thread 1] -> Linux CPU 7
0223
0224 AMD nomenclature for CMT systems::
0225
0226 [node 0] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 0
0227 -> [Compute Unit Core 1] -> Linux CPU 1
0228 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 2
0229 -> [Compute Unit Core 1] -> Linux CPU 3
0230
0231 [node 1] -> [Compute Unit 0] -> [Compute Unit Core 0] -> Linux CPU 4
0232 -> [Compute Unit Core 1] -> Linux CPU 5
0233 -> [Compute Unit 1] -> [Compute Unit Core 0] -> Linux CPU 6
0234 -> [Compute Unit Core 1] -> Linux CPU 7
