0001 ============================
0002 NUMA resource associativity
0003 ============================
0004
0005 Associativity represents the groupings of the various platform resources into
0006 domains of substantially similar mean performance relative to resources outside
0007 of that domain. Resources subsets of a given domain that exhibit better
0008 performance relative to each other than relative to other resources subsets
0009 are represented as being members of a sub-grouping domain. This performance
0010 characteristic is presented in terms of NUMA node distance within the Linux kernel.
0011 From the platform view, these groups are also referred to as domains.
0012
0013 PAPR interface currently supports different ways of communicating these resource
0014 grouping details to the OS. These are referred to as Form 0, Form 1 and Form2
0015 associativity grouping. Form 0 is the oldest format and is now considered deprecated.
0016
0017 Hypervisor indicates the type/form of associativity used via "ibm,architecture-vec-5 property".
0018 Bit 0 of byte 5 in the "ibm,architecture-vec-5" property indicates usage of Form 0 or Form 1.
0019 A value of 1 indicates the usage of Form 1 associativity. For Form 2 associativity
0020 bit 2 of byte 5 in the "ibm,architecture-vec-5" property is used.
0021
0022 Form 0
0023 ------
0024 Form 0 associativity supports only two NUMA distances (LOCAL and REMOTE).
0025
0026 Form 1
0027 ------
0028 With Form 1 a combination of ibm,associativity-reference-points, and ibm,associativity
0029 device tree properties are used to determine the NUMA distance between resource groups/domains.
0030
0031 The “ibm,associativity” property contains a list of one or more numbers (domainID)
0032 representing the resource’s platform grouping domains.
0033
0034 The “ibm,associativity-reference-points” property contains a list of one or more numbers
0035 (domainID index) that represents the 1 based ordinal in the associativity lists.
0036 The list of domainID indexes represents an increasing hierarchy of resource grouping.
0037
0038 ex:
0039 { primary domainID index, secondary domainID index, tertiary domainID index.. }
0040
0041 Linux kernel uses the domainID at the primary domainID index as the NUMA node id.
0042 Linux kernel computes NUMA distance between two domains by recursively comparing
0043 if they belong to the same higher-level domains. For mismatch at every higher
0044 level of the resource group, the kernel doubles the NUMA distance between the
0045 comparing domains.
0046
0047 Form 2
0048 -------
0049 Form 2 associativity format adds separate device tree properties representing NUMA node distance
0050 thereby making the node distance computation flexible. Form 2 also allows flexible primary
0051 domain numbering. With numa distance computation now detached from the index value in
0052 "ibm,associativity-reference-points" property, Form 2 allows a large number of primary domain
0053 ids at the same domainID index representing resource groups of different performance/latency
0054 characteristics.
0055
0056 Hypervisor indicates the usage of FORM2 associativity using bit 2 of byte 5 in the
0057 "ibm,architecture-vec-5" property.
0058
0059 "ibm,numa-lookup-index-table" property contains a list of one or more numbers representing
0060 the domainIDs present in the system. The offset of the domainID in this property is
0061 used as an index while computing numa distance information via "ibm,numa-distance-table".
0062
0063 prop-encoded-array: The number N of the domainIDs encoded as with encode-int, followed by
0064 N domainID encoded as with encode-int
0065
0066 For ex:
0067 "ibm,numa-lookup-index-table" = {4, 0, 8, 250, 252}. The offset of domainID 8 (2) is used when
0068 computing the distance of domain 8 from other domains present in the system. For the rest of
0069 this document, this offset will be referred to as domain distance offset.
0070
0071 "ibm,numa-distance-table" property contains a list of one or more numbers representing the NUMA
0072 distance between resource groups/domains present in the system.
0073
0074 prop-encoded-array: The number N of the distance values encoded as with encode-int, followed by
0075 N distance values encoded as with encode-bytes. The max distance value we could encode is 255.
0076 The number N must be equal to the square of m where m is the number of domainIDs in the
0077 numa-lookup-index-table.
0078
0079 For ex:
0080 ibm,numa-lookup-index-table = <3 0 8 40>;
0081 ibm,numa-distace-table = <9>, /bits/ 8 < 10 20 80 20 10 160 80 160 10>;
0082
0083 ::
0084
0085 | 0 8 40
0086 --|------------
0087 |
0088 0 | 10 20 80
0089 |
0090 8 | 20 10 160
0091 |
0092 40| 80 160 10
0093
0094 A possible "ibm,associativity" property for resources in node 0, 8 and 40
0095
0096 { 3, 6, 7, 0 }
0097 { 3, 6, 9, 8 }
0098 { 3, 6, 7, 40}
0099
0100 With "ibm,associativity-reference-points" { 0x3 }
0101
0102 "ibm,lookup-index-table" helps in having a compact representation of distance matrix.
0103 Since domainID can be sparse, the matrix of distances can also be effectively sparse.
0104 With "ibm,lookup-index-table" we can achieve a compact representation of
0105 distance information.
