0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 =================
0004 PCI NTB Function
0005 =================
0006
0007 :Author: Kishon Vijay Abraham I <kishon@ti.com>
0008
0009 PCI Non-Transparent Bridges (NTB) allow two host systems to communicate
0010 with each other by exposing each host as a device to the other host.
0011 NTBs typically support the ability to generate interrupts on the remote
0012 machine, expose memory ranges as BARs, and perform DMA. They also support
0013 scratchpads, which are areas of memory within the NTB that are accessible
0014 from both machines.
0015
0016 PCI NTB Function allows two different systems (or hosts) to communicate
0017 with each other by configuring the endpoint instances in such a way that
0018 transactions from one system are routed to the other system.
0019
0020 In the below diagram, PCI NTB function configures the SoC with multiple
0021 PCI Endpoint (EP) instances in such a way that transactions from one EP
0022 controller are routed to the other EP controller. Once PCI NTB function
0023 configures the SoC with multiple EP instances, HOST1 and HOST2 can
0024 communicate with each other using SoC as a bridge.
0025
0026 .. code-block:: text
0027
0028 +-------------+ +-------------+
0029 | | | |
0030 | HOST1 | | HOST2 |
0031 | | | |
0032 +------^------+ +------^------+
0033 | |
0034 | |
0035 +---------|-------------------------------------------------|---------+
0036 | +------v------+ +------v------+ |
0037 | | | | | |
0038 | | EP | | EP | |
0039 | | CONTROLLER1 | | CONTROLLER2 | |
0040 | | <-----------------------------------> | |
0041 | | | | | |
0042 | | | | | |
0043 | | | SoC With Multiple EP Instances | | |
0044 | | | (Configured using NTB Function) | | |
0045 | +-------------+ +-------------+ |
0046 +---------------------------------------------------------------------+
0047
0048 Constructs used for Implementing NTB
0049 ====================================
0050
0051 1) Config Region
0052 2) Self Scratchpad Registers
0053 3) Peer Scratchpad Registers
0054 4) Doorbell (DB) Registers
0055 5) Memory Window (MW)
0056
0057
0058 Config Region:
0059 --------------
0060
0061 Config Region is a construct that is specific to NTB implemented using NTB
0062 Endpoint Function Driver. The host and endpoint side NTB function driver will
0063 exchange information with each other using this region. Config Region has
0064 Control/Status Registers for configuring the Endpoint Controller. Host can
0065 write into this region for configuring the outbound Address Translation Unit
0066 (ATU) and to indicate the link status. Endpoint can indicate the status of
0067 commands issued by host in this region. Endpoint can also indicate the
0068 scratchpad offset and number of memory windows to the host using this region.
0069
0070 The format of Config Region is given below. All the fields here are 32 bits.
0071
0072 .. code-block:: text
0073
0074 +------------------------+
0075 | COMMAND |
0076 +------------------------+
0077 | ARGUMENT |
0078 +------------------------+
0079 | STATUS |
0080 +------------------------+
0081 | TOPOLOGY |
0082 +------------------------+
0083 | ADDRESS (LOWER 32) |
0084 +------------------------+
0085 | ADDRESS (UPPER 32) |
0086 +------------------------+
0087 | SIZE |
0088 +------------------------+
0089 | NO OF MEMORY WINDOW |
0090 +------------------------+
0091 | MEMORY WINDOW1 OFFSET |
0092 +------------------------+
0093 | SPAD OFFSET |
0094 +------------------------+
0095 | SPAD COUNT |
0096 +------------------------+
0097 | DB ENTRY SIZE |
0098 +------------------------+
0099 | DB DATA |
0100 +------------------------+
0101 | : |
0102 +------------------------+
0103 | : |
0104 +------------------------+
0105 | DB DATA |
0106 +------------------------+
0107
0108
0109 COMMAND:
0110
0111 NTB function supports three commands:
0112
0113 CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before
0114 invoking this command, the host should allocate and initialize
0115 MSI/MSI-X vectors (i.e., initialize the MSI/MSI-X Capability in the
0116 Endpoint). The endpoint on receiving this command will configure
0117 the outbound ATU such that transactions to Doorbell BAR will be routed
0118 to the MSI/MSI-X address programmed by the host. The ARGUMENT
0119 register should be populated with number of DBs to configure (in the
0120 lower 16 bits) and if MSI or MSI-X should be configured (BIT 16).
0121
0122 CMD_CONFIGURE_MW (0x2): Command to configure memory window (MW). The
0123 host invokes this command after allocating a buffer that can be
0124 accessed by remote host. The allocated address should be programmed
0125 in the ADDRESS register (64 bit), the size should be programmed in
0126 the SIZE register and the memory window index should be programmed
0127 in the ARGUMENT register. The endpoint on receiving this command
0128 will configure the outbound ATU such that transactions to MW BAR
0129 are routed to the address provided by the host.
0130
0131 CMD_LINK_UP (0x3): Command to indicate an NTB application is
0132 bound to the EP device on the host side. Once the endpoint
0133 receives this command from both the hosts, the endpoint will
0134 raise a LINK_UP event to both the hosts to indicate the host
0135 NTB applications can start communicating with each other.
0136
0137 ARGUMENT:
0138
0139 The value of this register is based on the commands issued in
0140 command register. See COMMAND section for more information.
0141
0142 TOPOLOGY:
0143
0144 Set to NTB_TOPO_B2B_USD for Primary interface
0145 Set to NTB_TOPO_B2B_DSD for Secondary interface
0146
0147 ADDRESS/SIZE:
0148
0149 Address and Size to be used while configuring the memory window.
0150 See "CMD_CONFIGURE_MW" for more info.
0151
0152 MEMORY WINDOW1 OFFSET:
0153
0154 Memory Window 1 and Doorbell registers are packed together in the
0155 same BAR. The initial portion of the region will have doorbell
0156 registers and the latter portion of the region is for memory window 1.
0157 This register will specify the offset of the memory window 1.
0158
0159 NO OF MEMORY WINDOW:
0160
0161 Specifies the number of memory windows supported by the NTB device.
0162
0163 SPAD OFFSET:
0164
0165 Self scratchpad region and config region are packed together in the
0166 same BAR. The initial portion of the region will have config region
0167 and the latter portion of the region is for self scratchpad. This
0168 register will specify the offset of the self scratchpad registers.
0169
0170 SPAD COUNT:
0171
0172 Specifies the number of scratchpad registers supported by the NTB
0173 device.
0174
0175 DB ENTRY SIZE:
0176
0177 Used to determine the offset within the DB BAR that should be written
0178 in order to raise doorbell. EPF NTB can use either MSI or MSI-X to
0179 ring doorbell (MSI-X support will be added later). MSI uses same
0180 address for all the interrupts and MSI-X can provide different
0181 addresses for different interrupts. The MSI/MSI-X address is provided
0182 by the host and the address it gives is based on the MSI/MSI-X
0183 implementation supported by the host. For instance, ARM platform
0184 using GIC ITS will have the same MSI-X address for all the interrupts.
0185 In order to support all the combinations and use the same mechanism
0186 for both MSI and MSI-X, EPF NTB allocates a separate region in the
0187 Outbound Address Space for each of the interrupts. This region will
0188 be mapped to the MSI/MSI-X address provided by the host. If a host
0189 provides the same address for all the interrupts, all the regions
0190 will be translated to the same address. If a host provides different
0191 addresses, the regions will be translated to different addresses. This
0192 will ensure there is no difference while raising the doorbell.
0193
0194 DB DATA:
0195
0196 EPF NTB supports 32 interrupts, so there are 32 DB DATA registers.
0197 This holds the MSI/MSI-X data that has to be written to MSI address
0198 for raising doorbell interrupt. This will be populated by EPF NTB
0199 while invoking CMD_CONFIGURE_DOORBELL.
0200
0201 Scratchpad Registers:
0202 ---------------------
0203
0204 Each host has its own register space allocated in the memory of NTB endpoint
0205 controller. They are both readable and writable from both sides of the bridge.
0206 They are used by applications built over NTB and can be used to pass control
0207 and status information between both sides of a device.
0208
0209 Scratchpad registers has 2 parts
0210 1) Self Scratchpad: Host's own register space
0211 2) Peer Scratchpad: Remote host's register space.
0212
0213 Doorbell Registers:
0214 -------------------
0215
0216 Doorbell Registers are used by the hosts to interrupt each other.
0217
0218 Memory Window:
0219 --------------
0220
0221 Actual transfer of data between the two hosts will happen using the
0222 memory window.
0223
0224 Modeling Constructs:
0225 ====================
0226
0227 There are 5 or more distinct regions (config, self scratchpad, peer
0228 scratchpad, doorbell, one or more memory windows) to be modeled to achieve
0229 NTB functionality. At least one memory window is required while more than
0230 one is permitted. All these regions should be mapped to BARs for hosts to
0231 access these regions.
0232
0233 If one 32-bit BAR is allocated for each of these regions, the scheme would
0234 look like this:
0235
0236 ====== ===============
0237 BAR NO CONSTRUCTS USED
0238 ====== ===============
0239 BAR0 Config Region
0240 BAR1 Self Scratchpad
0241 BAR2 Peer Scratchpad
0242 BAR3 Doorbell
0243 BAR4 Memory Window 1
0244 BAR5 Memory Window 2
0245 ====== ===============
0246
0247 However if we allocate a separate BAR for each of the regions, there would not
0248 be enough BARs for all the regions in a platform that supports only 64-bit
0249 BARs.
0250
0251 In order to be supported by most of the platforms, the regions should be
0252 packed and mapped to BARs in a way that provides NTB functionality and
0253 also makes sure the host doesn't access any region that it is not supposed
0254 to.
0255
0256 The following scheme is used in EPF NTB Function:
0257
0258 ====== ===============================
0259 BAR NO CONSTRUCTS USED
0260 ====== ===============================
0261 BAR0 Config Region + Self Scratchpad
0262 BAR1 Peer Scratchpad
0263 BAR2 Doorbell + Memory Window 1
0264 BAR3 Memory Window 2
0265 BAR4 Memory Window 3
0266 BAR5 Memory Window 4
0267 ====== ===============================
0268
0269 With this scheme, for the basic NTB functionality 3 BARs should be sufficient.
0270
0271 Modeling Config/Scratchpad Region:
0272 ----------------------------------
0273
0274 .. code-block:: text
0275
0276 +-----------------+------->+------------------+ +-----------------+
0277 | BAR0 | | CONFIG REGION | | BAR0 |
0278 +-----------------+----+ +------------------+<-------+-----------------+
0279 | BAR1 | | |SCRATCHPAD REGION | | BAR1 |
0280 +-----------------+ +-->+------------------+<-------+-----------------+
0281 | BAR2 | Local Memory | BAR2 |
0282 +-----------------+ +-----------------+
0283 | BAR3 | | BAR3 |
0284 +-----------------+ +-----------------+
0285 | BAR4 | | BAR4 |
0286 +-----------------+ +-----------------+
0287 | BAR5 | | BAR5 |
0288 +-----------------+ +-----------------+
0289 EP CONTROLLER 1 EP CONTROLLER 2
0290
0291 Above diagram shows Config region + Scratchpad region for HOST1 (connected to
0292 EP controller 1) allocated in local memory. The HOST1 can access the config
0293 region and scratchpad region (self scratchpad) using BAR0 of EP controller 1.
0294 The peer host (HOST2 connected to EP controller 2) can also access this
0295 scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This
0296 diagram shows the case where Config region and Scratchpad regions are allocated
0297 for HOST1, however the same is applicable for HOST2.
0298
0299 Modeling Doorbell/Memory Window 1:
0300 ----------------------------------
0301
0302 .. code-block:: text
0303
0304 +-----------------+ +----->+----------------+-----------+-----------------+
0305 | BAR0 | | | Doorbell 1 +-----------> MSI-X ADDRESS 1 |
0306 +-----------------+ | +----------------+ +-----------------+
0307 | BAR1 | | | Doorbell 2 +---------+ | |
0308 +-----------------+----+ +----------------+ | | |
0309 | BAR2 | | Doorbell 3 +-------+ | +-----------------+
0310 +-----------------+----+ +----------------+ | +-> MSI-X ADDRESS 2 |
0311 | BAR3 | | | Doorbell 4 +-----+ | +-----------------+
0312 +-----------------+ | |----------------+ | | | |
0313 | BAR4 | | | | | | +-----------------+
0314 +-----------------+ | | MW1 +---+ | +-->+ MSI-X ADDRESS 3||
0315 | BAR5 | | | | | | +-----------------+
0316 +-----------------+ +----->-----------------+ | | | |
0317 EP CONTROLLER 1 | | | | +-----------------+
0318 | | | +---->+ MSI-X ADDRESS 4 |
0319 +----------------+ | +-----------------+
0320 EP CONTROLLER 2 | | |
0321 (OB SPACE) | | |
0322 +-------> MW1 |
0323 | |
0324 | |
0325 +-----------------+
0326 | |
0327 | |
0328 | |
0329 | |
0330 | |
0331 +-----------------+
0332 PCI Address Space
0333 (Managed by HOST2)
0334
0335 Above diagram shows how the doorbell and memory window 1 is mapped so that
0336 HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access
0337 buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and
0338 memory window 1 regions are allocated in EP controller 2 outbound (OB) address
0339 space. Allocating and configuring BARs for doorbell and memory window1
0340 is done during the initialization phase of NTB endpoint function driver.
0341 Mapping from EP controller 2 OB space to PCI address space is done when HOST2
0342 sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL.
0343
0344 Modeling Optional Memory Windows:
0345 ---------------------------------
0346
0347 This is modeled the same was as MW1 but each of the additional memory windows
0348 is mapped to separate BARs.
