0001 .. include:: <isonum.txt>
0002
0003 =========================================================
0004 DPAA2 (Data Path Acceleration Architecture Gen2) Overview
0005 =========================================================
0006
0007 :Copyright: |copy| 2015 Freescale Semiconductor Inc.
0008 :Copyright: |copy| 2018 NXP
0009
0010 This document provides an overview of the Freescale DPAA2 architecture
0011 and how it is integrated into the Linux kernel.
0012
0013 Introduction
0014 ============
0015
0016 DPAA2 is a hardware architecture designed for high-speeed network
0017 packet processing. DPAA2 consists of sophisticated mechanisms for
0018 processing Ethernet packets, queue management, buffer management,
0019 autonomous L2 switching, virtual Ethernet bridging, and accelerator
0020 (e.g. crypto) sharing.
0021
0022 A DPAA2 hardware component called the Management Complex (or MC) manages the
0023 DPAA2 hardware resources. The MC provides an object-based abstraction for
0024 software drivers to use the DPAA2 hardware.
0025 The MC uses DPAA2 hardware resources such as queues, buffer pools, and
0026 network ports to create functional objects/devices such as network
0027 interfaces, an L2 switch, or accelerator instances.
0028 The MC provides memory-mapped I/O command interfaces (MC portals)
0029 which DPAA2 software drivers use to operate on DPAA2 objects.
0030
0031 The diagram below shows an overview of the DPAA2 resource management
0032 architecture::
0033
0034 +--------------------------------------+
0035 | OS |
0036 | DPAA2 drivers |
0037 | | |
0038 +-----------------------------|--------+
0039 |
0040 | (create,discover,connect
0041 | config,use,destroy)
0042 |
0043 DPAA2 |
0044 +------------------------| mc portal |-+
0045 | | |
0046 | +- - - - - - - - - - - - -V- - -+ |
0047 | | | |
0048 | | Management Complex (MC) | |
0049 | | | |
0050 | +- - - - - - - - - - - - - - - -+ |
0051 | |
0052 | Hardware Hardware |
0053 | Resources Objects |
0054 | --------- ------- |
0055 | -queues -DPRC |
0056 | -buffer pools -DPMCP |
0057 | -Eth MACs/ports -DPIO |
0058 | -network interface -DPNI |
0059 | profiles -DPMAC |
0060 | -queue portals -DPBP |
0061 | -MC portals ... |
0062 | ... |
0063 | |
0064 +--------------------------------------+
0065
0066
0067 The MC mediates operations such as create, discover,
0068 connect, configuration, and destroy. Fast-path operations
0069 on data, such as packet transmit/receive, are not mediated by
0070 the MC and are done directly using memory mapped regions in
0071 DPIO objects.
0072
0073 Overview of DPAA2 Objects
0074 =========================
0075
0076 The section provides a brief overview of some key DPAA2 objects.
0077 A simple scenario is described illustrating the objects involved
0078 in creating a network interfaces.
0079
0080 DPRC (Datapath Resource Container)
0081 ----------------------------------
0082
0083 A DPRC is a container object that holds all the other
0084 types of DPAA2 objects. In the example diagram below there
0085 are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
0086 in the container.
0087
0088 ::
0089
0090 +---------------------------------------------------------+
0091 | DPRC |
0092 | |
0093 | +-------+ +-------+ +-------+ +-------+ +-------+ |
0094 | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | |
0095 | +-------+ +-------+ +-------+ +---+---+ +---+---+ |
0096 | | DPMCP | | DPIO | |
0097 | +-------+ +-------+ |
0098 | | DPMCP | |
0099 | +-------+ |
0100 | |
0101 +---------------------------------------------------------+
0102
0103 From the point of view of an OS, a DPRC behaves similar to a plug and
0104 play bus, like PCI. DPRC commands can be used to enumerate the contents
0105 of the DPRC, discover the hardware objects present (including mappable
0106 regions and interrupts).
0107
0108 ::
0109
0110 DPRC.1 (bus)
0111 |
0112 +--+--------+-------+-------+-------+
0113 | | | | |
0114 DPMCP.1 DPIO.1 DPBP.1 DPNI.1 DPMAC.1
0115 DPMCP.2 DPIO.2
0116 DPMCP.3
0117
0118 Hardware objects can be created and destroyed dynamically, providing
0119 the ability to hot plug/unplug objects in and out of the DPRC.
0120
0121 A DPRC has a mappable MMIO region (an MC portal) that can be used
0122 to send MC commands. It has an interrupt for status events (like
0123 hotplug).
0124 All objects in a container share the same hardware "isolation context".
0125 This means that with respect to an IOMMU the isolation granularity
0126 is at the DPRC (container) level, not at the individual object
0127 level.
0128
0129 DPRCs can be defined statically and populated with objects
0130 via a config file passed to the MC when firmware starts it.
0131
0132 DPAA2 Objects for an Ethernet Network Interface
0133 -----------------------------------------------
0134
0135 A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
0136 queuing mechanisms, configuration mechanisms, buffer management,
0137 physical ports, and interrupts. DPAA2 uses a more granular approach
0138 utilizing multiple hardware objects. Each object provides specialized
0139 functions. Groups of these objects are used by software to provide
0140 Ethernet network interface functionality. This approach provides
0141 efficient use of finite hardware resources, flexibility, and
0142 performance advantages.
0143
0144 The diagram below shows the objects needed for a simple
0145 network interface configuration on a system with 2 CPUs.
0146
0147 ::
0148
0149 +---+---+ +---+---+
0150 CPU0 CPU1
0151 +---+---+ +---+---+
0152 | |
0153 +---+---+ +---+---+
0154 DPIO DPIO
0155 +---+---+ +---+---+
0156 \ /
0157 \ /
0158 \ /
0159 +---+---+
0160 DPNI --- DPBP,DPMCP
0161 +---+---+
0162 |
0163 |
0164 +---+---+
0165 DPMAC
0166 +---+---+
0167 |
0168 port/PHY
0169
0170 Below the objects are described. For each object a brief description
0171 is provided along with a summary of the kinds of operations the object
0172 supports and a summary of key resources of the object (MMIO regions
0173 and IRQs).
0174
0175 DPMAC (Datapath Ethernet MAC)
0176 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0177 Represents an Ethernet MAC, a hardware device that connects to an Ethernet
0178 PHY and allows physical transmission and reception of Ethernet frames.
0179
0180 - MMIO regions: none
0181 - IRQs: DPNI link change
0182 - commands: set link up/down, link config, get stats,
0183 IRQ config, enable, reset
0184
0185 DPNI (Datapath Network Interface)
0186 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
0187 Contains TX/RX queues, network interface configuration, and RX buffer pool
0188 configuration mechanisms. The TX/RX queues are in memory and are identified
0189 by queue number.
0190
0191 - MMIO regions: none
0192 - IRQs: link state
0193 - commands: port config, offload config, queue config,
0194 parse/classify config, IRQ config, enable, reset
0195
0196 DPIO (Datapath I/O)
0197 ~~~~~~~~~~~~~~~~~~~
0198 Provides interfaces to enqueue and dequeue
0199 packets and do hardware buffer pool management operations. The DPAA2
0200 architecture separates the mechanism to access queues (the DPIO object)
0201 from the queues themselves. The DPIO provides an MMIO interface to
0202 enqueue/dequeue packets. To enqueue something a descriptor is written
0203 to the DPIO MMIO region, which includes the target queue number.
0204 There will typically be one DPIO assigned to each CPU. This allows all
0205 CPUs to simultaneously perform enqueue/dequeued operations. DPIOs are
0206 expected to be shared by different DPAA2 drivers.
0207
0208 - MMIO regions: queue operations, buffer management
0209 - IRQs: data availability, congestion notification, buffer
0210 pool depletion
0211 - commands: IRQ config, enable, reset
0212
0213 DPBP (Datapath Buffer Pool)
0214 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
0215 Represents a hardware buffer pool.
0216
0217 - MMIO regions: none
0218 - IRQs: none
0219 - commands: enable, reset
0220
0221 DPMCP (Datapath MC Portal)
0222 ~~~~~~~~~~~~~~~~~~~~~~~~~~
0223 Provides an MC command portal.
0224 Used by drivers to send commands to the MC to manage
0225 objects.
0226
0227 - MMIO regions: MC command portal
0228 - IRQs: command completion
0229 - commands: IRQ config, enable, reset
0230
0231 Object Connections
0232 ==================
0233 Some objects have explicit relationships that must
0234 be configured:
0235
0236 - DPNI <--> DPMAC
0237 - DPNI <--> DPNI
0238 - DPNI <--> L2-switch-port
0239
0240 A DPNI must be connected to something such as a DPMAC,
0241 another DPNI, or L2 switch port. The DPNI connection
0242 is made via a DPRC command.
0243
0244 ::
0245
0246 +-------+ +-------+
0247 | DPNI | | DPMAC |
0248 +---+---+ +---+---+
0249 | |
0250 +==========+
0251
0252 - DPNI <--> DPBP
0253
0254 A network interface requires a 'buffer pool' (DPBP
0255 object) which provides a list of pointers to memory
0256 where received Ethernet data is to be copied. The
0257 Ethernet driver configures the DPBPs associated with
0258 the network interface.
0259
0260 Interrupts
0261 ==========
0262 All interrupts generated by DPAA2 objects are message
0263 interrupts. At the hardware level message interrupts
0264 generated by devices will normally have 3 components--
0265 1) a non-spoofable 'device-id' expressed on the hardware
0266 bus, 2) an address, 3) a data value.
0267
0268 In the case of DPAA2 devices/objects, all objects in the
0269 same container/DPRC share the same 'device-id'.
0270 For ARM-based SoC this is the same as the stream ID.
0271
0272
0273 DPAA2 Linux Drivers Overview
0274 ============================
0275
0276 This section provides an overview of the Linux kernel drivers for
0277 DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
0278 drivers and 2) functional object drivers (such as Ethernet).
0279
0280 As described previously, a DPRC is a container that holds the other
0281 types of DPAA2 objects. It is functionally similar to a plug-and-play
0282 bus controller.
0283 Each object in the DPRC is a Linux "device" and is bound to a driver.
0284 The diagram below shows the Linux drivers involved in a networking
0285 scenario and the objects bound to each driver. A brief description
0286 of each driver follows.
0287
0288 ::
0289
0290 +------------+
0291 | OS Network |
0292 | Stack |
0293 +------------+ +------------+
0294 | Allocator |. . . . . . . | Ethernet |
0295 |(DPMCP,DPBP)| | (DPNI) |
0296 +-.----------+ +---+---+----+
0297 . . ^ |
0298 . . <data avail, | | <enqueue,
0299 . . tx confirm> | | dequeue>
0300 +-------------+ . | |
0301 | DPRC driver | . +---+---V----+ +---------+
0302 | (DPRC) | . . . . . .| DPIO driver| | MAC |
0303 +----------+--+ | (DPIO) | | (DPMAC) |
0304 | +------+-----+ +-----+---+
0305 |<dev add/remove> | |
0306 | | |
0307 +--------+----------+ | +--+---+
0308 | MC-bus driver | | | PHY |
0309 | | | |driver|
0310 | /bus/fsl-mc | | +--+---+
0311 +-------------------+ | |
0312 | |
0313 ========================= HARDWARE =========|=================|======
0314 DPIO |
0315 | |
0316 DPNI---DPBP |
0317 | |
0318 DPMAC |
0319 | |
0320 PHY ---------------+
0321 ============================================|========================
0322
0323 A brief description of each driver is provided below.
0324
0325 MC-bus driver
0326 -------------
0327 The MC-bus driver is a platform driver and is probed from a
0328 node in the device tree (compatible "fsl,qoriq-mc") passed in by boot
0329 firmware. It is responsible for bootstrapping the DPAA2 kernel
0330 infrastructure.
0331 Key functions include:
0332
0333 - registering a new bus type named "fsl-mc" with the kernel,
0334 and implementing bus call-backs (e.g. match/uevent/dev_groups)
0335 - implementing APIs for DPAA2 driver registration and for device
0336 add/remove
0337 - creates an MSI IRQ domain
0338 - doing a 'device add' to expose the 'root' DPRC, in turn triggering
0339 a bind of the root DPRC to the DPRC driver
0340
0341 The binding for the MC-bus device-tree node can be consulted at
0342 *Documentation/devicetree/bindings/misc/fsl,qoriq-mc.txt*.
0343 The sysfs bind/unbind interfaces for the MC-bus can be consulted at
0344 *Documentation/ABI/testing/sysfs-bus-fsl-mc*.
0345
0346 DPRC driver
0347 -----------
0348 The DPRC driver is bound to DPRC objects and does runtime management
0349 of a bus instance. It performs the initial bus scan of the DPRC
0350 and handles interrupts for container events such as hot plug by
0351 re-scanning the DPRC.
0352
0353 Allocator
0354 ---------
0355 Certain objects such as DPMCP and DPBP are generic and fungible,
0356 and are intended to be used by other drivers. For example,
0357 the DPAA2 Ethernet driver needs:
0358
0359 - DPMCPs to send MC commands, to configure network interfaces
0360 - DPBPs for network buffer pools
0361
0362 The allocator driver registers for these allocatable object types
0363 and those objects are bound to the allocator when the bus is probed.
0364 The allocator maintains a pool of objects that are available for
0365 allocation by other DPAA2 drivers.
0366
0367 DPIO driver
0368 -----------
0369 The DPIO driver is bound to DPIO objects and provides services that allow
0370 other drivers such as the Ethernet driver to enqueue and dequeue data for
0371 their respective objects.
0372 Key services include:
0373
0374 - data availability notifications
0375 - hardware queuing operations (enqueue and dequeue of data)
0376 - hardware buffer pool management
0377
0378 To transmit a packet the Ethernet driver puts data on a queue and
0379 invokes a DPIO API. For receive, the Ethernet driver registers
0380 a data availability notification callback. To dequeue a packet
0381 a DPIO API is used.
0382 There is typically one DPIO object per physical CPU for optimum
0383 performance, allowing different CPUs to simultaneously enqueue
0384 and dequeue data.
0385
0386 The DPIO driver operates on behalf of all DPAA2 drivers
0387 active in the kernel-- Ethernet, crypto, compression,
0388 etc.
0389
0390 Ethernet driver
0391 ---------------
0392 The Ethernet driver is bound to a DPNI and implements the kernel
0393 interfaces needed to connect the DPAA2 network interface to
0394 the network stack.
0395 Each DPNI corresponds to a Linux network interface.
0396
0397 MAC driver
0398 ----------
0399 An Ethernet PHY is an off-chip, board specific component and is managed
0400 by the appropriate PHY driver via an mdio bus. The MAC driver
0401 plays a role of being a proxy between the PHY driver and the
0402 MC. It does this proxy via the MC commands to a DPMAC object.
0403 If the PHY driver signals a link change, the MAC driver notifies
0404 the MC via a DPMAC command. If a network interface is brought
0405 up or down, the MC notifies the DPMAC driver via an interrupt and
0406 the driver can take appropriate action.
