0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ==============================
0004 The QorIQ DPAA Ethernet Driver
0005 ==============================
0006
0007 Authors:
0008 - Madalin Bucur <madalin.bucur@nxp.com>
0009 - Camelia Groza <camelia.groza@nxp.com>
0010
0011 .. Contents
0012
0013 - DPAA Ethernet Overview
0014 - DPAA Ethernet Supported SoCs
0015 - Configuring DPAA Ethernet in your kernel
0016 - DPAA Ethernet Frame Processing
0017 - DPAA Ethernet Features
0018 - DPAA IRQ Affinity and Receive Side Scaling
0019 - Debugging
0020
0021 DPAA Ethernet Overview
0022 ======================
0023
0024 DPAA stands for Data Path Acceleration Architecture and it is a
0025 set of networking acceleration IPs that are available on several
0026 generations of SoCs, both on PowerPC and ARM64.
0027
0028 The Freescale DPAA architecture consists of a series of hardware blocks
0029 that support Ethernet connectivity. The Ethernet driver depends upon the
0030 following drivers in the Linux kernel:
0031
0032 - Peripheral Access Memory Unit (PAMU) (* needed only for PPC platforms)
0033 drivers/iommu/fsl_*
0034 - Frame Manager (FMan)
0035 drivers/net/ethernet/freescale/fman
0036 - Queue Manager (QMan), Buffer Manager (BMan)
0037 drivers/soc/fsl/qbman
0038
0039 A simplified view of the dpaa_eth interfaces mapped to FMan MACs::
0040
0041 dpaa_eth /eth0\ ... /ethN\
0042 driver | | | |
0043 ------------- ---- ----------- ---- -------------
0044 -Ports / Tx Rx \ ... / Tx Rx \
0045 FMan | | | |
0046 -MACs | MAC0 | | MACN |
0047 / dtsec0 \ ... / dtsecN \ (or tgec)
0048 / \ / \(or memac)
0049 --------- -------------- --- -------------- ---------
0050 FMan, FMan Port, FMan SP, FMan MURAM drivers
0051 ---------------------------------------------------------
0052 FMan HW blocks: MURAM, MACs, Ports, SP
0053 ---------------------------------------------------------
0054
0055 The dpaa_eth relation to the QMan, BMan and FMan::
0056
0057 ________________________________
0058 dpaa_eth / eth0 \
0059 driver / \
0060 --------- -^- -^- -^- --- ---------
0061 QMan driver / \ / \ / \ \ / | BMan |
0062 |Rx | |Rx | |Tx | |Tx | | driver |
0063 --------- |Dfl| |Err| |Cnf| |FQs| | |
0064 QMan HW |FQ | |FQ | |FQs| | | | |
0065 / \ / \ / \ \ / | |
0066 --------- --- --- --- -v- ---------
0067 | FMan QMI | |
0068 | FMan HW FMan BMI | BMan HW |
0069 ----------------------- --------
0070
0071 where the acronyms used above (and in the code) are:
0072
0073 =============== ===========================================================
0074 DPAA Data Path Acceleration Architecture
0075 FMan DPAA Frame Manager
0076 QMan DPAA Queue Manager
0077 BMan DPAA Buffers Manager
0078 QMI QMan interface in FMan
0079 BMI BMan interface in FMan
0080 FMan SP FMan Storage Profiles
0081 MURAM Multi-user RAM in FMan
0082 FQ QMan Frame Queue
0083 Rx Dfl FQ default reception FQ
0084 Rx Err FQ Rx error frames FQ
0085 Tx Cnf FQ Tx confirmation FQs
0086 Tx FQs transmission frame queues
0087 dtsec datapath three speed Ethernet controller (10/100/1000 Mbps)
0088 tgec ten gigabit Ethernet controller (10 Gbps)
0089 memac multirate Ethernet MAC (10/100/1000/10000)
0090 =============== ===========================================================
0091
0092 DPAA Ethernet Supported SoCs
0093 ============================
0094
0095 The DPAA drivers enable the Ethernet controllers present on the following SoCs:
0096
0097 PPC
0098 - P1023
0099 - P2041
0100 - P3041
0101 - P4080
0102 - P5020
0103 - P5040
0104 - T1023
0105 - T1024
0106 - T1040
0107 - T1042
0108 - T2080
0109 - T4240
0110 - B4860
0111
0112 ARM
0113 - LS1043A
0114 - LS1046A
0115
0116 Configuring DPAA Ethernet in your kernel
0117 ========================================
0118
0119 To enable the DPAA Ethernet driver, the following Kconfig options are required::
0120
0121 # common for arch/arm64 and arch/powerpc platforms
0122 CONFIG_FSL_DPAA=y
0123 CONFIG_FSL_FMAN=y
0124 CONFIG_FSL_DPAA_ETH=y
0125 CONFIG_FSL_XGMAC_MDIO=y
0126
0127 # for arch/powerpc only
0128 CONFIG_FSL_PAMU=y
0129
0130 # common options needed for the PHYs used on the RDBs
0131 CONFIG_VITESSE_PHY=y
0132 CONFIG_REALTEK_PHY=y
0133 CONFIG_AQUANTIA_PHY=y
0134
0135 DPAA Ethernet Frame Processing
0136 ==============================
0137
0138 On Rx, buffers for the incoming frames are retrieved from the buffers found
0139 in the dedicated interface buffer pool. The driver initializes and seeds these
0140 with one page buffers.
0141
0142 On Tx, all transmitted frames are returned to the driver through Tx
0143 confirmation frame queues. The driver is then responsible for freeing the
0144 buffers. In order to do this properly, a backpointer is added to the buffer
0145 before transmission that points to the skb. When the buffer returns to the
0146 driver on a confirmation FQ, the skb can be correctly consumed.
0147
0148 DPAA Ethernet Features
0149 ======================
0150
0151 Currently the DPAA Ethernet driver enables the basic features required for
0152 a Linux Ethernet driver. The support for advanced features will be added
0153 gradually.
0154
0155 The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx
0156 checksum offload feature is enabled by default and cannot be controlled through
0157 ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS
0158 provides a big performance boost for the forwarding scenarios, allowing
0159 different traffic flows received by one interface to be processed by different
0160 CPUs in parallel.
0161
0162 The driver has support for multiple prioritized Tx traffic classes. Priorities
0163 range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with
0164 strict priority levels. Each traffic class contains NR_CPU TX queues. By
0165 default, only one traffic class is enabled and the lowest priority Tx queues
0166 are used. Higher priority traffic classes can be enabled with the mqprio
0167 qdisc. For example, all four traffic classes are enabled on an interface with
0168 the following command. Furthermore, skb priority levels are mapped to traffic
0169 classes as follows:
0170
0171 * priorities 0 to 3 - traffic class 0 (low priority)
0172 * priorities 4 to 7 - traffic class 1 (medium-low priority)
0173 * priorities 8 to 11 - traffic class 2 (medium-high priority)
0174 * priorities 12 to 15 - traffic class 3 (high priority)
0175
0176 ::
0177
0178 tc qdisc add dev <int> root handle 1: \
0179 mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1
0180
0181 DPAA IRQ Affinity and Receive Side Scaling
0182 ==========================================
0183
0184 Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation
0185 queues is seen by the CPU as ingress traffic on a certain portal.
0186 The DPAA QMan portal interrupts are affined each to a certain CPU.
0187 The same portal interrupt services all the QMan portal consumers.
0188
0189 By default the DPAA Ethernet driver enables RSS, making use of the
0190 DPAA FMan Parser and Keygen blocks to distribute traffic on 128
0191 hardware frame queues using a hash on IP v4/v6 source and destination
0192 and L4 source and destination ports, in present in the received frame.
0193 When RSS is disabled, all traffic received by a certain interface is
0194 received on the default Rx frame queue. The default DPAA Rx frame
0195 queues are configured to put the received traffic into a pool channel
0196 that allows any available CPU portal to dequeue the ingress traffic.
0197 The default frame queues have the HOLDACTIVE option set, ensuring that
0198 traffic bursts from a certain queue are serviced by the same CPU.
0199 This ensures a very low rate of frame reordering. A drawback of this
0200 is that only one CPU at a time can service the traffic received by a
0201 certain interface when RSS is not enabled.
0202
0203 To implement RSS, the DPAA Ethernet driver allocates an extra set of
0204 128 Rx frame queues that are configured to dedicated channels, in a
0205 round-robin manner. The mapping of the frame queues to CPUs is now
0206 hardcoded, there is no indirection table to move traffic for a certain
0207 FQ (hash result) to another CPU. The ingress traffic arriving on one
0208 of these frame queues will arrive at the same portal and will always
0209 be processed by the same CPU. This ensures intra-flow order preservation
0210 and workload distribution for multiple traffic flows.
0211
0212 RSS can be turned off for a certain interface using ethtool, i.e.::
0213
0214 # ethtool -N fm1-mac9 rx-flow-hash tcp4 ""
0215
0216 To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6::
0217
0218 # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn
0219
0220 There is no independent control for individual protocols, any command
0221 run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is
0222 going to control the rx-flow-hashing for all protocols on that interface.
0223
0224 Besides using the FMan Keygen computed hash for spreading traffic on the
0225 128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when
0226 the NETIF_F_RXHASH feature is on (active by default). This can be turned
0227 on or off through ethtool, i.e.::
0228
0229 # ethtool -K fm1-mac9 rx-hashing off
0230 # ethtool -k fm1-mac9 | grep hash
0231 receive-hashing: off
0232 # ethtool -K fm1-mac9 rx-hashing on
0233 Actual changes:
0234 receive-hashing: on
0235 # ethtool -k fm1-mac9 | grep hash
0236 receive-hashing: on
0237
0238 Please note that Rx hashing depends upon the rx-flow-hashing being on
0239 for that interface - turning off rx-flow-hashing will also disable the
0240 rx-hashing (without ethtool reporting it as off as that depends on the
0241 NETIF_F_RXHASH feature flag).
0242
0243 Debugging
0244 =========
0245
0246 The following statistics are exported for each interface through ethtool:
0247
0248 - interrupt count per CPU
0249 - Rx packets count per CPU
0250 - Tx packets count per CPU
0251 - Tx confirmed packets count per CPU
0252 - Tx S/G frames count per CPU
0253 - Tx error count per CPU
0254 - Rx error count per CPU
0255 - Rx error count per type
0256 - congestion related statistics:
0257
0258 - congestion status
0259 - time spent in congestion
0260 - number of time the device entered congestion
0261 - dropped packets count per cause
0262
0263 The driver also exports the following information in sysfs:
0264
0265 - the FQ IDs for each FQ type
0266 /sys/devices/platform/soc/<addr>.fman/<addr>.ethernet/dpaa-ethernet.<id>/net/fm<nr>-mac<nr>/fqids
0267
0268 - the ID of the buffer pool in use
0269 /sys/devices/platform/soc/<addr>.fman/<addr>.ethernet/dpaa-ethernet.<id>/net/fm<nr>-mac<nr>/bpids
