0001 .. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
0002
0003 =============================================
0004 Netronome Flow Processor (NFP) Kernel Drivers
0005 =============================================
0006
0007 Copyright (c) 2019, Netronome Systems, Inc.
0008
0009 Contents
0010 ========
0011
0012 - `Overview`_
0013 - `Acquiring Firmware`_
0014
0015 Overview
0016 ========
0017
0018 This driver supports Netronome's line of Flow Processor devices,
0019 including the NFP4000, NFP5000, and NFP6000 models, which are also
0020 incorporated in the company's family of Agilio SmartNICs. The SR-IOV
0021 physical and virtual functions for these devices are supported by
0022 the driver.
0023
0024 Acquiring Firmware
0025 ==================
0026
0027 The NFP4000 and NFP6000 devices require application specific firmware
0028 to function. Application firmware can be located either on the host file system
0029 or in the device flash (if supported by management firmware).
0030
0031 Firmware files on the host filesystem contain card type (`AMDA-*` string), media
0032 config etc. They should be placed in `/lib/firmware/netronome` directory to
0033 load firmware from the host file system.
0034
0035 Firmware for basic NIC operation is available in the upstream
0036 `linux-firmware.git` repository.
0037
0038 Firmware in NVRAM
0039 -----------------
0040
0041 Recent versions of management firmware supports loading application
0042 firmware from flash when the host driver gets probed. The firmware loading
0043 policy configuration may be used to configure this feature appropriately.
0044
0045 Devlink or ethtool can be used to update the application firmware on the device
0046 flash by providing the appropriate `nic_AMDA*.nffw` file to the respective
0047 command. Users need to take care to write the correct firmware image for the
0048 card and media configuration to flash.
0049
0050 Available storage space in flash depends on the card being used.
0051
0052 Dealing with multiple projects
0053 ------------------------------
0054
0055 NFP hardware is fully programmable therefore there can be different
0056 firmware images targeting different applications.
0057
0058 When using application firmware from host, we recommend placing
0059 actual firmware files in application-named subdirectories in
0060 `/lib/firmware/netronome` and linking the desired files, e.g.::
0061
0062 $ tree /lib/firmware/netronome/
0063 /lib/firmware/netronome/
0064 ├── bpf
0065 │ ├── nic_AMDA0081-0001_1x40.nffw
0066 │ └── nic_AMDA0081-0001_4x10.nffw
0067 ├── flower
0068 │ ├── nic_AMDA0081-0001_1x40.nffw
0069 │ └── nic_AMDA0081-0001_4x10.nffw
0070 ├── nic
0071 │ ├── nic_AMDA0081-0001_1x40.nffw
0072 │ └── nic_AMDA0081-0001_4x10.nffw
0073 ├── nic_AMDA0081-0001_1x40.nffw -> bpf/nic_AMDA0081-0001_1x40.nffw
0074 └── nic_AMDA0081-0001_4x10.nffw -> bpf/nic_AMDA0081-0001_4x10.nffw
0075
0076 3 directories, 8 files
0077
0078 You may need to use hard instead of symbolic links on distributions
0079 which use old `mkinitrd` command instead of `dracut` (e.g. Ubuntu).
0080
0081 After changing firmware files you may need to regenerate the initramfs
0082 image. Initramfs contains drivers and firmware files your system may
0083 need to boot. Refer to the documentation of your distribution to find
0084 out how to update initramfs. Good indication of stale initramfs
0085 is system loading wrong driver or firmware on boot, but when driver is
0086 later reloaded manually everything works correctly.
0087
0088 Selecting firmware per device
0089 -----------------------------
0090
0091 Most commonly all cards on the system use the same type of firmware.
0092 If you want to load specific firmware image for a specific card, you
0093 can use either the PCI bus address or serial number. Driver will print
0094 which files it's looking for when it recognizes a NFP device::
0095
0096 nfp: Looking for firmware file in order of priority:
0097 nfp: netronome/serial-00-12-34-aa-bb-cc-10-ff.nffw: not found
0098 nfp: netronome/pci-0000:02:00.0.nffw: not found
0099 nfp: netronome/nic_AMDA0081-0001_1x40.nffw: found, loading...
0100
0101 In this case if file (or link) called *serial-00-12-34-aa-bb-5d-10-ff.nffw*
0102 or *pci-0000:02:00.0.nffw* is present in `/lib/firmware/netronome` this
0103 firmware file will take precedence over `nic_AMDA*` files.
0104
0105 Note that `serial-*` and `pci-*` files are **not** automatically included
0106 in initramfs, you will have to refer to documentation of appropriate tools
0107 to find out how to include them.
0108
0109 Firmware loading policy
0110 -----------------------
0111
0112 Firmware loading policy is controlled via three HWinfo parameters
0113 stored as key value pairs in the device flash:
0114
0115 app_fw_from_flash
0116 Defines which firmware should take precedence, 'Disk' (0), 'Flash' (1) or
0117 the 'Preferred' (2) firmware. When 'Preferred' is selected, the management
0118 firmware makes the decision over which firmware will be loaded by comparing
0119 versions of the flash firmware and the host supplied firmware.
0120 This variable is configurable using the 'fw_load_policy'
0121 devlink parameter.
0122
0123 abi_drv_reset
0124 Defines if the driver should reset the firmware when
0125 the driver is probed, either 'Disk' (0) if firmware was found on disk,
0126 'Always' (1) reset or 'Never' (2) reset. Note that the device is always
0127 reset on driver unload if firmware was loaded when the driver was probed.
0128 This variable is configurable using the 'reset_dev_on_drv_probe'
0129 devlink parameter.
0130
0131 abi_drv_load_ifc
0132 Defines a list of PF devices allowed to load FW on the device.
0133 This variable is not currently user configurable.
0134
0135 Statistics
0136 ==========
0137
0138 Following device statistics are available through the ``ethtool -S`` interface:
0139
0140 .. flat-table:: NFP device statistics
0141 :header-rows: 1
0142 :widths: 3 1 11
0143
0144 * - Name
0145 - ID
0146 - Meaning
0147
0148 * - dev_rx_discards
0149 - 1
0150 - Packet can be discarded on the RX path for one of the following reasons:
0151
0152 * The NIC is not in promisc mode, and the destination MAC address
0153 doesn't match the interfaces' MAC address.
0154 * The received packet is larger than the max buffer size on the host.
0155 I.e. it exceeds the Layer 3 MRU.
0156 * There is no freelist descriptor available on the host for the packet.
0157 It is likely that the NIC couldn't cache one in time.
0158 * A BPF program discarded the packet.
0159 * The datapath drop action was executed.
0160 * The MAC discarded the packet due to lack of ingress buffer space
0161 on the NIC.
0162
0163 * - dev_rx_errors
0164 - 2
0165 - A packet can be counted (and dropped) as RX error for the following
0166 reasons:
0167
0168 * A problem with the VEB lookup (only when SR-IOV is used).
0169 * A physical layer problem that causes Ethernet errors, like FCS or
0170 alignment errors. The cause is usually faulty cables or SFPs.
0171
0172 * - dev_rx_bytes
0173 - 3
0174 - Total number of bytes received.
0175
0176 * - dev_rx_uc_bytes
0177 - 4
0178 - Unicast bytes received.
0179
0180 * - dev_rx_mc_bytes
0181 - 5
0182 - Multicast bytes received.
0183
0184 * - dev_rx_bc_bytes
0185 - 6
0186 - Broadcast bytes received.
0187
0188 * - dev_rx_pkts
0189 - 7
0190 - Total number of packets received.
0191
0192 * - dev_rx_mc_pkts
0193 - 8
0194 - Multicast packets received.
0195
0196 * - dev_rx_bc_pkts
0197 - 9
0198 - Broadcast packets received.
0199
0200 * - dev_tx_discards
0201 - 10
0202 - A packet can be discarded in the TX direction if the MAC is
0203 being flow controlled and the NIC runs out of TX queue space.
0204
0205 * - dev_tx_errors
0206 - 11
0207 - A packet can be counted as TX error (and dropped) for one for the
0208 following reasons:
0209
0210 * The packet is an LSO segment, but the Layer 3 or Layer 4 offset
0211 could not be determined. Therefore LSO could not continue.
0212 * An invalid packet descriptor was received over PCIe.
0213 * The packet Layer 3 length exceeds the device MTU.
0214 * An error on the MAC/physical layer. Usually due to faulty cables or
0215 SFPs.
0216 * A CTM buffer could not be allocated.
0217 * The packet offset was incorrect and could not be fixed by the NIC.
0218
0219 * - dev_tx_bytes
0220 - 12
0221 - Total number of bytes transmitted.
0222
0223 * - dev_tx_uc_bytes
0224 - 13
0225 - Unicast bytes transmitted.
0226
0227 * - dev_tx_mc_bytes
0228 - 14
0229 - Multicast bytes transmitted.
0230
0231 * - dev_tx_bc_bytes
0232 - 15
0233 - Broadcast bytes transmitted.
0234
0235 * - dev_tx_pkts
0236 - 16
0237 - Total number of packets transmitted.
0238
0239 * - dev_tx_mc_pkts
0240 - 17
0241 - Multicast packets transmitted.
0242
0243 * - dev_tx_bc_pkts
0244 - 18
0245 - Broadcast packets transmitted.
0246
0247 Note that statistics unknown to the driver will be displayed as
0248 ``dev_unknown_stat$ID``, where ``$ID`` refers to the second column
0249 above.
