0001 =================================================================
0002 Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC)
0003 =================================================================
0004
0005 Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC) feature
0006 supports Ethernet functionality over Omni-Path fabric by encapsulating
0007 the Ethernet packets between HFI nodes.
0008
0009 Architecture
0010 =============
0011 The patterns of exchanges of Omni-Path encapsulated Ethernet packets
0012 involves one or more virtual Ethernet switches overlaid on the Omni-Path
0013 fabric topology. A subset of HFI nodes on the Omni-Path fabric are
0014 permitted to exchange encapsulated Ethernet packets across a particular
0015 virtual Ethernet switch. The virtual Ethernet switches are logical
0016 abstractions achieved by configuring the HFI nodes on the fabric for
0017 header generation and processing. In the simplest configuration all HFI
0018 nodes across the fabric exchange encapsulated Ethernet packets over a
0019 single virtual Ethernet switch. A virtual Ethernet switch, is effectively
0020 an independent Ethernet network. The configuration is performed by an
0021 Ethernet Manager (EM) which is part of the trusted Fabric Manager (FM)
0022 application. HFI nodes can have multiple VNICs each connected to a
0023 different virtual Ethernet switch. The below diagram presents a case
0024 of two virtual Ethernet switches with two HFI nodes::
0025
0026 +-------------------+
0027 | Subnet/ |
0028 | Ethernet |
0029 | Manager |
0030 +-------------------+
0031 / /
0032 / /
0033 / /
0034 / /
0035 +-----------------------------+ +------------------------------+
0036 | Virtual Ethernet Switch | | Virtual Ethernet Switch |
0037 | +---------+ +---------+ | | +---------+ +---------+ |
0038 | | VPORT | | VPORT | | | | VPORT | | VPORT | |
0039 +--+---------+----+---------+-+ +-+---------+----+---------+---+
0040 | \ / |
0041 | \ / |
0042 | \/ |
0043 | / \ |
0044 | / \ |
0045 +-----------+------------+ +-----------+------------+
0046 | VNIC | VNIC | | VNIC | VNIC |
0047 +-----------+------------+ +-----------+------------+
0048 | HFI | | HFI |
0049 +------------------------+ +------------------------+
0050
0051
0052 The Omni-Path encapsulated Ethernet packet format is as described below.
0053
0054 ==================== ================================
0055 Bits Field
0056 ==================== ================================
0057 Quad Word 0:
0058 0-19 SLID (lower 20 bits)
0059 20-30 Length (in Quad Words)
0060 31 BECN bit
0061 32-51 DLID (lower 20 bits)
0062 52-56 SC (Service Class)
0063 57-59 RC (Routing Control)
0064 60 FECN bit
0065 61-62 L2 (=10, 16B format)
0066 63 LT (=1, Link Transfer Head Flit)
0067
0068 Quad Word 1:
0069 0-7 L4 type (=0x78 ETHERNET)
0070 8-11 SLID[23:20]
0071 12-15 DLID[23:20]
0072 16-31 PKEY
0073 32-47 Entropy
0074 48-63 Reserved
0075
0076 Quad Word 2:
0077 0-15 Reserved
0078 16-31 L4 header
0079 32-63 Ethernet Packet
0080
0081 Quad Words 3 to N-1:
0082 0-63 Ethernet packet (pad extended)
0083
0084 Quad Word N (last):
0085 0-23 Ethernet packet (pad extended)
0086 24-55 ICRC
0087 56-61 Tail
0088 62-63 LT (=01, Link Transfer Tail Flit)
0089 ==================== ================================
0090
0091 Ethernet packet is padded on the transmit side to ensure that the VNIC OPA
0092 packet is quad word aligned. The 'Tail' field contains the number of bytes
0093 padded. On the receive side the 'Tail' field is read and the padding is
0094 removed (along with ICRC, Tail and OPA header) before passing packet up
0095 the network stack.
0096
0097 The L4 header field contains the virtual Ethernet switch id the VNIC port
0098 belongs to. On the receive side, this field is used to de-multiplex the
0099 received VNIC packets to different VNIC ports.
0100
0101 Driver Design
0102 ==============
0103 Intel OPA VNIC software design is presented in the below diagram.
0104 OPA VNIC functionality has a HW dependent component and a HW
0105 independent component.
0106
0107 The support has been added for IB device to allocate and free the RDMA
0108 netdev devices. The RDMA netdev supports interfacing with the network
0109 stack thus creating standard network interfaces. OPA_VNIC is an RDMA
0110 netdev device type.
0111
0112 The HW dependent VNIC functionality is part of the HFI1 driver. It
0113 implements the verbs to allocate and free the OPA_VNIC RDMA netdev.
0114 It involves HW resource allocation/management for VNIC functionality.
0115 It interfaces with the network stack and implements the required
0116 net_device_ops functions. It expects Omni-Path encapsulated Ethernet
0117 packets in the transmit path and provides HW access to them. It strips
0118 the Omni-Path header from the received packets before passing them up
0119 the network stack. It also implements the RDMA netdev control operations.
0120
0121 The OPA VNIC module implements the HW independent VNIC functionality.
0122 It consists of two parts. The VNIC Ethernet Management Agent (VEMA)
0123 registers itself with IB core as an IB client and interfaces with the
0124 IB MAD stack. It exchanges the management information with the Ethernet
0125 Manager (EM) and the VNIC netdev. The VNIC netdev part allocates and frees
0126 the OPA_VNIC RDMA netdev devices. It overrides the net_device_ops functions
0127 set by HW dependent VNIC driver where required to accommodate any control
0128 operation. It also handles the encapsulation of Ethernet packets with an
0129 Omni-Path header in the transmit path. For each VNIC interface, the
0130 information required for encapsulation is configured by the EM via VEMA MAD
0131 interface. It also passes any control information to the HW dependent driver
0132 by invoking the RDMA netdev control operations::
0133
0134 +-------------------+ +----------------------+
0135 | | | Linux |
0136 | IB MAD | | Network |
0137 | | | Stack |
0138 +-------------------+ +----------------------+
0139 | | |
0140 | | |
0141 +----------------------------+ |
0142 | | |
0143 | OPA VNIC Module | |
0144 | (OPA VNIC RDMA Netdev | |
0145 | & EMA functions) | |
0146 | | |
0147 +----------------------------+ |
0148 | |
0149 | |
0150 +------------------+ |
0151 | IB core | |
0152 +------------------+ |
0153 | |
0154 | |
0155 +--------------------------------------------+
0156 | |
0157 | HFI1 Driver with VNIC support |
0158 | |
0159 +--------------------------------------------+
