0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 ================
0004 SCSI FC Tansport
0005 ================
0006
0007 Date: 11/18/2008
0008
0009 Kernel Revisions for features::
0010
0011 rports : <<TBS>>
0012 vports : 2.6.22
0013 bsg support : 2.6.30 (?TBD?)
0014
0015
0016 Introduction
0017 ============
0018 This file documents the features and components of the SCSI FC Transport.
0019 It also provides documents the API between the transport and FC LLDDs.
0020
0021 The FC transport can be found at::
0022
0023 drivers/scsi/scsi_transport_fc.c
0024 include/scsi/scsi_transport_fc.h
0025 include/scsi/scsi_netlink_fc.h
0026 include/scsi/scsi_bsg_fc.h
0027
0028 This file is found at Documentation/scsi/scsi_fc_transport.rst
0029
0030
0031 FC Remote Ports (rports)
0032 ========================
0033 << To Be Supplied >>
0034
0035
0036 FC Virtual Ports (vports)
0037 =========================
0038
0039 Overview
0040 --------
0041
0042 New FC standards have defined mechanisms which allows for a single physical
0043 port to appear on as multiple communication ports. Using the N_Port Id
0044 Virtualization (NPIV) mechanism, a point-to-point connection to a Fabric
0045 can be assigned more than 1 N_Port_ID. Each N_Port_ID appears as a
0046 separate port to other endpoints on the fabric, even though it shares one
0047 physical link to the switch for communication. Each N_Port_ID can have a
0048 unique view of the fabric based on fabric zoning and array lun-masking
0049 (just like a normal non-NPIV adapter). Using the Virtual Fabric (VF)
0050 mechanism, adding a fabric header to each frame allows the port to
0051 interact with the Fabric Port to join multiple fabrics. The port will
0052 obtain an N_Port_ID on each fabric it joins. Each fabric will have its
0053 own unique view of endpoints and configuration parameters. NPIV may be
0054 used together with VF so that the port can obtain multiple N_Port_IDs
0055 on each virtual fabric.
0056
0057 The FC transport is now recognizing a new object - a vport. A vport is
0058 an entity that has a world-wide unique World Wide Port Name (wwpn) and
0059 World Wide Node Name (wwnn). The transport also allows for the FC4's to
0060 be specified for the vport, with FCP_Initiator being the primary role
0061 expected. Once instantiated by one of the above methods, it will have a
0062 distinct N_Port_ID and view of fabric endpoints and storage entities.
0063 The fc_host associated with the physical adapter will export the ability
0064 to create vports. The transport will create the vport object within the
0065 Linux device tree, and instruct the fc_host's driver to instantiate the
0066 virtual port. Typically, the driver will create a new scsi_host instance
0067 on the vport, resulting in a unique <H,C,T,L> namespace for the vport.
0068 Thus, whether a FC port is based on a physical port or on a virtual port,
0069 each will appear as a unique scsi_host with its own target and lun space.
0070
0071 .. Note::
0072 At this time, the transport is written to create only NPIV-based
0073 vports. However, consideration was given to VF-based vports and it
0074 should be a minor change to add support if needed. The remaining
0075 discussion will concentrate on NPIV.
0076
0077 .. Note::
0078 World Wide Name assignment (and uniqueness guarantees) are left
0079 up to an administrative entity controlling the vport. For example,
0080 if vports are to be associated with virtual machines, a XEN mgmt
0081 utility would be responsible for creating wwpn/wwnn's for the vport,
0082 using its own naming authority and OUI. (Note: it already does this
0083 for virtual MAC addresses).
0084
0085
0086 Device Trees and Vport Objects:
0087 -------------------------------
0088
0089 Today, the device tree typically contains the scsi_host object,
0090 with rports and scsi target objects underneath it. Currently the FC
0091 transport creates the vport object and places it under the scsi_host
0092 object corresponding to the physical adapter. The LLDD will allocate
0093 a new scsi_host for the vport and link its object under the vport.
0094 The remainder of the tree under the vports scsi_host is the same
0095 as the non-NPIV case. The transport is written currently to easily
0096 allow the parent of the vport to be something other than the scsi_host.
0097 This could be used in the future to link the object onto a vm-specific
0098 device tree. If the vport's parent is not the physical port's scsi_host,
0099 a symbolic link to the vport object will be placed in the physical
0100 port's scsi_host.
0101
0102 Here's what to expect in the device tree :
0103
0104 The typical Physical Port's Scsi_Host::
0105
0106 /sys/devices/.../host17/
0107
0108 and it has the typical descendant tree::
0109
0110 /sys/devices/.../host17/rport-17:0-0/target17:0:0/17:0:0:0:
0111
0112 and then the vport is created on the Physical Port::
0113
0114 /sys/devices/.../host17/vport-17:0-0
0115
0116 and the vport's Scsi_Host is then created::
0117
0118 /sys/devices/.../host17/vport-17:0-0/host18
0119
0120 and then the rest of the tree progresses, such as::
0121
0122 /sys/devices/.../host17/vport-17:0-0/host18/rport-18:0-0/target18:0:0/18:0:0:0:
0123
0124 Here's what to expect in the sysfs tree::
0125
0126 scsi_hosts:
0127 /sys/class/scsi_host/host17 physical port's scsi_host
0128 /sys/class/scsi_host/host18 vport's scsi_host
0129 fc_hosts:
0130 /sys/class/fc_host/host17 physical port's fc_host
0131 /sys/class/fc_host/host18 vport's fc_host
0132 fc_vports:
0133 /sys/class/fc_vports/vport-17:0-0 the vport's fc_vport
0134 fc_rports:
0135 /sys/class/fc_remote_ports/rport-17:0-0 rport on the physical port
0136 /sys/class/fc_remote_ports/rport-18:0-0 rport on the vport
0137
0138
0139 Vport Attributes
0140 ----------------
0141
0142 The new fc_vport class object has the following attributes
0143
0144 node_name: Read_Only
0145 The WWNN of the vport
0146
0147 port_name: Read_Only
0148 The WWPN of the vport
0149
0150 roles: Read_Only
0151 Indicates the FC4 roles enabled on the vport.
0152
0153 symbolic_name: Read_Write
0154 A string, appended to the driver's symbolic port name string, which
0155 is registered with the switch to identify the vport. For example,
0156 a hypervisor could set this string to "Xen Domain 2 VM 5 Vport 2",
0157 and this set of identifiers can be seen on switch management screens
0158 to identify the port.
0159
0160 vport_delete: Write_Only
0161 When written with a "1", will tear down the vport.
0162
0163 vport_disable: Write_Only
0164 When written with a "1", will transition the vport to a disabled.
0165 state. The vport will still be instantiated with the Linux kernel,
0166 but it will not be active on the FC link.
0167 When written with a "0", will enable the vport.
0168
0169 vport_last_state: Read_Only
0170 Indicates the previous state of the vport. See the section below on
0171 "Vport States".
0172
0173 vport_state: Read_Only
0174 Indicates the state of the vport. See the section below on
0175 "Vport States".
0176
0177 vport_type: Read_Only
0178 Reflects the FC mechanism used to create the virtual port.
0179 Only NPIV is supported currently.
0180
0181
0182 For the fc_host class object, the following attributes are added for vports:
0183
0184 max_npiv_vports: Read_Only
0185 Indicates the maximum number of NPIV-based vports that the
0186 driver/adapter can support on the fc_host.
0187
0188 npiv_vports_inuse: Read_Only
0189 Indicates how many NPIV-based vports have been instantiated on the
0190 fc_host.
0191
0192 vport_create: Write_Only
0193 A "simple" create interface to instantiate a vport on an fc_host.
0194 A "<WWPN>:<WWNN>" string is written to the attribute. The transport
0195 then instantiates the vport object and calls the LLDD to create the
0196 vport with the role of FCP_Initiator. Each WWN is specified as 16
0197 hex characters and may *not* contain any prefixes (e.g. 0x, x, etc).
0198
0199 vport_delete: Write_Only
0200 A "simple" delete interface to teardown a vport. A "<WWPN>:<WWNN>"
0201 string is written to the attribute. The transport will locate the
0202 vport on the fc_host with the same WWNs and tear it down. Each WWN
0203 is specified as 16 hex characters and may *not* contain any prefixes
0204 (e.g. 0x, x, etc).
0205
0206
0207 Vport States
0208 ------------
0209
0210 Vport instantiation consists of two parts:
0211
0212 - Creation with the kernel and LLDD. This means all transport and
0213 driver data structures are built up, and device objects created.
0214 This is equivalent to a driver "attach" on an adapter, which is
0215 independent of the adapter's link state.
0216 - Instantiation of the vport on the FC link via ELS traffic, etc.
0217 This is equivalent to a "link up" and successful link initialization.
0218
0219 Further information can be found in the interfaces section below for
0220 Vport Creation.
0221
0222 Once a vport has been instantiated with the kernel/LLDD, a vport state
0223 can be reported via the sysfs attribute. The following states exist:
0224
0225 FC_VPORT_UNKNOWN - Unknown
0226 An temporary state, typically set only while the vport is being
0227 instantiated with the kernel and LLDD.
0228
0229 FC_VPORT_ACTIVE - Active
0230 The vport has been successfully been created on the FC link.
0231 It is fully functional.
0232
0233 FC_VPORT_DISABLED - Disabled
0234 The vport instantiated, but "disabled". The vport is not instantiated
0235 on the FC link. This is equivalent to a physical port with the
0236 link "down".
0237
0238 FC_VPORT_LINKDOWN - Linkdown
0239 The vport is not operational as the physical link is not operational.
0240
0241 FC_VPORT_INITIALIZING - Initializing
0242 The vport is in the process of instantiating on the FC link.
0243 The LLDD will set this state just prior to starting the ELS traffic
0244 to create the vport. This state will persist until the vport is
0245 successfully created (state becomes FC_VPORT_ACTIVE) or it fails
0246 (state is one of the values below). As this state is transitory,
0247 it will not be preserved in the "vport_last_state".
0248
0249 FC_VPORT_NO_FABRIC_SUPP - No Fabric Support
0250 The vport is not operational. One of the following conditions were
0251 encountered:
0252
0253 - The FC topology is not Point-to-Point
0254 - The FC port is not connected to an F_Port
0255 - The F_Port has indicated that NPIV is not supported.
0256
0257 FC_VPORT_NO_FABRIC_RSCS - No Fabric Resources
0258 The vport is not operational. The Fabric failed FDISC with a status
0259 indicating that it does not have sufficient resources to complete
0260 the operation.
0261
0262 FC_VPORT_FABRIC_LOGOUT - Fabric Logout
0263 The vport is not operational. The Fabric has LOGO'd the N_Port_ID
0264 associated with the vport.
0265
0266 FC_VPORT_FABRIC_REJ_WWN - Fabric Rejected WWN
0267 The vport is not operational. The Fabric failed FDISC with a status
0268 indicating that the WWN's are not valid.
0269
0270 FC_VPORT_FAILED - VPort Failed
0271 The vport is not operational. This is a catchall for all other
0272 error conditions.
0273
0274
0275 The following state table indicates the different state transitions:
0276
0277 +------------------+--------------------------------+---------------------+
0278 | State | Event | New State |
0279 +==================+================================+=====================+
0280 | n/a | Initialization | Unknown |
0281 +------------------+--------------------------------+---------------------+
0282 | Unknown: | Link Down | Linkdown |
0283 | +--------------------------------+---------------------+
0284 | | Link Up & Loop | No Fabric Support |
0285 | +--------------------------------+---------------------+
0286 | | Link Up & no Fabric | No Fabric Support |
0287 | +--------------------------------+---------------------+
0288 | | Link Up & FLOGI response | No Fabric Support |
0289 | | indicates no NPIV support | |
0290 | +--------------------------------+---------------------+
0291 | | Link Up & FDISC being sent | Initializing |
0292 | +--------------------------------+---------------------+
0293 | | Disable request | Disable |
0294 +------------------+--------------------------------+---------------------+
0295 | Linkdown: | Link Up | Unknown |
0296 +------------------+--------------------------------+---------------------+
0297 | Initializing: | FDISC ACC | Active |
0298 | +--------------------------------+---------------------+
0299 | | FDISC LS_RJT w/ no resources | No Fabric Resources |
0300 | +--------------------------------+---------------------+
0301 | | FDISC LS_RJT w/ invalid | Fabric Rejected WWN |
0302 | | pname or invalid nport_id | |
0303 | +--------------------------------+---------------------+
0304 | | FDISC LS_RJT failed for | Vport Failed |
0305 | | other reasons | |
0306 | +--------------------------------+---------------------+
0307 | | Link Down | Linkdown |
0308 | +--------------------------------+---------------------+
0309 | | Disable request | Disable |
0310 +------------------+--------------------------------+---------------------+
0311 | Disable: | Enable request | Unknown |
0312 +------------------+--------------------------------+---------------------+
0313 | Active: | LOGO received from fabric | Fabric Logout |
0314 | +--------------------------------+---------------------+
0315 | | Link Down | Linkdown |
0316 | +--------------------------------+---------------------+
0317 | | Disable request | Disable |
0318 +------------------+--------------------------------+---------------------+
0319 | Fabric Logout: | Link still up | Unknown |
0320 +------------------+--------------------------------+---------------------+
0321
0322 The following 4 error states all have the same transitions::
0323
0324 No Fabric Support:
0325 No Fabric Resources:
0326 Fabric Rejected WWN:
0327 Vport Failed:
0328 Disable request Disable
0329 Link goes down Linkdown
0330
0331
0332 Transport <-> LLDD Interfaces
0333 -----------------------------
0334
0335 Vport support by LLDD:
0336
0337 The LLDD indicates support for vports by supplying a vport_create()
0338 function in the transport template. The presence of this function will
0339 cause the creation of the new attributes on the fc_host. As part of
0340 the physical port completing its initialization relative to the
0341 transport, it should set the max_npiv_vports attribute to indicate the
0342 maximum number of vports the driver and/or adapter supports.
0343
0344
0345 Vport Creation:
0346
0347 The LLDD vport_create() syntax is::
0348
0349 int vport_create(struct fc_vport *vport, bool disable)
0350
0351 where:
0352
0353 ======= ===========================================================
0354 vport Is the newly allocated vport object
0355 disable If "true", the vport is to be created in a disabled stated.
0356 If "false", the vport is to be enabled upon creation.
0357 ======= ===========================================================
0358
0359 When a request is made to create a new vport (via sgio/netlink, or the
0360 vport_create fc_host attribute), the transport will validate that the LLDD
0361 can support another vport (e.g. max_npiv_vports > npiv_vports_inuse).
0362 If not, the create request will be failed. If space remains, the transport
0363 will increment the vport count, create the vport object, and then call the
0364 LLDD's vport_create() function with the newly allocated vport object.
0365
0366 As mentioned above, vport creation is divided into two parts:
0367
0368 - Creation with the kernel and LLDD. This means all transport and
0369 driver data structures are built up, and device objects created.
0370 This is equivalent to a driver "attach" on an adapter, which is
0371 independent of the adapter's link state.
0372 - Instantiation of the vport on the FC link via ELS traffic, etc.
0373 This is equivalent to a "link up" and successful link initialization.
0374
0375 The LLDD's vport_create() function will not synchronously wait for both
0376 parts to be fully completed before returning. It must validate that the
0377 infrastructure exists to support NPIV, and complete the first part of
0378 vport creation (data structure build up) before returning. We do not
0379 hinge vport_create() on the link-side operation mainly because:
0380
0381 - The link may be down. It is not a failure if it is. It simply
0382 means the vport is in an inoperable state until the link comes up.
0383 This is consistent with the link bouncing post vport creation.
0384 - The vport may be created in a disabled state.
0385 - This is consistent with a model where: the vport equates to a
0386 FC adapter. The vport_create is synonymous with driver attachment
0387 to the adapter, which is independent of link state.
0388
0389 .. Note::
0390
0391 special error codes have been defined to delineate infrastructure
0392 failure cases for quicker resolution.
0393
0394 The expected behavior for the LLDD's vport_create() function is:
0395
0396 - Validate Infrastructure:
0397
0398 - If the driver or adapter cannot support another vport, whether
0399 due to improper firmware, (a lie about) max_npiv, or a lack of
0400 some other resource - return VPCERR_UNSUPPORTED.
0401 - If the driver validates the WWN's against those already active on
0402 the adapter and detects an overlap - return VPCERR_BAD_WWN.
0403 - If the driver detects the topology is loop, non-fabric, or the
0404 FLOGI did not support NPIV - return VPCERR_NO_FABRIC_SUPP.
0405
0406 - Allocate data structures. If errors are encountered, such as out
0407 of memory conditions, return the respective negative Exxx error code.
0408 - If the role is FCP Initiator, the LLDD is to :
0409
0410 - Call scsi_host_alloc() to allocate a scsi_host for the vport.
0411 - Call scsi_add_host(new_shost, &vport->dev) to start the scsi_host
0412 and bind it as a child of the vport device.
0413 - Initializes the fc_host attribute values.
0414
0415 - Kick of further vport state transitions based on the disable flag and
0416 link state - and return success (zero).
0417
0418 LLDD Implementers Notes:
0419
0420 - It is suggested that there be a different fc_function_templates for
0421 the physical port and the virtual port. The physical port's template
0422 would have the vport_create, vport_delete, and vport_disable functions,
0423 while the vports would not.
0424 - It is suggested that there be different scsi_host_templates
0425 for the physical port and virtual port. Likely, there are driver
0426 attributes, embedded into the scsi_host_template, that are applicable
0427 for the physical port only (link speed, topology setting, etc). This
0428 ensures that the attributes are applicable to the respective scsi_host.
0429
0430
0431 Vport Disable/Enable:
0432
0433 The LLDD vport_disable() syntax is::
0434
0435 int vport_disable(struct fc_vport *vport, bool disable)
0436
0437 where:
0438
0439 ======= =======================================
0440 vport Is vport to be enabled or disabled
0441 disable If "true", the vport is to be disabled.
0442 If "false", the vport is to be enabled.
0443 ======= =======================================
0444
0445 When a request is made to change the disabled state on a vport, the
0446 transport will validate the request against the existing vport state.
0447 If the request is to disable and the vport is already disabled, the
0448 request will fail. Similarly, if the request is to enable, and the
0449 vport is not in a disabled state, the request will fail. If the request
0450 is valid for the vport state, the transport will call the LLDD to
0451 change the vport's state.
0452
0453 Within the LLDD, if a vport is disabled, it remains instantiated with
0454 the kernel and LLDD, but it is not active or visible on the FC link in
0455 any way. (see Vport Creation and the 2 part instantiation discussion).
0456 The vport will remain in this state until it is deleted or re-enabled.
0457 When enabling a vport, the LLDD reinstantiates the vport on the FC
0458 link - essentially restarting the LLDD statemachine (see Vport States
0459 above).
0460
0461
0462 Vport Deletion:
0463
0464 The LLDD vport_delete() syntax is::
0465
0466 int vport_delete(struct fc_vport *vport)
0467
0468 where:
0469
0470 vport: Is vport to delete
0471
0472 When a request is made to delete a vport (via sgio/netlink, or via the
0473 fc_host or fc_vport vport_delete attributes), the transport will call
0474 the LLDD to terminate the vport on the FC link, and teardown all other
0475 datastructures and references. If the LLDD completes successfully,
0476 the transport will teardown the vport objects and complete the vport
0477 removal. If the LLDD delete request fails, the vport object will remain,
0478 but will be in an indeterminate state.
0479
0480 Within the LLDD, the normal code paths for a scsi_host teardown should
0481 be followed. E.g. If the vport has a FCP Initiator role, the LLDD
0482 will call fc_remove_host() for the vports scsi_host, followed by
0483 scsi_remove_host() and scsi_host_put() for the vports scsi_host.
0484
0485
0486 Other:
0487 fc_host port_type attribute:
0488 There is a new fc_host port_type value - FC_PORTTYPE_NPIV. This value
0489 must be set on all vport-based fc_hosts. Normally, on a physical port,
0490 the port_type attribute would be set to NPORT, NLPORT, etc based on the
0491 topology type and existence of the fabric. As this is not applicable to
0492 a vport, it makes more sense to report the FC mechanism used to create
0493 the vport.
0494
0495 Driver unload:
0496 FC drivers are required to call fc_remove_host() prior to calling
0497 scsi_remove_host(). This allows the fc_host to tear down all remote
0498 ports prior the scsi_host being torn down. The fc_remove_host() call
0499 was updated to remove all vports for the fc_host as well.
0500
0501
0502 Transport supplied functions
0503 ----------------------------
0504
0505 The following functions are supplied by the FC-transport for use by LLDs.
0506
0507 ================== =========================
0508 fc_vport_create create a vport
0509 fc_vport_terminate detach and remove a vport
0510 ================== =========================
0511
0512 Details::
0513
0514 /**
0515 * fc_vport_create - Admin App or LLDD requests creation of a vport
0516 * @shost: scsi host the virtual port is connected to.
0517 * @ids: The world wide names, FC4 port roles, etc for
0518 * the virtual port.
0519 *
0520 * Notes:
0521 * This routine assumes no locks are held on entry.
0522 */
0523 struct fc_vport *
0524 fc_vport_create(struct Scsi_Host *shost, struct fc_vport_identifiers *ids)
0525
0526 /**
0527 * fc_vport_terminate - Admin App or LLDD requests termination of a vport
0528 * @vport: fc_vport to be terminated
0529 *
0530 * Calls the LLDD vport_delete() function, then deallocates and removes
0531 * the vport from the shost and object tree.
0532 *
0533 * Notes:
0534 * This routine assumes no locks are held on entry.
0535 */
0536 int
0537 fc_vport_terminate(struct fc_vport *vport)
0538
0539
0540 FC BSG support (CT & ELS passthru, and more)
0541 ============================================
0542
0543 << To Be Supplied >>
0544
0545
0546
0547
0548
0549 Credits
0550 =======
0551 The following people have contributed to this document:
0552
0553
0554
0555
0556
0557
0558 James Smart
0559 james.smart@emulex.com
0560
