0001 .. SPDX-License-Identifier: GPL-2.0
0002
0003 =========
0004 SAS Layer
0005 =========
0006
0007 The SAS Layer is a management infrastructure which manages
0008 SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The
0009 layout is as follows: while SCSI Core is concerned with
0010 SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
0011 phy/OOB/link management, the SAS layer is concerned with:
0012
0013 * SAS Phy/Port/HA event management (LLDD generates,
0014 SAS Layer processes),
0015 * SAS Port management (creation/destruction),
0016 * SAS Domain discovery and revalidation,
0017 * SAS Domain device management,
0018 * SCSI Host registration/unregistration,
0019 * Device registration with SCSI Core (SAS) or libata
0020 (SATA), and
0021 * Expander management and exporting expander control
0022 to user space.
0023
0024 A SAS LLDD is a PCI device driver. It is concerned with
0025 phy/OOB management, and vendor specific tasks and generates
0026 events to the SAS layer.
0027
0028 The SAS Layer does most SAS tasks as outlined in the SAS 1.1
0029 spec.
0030
0031 The sas_ha_struct describes the SAS LLDD to the SAS layer.
0032 Most of it is used by the SAS Layer but a few fields need to
0033 be initialized by the LLDDs.
0034
0035 After initializing your hardware, from the probe() function
0036 you call sas_register_ha(). It will register your LLDD with
0037 the SCSI subsystem, creating a SCSI host and it will
0038 register your SAS driver with the sysfs SAS tree it creates.
0039 It will then return. Then you enable your phys to actually
0040 start OOB (at which point your driver will start calling the
0041 notify_* event callbacks).
0042
0043 Structure descriptions
0044 ======================
0045
0046 ``struct sas_phy``
0047 ------------------
0048
0049 Normally this is statically embedded to your driver's
0050 phy structure::
0051
0052 struct my_phy {
0053 blah;
0054 struct sas_phy sas_phy;
0055 bleh;
0056 };
0057
0058 And then all the phys are an array of my_phy in your HA
0059 struct (shown below).
0060
0061 Then as you go along and initialize your phys you also
0062 initialize the sas_phy struct, along with your own
0063 phy structure.
0064
0065 In general, the phys are managed by the LLDD and the ports
0066 are managed by the SAS layer. So the phys are initialized
0067 and updated by the LLDD and the ports are initialized and
0068 updated by the SAS layer.
0069
0070 There is a scheme where the LLDD can RW certain fields,
0071 and the SAS layer can only read such ones, and vice versa.
0072 The idea is to avoid unnecessary locking.
0073
0074 enabled
0075 - must be set (0/1)
0076
0077 id
0078 - must be set [0,MAX_PHYS)]
0079
0080 class, proto, type, role, oob_mode, linkrate
0081 - must be set
0082
0083 oob_mode
0084 - you set this when OOB has finished and then notify
0085 the SAS Layer.
0086
0087 sas_addr
0088 - this normally points to an array holding the sas
0089 address of the phy, possibly somewhere in your my_phy
0090 struct.
0091
0092 attached_sas_addr
0093 - set this when you (LLDD) receive an
0094 IDENTIFY frame or a FIS frame, _before_ notifying the SAS
0095 layer. The idea is that sometimes the LLDD may want to fake
0096 or provide a different SAS address on that phy/port and this
0097 allows it to do this. At best you should copy the sas
0098 address from the IDENTIFY frame or maybe generate a SAS
0099 address for SATA directly attached devices. The Discover
0100 process may later change this.
0101
0102 frame_rcvd
0103 - this is where you copy the IDENTIFY/FIS frame
0104 when you get it; you lock, copy, set frame_rcvd_size and
0105 unlock the lock, and then call the event. It is a pointer
0106 since there's no way to know your hw frame size _exactly_,
0107 so you define the actual array in your phy struct and let
0108 this pointer point to it. You copy the frame from your
0109 DMAable memory to that area holding the lock.
0110
0111 sas_prim
0112 - this is where primitives go when they're
0113 received. See sas.h. Grab the lock, set the primitive,
0114 release the lock, notify.
0115
0116 port
0117 - this points to the sas_port if the phy belongs
0118 to a port -- the LLDD only reads this. It points to the
0119 sas_port this phy is part of. Set by the SAS Layer.
0120
0121 ha
0122 - may be set; the SAS layer sets it anyway.
0123
0124 lldd_phy
0125 - you should set this to point to your phy so you
0126 can find your way around faster when the SAS layer calls one
0127 of your callbacks and passes you a phy. If the sas_phy is
0128 embedded you can also use container_of -- whatever you
0129 prefer.
0130
0131
0132 ``struct sas_port``
0133 -------------------
0134
0135 The LLDD doesn't set any fields of this struct -- it only
0136 reads them. They should be self explanatory.
0137
0138 phy_mask is 32 bit, this should be enough for now, as I
0139 haven't heard of a HA having more than 8 phys.
0140
0141 lldd_port
0142 - I haven't found use for that -- maybe other
0143 LLDD who wish to have internal port representation can make
0144 use of this.
0145
0146 ``struct sas_ha_struct``
0147 ------------------------
0148
0149 It normally is statically declared in your own LLDD
0150 structure describing your adapter::
0151
0152 struct my_sas_ha {
0153 blah;
0154 struct sas_ha_struct sas_ha;
0155 struct my_phy phys[MAX_PHYS];
0156 struct sas_port sas_ports[MAX_PHYS]; /* (1) */
0157 bleh;
0158 };
0159
0160 (1) If your LLDD doesn't have its own port representation.
0161
0162 What needs to be initialized (sample function given below).
0163
0164 pcidev
0165 ^^^^^^
0166
0167 sas_addr
0168 - since the SAS layer doesn't want to mess with
0169 memory allocation, etc, this points to statically
0170 allocated array somewhere (say in your host adapter
0171 structure) and holds the SAS address of the host
0172 adapter as given by you or the manufacturer, etc.
0173
0174 sas_port
0175 ^^^^^^^^
0176
0177 sas_phy
0178 - an array of pointers to structures. (see
0179 note above on sas_addr).
0180 These must be set. See more notes below.
0181
0182 num_phys
0183 - the number of phys present in the sas_phy array,
0184 and the number of ports present in the sas_port
0185 array. There can be a maximum num_phys ports (one per
0186 port) so we drop the num_ports, and only use
0187 num_phys.
0188
0189 The event interface::
0190
0191 /* LLDD calls these to notify the class of an event. */
0192 void sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t);
0193 void sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t);
0194
0195 The port notification::
0196
0197 /* The class calls these to notify the LLDD of an event. */
0198 void (*lldd_port_formed)(struct sas_phy *);
0199 void (*lldd_port_deformed)(struct sas_phy *);
0200
0201 If the LLDD wants notification when a port has been formed
0202 or deformed it sets those to a function satisfying the type.
0203
0204 A SAS LLDD should also implement at least one of the Task
0205 Management Functions (TMFs) described in SAM::
0206
0207 /* Task Management Functions. Must be called from process context. */
0208 int (*lldd_abort_task)(struct sas_task *);
0209 int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
0210 int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
0211 int (*lldd_I_T_nexus_reset)(struct domain_device *);
0212 int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
0213 int (*lldd_query_task)(struct sas_task *);
0214
0215 For more information please read SAM from T10.org.
0216
0217 Port and Adapter management::
0218
0219 /* Port and Adapter management */
0220 int (*lldd_clear_nexus_port)(struct sas_port *);
0221 int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
0222
0223 A SAS LLDD should implement at least one of those.
0224
0225 Phy management::
0226
0227 /* Phy management */
0228 int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
0229
0230 lldd_ha
0231 - set this to point to your HA struct. You can also
0232 use container_of if you embedded it as shown above.
0233
0234 A sample initialization and registration function
0235 can look like this (called last thing from probe())
0236 *but* before you enable the phys to do OOB::
0237
0238 static int register_sas_ha(struct my_sas_ha *my_ha)
0239 {
0240 int i;
0241 static struct sas_phy *sas_phys[MAX_PHYS];
0242 static struct sas_port *sas_ports[MAX_PHYS];
0243
0244 my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
0245
0246 for (i = 0; i < MAX_PHYS; i++) {
0247 sas_phys[i] = &my_ha->phys[i].sas_phy;
0248 sas_ports[i] = &my_ha->sas_ports[i];
0249 }
0250
0251 my_ha->sas_ha.sas_phy = sas_phys;
0252 my_ha->sas_ha.sas_port = sas_ports;
0253 my_ha->sas_ha.num_phys = MAX_PHYS;
0254
0255 my_ha->sas_ha.lldd_port_formed = my_port_formed;
0256
0257 my_ha->sas_ha.lldd_dev_found = my_dev_found;
0258 my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
0259
0260 my_ha->sas_ha.lldd_execute_task = my_execute_task;
0261
0262 my_ha->sas_ha.lldd_abort_task = my_abort_task;
0263 my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
0264 my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
0265 my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
0266 my_ha->sas_ha.lldd_lu_reset = my_lu_reset;
0267 my_ha->sas_ha.lldd_query_task = my_query_task;
0268
0269 my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
0270 my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
0271
0272 my_ha->sas_ha.lldd_control_phy = my_control_phy;
0273
0274 return sas_register_ha(&my_ha->sas_ha);
0275 }
0276
0277 (2) SAS 1.1 does not define I_T Nexus Reset TMF.
0278
0279 Events
0280 ======
0281
0282 Events are **the only way** a SAS LLDD notifies the SAS layer
0283 of anything. There is no other method or way a LLDD to tell
0284 the SAS layer of anything happening internally or in the SAS
0285 domain.
0286
0287 Phy events::
0288
0289 PHYE_LOSS_OF_SIGNAL, (C)
0290 PHYE_OOB_DONE,
0291 PHYE_OOB_ERROR, (C)
0292 PHYE_SPINUP_HOLD.
0293
0294 Port events, passed on a _phy_::
0295
0296 PORTE_BYTES_DMAED, (M)
0297 PORTE_BROADCAST_RCVD, (E)
0298 PORTE_LINK_RESET_ERR, (C)
0299 PORTE_TIMER_EVENT, (C)
0300 PORTE_HARD_RESET.
0301
0302 Host Adapter event:
0303 HAE_RESET
0304
0305 A SAS LLDD should be able to generate
0306
0307 - at least one event from group C (choice),
0308 - events marked M (mandatory) are mandatory (only one),
0309 - events marked E (expander) if it wants the SAS layer
0310 to handle domain revalidation (only one such).
0311 - Unmarked events are optional.
0312
0313 Meaning:
0314
0315 HAE_RESET
0316 - when your HA got internal error and was reset.
0317
0318 PORTE_BYTES_DMAED
0319 - on receiving an IDENTIFY/FIS frame
0320
0321 PORTE_BROADCAST_RCVD
0322 - on receiving a primitive
0323
0324 PORTE_LINK_RESET_ERR
0325 - timer expired, loss of signal, loss of DWS, etc. [1]_
0326
0327 PORTE_TIMER_EVENT
0328 - DWS reset timeout timer expired [1]_
0329
0330 PORTE_HARD_RESET
0331 - Hard Reset primitive received.
0332
0333 PHYE_LOSS_OF_SIGNAL
0334 - the device is gone [1]_
0335
0336 PHYE_OOB_DONE
0337 - OOB went fine and oob_mode is valid
0338
0339 PHYE_OOB_ERROR
0340 - Error while doing OOB, the device probably
0341 got disconnected. [1]_
0342
0343 PHYE_SPINUP_HOLD
0344 - SATA is present, COMWAKE not sent.
0345
0346 .. [1] should set/clear the appropriate fields in the phy,
0347 or alternatively call the inlined sas_phy_disconnected()
0348 which is just a helper, from their tasklet.
0349
0350 The Execute Command SCSI RPC::
0351
0352 int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);
0353
0354 Used to queue a task to the SAS LLDD. @task is the task to be executed.
0355 @gfp_mask is the gfp_mask defining the context of the caller.
0356
0357 This function should implement the Execute Command SCSI RPC,
0358
0359 That is, when lldd_execute_task() is called, the command
0360 go out on the transport *immediately*. There is *no*
0361 queuing of any sort and at any level in a SAS LLDD.
0362
0363 Returns:
0364
0365 * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
0366 * 0, the task(s) were queued.
0367
0368 ::
0369
0370 struct sas_task {
0371 dev -- the device this task is destined to
0372 task_proto -- _one_ of enum sas_proto
0373 scatter -- pointer to scatter gather list array
0374 num_scatter -- number of elements in scatter
0375 total_xfer_len -- total number of bytes expected to be transferred
0376 data_dir -- PCI_DMA_...
0377 task_done -- callback when the task has finished execution
0378 };
0379
0380 Discovery
0381 =========
0382
0383 The sysfs tree has the following purposes:
0384
0385 a) It shows you the physical layout of the SAS domain at
0386 the current time, i.e. how the domain looks in the
0387 physical world right now.
0388 b) Shows some device parameters _at_discovery_time_.
0389
0390 This is a link to the tree(1) program, very useful in
0391 viewing the SAS domain:
0392 ftp://mama.indstate.edu/linux/tree/
0393
0394 I expect user space applications to actually create a
0395 graphical interface of this.
0396
0397 That is, the sysfs domain tree doesn't show or keep state if
0398 you e.g., change the meaning of the READY LED MEANING
0399 setting, but it does show you the current connection status
0400 of the domain device.
0401
0402 Keeping internal device state changes is responsibility of
0403 upper layers (Command set drivers) and user space.
0404
0405 When a device or devices are unplugged from the domain, this
0406 is reflected in the sysfs tree immediately, and the device(s)
0407 removed from the system.
0408
0409 The structure domain_device describes any device in the SAS
0410 domain. It is completely managed by the SAS layer. A task
0411 points to a domain device, this is how the SAS LLDD knows
0412 where to send the task(s) to. A SAS LLDD only reads the
0413 contents of the domain_device structure, but it never creates
0414 or destroys one.
0415
0416 Expander management from User Space
0417 ===================================
0418
0419 In each expander directory in sysfs, there is a file called
0420 "smp_portal". It is a binary sysfs attribute file, which
0421 implements an SMP portal (Note: this is *NOT* an SMP port),
0422 to which user space applications can send SMP requests and
0423 receive SMP responses.
0424
0425 Functionality is deceptively simple:
0426
0427 1. Build the SMP frame you want to send. The format and layout
0428 is described in the SAS spec. Leave the CRC field equal 0.
0429
0430 open(2)
0431
0432 2. Open the expander's SMP portal sysfs file in RW mode.
0433
0434 write(2)
0435
0436 3. Write the frame you built in 1.
0437
0438 read(2)
0439
0440 4. Read the amount of data you expect to receive for the frame you built.
0441 If you receive different amount of data you expected to receive,
0442 then there was some kind of error.
0443
0444 close(2)
0445
0446 All this process is shown in detail in the function do_smp_func()
0447 and its callers, in the file "expander_conf.c".
0448
0449 The kernel functionality is implemented in the file
0450 "sas_expander.c".
0451
0452 The program "expander_conf.c" implements this. It takes one
0453 argument, the sysfs file name of the SMP portal to the
0454 expander, and gives expander information, including routing
0455 tables.
0456
0457 The SMP portal gives you complete control of the expander,
0458 so please be careful.
