Back to home page

LXR

 
 

    


0001 The cluster MD is a shared-device RAID for a cluster.
0002 
0003 
0004 1. On-disk format
0005 
0006 Separate write-intent-bitmaps are used for each cluster node.
0007 The bitmaps record all writes that may have been started on that node,
0008 and may not yet have finished. The on-disk layout is:
0009 
0010 0                    4k                     8k                    12k
0011 -------------------------------------------------------------------
0012 | idle                | md super            | bm super [0] + bits |
0013 | bm bits[0, contd]   | bm super[1] + bits  | bm bits[1, contd]   |
0014 | bm super[2] + bits  | bm bits [2, contd]  | bm super[3] + bits  |
0015 | bm bits [3, contd]  |                     |                     |
0016 
0017 During "normal" functioning we assume the filesystem ensures that only
0018 one node writes to any given block at a time, so a write request will
0019 
0020  - set the appropriate bit (if not already set)
0021  - commit the write to all mirrors
0022  - schedule the bit to be cleared after a timeout.
0023 
0024 Reads are just handled normally. It is up to the filesystem to ensure
0025 one node doesn't read from a location where another node (or the same
0026 node) is writing.
0027 
0028 
0029 2. DLM Locks for management
0030 
0031 There are three groups of locks for managing the device:
0032 
0033 2.1 Bitmap lock resource (bm_lockres)
0034 
0035  The bm_lockres protects individual node bitmaps. They are named in
0036  the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
0037  node joins the cluster, it acquires the lock in PW mode and it stays
0038  so during the lifetime the node is part of the cluster. The lock
0039  resource number is based on the slot number returned by the DLM
0040  subsystem. Since DLM starts node count from one and bitmap slots
0041  start from zero, one is subtracted from the DLM slot number to arrive
0042  at the bitmap slot number.
0043 
0044  The LVB of the bitmap lock for a particular node records the range
0045  of sectors that are being re-synced by that node.  No other
0046  node may write to those sectors.  This is used when a new nodes
0047  joins the cluster.
0048 
0049 2.2 Message passing locks
0050 
0051  Each node has to communicate with other nodes when starting or ending
0052  resync, and for metadata superblock updates.  This communication is
0053  managed through three locks: "token", "message", and "ack", together
0054  with the Lock Value Block (LVB) of one of the "message" lock.
0055 
0056 2.3 new-device management
0057 
0058  A single lock: "no-new-dev" is used to co-ordinate the addition of
0059  new devices - this must be synchronized across the array.
0060  Normally all nodes hold a concurrent-read lock on this device.
0061 
0062 3. Communication
0063 
0064  Messages can be broadcast to all nodes, and the sender waits for all
0065  other nodes to acknowledge the message before proceeding.  Only one
0066  message can be processed at a time.
0067 
0068 3.1 Message Types
0069 
0070  There are six types of messages which are passed:
0071 
0072  3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
0073    been updated, and the node must re-read the md superblock. This is
0074    performed synchronously. It is primarily used to signal device
0075    failure.
0076 
0077  3.1.2 RESYNCING: informs other nodes that a resync is initiated or
0078    ended so that each node may suspend or resume the region.  Each
0079    RESYNCING message identifies a range of the devices that the
0080    sending node is about to resync. This over-rides any pervious
0081    notification from that node: only one ranged can be resynced at a
0082    time per-node.
0083 
0084  3.1.3 NEWDISK: informs other nodes that a device is being added to
0085    the array. Message contains an identifier for that device.  See
0086    below for further details.
0087 
0088  3.1.4 REMOVE: A failed or spare device is being removed from the
0089    array. The slot-number of the device is included in the message.
0090 
0091  3.1.5 RE_ADD: A failed device is being re-activated - the assumption
0092    is that it has been determined to be working again.
0093 
0094  3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
0095    isn't clean, then another node is informed to take the ownership of
0096    resync.
0097 
0098 3.2 Communication mechanism
0099 
0100  The DLM LVB is used to communicate within nodes of the cluster. There
0101  are three resources used for the purpose:
0102 
0103   3.2.1 token: The resource which protects the entire communication
0104    system. The node having the token resource is allowed to
0105    communicate.
0106 
0107   3.2.2 message: The lock resource which carries the data to
0108    communicate.
0109 
0110   3.2.3 ack: The resource, acquiring which means the message has been
0111    acknowledged by all nodes in the cluster. The BAST of the resource
0112    is used to inform the receiving node that a node wants to
0113    communicate.
0114 
0115 The algorithm is:
0116 
0117  1. receive status - all nodes have concurrent-reader lock on "ack".
0118 
0119    sender                         receiver                 receiver
0120    "ack":CR                       "ack":CR                 "ack":CR
0121 
0122  2. sender get EX on "token"
0123     sender get EX on "message"
0124     sender                        receiver                 receiver
0125     "token":EX                    "ack":CR                 "ack":CR
0126     "message":EX
0127     "ack":CR
0128 
0129     Sender checks that it still needs to send a message. Messages
0130     received or other events that happened while waiting for the
0131     "token" may have made this message inappropriate or redundant.
0132 
0133  3. sender writes LVB.
0134     sender down-convert "message" from EX to CW
0135     sender try to get EX of "ack"
0136     [ wait until all receivers have *processed* the "message" ]
0137 
0138                                      [ triggered by bast of "ack" ]
0139                                      receiver get CR on "message"
0140                                      receiver read LVB
0141                                      receiver processes the message
0142                                      [ wait finish ]
0143                                      receiver releases "ack"
0144                                      receiver tries to get PR on "message"
0145 
0146    sender                         receiver                  receiver
0147    "token":EX                     "message":CR              "message":CR
0148    "message":CW
0149    "ack":EX
0150 
0151  4. triggered by grant of EX on "ack" (indicating all receivers
0152     have processed message)
0153     sender down-converts "ack" from EX to CR
0154     sender releases "message"
0155     sender releases "token"
0156                                receiver upconvert to PR on "message"
0157                                receiver get CR of "ack"
0158                                receiver release "message"
0159 
0160    sender                      receiver                   receiver
0161    "ack":CR                    "ack":CR                   "ack":CR
0162 
0163 
0164 4. Handling Failures
0165 
0166 4.1 Node Failure
0167 
0168  When a node fails, the DLM informs the cluster with the slot
0169  number. The node starts a cluster recovery thread. The cluster
0170  recovery thread:
0171 
0172         - acquires the bitmap<number> lock of the failed node
0173         - opens the bitmap
0174         - reads the bitmap of the failed node
0175         - copies the set bitmap to local node
0176         - cleans the bitmap of the failed node
0177         - releases bitmap<number> lock of the failed node
0178         - initiates resync of the bitmap on the current node
0179                 md_check_recovery is invoked within recover_bitmaps,
0180                 then md_check_recovery -> metadata_update_start/finish,
0181                 it will lock the communication by lock_comm.
0182                 Which means when one node is resyncing it blocks all
0183                 other nodes from writing anywhere on the array.
0184 
0185  The resync process is the regular md resync. However, in a clustered
0186  environment when a resync is performed, it needs to tell other nodes
0187  of the areas which are suspended. Before a resync starts, the node
0188  send out RESYNCING with the (lo,hi) range of the area which needs to
0189  be suspended. Each node maintains a suspend_list, which contains the
0190  list of ranges which are currently suspended. On receiving RESYNCING,
0191  the node adds the range to the suspend_list. Similarly, when the node
0192  performing resync finishes, it sends RESYNCING with an empty range to
0193  other nodes and other nodes remove the corresponding entry from the
0194  suspend_list.
0195 
0196  A helper function, ->area_resyncing() can be used to check if a
0197  particular I/O range should be suspended or not.
0198 
0199 4.2 Device Failure
0200 
0201  Device failures are handled and communicated with the metadata update
0202  routine.  When a node detects a device failure it does not allow
0203  any further writes to that device until the failure has been
0204  acknowledged by all other nodes.
0205 
0206 5. Adding a new Device
0207 
0208  For adding a new device, it is necessary that all nodes "see" the new
0209  device to be added. For this, the following algorithm is used:
0210 
0211     1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
0212        ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
0213     2. Node 1 sends a NEWDISK message with uuid and slot number
0214     3. Other nodes issue kobject_uevent_env with uuid and slot number
0215        (Steps 4,5 could be a udev rule)
0216     4. In userspace, the node searches for the disk, perhaps
0217        using blkid -t SUB_UUID=""
0218     5. Other nodes issue either of the following depending on whether
0219        the disk was found:
0220        ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
0221              disc.number set to slot number)
0222        ioctl(CLUSTERED_DISK_NACK)
0223     6. Other nodes drop lock on "no-new-devs" (CR) if device is found
0224     7. Node 1 attempts EX lock on "no-new-dev"
0225     8. If node 1 gets the lock, it sends METADATA_UPDATED after
0226        unmarking the disk as SpareLocal
0227     9. If not (get "no-new-dev" lock), it fails the operation and sends
0228        METADATA_UPDATED.
0229    10. Other nodes get the information whether a disk is added or not
0230        by the following METADATA_UPDATED.
0231 
0232 6. Module interface.
0233 
0234  There are 17 call-backs which the md core can make to the cluster
0235  module.  Understanding these can give a good overview of the whole
0236  process.
0237 
0238 6.1 join(nodes) and leave()
0239 
0240  These are called when an array is started with a clustered bitmap,
0241  and when the array is stopped.  join() ensures the cluster is
0242  available and initializes the various resources.
0243  Only the first 'nodes' nodes in the cluster can use the array.
0244 
0245 6.2 slot_number()
0246 
0247  Reports the slot number advised by the cluster infrastructure.
0248  Range is from 0 to nodes-1.
0249 
0250 6.3 resync_info_update()
0251 
0252  This updates the resync range that is stored in the bitmap lock.
0253  The starting point is updated as the resync progresses.  The
0254  end point is always the end of the array.
0255  It does *not* send a RESYNCING message.
0256 
0257 6.4 resync_start(), resync_finish()
0258 
0259  These are called when resync/recovery/reshape starts or stops.
0260  They update the resyncing range in the bitmap lock and also
0261  send a RESYNCING message.  resync_start reports the whole
0262  array as resyncing, resync_finish reports none of it.
0263 
0264  resync_finish() also sends a BITMAP_NEEDS_SYNC message which
0265  allows some other node to take over.
0266 
0267 6.5 metadata_update_start(), metadata_update_finish(),
0268     metadata_update_cancel().
0269 
0270  metadata_update_start is used to get exclusive access to
0271  the metadata.  If a change is still needed once that access is
0272  gained, metadata_update_finish() will send a METADATA_UPDATE
0273  message to all other nodes, otherwise metadata_update_cancel()
0274  can be used to release the lock.
0275 
0276 6.6 area_resyncing()
0277 
0278  This combines two elements of functionality.
0279 
0280  Firstly, it will check if any node is currently resyncing
0281  anything in a given range of sectors.  If any resync is found,
0282  then the caller will avoid writing or read-balancing in that
0283  range.
0284 
0285  Secondly, while node recovery is happening it reports that
0286  all areas are resyncing for READ requests.  This avoids races
0287  between the cluster-filesystem and the cluster-RAID handling
0288  a node failure.
0289 
0290 6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()
0291 
0292  These are used to manage the new-disk protocol described above.
0293  When a new device is added, add_new_disk_start() is called before
0294  it is bound to the array and, if that succeeds, add_new_disk_finish()
0295  is called the device is fully added.
0296 
0297  When a device is added in acknowledgement to a previous
0298  request, or when the device is declared "unavailable",
0299  new_disk_ack() is called.
0300 
0301 6.8 remove_disk()
0302 
0303  This is called when a spare or failed device is removed from
0304  the array.  It causes a REMOVE message to be send to other nodes.
0305 
0306 6.9 gather_bitmaps()
0307 
0308  This sends a RE_ADD message to all other nodes and then
0309  gathers bitmap information from all bitmaps.  This combined
0310  bitmap is then used to recovery the re-added device.
0311 
0312 6.10 lock_all_bitmaps() and unlock_all_bitmaps()
0313 
0314  These are called when change bitmap to none. If a node plans
0315  to clear the cluster raid's bitmap, it need to make sure no other
0316  nodes are using the raid which is achieved by lock all bitmap
0317  locks within the cluster, and also those locks are unlocked
0318  accordingly.
0319 
0320 7. Unsupported features
0321 
0322 There are somethings which are not supported by cluster MD yet.
0323 
0324 - update size and change array_sectors.