0001 .. SPDX-License-Identifier: GPL-2.0-only
0002
0003 ========
0004 dm-clone
0005 ========
0006
0007 Introduction
0008 ============
0009
0010 dm-clone is a device mapper target which produces a one-to-one copy of an
0011 existing, read-only source device into a writable destination device: It
0012 presents a virtual block device which makes all data appear immediately, and
0013 redirects reads and writes accordingly.
0014
0015 The main use case of dm-clone is to clone a potentially remote, high-latency,
0016 read-only, archival-type block device into a writable, fast, primary-type device
0017 for fast, low-latency I/O. The cloned device is visible/mountable immediately
0018 and the copy of the source device to the destination device happens in the
0019 background, in parallel with user I/O.
0020
0021 For example, one could restore an application backup from a read-only copy,
0022 accessible through a network storage protocol (NBD, Fibre Channel, iSCSI, AoE,
0023 etc.), into a local SSD or NVMe device, and start using the device immediately,
0024 without waiting for the restore to complete.
0025
0026 When the cloning completes, the dm-clone table can be removed altogether and be
0027 replaced, e.g., by a linear table, mapping directly to the destination device.
0028
0029 The dm-clone target reuses the metadata library used by the thin-provisioning
0030 target.
0031
0032 Glossary
0033 ========
0034
0035 Hydration
0036 The process of filling a region of the destination device with data from
0037 the same region of the source device, i.e., copying the region from the
0038 source to the destination device.
0039
0040 Once a region gets hydrated we redirect all I/O regarding it to the destination
0041 device.
0042
0043 Design
0044 ======
0045
0046 Sub-devices
0047 -----------
0048
0049 The target is constructed by passing three devices to it (along with other
0050 parameters detailed later):
0051
0052 1. A source device - the read-only device that gets cloned and source of the
0053 hydration.
0054
0055 2. A destination device - the destination of the hydration, which will become a
0056 clone of the source device.
0057
0058 3. A small metadata device - it records which regions are already valid in the
0059 destination device, i.e., which regions have already been hydrated, or have
0060 been written to directly, via user I/O.
0061
0062 The size of the destination device must be at least equal to the size of the
0063 source device.
0064
0065 Regions
0066 -------
0067
0068 dm-clone divides the source and destination devices in fixed sized regions.
0069 Regions are the unit of hydration, i.e., the minimum amount of data copied from
0070 the source to the destination device.
0071
0072 The region size is configurable when you first create the dm-clone device. The
0073 recommended region size is the same as the file system block size, which usually
0074 is 4KB. The region size must be between 8 sectors (4KB) and 2097152 sectors
0075 (1GB) and a power of two.
0076
0077 Reads and writes from/to hydrated regions are serviced from the destination
0078 device.
0079
0080 A read to a not yet hydrated region is serviced directly from the source device.
0081
0082 A write to a not yet hydrated region will be delayed until the corresponding
0083 region has been hydrated and the hydration of the region starts immediately.
0084
0085 Note that a write request with size equal to region size will skip copying of
0086 the corresponding region from the source device and overwrite the region of the
0087 destination device directly.
0088
0089 Discards
0090 --------
0091
0092 dm-clone interprets a discard request to a range that hasn't been hydrated yet
0093 as a hint to skip hydration of the regions covered by the request, i.e., it
0094 skips copying the region's data from the source to the destination device, and
0095 only updates its metadata.
0096
0097 If the destination device supports discards, then by default dm-clone will pass
0098 down discard requests to it.
0099
0100 Background Hydration
0101 --------------------
0102
0103 dm-clone copies continuously from the source to the destination device, until
0104 all of the device has been copied.
0105
0106 Copying data from the source to the destination device uses bandwidth. The user
0107 can set a throttle to prevent more than a certain amount of copying occurring at
0108 any one time. Moreover, dm-clone takes into account user I/O traffic going to
0109 the devices and pauses the background hydration when there is I/O in-flight.
0110
0111 A message `hydration_threshold <#regions>` can be used to set the maximum number
0112 of regions being copied, the default being 1 region.
0113
0114 dm-clone employs dm-kcopyd for copying portions of the source device to the
0115 destination device. By default, we issue copy requests of size equal to the
0116 region size. A message `hydration_batch_size <#regions>` can be used to tune the
0117 size of these copy requests. Increasing the hydration batch size results in
0118 dm-clone trying to batch together contiguous regions, so we copy the data in
0119 batches of this many regions.
0120
0121 When the hydration of the destination device finishes, a dm event will be sent
0122 to user space.
0123
0124 Updating on-disk metadata
0125 -------------------------
0126
0127 On-disk metadata is committed every time a FLUSH or FUA bio is written. If no
0128 such requests are made then commits will occur every second. This means the
0129 dm-clone device behaves like a physical disk that has a volatile write cache. If
0130 power is lost you may lose some recent writes. The metadata should always be
0131 consistent in spite of any crash.
0132
0133 Target Interface
0134 ================
0135
0136 Constructor
0137 -----------
0138
0139 ::
0140
0141 clone <metadata dev> <destination dev> <source dev> <region size>
0142 [<#feature args> [<feature arg>]* [<#core args> [<core arg>]*]]
0143
0144 ================ ==============================================================
0145 metadata dev Fast device holding the persistent metadata
0146 destination dev The destination device, where the source will be cloned
0147 source dev Read only device containing the data that gets cloned
0148 region size The size of a region in sectors
0149
0150 #feature args Number of feature arguments passed
0151 feature args no_hydration or no_discard_passdown
0152
0153 #core args An even number of arguments corresponding to key/value pairs
0154 passed to dm-clone
0155 core args Key/value pairs passed to dm-clone, e.g. `hydration_threshold
0156 256`
0157 ================ ==============================================================
0158
0159 Optional feature arguments are:
0160
0161 ==================== =========================================================
0162 no_hydration Create a dm-clone instance with background hydration
0163 disabled
0164 no_discard_passdown Disable passing down discards to the destination device
0165 ==================== =========================================================
0166
0167 Optional core arguments are:
0168
0169 ================================ ==============================================
0170 hydration_threshold <#regions> Maximum number of regions being copied from
0171 the source to the destination device at any
0172 one time, during background hydration.
0173 hydration_batch_size <#regions> During background hydration, try to batch
0174 together contiguous regions, so we copy data
0175 from the source to the destination device in
0176 batches of this many regions.
0177 ================================ ==============================================
0178
0179 Status
0180 ------
0181
0182 ::
0183
0184 <metadata block size> <#used metadata blocks>/<#total metadata blocks>
0185 <region size> <#hydrated regions>/<#total regions> <#hydrating regions>
0186 <#feature args> <feature args>* <#core args> <core args>*
0187 <clone metadata mode>
0188
0189 ======================= =======================================================
0190 metadata block size Fixed block size for each metadata block in sectors
0191 #used metadata blocks Number of metadata blocks used
0192 #total metadata blocks Total number of metadata blocks
0193 region size Configurable region size for the device in sectors
0194 #hydrated regions Number of regions that have finished hydrating
0195 #total regions Total number of regions to hydrate
0196 #hydrating regions Number of regions currently hydrating
0197 #feature args Number of feature arguments to follow
0198 feature args Feature arguments, e.g. `no_hydration`
0199 #core args Even number of core arguments to follow
0200 core args Key/value pairs for tuning the core, e.g.
0201 `hydration_threshold 256`
0202 clone metadata mode ro if read-only, rw if read-write
0203
0204 In serious cases where even a read-only mode is deemed
0205 unsafe no further I/O will be permitted and the status
0206 will just contain the string 'Fail'. If the metadata
0207 mode changes, a dm event will be sent to user space.
0208 ======================= =======================================================
0209
0210 Messages
0211 --------
0212
0213 `disable_hydration`
0214 Disable the background hydration of the destination device.
0215
0216 `enable_hydration`
0217 Enable the background hydration of the destination device.
0218
0219 `hydration_threshold <#regions>`
0220 Set background hydration threshold.
0221
0222 `hydration_batch_size <#regions>`
0223 Set background hydration batch size.
0224
0225 Examples
0226 ========
0227
0228 Clone a device containing a file system
0229 ---------------------------------------
0230
0231 1. Create the dm-clone device.
0232
0233 ::
0234
0235 dmsetup create clone --table "0 1048576000 clone $metadata_dev $dest_dev \
0236 $source_dev 8 1 no_hydration"
0237
0238 2. Mount the device and trim the file system. dm-clone interprets the discards
0239 sent by the file system and it will not hydrate the unused space.
0240
0241 ::
0242
0243 mount /dev/mapper/clone /mnt/cloned-fs
0244 fstrim /mnt/cloned-fs
0245
0246 3. Enable background hydration of the destination device.
0247
0248 ::
0249
0250 dmsetup message clone 0 enable_hydration
0251
0252 4. When the hydration finishes, we can replace the dm-clone table with a linear
0253 table.
0254
0255 ::
0256
0257 dmsetup suspend clone
0258 dmsetup load clone --table "0 1048576000 linear $dest_dev 0"
0259 dmsetup resume clone
0260
0261 The metadata device is no longer needed and can be safely discarded or reused
0262 for other purposes.
0263
0264 Known issues
0265 ============
0266
0267 1. We redirect reads, to not-yet-hydrated regions, to the source device. If
0268 reading the source device has high latency and the user repeatedly reads from
0269 the same regions, this behaviour could degrade performance. We should use
0270 these reads as hints to hydrate the relevant regions sooner. Currently, we
0271 rely on the page cache to cache these regions, so we hopefully don't end up
0272 reading them multiple times from the source device.
0273
0274 2. Release in-core resources, i.e., the bitmaps tracking which regions are
0275 hydrated, after the hydration has finished.
0276
0277 3. During background hydration, if we fail to read the source or write to the
0278 destination device, we print an error message, but the hydration process
0279 continues indefinitely, until it succeeds. We should stop the background
0280 hydration after a number of failures and emit a dm event for user space to
0281 notice.
0282
0283 Why not...?
0284 ===========
0285
0286 We explored the following alternatives before implementing dm-clone:
0287
0288 1. Use dm-cache with cache size equal to the source device and implement a new
0289 cloning policy:
0290
0291 * The resulting cache device is not a one-to-one mirror of the source device
0292 and thus we cannot remove the cache device once cloning completes.
0293
0294 * dm-cache writes to the source device, which violates our requirement that
0295 the source device must be treated as read-only.
0296
0297 * Caching is semantically different from cloning.
0298
0299 2. Use dm-snapshot with a COW device equal to the source device:
0300
0301 * dm-snapshot stores its metadata in the COW device, so the resulting device
0302 is not a one-to-one mirror of the source device.
0303
0304 * No background copying mechanism.
0305
0306 * dm-snapshot needs to commit its metadata whenever a pending exception
0307 completes, to ensure snapshot consistency. In the case of cloning, we don't
0308 need to be so strict and can rely on committing metadata every time a FLUSH
0309 or FUA bio is written, or periodically, like dm-thin and dm-cache do. This
0310 improves the performance significantly.
0311
0312 3. Use dm-mirror: The mirror target has a background copying/mirroring
0313 mechanism, but it writes to all mirrors, thus violating our requirement that
0314 the source device must be treated as read-only.
0315
0316 4. Use dm-thin's external snapshot functionality. This approach is the most
0317 promising among all alternatives, as the thinly-provisioned volume is a
0318 one-to-one mirror of the source device and handles reads and writes to
0319 un-provisioned/not-yet-cloned areas the same way as dm-clone does.
0320
0321 Still:
0322
0323 * There is no background copying mechanism, though one could be implemented.
0324
0325 * Most importantly, we want to support arbitrary block devices as the
0326 destination of the cloning process and not restrict ourselves to
0327 thinly-provisioned volumes. Thin-provisioning has an inherent metadata
0328 overhead, for maintaining the thin volume mappings, which significantly
0329 degrades performance.
0330
0331 Moreover, cloning a device shouldn't force the use of thin-provisioning. On
0332 the other hand, if we wish to use thin provisioning, we can just use a thin
0333 LV as dm-clone's destination device.
