OSCL-LXR linux-6.0.1/​Documentation/​virt/​hyperv/​vmbus.rst	Source navigation Diff markup Identifier search General search
	Version: linux-6.0.1 spark-3.0.0 hadoop-3.2.0-src hadoop-3.3.0-src spark-2.4.7 linux-4.15.13 hadoop-2.7.4-src Revert   Change

 .. SPDX-License-Identifier: GPL-2.0
 
 VMbus
 =====
 VMbus is a software construct provided by Hyper-V to guest VMs.  It
 consists of a control path and common facilities used by synthetic
 devices that Hyper-V presents to guest VMs.   The control path is
 used to offer synthetic devices to the guest VM and, in some cases,
 to rescind those devices.   The common facilities include software
 channels for communicating between the device driver in the guest VM
 and the synthetic device implementation that is part of Hyper-V, and
 signaling primitives to allow Hyper-V and the guest to interrupt
 each other.
 
 VMbus is modeled in Linux as a bus, with the expected /sys/bus/vmbus
 entry in a running Linux guest.  The VMbus driver (drivers/hv/vmbus_drv.c)
 establishes the VMbus control path with the Hyper-V host, then
 registers itself as a Linux bus driver.  It implements the standard
 bus functions for adding and removing devices to/from the bus.
 
 Most synthetic devices offered by Hyper-V have a corresponding Linux
 device driver.  These devices include:
 
 * SCSI controller
 * NIC
 * Graphics frame buffer
 * Keyboard
 * Mouse
 * PCI device pass-thru
 * Heartbeat
 * Time Sync
 * Shutdown
 * Memory balloon
 * Key/Value Pair (KVP) exchange with Hyper-V
 * Hyper-V online backup (a.k.a. VSS)
 
 Guest VMs may have multiple instances of the synthetic SCSI
 controller, synthetic NIC, and PCI pass-thru devices.  Other
 synthetic devices are limited to a single instance per VM.  Not
 listed above are a small number of synthetic devices offered by
 Hyper-V that are used only by Windows guests and for which Linux
 does not have a driver.
 
 Hyper-V uses the terms "VSP" and "VSC" in describing synthetic
 devices.  "VSP" refers to the Hyper-V code that implements a
 particular synthetic device, while "VSC" refers to the driver for
 the device in the guest VM.  For example, the Linux driver for the
 synthetic NIC is referred to as "netvsc" and the Linux driver for
 the synthetic SCSI controller is "storvsc".  These drivers contain
 functions with names like "storvsc_connect_to_vsp".
 
 VMbus channels
 --------------
 An instance of a synthetic device uses VMbus channels to communicate
 between the VSP and the VSC.  Channels are bi-directional and used
 for passing messages.   Most synthetic devices use a single channel,
 but the synthetic SCSI controller and synthetic NIC may use multiple
 channels to achieve higher performance and greater parallelism.
 
 Each channel consists of two ring buffers.  These are classic ring
 buffers from a university data structures textbook.  If the read
 and writes pointers are equal, the ring buffer is considered to be
 empty, so a full ring buffer always has at least one byte unused.
 The "in" ring buffer is for messages from the Hyper-V host to the
 guest, and the "out" ring buffer is for messages from the guest to
 the Hyper-V host.  In Linux, the "in" and "out" designations are as
 viewed by the guest side.  The ring buffers are memory that is
 shared between the guest and the host, and they follow the standard
 paradigm where the memory is allocated by the guest, with the list
 of GPAs that make up the ring buffer communicated to the host.  Each
 ring buffer consists of a header page (4 Kbytes) with the read and
 write indices and some control flags, followed by the memory for the
 actual ring.  The size of the ring is determined by the VSC in the
 guest and is specific to each synthetic device.   The list of GPAs
 making up the ring is communicated to the Hyper-V host over the
 VMbus control path as a GPA Descriptor List (GPADL).  See function
 vmbus_establish_gpadl().
 
 Each ring buffer is mapped into contiguous Linux kernel virtual
 space in three parts:  1) the 4 Kbyte header page, 2) the memory
 that makes up the ring itself, and 3) a second mapping of the memory
 that makes up the ring itself.  Because (2) and (3) are contiguous
 in kernel virtual space, the code that copies data to and from the
 ring buffer need not be concerned with ring buffer wrap-around.
 Once a copy operation has completed, the read or write index may
 need to be reset to point back into the first mapping, but the
 actual data copy does not need to be broken into two parts.  This
 approach also allows complex data structures to be easily accessed
 directly in the ring without handling wrap-around.
 
 On arm64 with page sizes > 4 Kbytes, the header page must still be
 passed to Hyper-V as a 4 Kbyte area.  But the memory for the actual
 ring must be aligned to PAGE_SIZE and have a size that is a multiple
 of PAGE_SIZE so that the duplicate mapping trick can be done.  Hence
 a portion of the header page is unused and not communicated to
 Hyper-V.  This case is handled by vmbus_establish_gpadl().
 
 Hyper-V enforces a limit on the aggregate amount of guest memory
 that can be shared with the host via GPADLs.  This limit ensures
 that a rogue guest can't force the consumption of excessive host
 resources.  For Windows Server 2019 and later, this limit is
 approximately 1280 Mbytes.  For versions prior to Windows Server
 2019, the limit is approximately 384 Mbytes.
 
 VMbus messages
 --------------
 All VMbus messages have a standard header that includes the message
 length, the offset of the message payload, some flags, and a
 transactionID.  The portion of the message after the header is
 unique to each VSP/VSC pair.
 
 Messages follow one of two patterns:
 
 * Unidirectional:  Either side sends a message and does not
   expect a response message
 * Request/response:  One side (usually the guest) sends a message
   and expects a response
 
 The transactionID (a.k.a. "requestID") is for matching requests &
 responses.  Some synthetic devices allow multiple requests to be in-
 flight simultaneously, so the guest specifies a transactionID when
 sending a request.  Hyper-V sends back the same transactionID in the
 matching response.
 
 Messages passed between the VSP and VSC are control messages.  For
 example, a message sent from the storvsc driver might be "execute
 this SCSI command".   If a message also implies some data transfer
 between the guest and the Hyper-V host, the actual data to be
 transferred may be embedded with the control message, or it may be
 specified as a separate data buffer that the Hyper-V host will
 access as a DMA operation.  The former case is used when the size of
 the data is small and the cost of copying the data to and from the
 ring buffer is minimal.  For example, time sync messages from the
 Hyper-V host to the guest contain the actual time value.  When the
 data is larger, a separate data buffer is used.  In this case, the
 control message contains a list of GPAs that describe the data
 buffer.  For example, the storvsc driver uses this approach to
 specify the data buffers to/from which disk I/O is done.
 
 Three functions exist to send VMbus messages:
 
 1. vmbus_sendpacket():  Control-only messages and messages with
    embedded data -- no GPAs
 2. vmbus_sendpacket_pagebuffer(): Message with list of GPAs
    identifying data to transfer.  An offset and length is
    associated with each GPA so that multiple discontinuous areas
    of guest memory can be targeted.
 3. vmbus_sendpacket_mpb_desc(): Message with list of GPAs
    identifying data to transfer.  A single offset and length is
    associated with a list of GPAs.  The GPAs must describe a
    single logical area of guest memory to be targeted.
 
 Historically, Linux guests have trusted Hyper-V to send well-formed
 and valid messages, and Linux drivers for synthetic devices did not
 fully validate messages.  With the introduction of processor
 technologies that fully encrypt guest memory and that allow the
 guest to not trust the hypervisor (AMD SNP-SEV, Intel TDX), trusting
 the Hyper-V host is no longer a valid assumption.  The drivers for
 VMbus synthetic devices are being updated to fully validate any
 values read from memory that is shared with Hyper-V, which includes
 messages from VMbus devices.  To facilitate such validation,
 messages read by the guest from the "in" ring buffer are copied to a
 temporary buffer that is not shared with Hyper-V.  Validation is
 performed in this temporary buffer without the risk of Hyper-V
 maliciously modifying the message after it is validated but before
 it is used.
 
 VMbus interrupts
 ----------------
 VMbus provides a mechanism for the guest to interrupt the host when
 the guest has queued new messages in a ring buffer.  The host
 expects that the guest will send an interrupt only when an "out"
 ring buffer transitions from empty to non-empty.  If the guest sends
 interrupts at other times, the host deems such interrupts to be
 unnecessary.  If a guest sends an excessive number of unnecessary
 interrupts, the host may throttle that guest by suspending its
 execution for a few seconds to prevent a denial-of-service attack.
 
 Similarly, the host will interrupt the guest when it sends a new
 message on the VMbus control path, or when a VMbus channel "in" ring
 buffer transitions from empty to non-empty.  Each CPU in the guest
 may receive VMbus interrupts, so they are best modeled as per-CPU
 interrupts in Linux.  This model works well on arm64 where a single
 per-CPU IRQ is allocated for VMbus.  Since x86/x64 lacks support for
 per-CPU IRQs, an x86 interrupt vector is statically allocated (see
 HYPERVISOR_CALLBACK_VECTOR) across all CPUs and explicitly coded to
 call the VMbus interrupt service routine.  These interrupts are
 visible in /proc/interrupts on the "HYP" line.
 
 The guest CPU that a VMbus channel will interrupt is selected by the
 guest when the channel is created, and the host is informed of that
 selection.  VMbus devices are broadly grouped into two categories:
 
 1. "Slow" devices that need only one VMbus channel.  The devices
    (such as keyboard, mouse, heartbeat, and timesync) generate
    relatively few interrupts.  Their VMbus channels are all
    assigned to interrupt the VMBUS_CONNECT_CPU, which is always
    CPU 0.
 
 2. "High speed" devices that may use multiple VMbus channels for
    higher parallelism and performance.  These devices include the
    synthetic SCSI controller and synthetic NIC.  Their VMbus
    channels interrupts are assigned to CPUs that are spread out
    among the available CPUs in the VM so that interrupts on
    multiple channels can be processed in parallel.
 
 The assignment of VMbus channel interrupts to CPUs is done in the
 function init_vp_index().  This assignment is done outside of the
 normal Linux interrupt affinity mechanism, so the interrupts are
 neither "unmanaged" nor "managed" interrupts.
 
 The CPU that a VMbus channel will interrupt can be seen in
 /sys/bus/vmbus/devices/<deviceGUID>/ channels/<channelRelID>/cpu.
 When running on later versions of Hyper-V, the CPU can be changed
 by writing a new value to this sysfs entry.  Because the interrupt
 assignment is done outside of the normal Linux affinity mechanism,
 there are no entries in /proc/irq corresponding to individual
 VMbus channel interrupts.
 
 An online CPU in a Linux guest may not be taken offline if it has
 VMbus channel interrupts assigned to it.  Any such channel
 interrupts must first be manually reassigned to another CPU as
 described above.  When no channel interrupts are assigned to the
 CPU, it can be taken offline.
 
 When a guest CPU receives a VMbus interrupt from the host, the
 function vmbus_isr() handles the interrupt.  It first checks for
 channel interrupts by calling vmbus_chan_sched(), which looks at a
 bitmap setup by the host to determine which channels have pending
 interrupts on this CPU.  If multiple channels have pending
 interrupts for this CPU, they are processed sequentially.  When all
 channel interrupts have been processed, vmbus_isr() checks for and
 processes any message received on the VMbus control path.
 
 The VMbus channel interrupt handling code is designed to work
 correctly even if an interrupt is received on a CPU other than the
 CPU assigned to the channel.  Specifically, the code does not use
 CPU-based exclusion for correctness.  In normal operation, Hyper-V
 will interrupt the assigned CPU.  But when the CPU assigned to a
 channel is being changed via sysfs, the guest doesn't know exactly
 when Hyper-V will make the transition.  The code must work correctly
 even if there is a time lag before Hyper-V starts interrupting the
 new CPU.  See comments in target_cpu_store().
 
 VMbus device creation/deletion
 ------------------------------
 Hyper-V and the Linux guest have a separate message-passing path
 that is used for synthetic device creation and deletion. This
 path does not use a VMbus channel.  See vmbus_post_msg() and
 vmbus_on_msg_dpc().
 
 The first step is for the guest to connect to the generic
 Hyper-V VMbus mechanism.  As part of establishing this connection,
 the guest and Hyper-V agree on a VMbus protocol version they will
 use.  This negotiation allows newer Linux kernels to run on older
 Hyper-V versions, and vice versa.
 
 The guest then tells Hyper-V to "send offers".  Hyper-V sends an
 offer message to the guest for each synthetic device that the VM
 is configured to have. Each VMbus device type has a fixed GUID
 known as the "class ID", and each VMbus device instance is also
 identified by a GUID. The offer message from Hyper-V contains
 both GUIDs to uniquely (within the VM) identify the device.
 There is one offer message for each device instance, so a VM with
 two synthetic NICs will get two offers messages with the NIC
 class ID. The ordering of offer messages can vary from boot-to-boot
 and must not be assumed to be consistent in Linux code. Offer
 messages may also arrive long after Linux has initially booted
 because Hyper-V supports adding devices, such as synthetic NICs,
 to running VMs. A new offer message is processed by
 vmbus_process_offer(), which indirectly invokes vmbus_add_channel_work().
 
 Upon receipt of an offer message, the guest identifies the device
 type based on the class ID, and invokes the correct driver to set up
 the device.  Driver/device matching is performed using the standard
 Linux mechanism.
 
 The device driver probe function opens the primary VMbus channel to
 the corresponding VSP. It allocates guest memory for the channel
 ring buffers and shares the ring buffer with the Hyper-V host by
 giving the host a list of GPAs for the ring buffer memory.  See
 vmbus_establish_gpadl().
 
 Once the ring buffer is set up, the device driver and VSP exchange
 setup messages via the primary channel.  These messages may include
 negotiating the device protocol version to be used between the Linux
 VSC and the VSP on the Hyper-V host.  The setup messages may also
 include creating additional VMbus channels, which are somewhat
 mis-named as "sub-channels" since they are functionally
 equivalent to the primary channel once they are created.
 
 Finally, the device driver may create entries in /dev as with
 any device driver.
 
 The Hyper-V host can send a "rescind" message to the guest to
 remove a device that was previously offered. Linux drivers must
 handle such a rescind message at any time. Rescinding a device
 invokes the device driver "remove" function to cleanly shut
 down the device and remove it. Once a synthetic device is
 rescinded, neither Hyper-V nor Linux retains any state about
 its previous existence. Such a device might be re-added later,
 in which case it is treated as an entirely new device. See
 vmbus_onoffer_rescind().

This page was automatically generated by the 2.1.0 LXR engine. The LXR team