Back to home page

LXR

 
 

    


0001 /*
0002  * VFIO Mediated devices
0003  *
0004  * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
0005  *     Author: Neo Jia <cjia@nvidia.com>
0006  *             Kirti Wankhede <kwankhede@nvidia.com>
0007  *
0008  * This program is free software; you can redistribute it and/or modify
0009  * it under the terms of the GNU General Public License version 2 as
0010  * published by the Free Software Foundation.
0011  */
0012 
0013 Virtual Function I/O (VFIO) Mediated devices[1]
0014 ===============================================
0015 
0016 The number of use cases for virtualizing DMA devices that do not have built-in
0017 SR_IOV capability is increasing. Previously, to virtualize such devices,
0018 developers had to create their own management interfaces and APIs, and then
0019 integrate them with user space software. To simplify integration with user space
0020 software, we have identified common requirements and a unified management
0021 interface for such devices.
0022 
0023 The VFIO driver framework provides unified APIs for direct device access. It is
0024 an IOMMU/device-agnostic framework for exposing direct device access to user
0025 space in a secure, IOMMU-protected environment. This framework is used for
0026 multiple devices, such as GPUs, network adapters, and compute accelerators. With
0027 direct device access, virtual machines or user space applications have direct
0028 access to the physical device. This framework is reused for mediated devices.
0029 
0030 The mediated core driver provides a common interface for mediated device
0031 management that can be used by drivers of different devices. This module
0032 provides a generic interface to perform these operations:
0033 
0034 * Create and destroy a mediated device
0035 * Add a mediated device to and remove it from a mediated bus driver
0036 * Add a mediated device to and remove it from an IOMMU group
0037 
0038 The mediated core driver also provides an interface to register a bus driver.
0039 For example, the mediated VFIO mdev driver is designed for mediated devices and
0040 supports VFIO APIs. The mediated bus driver adds a mediated device to and
0041 removes it from a VFIO group.
0042 
0043 The following high-level block diagram shows the main components and interfaces
0044 in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM
0045 devices as examples, as these devices are the first devices to use this module.
0046 
0047      +---------------+
0048      |               |
0049      | +-----------+ |  mdev_register_driver() +--------------+
0050      | |           | +<------------------------+              |
0051      | |  mdev     | |                         |              |
0052      | |  bus      | +------------------------>+ vfio_mdev.ko |<-> VFIO user
0053      | |  driver   | |     probe()/remove()    |              |    APIs
0054      | |           | |                         +--------------+
0055      | +-----------+ |
0056      |               |
0057      |  MDEV CORE    |
0058      |   MODULE      |
0059      |   mdev.ko     |
0060      | +-----------+ |  mdev_register_device() +--------------+
0061      | |           | +<------------------------+              |
0062      | |           | |                         |  nvidia.ko   |<-> physical
0063      | |           | +------------------------>+              |    device
0064      | |           | |        callbacks        +--------------+
0065      | | Physical  | |
0066      | |  device   | |  mdev_register_device() +--------------+
0067      | | interface | |<------------------------+              |
0068      | |           | |                         |  i915.ko     |<-> physical
0069      | |           | +------------------------>+              |    device
0070      | |           | |        callbacks        +--------------+
0071      | |           | |
0072      | |           | |  mdev_register_device() +--------------+
0073      | |           | +<------------------------+              |
0074      | |           | |                         | ccw_device.ko|<-> physical
0075      | |           | +------------------------>+              |    device
0076      | |           | |        callbacks        +--------------+
0077      | +-----------+ |
0078      +---------------+
0079 
0080 
0081 Registration Interfaces
0082 =======================
0083 
0084 The mediated core driver provides the following types of registration
0085 interfaces:
0086 
0087 * Registration interface for a mediated bus driver
0088 * Physical device driver interface
0089 
0090 Registration Interface for a Mediated Bus Driver
0091 ------------------------------------------------
0092 
0093 The registration interface for a mediated bus driver provides the following
0094 structure to represent a mediated device's driver:
0095 
0096      /*
0097       * struct mdev_driver [2] - Mediated device's driver
0098       * @name: driver name
0099       * @probe: called when new device created
0100       * @remove: called when device removed
0101       * @driver: device driver structure
0102       */
0103      struct mdev_driver {
0104              const char *name;
0105              int  (*probe)  (struct device *dev);
0106              void (*remove) (struct device *dev);
0107              struct device_driver    driver;
0108      };
0109 
0110 A mediated bus driver for mdev should use this structure in the function calls
0111 to register and unregister itself with the core driver:
0112 
0113 * Register:
0114 
0115   extern int  mdev_register_driver(struct mdev_driver *drv,
0116                                    struct module *owner);
0117 
0118 * Unregister:
0119 
0120   extern void mdev_unregister_driver(struct mdev_driver *drv);
0121 
0122 The mediated bus driver is responsible for adding mediated devices to the VFIO
0123 group when devices are bound to the driver and removing mediated devices from
0124 the VFIO when devices are unbound from the driver.
0125 
0126 
0127 Physical Device Driver Interface
0128 --------------------------------
0129 
0130 The physical device driver interface provides the mdev_parent_ops[3] structure
0131 to define the APIs to manage work in the mediated core driver that is related
0132 to the physical device.
0133 
0134 The structures in the mdev_parent_ops structure are as follows:
0135 
0136 * dev_attr_groups: attributes of the parent device
0137 * mdev_attr_groups: attributes of the mediated device
0138 * supported_config: attributes to define supported configurations
0139 
0140 The functions in the mdev_parent_ops structure are as follows:
0141 
0142 * create: allocate basic resources in a driver for a mediated device
0143 * remove: free resources in a driver when a mediated device is destroyed
0144 
0145 The callbacks in the mdev_parent_ops structure are as follows:
0146 
0147 * open: open callback of mediated device
0148 * close: close callback of mediated device
0149 * ioctl: ioctl callback of mediated device
0150 * read : read emulation callback
0151 * write: write emulation callback
0152 * mmap: mmap emulation callback
0153 
0154 A driver should use the mdev_parent_ops structure in the function call to
0155 register itself with the mdev core driver:
0156 
0157 extern int  mdev_register_device(struct device *dev,
0158                                  const struct mdev_parent_ops *ops);
0159 
0160 However, the mdev_parent_ops structure is not required in the function call
0161 that a driver should use to unregister itself with the mdev core driver:
0162 
0163 extern void mdev_unregister_device(struct device *dev);
0164 
0165 
0166 Mediated Device Management Interface Through sysfs
0167 ==================================================
0168 
0169 The management interface through sysfs enables user space software, such as
0170 libvirt, to query and configure mediated devices in a hardware-agnostic fashion.
0171 This management interface provides flexibility to the underlying physical
0172 device's driver to support features such as:
0173 
0174 * Mediated device hot plug
0175 * Multiple mediated devices in a single virtual machine
0176 * Multiple mediated devices from different physical devices
0177 
0178 Links in the mdev_bus Class Directory
0179 -------------------------------------
0180 The /sys/class/mdev_bus/ directory contains links to devices that are registered
0181 with the mdev core driver.
0182 
0183 Directories and files under the sysfs for Each Physical Device
0184 --------------------------------------------------------------
0185 
0186 |- [parent physical device]
0187 |--- Vendor-specific-attributes [optional]
0188 |--- [mdev_supported_types]
0189 |     |--- [<type-id>]
0190 |     |   |--- create
0191 |     |   |--- name
0192 |     |   |--- available_instances
0193 |     |   |--- device_api
0194 |     |   |--- description
0195 |     |   |--- [devices]
0196 |     |--- [<type-id>]
0197 |     |   |--- create
0198 |     |   |--- name
0199 |     |   |--- available_instances
0200 |     |   |--- device_api
0201 |     |   |--- description
0202 |     |   |--- [devices]
0203 |     |--- [<type-id>]
0204 |          |--- create
0205 |          |--- name
0206 |          |--- available_instances
0207 |          |--- device_api
0208 |          |--- description
0209 |          |--- [devices]
0210 
0211 * [mdev_supported_types]
0212 
0213   The list of currently supported mediated device types and their details.
0214 
0215   [<type-id>], device_api, and available_instances are mandatory attributes
0216   that should be provided by vendor driver.
0217 
0218 * [<type-id>]
0219 
0220   The [<type-id>] name is created by adding the the device driver string as a
0221   prefix to the string provided by the vendor driver. This format of this name
0222   is as follows:
0223 
0224         sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name);
0225 
0226   (or using mdev_parent_dev(mdev) to arrive at the parent device outside
0227    of the core mdev code)
0228 
0229 * device_api
0230 
0231   This attribute should show which device API is being created, for example,
0232   "vfio-pci" for a PCI device.
0233 
0234 * available_instances
0235 
0236   This attribute should show the number of devices of type <type-id> that can be
0237   created.
0238 
0239 * [device]
0240 
0241   This directory contains links to the devices of type <type-id> that have been
0242 created.
0243 
0244 * name
0245 
0246   This attribute should show human readable name. This is optional attribute.
0247 
0248 * description
0249 
0250   This attribute should show brief features/description of the type. This is
0251   optional attribute.
0252 
0253 Directories and Files Under the sysfs for Each mdev Device
0254 ----------------------------------------------------------
0255 
0256 |- [parent phy device]
0257 |--- [$MDEV_UUID]
0258          |--- remove
0259          |--- mdev_type {link to its type}
0260          |--- vendor-specific-attributes [optional]
0261 
0262 * remove (write only)
0263 Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can
0264 fail the remove() callback if that device is active and the vendor driver
0265 doesn't support hot unplug.
0266 
0267 Example:
0268         # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove
0269 
0270 Mediated device Hot plug:
0271 ------------------------
0272 
0273 Mediated devices can be created and assigned at runtime. The procedure to hot
0274 plug a mediated device is the same as the procedure to hot plug a PCI device.
0275 
0276 Translation APIs for Mediated Devices
0277 =====================================
0278 
0279 The following APIs are provided for translating user pfn to host pfn in a VFIO
0280 driver:
0281 
0282 extern int vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
0283                           int npage, int prot, unsigned long *phys_pfn);
0284 
0285 extern int vfio_unpin_pages(struct device *dev, unsigned long *user_pfn,
0286                             int npage);
0287 
0288 These functions call back into the back-end IOMMU module by using the pin_pages
0289 and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently
0290 these callbacks are supported in the TYPE1 IOMMU module. To enable them for
0291 other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide
0292 these two callback functions.
0293 
0294 Using the Sample Code
0295 =====================
0296 
0297 mtty.c in samples/vfio-mdev/ directory is a sample driver program to
0298 demonstrate how to use the mediated device framework.
0299 
0300 The sample driver creates an mdev device that simulates a serial port over a PCI
0301 card.
0302 
0303 1. Build and load the mtty.ko module.
0304 
0305    This step creates a dummy device, /sys/devices/virtual/mtty/mtty/
0306 
0307    Files in this device directory in sysfs are similar to the following:
0308 
0309    # tree /sys/devices/virtual/mtty/mtty/
0310       /sys/devices/virtual/mtty/mtty/
0311       |-- mdev_supported_types
0312       |   |-- mtty-1
0313       |   |   |-- available_instances
0314       |   |   |-- create
0315       |   |   |-- device_api
0316       |   |   |-- devices
0317       |   |   `-- name
0318       |   `-- mtty-2
0319       |       |-- available_instances
0320       |       |-- create
0321       |       |-- device_api
0322       |       |-- devices
0323       |       `-- name
0324       |-- mtty_dev
0325       |   `-- sample_mtty_dev
0326       |-- power
0327       |   |-- autosuspend_delay_ms
0328       |   |-- control
0329       |   |-- runtime_active_time
0330       |   |-- runtime_status
0331       |   `-- runtime_suspended_time
0332       |-- subsystem -> ../../../../class/mtty
0333       `-- uevent
0334 
0335 2. Create a mediated device by using the dummy device that you created in the
0336    previous step.
0337 
0338    # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" >      \
0339               /sys/devices/virtual/mtty/mtty/mdev_supported_types/mtty-2/create
0340 
0341 3. Add parameters to qemu-kvm.
0342 
0343    -device vfio-pci,\
0344     sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001
0345 
0346 4. Boot the VM.
0347 
0348    In the Linux guest VM, with no hardware on the host, the device appears
0349    as  follows:
0350 
0351    # lspci -s 00:05.0 -xxvv
0352    00:05.0 Serial controller: Device 4348:3253 (rev 10) (prog-if 02 [16550])
0353            Subsystem: Device 4348:3253
0354            Physical Slot: 5
0355            Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
0356    Stepping- SERR- FastB2B- DisINTx-
0357            Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
0358    <TAbort- <MAbort- >SERR- <PERR- INTx-
0359            Interrupt: pin A routed to IRQ 10
0360            Region 0: I/O ports at c150 [size=8]
0361            Region 1: I/O ports at c158 [size=8]
0362            Kernel driver in use: serial
0363    00: 48 43 53 32 01 00 00 02 10 02 00 07 00 00 00 00
0364    10: 51 c1 00 00 59 c1 00 00 00 00 00 00 00 00 00 00
0365    20: 00 00 00 00 00 00 00 00 00 00 00 00 48 43 53 32
0366    30: 00 00 00 00 00 00 00 00 00 00 00 00 0a 01 00 00
0367 
0368    In the Linux guest VM, dmesg output for the device is as follows:
0369 
0370    serial 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ
0371 10
0372    0000:00:05.0: ttyS1 at I/O 0xc150 (irq = 10) is a 16550A
0373    0000:00:05.0: ttyS2 at I/O 0xc158 (irq = 10) is a 16550A
0374 
0375 
0376 5. In the Linux guest VM, check the serial ports.
0377 
0378    # setserial -g /dev/ttyS*
0379    /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4
0380    /dev/ttyS1, UART: 16550A, Port: 0xc150, IRQ: 10
0381    /dev/ttyS2, UART: 16550A, Port: 0xc158, IRQ: 10
0382 
0383 6. Using a minicom or any terminal enulation program, open port /dev/ttyS1 or
0384    /dev/ttyS2 with hardware flow control disabled.
0385 
0386 7. Type data on the minicom terminal or send data to the terminal emulation
0387    program and read the data.
0388 
0389    Data is loop backed from hosts mtty driver.
0390 
0391 8. Destroy the mediated device that you created.
0392 
0393    # echo 1 > /sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa1001/remove
0394 
0395 References
0396 ==========
0397 
0398 [1] See Documentation/vfio.txt for more information on VFIO.
0399 [2] struct mdev_driver in include/linux/mdev.h
0400 [3] struct mdev_parent_ops in include/linux/mdev.h
0401 [4] struct vfio_iommu_driver_ops in include/linux/vfio.h