0001 ========================================================
0002 OpenCAPI (Open Coherent Accelerator Processor Interface)
0003 ========================================================
0004
0005 OpenCAPI is an interface between processors and accelerators. It aims
0006 at being low-latency and high-bandwidth. The specification is
0007 developed by the `OpenCAPI Consortium <http://opencapi.org/>`_.
0008
0009 It allows an accelerator (which could be an FPGA, ASICs, ...) to access
0010 the host memory coherently, using virtual addresses. An OpenCAPI
0011 device can also host its own memory, that can be accessed from the
0012 host.
0013
0014 OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic
0015 evolution of 'cxl' (the driver for the IBM CAPI interface for
0016 powerpc), which was named that way to avoid confusion with the ISDN
0017 CAPI subsystem.
0018
0019
0020 High-level view
0021 ===============
0022
0023 OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to
0024 be implemented on top of a physical link. Any processor or device
0025 implementing the DL and TL can start sharing memory.
0026
0027 ::
0028
0029 +-----------+ +-------------+
0030 | | | |
0031 | | | Accelerated |
0032 | Processor | | Function |
0033 | | +--------+ | Unit | +--------+
0034 | |--| Memory | | (AFU) |--| Memory |
0035 | | +--------+ | | +--------+
0036 +-----------+ +-------------+
0037 | |
0038 +-----------+ +-------------+
0039 | TL | | TLX |
0040 +-----------+ +-------------+
0041 | |
0042 +-----------+ +-------------+
0043 | DL | | DLX |
0044 +-----------+ +-------------+
0045 | |
0046 | PHY |
0047 +---------------------------------------+
0048
0049
0050
0051 Device discovery
0052 ================
0053
0054 OpenCAPI relies on a PCI-like configuration space, implemented on the
0055 device. So the host can discover AFUs by querying the config space.
0056
0057 OpenCAPI devices in Linux are treated like PCI devices (with a few
0058 caveats). The firmware is expected to abstract the hardware as if it
0059 was a PCI link. A lot of the existing PCI infrastructure is reused:
0060 devices are scanned and BARs are assigned during the standard PCI
0061 enumeration. Commands like 'lspci' can therefore be used to see what
0062 devices are available.
0063
0064 The configuration space defines the AFU(s) that can be found on the
0065 physical adapter, such as its name, how many memory contexts it can
0066 work with, the size of its MMIO areas, ...
0067
0068
0069
0070 MMIO
0071 ====
0072
0073 OpenCAPI defines two MMIO areas for each AFU:
0074
0075 * the global MMIO area, with registers pertinent to the whole AFU.
0076 * a per-process MMIO area, which has a fixed size for each context.
0077
0078
0079
0080 AFU interrupts
0081 ==============
0082
0083 OpenCAPI includes the possibility for an AFU to send an interrupt to a
0084 host process. It is done through a 'intrp_req' defined in the
0085 Transaction Layer, specifying a 64-bit object handle which defines the
0086 interrupt.
0087
0088 The driver allows a process to allocate an interrupt and obtain its
0089 64-bit object handle, that can be passed to the AFU.
0090
0091
0092
0093 char devices
0094 ============
0095
0096 The driver creates one char device per AFU found on the physical
0097 device. A physical device may have multiple functions and each
0098 function can have multiple AFUs. At the time of this writing though,
0099 it has only been tested with devices exporting only one AFU.
0100
0101 Char devices can be found in /dev/ocxl/ and are named as:
0102 /dev/ocxl/<AFU name>.<location>.<index>
0103
0104 where <AFU name> is a max 20-character long name, as found in the
0105 config space of the AFU.
0106 <location> is added by the driver and can help distinguish devices
0107 when a system has more than one instance of the same OpenCAPI device.
0108 <index> is also to help distinguish AFUs in the unlikely case where a
0109 device carries multiple copies of the same AFU.
0110
0111
0112
0113 Sysfs class
0114 ===========
0115
0116 An ocxl class is added for the devices representing the AFUs. See
0117 /sys/class/ocxl. The layout is described in
0118 Documentation/ABI/testing/sysfs-class-ocxl
0119
0120
0121
0122 User API
0123 ========
0124
0125 open
0126 ----
0127
0128 Based on the AFU definition found in the config space, an AFU may
0129 support working with more than one memory context, in which case the
0130 associated char device may be opened multiple times by different
0131 processes.
0132
0133
0134 ioctl
0135 -----
0136
0137 OCXL_IOCTL_ATTACH:
0138
0139 Attach the memory context of the calling process to the AFU so that
0140 the AFU can access its memory.
0141
0142 OCXL_IOCTL_IRQ_ALLOC:
0143
0144 Allocate an AFU interrupt and return an identifier.
0145
0146 OCXL_IOCTL_IRQ_FREE:
0147
0148 Free a previously allocated AFU interrupt.
0149
0150 OCXL_IOCTL_IRQ_SET_FD:
0151
0152 Associate an event fd to an AFU interrupt so that the user process
0153 can be notified when the AFU sends an interrupt.
0154
0155 OCXL_IOCTL_GET_METADATA:
0156
0157 Obtains configuration information from the card, such at the size of
0158 MMIO areas, the AFU version, and the PASID for the current context.
0159
0160 OCXL_IOCTL_ENABLE_P9_WAIT:
0161
0162 Allows the AFU to wake a userspace thread executing 'wait'. Returns
0163 information to userspace to allow it to configure the AFU. Note that
0164 this is only available on POWER9.
0165
0166 OCXL_IOCTL_GET_FEATURES:
0167
0168 Reports on which CPU features that affect OpenCAPI are usable from
0169 userspace.
0170
0171
0172 mmap
0173 ----
0174
0175 A process can mmap the per-process MMIO area for interactions with the
0176 AFU.
