diff options
Diffstat (limited to 'Documentation/misc-devices')
| -rw-r--r-- | Documentation/misc-devices/index.rst | 1 | ||||
| -rw-r--r-- | Documentation/misc-devices/mic/index.rst | 16 | ||||
| -rw-r--r-- | Documentation/misc-devices/mic/mic_overview.rst | 85 | ||||
| -rw-r--r-- | Documentation/misc-devices/mic/scif_overview.rst | 108 | ||||
| -rw-r--r-- | Documentation/misc-devices/uacce.rst | 176 |
5 files changed, 386 insertions, 0 deletions
diff --git a/Documentation/misc-devices/index.rst b/Documentation/misc-devices/index.rst index f11c5daeada5..c1dcd2628911 100644 --- a/Documentation/misc-devices/index.rst +++ b/Documentation/misc-devices/index.rst @@ -20,4 +20,5 @@ fit into other categories. isl29003 lis3lv02d max6875 + mic/index xilinx_sdfec diff --git a/Documentation/misc-devices/mic/index.rst b/Documentation/misc-devices/mic/index.rst new file mode 100644 index 000000000000..3a8d06367ef1 --- /dev/null +++ b/Documentation/misc-devices/mic/index.rst @@ -0,0 +1,16 @@ +============================================= +Intel Many Integrated Core (MIC) architecture +============================================= + +.. toctree:: + :maxdepth: 1 + + mic_overview + scif_overview + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/misc-devices/mic/mic_overview.rst b/Documentation/misc-devices/mic/mic_overview.rst new file mode 100644 index 000000000000..17d956bdaf7c --- /dev/null +++ b/Documentation/misc-devices/mic/mic_overview.rst @@ -0,0 +1,85 @@ +====================================================== +Intel Many Integrated Core (MIC) architecture overview +====================================================== + +An Intel MIC X100 device is a PCIe form factor add-in coprocessor +card based on the Intel Many Integrated Core (MIC) architecture +that runs a Linux OS. It is a PCIe endpoint in a platform and therefore +implements the three required standard address spaces i.e. configuration, +memory and I/O. The host OS loads a device driver as is typical for +PCIe devices. The card itself runs a bootstrap after reset that +transfers control to the card OS downloaded from the host driver. The +host driver supports OSPM suspend and resume operations. It shuts down +the card during suspend and reboots the card OS during resume. +The card OS as shipped by Intel is a Linux kernel with modifications +for the X100 devices. + +Since it is a PCIe card, it does not have the ability to host hardware +devices for networking, storage and console. We provide these devices +on X100 coprocessors thus enabling a self-bootable equivalent +environment for applications. A key benefit of our solution is that it +leverages the standard virtio framework for network, disk and console +devices, though in our case the virtio framework is used across a PCIe +bus. A Virtio Over PCIe (VOP) driver allows creating user space +backends or devices on the host which are used to probe virtio drivers +for these devices on the MIC card. The existing VRINGH infrastructure +in the kernel is used to access virtio rings from the host. The card +VOP driver allows card virtio drivers to communicate with their user +space backends on the host via a device page. Ring 3 apps on the host +can add, remove and configure virtio devices. A thin MIC specific +virtio_config_ops is implemented which is borrowed heavily from +previous similar implementations in lguest and s390. + +MIC PCIe card has a dma controller with 8 channels. These channels are +shared between the host s/w and the card s/w. 0 to 3 are used by host +and 4 to 7 by card. As the dma device doesn't show up as PCIe device, +a virtual bus called mic bus is created and virtual dma devices are +created on it by the host/card drivers. On host the channels are private +and used only by the host driver to transfer data for the virtio devices. + +The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a +low level communications API across PCIe currently implemented for MIC. +More details are available at scif_overview.txt. + +The Coprocessor State Management (COSM) driver on the host allows for +boot, shutdown and reset of Intel MIC devices. It communicates with a COSM +"client" driver on the MIC cards over SCIF to perform these functions. + +Here is a block diagram of the various components described above. The +virtio backends are situated on the host rather than the card given better +single threaded performance for the host compared to MIC, the ability of +the host to initiate DMA's to/from the card using the MIC DMA engine and +the fact that the virtio block storage backend can only be on the host:: + + +----------+ | +----------+ + | Card OS | | | Host OS | + +----------+ | +----------+ + | + +-------+ +--------+ +------+ | +---------+ +--------+ +--------+ + | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio | + | Net | |Console | |Block | | |Net | |Console | |Block | + | Driver| |Driver | |Driver| | |backend | |backend | |backend | + +---+---+ +---+----+ +--+---+ | +---------+ +----+---+ +--------+ + | | | | | | | + | | | |User | | | + | | | |------|------------|--+------|------- + +---------+---------+ |Kernel | + | | | + +---------+ +---+----+ +------+ | +------+ +------+ +--+---+ +-------+ + |MIC DMA | | VOP | | SCIF | | | SCIF | | COSM | | VOP | |MIC DMA| + +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+ + | | | | | | | + +---+-----+ +---+----+ +--+---+ | +--+---+ +--+---+ +------+ +----+--+ + |MIC | | VOP | |SCIF | | |SCIF | | COSM | | VOP | | MIC | + |HW Bus | | HW Bus| |HW Bus| | |HW Bus| | Bus | |HW Bus| |HW Bus | + +---------+ +--------+ +--+---+ | +--+---+ +------+ +------+ +-------+ + | | | | | | | + | +-----------+--+ | | | +---------------+ | + | |Intel MIC | | | | |Intel MIC | | + | |Card Driver | | | | |Host Driver | | + +---+--------------+------+ | +----+---------------+-----+ + | | | + +-------------------------------------------------------------+ + | | + | PCIe Bus | + +-------------------------------------------------------------+ diff --git a/Documentation/misc-devices/mic/scif_overview.rst b/Documentation/misc-devices/mic/scif_overview.rst new file mode 100644 index 000000000000..4c8ad9e43706 --- /dev/null +++ b/Documentation/misc-devices/mic/scif_overview.rst @@ -0,0 +1,108 @@ +======================================== +Symmetric Communication Interface (SCIF) +======================================== + +The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low +level communications API across PCIe currently implemented for MIC. Currently +SCIF provides inter-node communication within a single host platform, where a +node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of +communicating over the PCIe bus while providing an API that is symmetric +across all the nodes in the PCIe network. An important design objective for SCIF +is to deliver the maximum possible performance given the communication +abilities of the hardware. SCIF has been used to implement an offload compiler +runtime and OFED support for MPI implementations for MIC coprocessors. + +SCIF API Components +=================== + +The SCIF API has the following parts: + +1. Connection establishment using a client server model +2. Byte stream messaging intended for short messages +3. Node enumeration to determine online nodes +4. Poll semantics for detection of incoming connections and messages +5. Memory registration to pin down pages +6. Remote memory mapping for low latency CPU accesses via mmap +7. Remote DMA (RDMA) for high bandwidth DMA transfers +8. Fence APIs for RDMA synchronization + +SCIF exposes the notion of a connection which can be used by peer processes on +nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A +process in a SCIF node initiates a SCIF connection to a peer process on a +different node via a SCIF "endpoint". SCIF endpoints support messaging APIs +which are similar to connection oriented socket APIs. Connected SCIF endpoints +can also register local memory which is followed by data transfer using either +DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and +kernel mode clients which are functionally equivalent. + +SCIF Performance for MIC +======================== + +DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus +SCIF shows the performance advantages of SCIF for HPC applications and +runtimes:: + + Comparison of TCP and SCIF based BW + + Throughput (GB/sec) + 8 + PCIe Bandwidth ****** + + TCP ###### + 7 + ************************************** SCIF %%%%%% + | %%%%%%%%%%%%%%%%%%% + 6 + %%%% + | %% + | %%% + 5 + %% + | %% + 4 + %% + | %% + 3 + %% + | % + 2 + %% + | %% + | % + 1 + + + ###################################### + 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- + 1 10 100 1000 10000 100000 + Transfer Size (KBytes) + +SCIF allows memory sharing via mmap(..) between processes on different PCIe +nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap +latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. + +SCIF has a user space library which is a thin IOCTL wrapper providing a user +space API similar to the kernel API in scif.h. The SCIF user space library +is distributed @ https://software.intel.com/en-us/mic-developer + +Here is some pseudo code for an example of how two applications on two PCIe +nodes would typically use the SCIF API:: + + Process A (on node A) Process B (on node B) + + /* get online node information */ + scif_get_node_ids(..) scif_get_node_ids(..) + scif_open(..) scif_open(..) + scif_bind(..) scif_bind(..) + scif_listen(..) + scif_accept(..) scif_connect(..) + /* SCIF connection established */ + + /* Send and receive short messages */ + scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..) + + /* Register memory */ + scif_register(..) scif_register(..) + + /* RDMA */ + scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..) + + /* Fence DMAs */ + scif_fence_signal(..) scif_fence_signal(..) + + mmap(..) mmap(..) + + /* Access remote registered memory */ + + /* Close the endpoints */ + scif_close(..) scif_close(..) diff --git a/Documentation/misc-devices/uacce.rst b/Documentation/misc-devices/uacce.rst new file mode 100644 index 000000000000..1db412e9b1a3 --- /dev/null +++ b/Documentation/misc-devices/uacce.rst @@ -0,0 +1,176 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Introduction of Uacce +--------------------- + +Uacce (Unified/User-space-access-intended Accelerator Framework) targets to +provide Shared Virtual Addressing (SVA) between accelerators and processes. +So accelerator can access any data structure of the main cpu. +This differs from the data sharing between cpu and io device, which share +only data content rather than address. +Because of the unified address, hardware and user space of process can +share the same virtual address in the communication. +Uacce takes the hardware accelerator as a heterogeneous processor, while +IOMMU share the same CPU page tables and as a result the same translation +from va to pa. + +:: + + __________________________ __________________________ + | | | | + | User application (CPU) | | Hardware Accelerator | + |__________________________| |__________________________| + + | | + | va | va + V V + __________ __________ + | | | | + | MMU | | IOMMU | + |__________| |__________| + | | + | | + V pa V pa + _______________________________________ + | | + | Memory | + |_______________________________________| + + + +Architecture +------------ + +Uacce is the kernel module, taking charge of iommu and address sharing. +The user drivers and libraries are called WarpDrive. + +The uacce device, built around the IOMMU SVA API, can access multiple +address spaces, including the one without PASID. + +A virtual concept, queue, is used for the communication. It provides a +FIFO-like interface. And it maintains a unified address space between the +application and all involved hardware. + +:: + + ___________________ ________________ + | | user API | | + | WarpDrive library | ------------> | user driver | + |___________________| |________________| + | | + | | + | queue fd | + | | + | | + v | + ___________________ _________ | + | | | | | mmap memory + | Other framework | | uacce | | r/w interface + | crypto/nic/others | |_________| | + |___________________| | + | | | + | register | register | + | | | + | | | + | _________________ __________ | + | | | | | | + ------------- | Device Driver | | IOMMU | | + |_________________| |__________| | + | | + | V + | ___________________ + | | | + -------------------------- | Device(Hardware) | + |___________________| + + +How does it work +---------------- + +Uacce uses mmap and IOMMU to play the trick. + +Uacce creates a chrdev for every device registered to it. New queue is +created when user application open the chrdev. The file descriptor is used +as the user handle of the queue. +The accelerator device present itself as an Uacce object, which exports as +a chrdev to the user space. The user application communicates with the +hardware by ioctl (as control path) or share memory (as data path). + +The control path to the hardware is via file operation, while data path is +via mmap space of the queue fd. + +The queue file address space: + +:: + + /** + * enum uacce_qfrt: qfrt type + * @UACCE_QFRT_MMIO: device mmio region + * @UACCE_QFRT_DUS: device user share region + */ + enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, + UACCE_QFRT_DUS = 1, + }; + +All regions are optional and differ from device type to type. +Each region can be mmapped only once, otherwise -EEXIST returns. + +The device mmio region is mapped to the hardware mmio space. It is generally +used for doorbell or other notification to the hardware. It is not fast enough +as data channel. + +The device user share region is used for share data buffer between user process +and device. + + +The Uacce register API +---------------------- + +The register API is defined in uacce.h. + +:: + + struct uacce_interface { + char name[UACCE_MAX_NAME_SIZE]; + unsigned int flags; + const struct uacce_ops *ops; + }; + +According to the IOMMU capability, uacce_interface flags can be: + +:: + + /** + * UACCE Device flags: + * UACCE_DEV_SVA: Shared Virtual Addresses + * Support PASID + * Support device page faults (PCI PRI or SMMU Stall) + */ + #define UACCE_DEV_SVA BIT(0) + + struct uacce_device *uacce_alloc(struct device *parent, + struct uacce_interface *interface); + int uacce_register(struct uacce_device *uacce); + void uacce_remove(struct uacce_device *uacce); + +uacce_register results can be: + +a. If uacce module is not compiled, ERR_PTR(-ENODEV) + +b. Succeed with the desired flags + +c. Succeed with the negotiated flags, for example + + uacce_interface.flags = UACCE_DEV_SVA but uacce->flags = ~UACCE_DEV_SVA + + So user driver need check return value as well as the negotiated uacce->flags. + + +The user driver +--------------- + +The queue file mmap space will need a user driver to wrap the communication +protocol. Uacce provides some attributes in sysfs for the user driver to +match the right accelerator accordingly. +More details in Documentation/ABI/testing/sysfs-driver-uacce. |