Mellanox GPUDirect RDMA User Manual
Contents
1. Mellanox TECHNOLOGIES Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox CORE Direct GPUDirect InfiniBridge InfiniHost InfiniScale Kotura Kotura logo Mellanox Connect Accelerate Outperform logo Mellanox Federal Systems Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX MetroX amp MLNX OS Open Ethernet logo PhyX ScalableHPC SwitchX TestX The Generation of Open Ethernet logo UFM Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd CyPU ExtendX FabricIT FPGADirect HPC X Mellanox Care Mellanox CloudX Mellanox NEO Mellanox Open Ethernet Mellanox PeerDirect NVMeDirect StPU Spectrum Switch IBTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 3878 Rev 1 2 Table of Contents Table off Contents eua ue AA ia 3 List f Tables eso a exce m EK Sat ead Ss Crane SG Kea did a clare RR one d E 4 Document Revision History 5 Chapter Overvi2. 1 tar gz Step 2 Change the working directory to be nvidia peer memory cd nvidia peer memory 1 0 0 Step3 Display the content of the README file and follow the installation instructions cat README txt Note On SLES OSes add nodeps gt To install GPUDirect RDMA for OpenMPI on Ubuntu Copy the tarball to a temporary directory tar xzf lt tarball gt cd lt extracted directory gt dpkg buildpackage us uc dpkg i lt path to generated deb files gt Example dpkg i nvidia peer memory 1 0 0 all deb dpkg i nvidia peer memory dkms 1 0 0 all deb Please make sure this kernel module is installed and loaded on each GPU InfiniBand com k pute nodes A gt To install GPUDirect RDMA for MVAPICH2 Step 1 Download gdrcopy library from https github com NVIDIA gdrcopy archive master zip and build it cd opt mvapich2 gdr 2 1 cuda7 0 gnu unzip master zip cd opt mvapich2 gdr 2 1 cuda7 0 gnu gdrcopy master make CUDA usr local cuda 7 0 all Step 2 Make sure gdrcopy is installed on all compute nodes and load the module on each GPU node cd opt mvapich2 gdr 2 1 cuda7 0 gnu gdrcopy master insmod sh 8 Mellanox Technologies J Rev 1 2 3 Benchmark Tests 3 1 Running GPUDirect with MVAPICH GDR 2 1 MVAPICH2 takes advantage of the new GPUDirect RDMA technology for inter node data movement on NVIDIA GPUs clusters with Mellanox InfiniBand interconnect MVAPICH GDR v2 1 can be downloaded from ht
3. GPUDirect RDMA is an API between IB CORE and peer memory clients such as NVIDIA Kepler class GPU s It provides access the HCA to read write peer memory data buffers as a result it allows RDMA based applications to use the peer device computing power with the RDMA interconnect without the need to copy data to host memory This capability is supported with Mellanox ConnectX amp 3 VPI and later or Connect IB InfiniBand adapters It will also work seemlessly using RoCE technology with the Mellanox ConnectX 3 and later VPI adapt ers 1 4 System Requirements The platform and server requirements for GPUDirect RDMA are detailed in the following table Table 2 GPUDirect RDMA System Requirements Platform Type and Version HCAs Mellanox ConnectX 3 Mellanox ConnectX 3 Pro e Mellanox Connect IB Mellanox ConnectX 4 NVIDIA Tesla K Series K10 K20 K40 K80 GPU Software Plugins MLNX OFED v2 1 x x x or later www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Linux SW Driv ers Plugin module to enable GPUDirect RDMA www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt GPUDirect RDMA on the left navigation pane NVIDIA Driver http www nvidia com Download index aspx lang en us NVIDIA CUDA Runtime and Toolkit https developer nvidia com cuda downloads NVIDIA Documentation http docs nvidia com cuda index html getting started g
4. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox GPUDirect RDMA User Manual Rev 1 2 www mellanox com Rev 1 2 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE
5. ew susu Z sae eee Ne 6 1 1 System Requirements 6 1 2 Important Notes a osos 530 S es an dione Sd D AN aR oi 6 Chapter 2 Installing GPUDirect 8 Chapter 3 Benchmark Tests 9 3 1 Running GPUDirect with MVAPICH GDR 2 1 9 3 2 Running GPUDirect with OpenMPI 1 10 0 10 Mellanox Technologies 3 J Rev 1 2 List of Tables Table 1 Document Revision History 5 Table 2 GPUDirect System Requirements 6 4 Mellanox Technologies J Rev 1 2 Document Revision History Table 1 Document Revision History Release Date Description 1 2 September 2015 Updated the following sections Section 1 1 System Requirements on page 6 Section 2 Installing GPUDirect RDMA on page 8 Section 3 1 Running GPUDirect RDMA with MVAPICH GDR 2 1 on page 9 Section 3 2 Running GPUDirect RDMA with OpenMPI 1 10 0 on page 10 1 1 December 18 2014 Updated Section 3 2 Running GPUDirect RDMA with MVAPICH GDR 2 0b on page 6 Added how to enable RoCE communication 1 0 May 19 2014 Initial release Mellanox Technologies 5 J Rev 1 2 Overview 1 Overview
6. h lib disable vt enable orterun prefix by default disable io romio enable picky with cuda usr local cuda 7 0 make make install The OSU benchmarks are CUDA enabled benchmarks that can downloaded from http mvapich cse ohio state edu benchmarks When building the OSU benchmarks you must verify that the proper flags are set to enable the CUDA part of the tests otherwise the tests will only run using the host memory instead which is the default Additionally make sure that the MPI libraries OpenMPI is installed prior to compiling the benchmarks export PATH path to openmpi 1 10 0 cuda7 0 bin PATH configure CC mpicc prefix path to osu benchmarks N enable cuda with cuda usr local cuda 7 0 make make install 10 Mellanox Technologies Rev 1 2 To run the OpenMPI that uses the flag that enables GPUDirect RDMA mpirun mca btl openib want cuda gdr 1 np 2 npernode 1 mca btl openib if include mlx 5 0 1 bind to core cpu set 19 x CUDA VISIBLE DEVICES 0 path to osu benchmarks osu latency d cuda D D OSU MPI CUDA Latency Test Send Buffer on DEVICE D and Receive Buffer on DEVICE D Size Latency us 0 1 08 1 3 83 2 998 4 3 84 8 3 83 16 3 83 32 31502 64 3 80 Please note that cpu set core number has to be a core number from the same socket that shares the same PCI slot with the GPU P If the flag for GPUDirect RDMA is not enabled it would result in much higher latenc
7. rect RDMA such as MV2 CUDA BLOCK SIZE can be found in the 2 user guideline Below is an example of enabling RoCE communication mpirun np 2 hostl host2 genv MV2 USE RoCE 1 genv MV2 DEFAULT GID INDEX 2 genv MV2 DEFAULT SER VICE LEVEL 3 genv MV2 USE CUDA 1 MV2 USE GPUDIRECT 1 opt mvapich2 gdr 2 1 cuda7 0 gnu libexec mvapich2 osu bw d cuda D D Where Parameter Description MV2 USE RoCE 1 Enables RoCE communication Mellanox Technologies 9 J Rev 1 2 Benchmark Tests Description MV2_DEFAULT_GID_INDEX lt gid index gt Selects the non default GID index using MV2 DEFAULT GID INDEX since all VLAN interfaces appear as additional GID indexes starting from 1 on the InfiniBand HCA side of the RoCE adapter You can select a non default GID index using run time parameter MV2 DE FAULT GID INDEX 11 84 and RoCE priority service level using MV2 DEFAULT SER VICE LEVEL MV2 DEFAULT SERVICE LEVEL service level Selects RoCE priority service level using MV2 DEFAULT SERVICE LEVEL 3 2 Running GPUDirect with OpenMPI 1 10 0 The GPUDirect RDMA support is available on OpenMPI 1 10 0 Unlike MVAPICH2 GDR which is available in the RPM format one can download the source code for OpenMPI and com pile using flags below to enable GPUDirect RDMA support configure prefix path to openmpi 1 10 0 cuda7 0 V with wrapper ldflags Wl rpat
8. tp mvapich cse ohio state edu download GPUDirect RDMA can be tested by running the micro benchmarks from Ohio State University OSU Below is an example of running one of the OSU benchmark which is already bundled with MVAPICH2 GDR v2 1 with GPUDirect RDMA mpirun np 2 hosti host2 genv MV2 CPU MAPPING 0 genv MV2 USE CUDA 1 genv MV2 USE GPUDIRECT 1 opt mvapich2 gdr 2 1 cuda7 0 gnu libexec mvapich2 osu bw d cuda D D OSU MPI CUDA Bandwidth Test Send Buffer on DEVICE D and Receive Buffer on DEVICE D Size Bandwidth MB s 2097152 6372 60 4194304 6388 63 Please note that MV2 CPU lt number gt has to be a core number from the same socket that shares the same PCI slot with the GPU The MV2 GPUDIRECT LIMIT is used to tune the hybrid design that uses pipelining and GPU Direct RDMA for maximum performance while overcoming P2P bandwidth bottlenecks seen on modern systems GPUDirect RDMA is used only for messages with size less than or equal to this limit Here is a list of runtime parameters that can be used for process to rail binding in case the system has multi rail configuration export MV2 USE CUDA 1 export MV2 USE GPUDIRECT 1 export MV2 RAIL SHARING POLICY FIXED MAPPING export MV2 PROCESS TO RAIL MAPPING mlx5 0 mlx5 1 export MV2 RAIL SHARING LARGE MSG THRESHOLD 1G export MV2 CPU BINDING LEVEL SOCKET export MV2 CPU BINDING POLICY SCATTER Additional tuning parameters related to CUDA and GPUDi
9. uides 1 2 Important Notes Once the NVIDIA software components are installed it is important to check that the GPUDirect kernel module is properly loaded on each of the compute systems where you plan to run the job that requires the GPUDirect RDMA feature To check service nv_peer mem status Or for some other flavors of Linux lsmod grep nv peer mem Usually this kernel module is set to load by default by the system startup service If not loaded GPUDirect RDMA would not work which would result in very high latency for message com munications One you start the module by either service nv peer mem start 6 Mellanox Technologies J Rev 1 2 Or for some other flavors of Linux modprobe nv_peer mem To achieve the best performance for GPUDirect RDMA it is required that both the HCA and the GPU be physically located on the same PCIe IO root complex To find out about the system architecture either review the system manual run lspci tv grep NVIDIA Mellanox Technologies 7 J Rev 1 2 Installing GPUDirect RDMA 2 Installing GPUDirect Please ensure that you have installed MLNX_OFED before trying to install GPUDirect RDMA MLNX OFED can be downloaded from ae www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Linux SW Drivers gt To install GPUDirect RDMA for OpenMPI excluding Ubuntu Step 1 Unzip the package untar nvidia peer memory 1 0
10. y for the above By default in OpenMPI 1 10 0 the GPUDirect RDMA will work for message sizes between 0 to 30KB For messages above that limit it will be switched to use asynchronous copies through the host memory instead Sometimes better application performance can be seen by adjusting that limit Here is an example of increasing to adjust the switch over point to above 64KB mca btl openib cuda rdma limit 65537 Mellanox Technologies 11 J
Download Pdf Manuals
Related Search