Mellanox GPUDirect RDMA User Manual
Contents
1. Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Hakidma 26 Sunnyvale CA 94085 Ofer Industrial Park U S A Yokneam 2069200 www mellanox com Israel Tel 408 970 3400 www mellanox com Fax 408 970 3403 Tel 972 0 74 723 7200 Fax 972 0 4 959 3245 O Copyright 2015 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CoolBox amp CORE Direct amp GPUDirect InfiniBridge InfiniHost InfiniScale Kotura Kotura logo MetroX MLNX OS PhyX ScalableHPC SwitchX TestX Virtual Protocol Interconnect Voltaire and Voltaire logo are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT HPC X Mellanox CloudX Mellanox Open Ethernet Mellanox PeerDirect Mellanox Virtual Modular Switch MetroDX Switch IB Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 3878 Rev 1 1 Table of Contents Table of Contemts 345 s s euo iu Ae eh RE 3 List f s a n Cx Sat ead tame SG ono e OK Cor eR RI GR UR RR 1 Document Revision History 2 Chapter T OVervieweilsvu rede pa hs son SSG SEER ROS EX eR Qe 3 1 1 System2. Mellanox TECHNOLOGIES Connect Accelerate Outperform Mellanox GPUDirect RDMA User Manual Rev 1 1 www mellanox com 1 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE
3. Requirements 3 1 2 Important Notes a ocot S Ree RH D RA E ee oR ce atl 3 Chapter2 Installing GPUDirect 4 Chapter3 Benchmark 5 3 1 Testing GPUDirect with CUDA Enabled Benchmark 5 3 2 Running GPUDirect with MVAPICH GDR 2 0b 5 3 3 Running GPUDirect RDMA with OpenMPI 1 7 4 6 Mellanox Technologies 3 J 1 1 List of Tables Table 1 Document Revision History 2 Table2 GPUDirect RDMA System Requirements 3 1 Mellanox Technologies J Rev 1 1 Document Revision History Table 1 Document Revision History Release Date Description 1 1 December 18 2014 Updated Section 3 2 Running GPUDirect RDMA with MVAPICH GDR 2 0b on page 5 Added how to enable RoCE communication 1 0 May 19 2014 Initial release Mellanox Technologies 2 J I Rev 1 1 Overview 1 Overview GPUDirect RDMA is an API between IB CORE and peer memory clients such as NVIDIA Kepler class GPU s It provides access for the HCA to read write peer memory data buffers as a result it allows RDMA based applications to use the peer device computing power with the RDMA interconnect without the need to copy data
4. at can downloaded from http mvapich cse ohio state edu benchmarks When building the OSU benchmarks you must verify that the proper flags are set to enable the CUDA part of the tests otherwise the tests will only run using the host memory instead which is the default configure CC path to mpicc V enable cuda V with cuda include path to cuda include Y with cuda libpath path to cuda lib make make install Running GPUDirect RDMA with MVAPICH GDR 2 0b 2 that takes advantage of the new GPUDirect RDMA technology for inter node data movement on NVIDIA GPUS clusters with Mellanox InfiniBand interconnect MVAPICH GDR 2 0b can be downloaded from http mvapich cse ohio state edu download mvapich2gdr Below is an example of running one of the OSU benchmark which enables GPUDirect RDMA gdr ops001 15 mpirun rsh np 2 ops001 ops002 MV2 USE CUDA 1 MV2 USE GPUDIRECT 1 home gdr osu micro benchmarks 4 2 mvapich2 mpi pt2pt osu bw d cuda D D OSU MPI CUDA Bandwidth Test v4 2 Send Buffer on DEVICE D and Receive Buffer on DEVICE D Size Bandwidth MB s 2097152 6372 60 4194304 6388 63 The MV2 GPUDIRECT LIMIT is used to tune the hybrid design that uses pipelining and GPU Direct RDMA for maximum performance while overcoming P2P bandwidth bottlenecks seen on modern systems GPUDirect RDMA is used only for messages with size less than or equal to this limit Here is a list of runtime parameters that can be us
5. cy for the above By default in OpenMPI 1 7 4 the GPUDirect RDMA will work for message sizes between 0 to 30KB For messages above that limit it will be switched to use asynchronous copies through the host memory instead Sometimes better application performance can be seen by adjusting that limit Here is an example of increasing to adjust the switch over point to above 64KB mca btl openib cuda rdma limit 65537 7 Mellanox Technologies
6. ed for process to rail binding in case the system has multi rail configuration export MV2 USE CUDA 1 export MV2 USE GPUDIRECT 1 export MV2 RAIL SHARING POLICY FIXED MAPPING export MV2 PROCESS TO RAIL MAPPING mlx5 0 mlx5 1 export MV2 RAIL SHARING LARGE MSG THRESHOLD 1G export MV2 CPU BINDING LEVEL SOCKET export MV2 CPU BINDING POLICY SCATTER Additional tuning parameters related to CUDA and GPUDirect RDMA such as MV2 CUDA BLOCK SIZE can be found in the README installed on the node opt mvapich2 gdr 2 0 gnu share doc mvapich2 gdr gnu 2 0 README GDR 5 Mellanox Technologies J Rev 1 1 Below is an example of enabling RoCE communication opt mvapich2 gdr 2 0 gnu bin mpirun_rsh np 2 test01 test02 MV2 USE RoCE 1 MV2 DEFAULT GID INDEX 2 MV2 DEFAULT SERVICE LEVEL 3 MV2 USE CUDA 1 MV2 USE GPUDIRECT 1 opt mvapich2 gdr 2 0 gnu libexec mvapich2 osu bw d cuda D D Where Parameter Description MV2 USE RoCE 1 Enables RoCE communication MV2 DEFAULT GID INDEX gid gt Selects the non default GID index using MV2 DEFAULT GID INDEX since all VLAN interfaces appear as additional GID indexes starting from 1 on the InfiniBand HCA side of the RoCE adapter You can select a non default GID index using run time parameter MV2 DEFAULT GID INDEX 11 84 and RoCE priority service level using MV2 DEFAULT SERVICE LEVEL MV2 DEFAULT SERVICE LEVEL service level Selects RoCE priority
7. m startup service If not loaded GPU Direct RDMA would not work which would result in very high latency for message communications One you start the module by either service nv peer mem start Or for some other flavors of Linux modprobe nv peer mem To achieve the best performance for GPUDirect RDMA it is required that both the HCA and the GPU be physically located on the same PCIe IO root complex To find out about the system architecture either review the system manual or run Ispci tv 3 Mellanox Technologies J Rev 1 1 2 Installing GPUDirect RDMA gt To install GPUDirect RDMA excluding ubuntu rpmbuild rebuild path to srpm rpm ivh path to generated binary rpm file Note On SLES OSes add To install GPUDirect RDMA on Ubuntu Copy the tarball to a temporary directory tar xzf tarball cd extracted directory dpkg buildpackage us uc dpkg i path to generated deb files Example dpkg i nvidia peer memory 1 0 0 all deb dpkg i nvidia peer memory dkms 1 0 0 all deb p Please make sure this kernel module is installed and loaded on each GPU InfiniBand com pute nodes Mellanox Technologies 4 J 1 1 Benchmark Tests 3 1 3 2 Benchmark Tests Testing GPUDirect RDMA with CUDA Enabled Benchmark GPUDirect RDMA be tested by running the micro benchmarks from Ohio State University OSU The OSU benchmarks 4 and above are CUDA enabled benchmarks th
8. service level using MV2 DEFAULT SERVICE LEVEL 3 3 Running GPUDirect RDMA with OpenMPI 1 7 4 The GPUDirect support is available on OpenMPI 1 7 4rc1 Unlike MVAPICH2 GDR which is available in the RPM format one can download the source code for OpenMPI and com pile using flags below to enable GPUDirect RDMA support co melliGlogin sand8 configure prefix path to openmpi 1 7 4rcl install Y with wrapper ldflags Wl rpath lib disable vt enable orterun prefix by default dis able io romio enable picky with cuda usr local cuda 5 5 V with cuda include usr local cuda 6 0 include V with cuda libpath usr local cuda 6 0 1ib64 co melliGlogin sand8 make make install To run the OpenMPI that uses the flag that enables GPUDirect RDMA gdr jupiter001 mpirun mca btl openib want cuda 1 np 2 npernode 1 x LD LIBRARY PATH mca btl openib if include mlx5 0 1 bind to core report bindings mca coll fca enable 0 x CUDA VISIBLE DEVICES 0 home co melll scratch osu micro benchmarks 4 2 install libexec osu micro benchmarks mpi pt2pt osu latency d cuda D D OSU MPI CUDA Latency Test v4 2 Send Buffer on DEVICE D and Receive Buffer on DEVICE D Size Latency us 0 1 08 1 2289 2 2280 4 3 84 8 3 83 16 2289 32 3 62 64 3 80 Mellanox Technologies 6 J 1 1 Benchmark Tests If the flag for GPUDirect RDMA is not enabled it would result in much higher laten
9. to host memory This capability is supported with Mellanox ConnectX 3 VPI or Connect IB InfiniBand adapters It will also work seem lessly using technology with the Mellanox ConnectX 3 VPI adapters 1 4 System Requirements The platform and server requirements for GPUDirect RDMA are detailed in the following table Table 2 GPUDirect RDMA System Requirements Platform Type and Version HCAs Mellanox ConnectX 3 Mellanox ConnectX 3 Pro Mellanox Connect IB NVIDIA Tesla K Series K10 K20 40 GPU Software Plugins MLNX OFED v2 1 x x x or later www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Linux SW Driv ers Plugin module to enable GPUDirect RDMA www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt GPUDirect RDMA NVIDIA Driver 331 20 or later http www nvidia com Download index aspx lang en us NVIDIA CUDA Runtime and Toolkit 6 0 https developer nvidia com cuda downloadsservice 1 2 Important Notes Once the hardware and software components are installed it is important to check that the GPUDirect kernel module is properly loaded on each of the compute systems where you plan to run the job that requires the GPUDirect RDMA feature To check service nv_peer mem status Or for some other flavors of Linux lsmod grep nv peer mem Usually this kernel module is set to load by default by the syste
Download Pdf Manuals
Related Search