MellanoX Messaging Library User Manual
Contents
1. MXM over the Ethernet fabric Step 1 Make sure the Ethernet port is active ibv devinfo ibv_devinfo displays the list of cards and ports in the system Please make sure in the gt d ibv_devinfo output that the desired port has Ethernet at the 1ink layer field and that its state is PORT ACTIVE Step 2 Specify the ports you would like to use if there is a non Ethernet active port in the card x MXM RDMA PORTS mlx4 0 1 or x MXM IB PORTS mlx4 0 1 2 6 Configuring MXM over Different Transports MXM v3 0 supports the following transports Intra node communication via Shared Memory with KNEM support Unreliable Datagram UD Reliable Connected RC SELF transport a single process communicates with itself Dynamically Connected Transport DC at beta level Note DC is supported on Connect IB HCAs with MLNX OFED v2 1 1 0 0 and higher To use DC set the following in the command line mpirun x MXM TLS self shm dc from the SHELL export MXM TLS self shm dc By default the transports TLS used are MxM_TLS self shm ud Mellanox Technologies 9 Rev 3 0 Configuring MXM 2 7 Configuring Service Level Support Service Level Support is currently at alpha level Please be aware that the content below is subject to change MXM v3 0 added support for Service Level to enable Quality of Service QoS If set every InfiniBand endpoint in MXM will generate a random Service Level SL within the given r2. GE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox amp Mellanox logo BridgeX ConnectX Connect IB amp CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 4113 Rev 3 0 Table of Content Tabl of Content uses dee cee nase Es Oren aa a ca s mex E Eu etd 3 List of Vables ss ss ued E AA E A eee eee Eu 4 Document Revision History ooooccocnnnr eee 5 Chapter 1 MellanoX Messaging Library oooooooooommom o 6 Ee I EE 6 1 2 System Requirements usce oue AA 6 Chapter 2 Configuring MXM 200 RR RII 7 2 1 Compiling Open MPI with MXM 2020002000008 7 2 2 Enabling MXM in Open MPI rra rasa 8 2 3 Tuning MXM Set
3. Mellanox TECHNOLOGIES Connect Accelerate Outperform MellanoX Messaging Library User Manual Rev 3 0 www mellanox com Rev 3 0 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMA
4. ange and use it for outbound communication Setting the value is done via the following environment parameter MXM IB NUM SLS Available Service Level values are 1 16 where the default is 1 10 Mellanox Technologies Rev 3 0 3 MXM Utilities 3 1 mxm_dump_config Enables viewing of all the environment parameters that MXM uses To see all the parameters run opt mellanox mxm bin mxm dump config f For further information please run opt mellanox mxm bin mxm dump config help For example to set the MXM TLS environment parameter run o hat export MXM TLS gt Environment parameters can be set by using the export command 3 2 mxm perftest A server client based application which is designed to test MXM s performance and sanity checks on MXM To run it two terminals are required to be opened one on the server side and one on the client side The working flow is as follow 1 The server listens to the request coming from the client 2 Once a connection is established between them MXM sends and receives messages between the two sides according to what the client requested 3 The results of the communications are printed out For further information please run opt mellanox mxm bin mxm perftest help Example From the server side run opt mellanox mxm bin mxm perftest from the client side run opt mellanox mxm bin mxm perftest server host name t send lat Among other pa
5. ewer MXM Step 1 Remove MXM rpm e mxm Step 2 Remove the pre compiled Open MPI rpm e mlnx openmpi gcc Step3 Install the new MXM and compile the Open MPI with it To run Open MPI without MXM run o mpirun mca mtl mxm lt gt p When upgrading to MXM v3 0 Open MPI compiled with the previous versions of the MXM should be recompiled with MXM v3 0 Mellanox Technologies 7 Rev 3 0 Configuring MXM 2 2 Enabling MXM in Open MPI As of MXM v2 1 MXM is automatically selected by Open MPI up to v1 6 when the Number of Processes NP is higher or equal to 128 gt To activate MXM for any NP run o mpirun mca mtl mxm np 0 other mpirun parameters gt From Open MPI v1 7 x MXM is selected when the number of processes is higher or equal to 0 i e by default 2 3 Tuning MXM Settings The default MXM settings are already optimized To check the available MXM parameters and their default values run the opt mellanox mxm bin mxm dump config f utility which is part of the MXM RPM MXM parameters can be modified in one of the following methods Modifying the default MXM parameters value as part of the mpirun mpirun x MXM UD RX MAX BUFFS 128000 lt gt Modifying the default MXM parameters value from SHELL export MXM UD RX MAX BUFFS 128000 G ILA E 2 4 Configuring Multi Rail Support Multi Rail support enables the user to use more than one of the active ports on the card by
6. hin the parallel communication libraries The latest MXM software can be downloaded from the Mellanox website 1 2 System Requirements Mellanox OFED 2 0 3 0 0 and later Open MPI 1 6 5 or later To download the Open MPI v1 6 5 which contains special patches added by Mellanox please refer to the Mellanox website Open MPI v1 7 4 or later To Open MPI v1 7 4 go to http www open mpi org software ompi v1 7 6 Mellanox Technologies Rev 3 0 2 Configuring MXM 2 1 Compiling Open MPI with MXM Step 1 Install MXM from anRPM rpm ihv mxm x y z 1 x86 64 rpm atarball tar jxf mxm x y z tar bz MXM will be installed automatically in the opt mellanox mxm folder Step 2 Enter Open MPI source directory and run cd OMPI HOME configure with mxm opt mellanox mxm other configure parameters make all amp amp make install oo Older versions of MLNX OFED come with pre installed older MXM and Open MPI versions Please uninstall any old MXM version prior to installing the latest MXM version in order to use it with older MLNX OFED versions Table 2 MLNX OFED and MXM Versions MLNX OFED Version MXM Version v1 5 3 3 1 0 and v2 0 3 0 0 MXM v1 x and Open MPI compiled with MXM vl x v2 0 3 0 0 and higher MXM v2 x 3 x and Open MPI compiled with MXM v2 x 3 x To check the version of MXM installed on your host run rpm qi mxm gt To upgrade MLNX OFED v1 5 3 3 1 0 or later with a n
7. mak ing a better use of the resources It provides a combined throughput among the used ports Multi Rail support in MXM v3 0 allows different processes on the same host to use different active ports Every process can only use one port as opposed to MXM v1 5 gt To configure dual rail support Specify the list of ports you would like to use to enable multi rail support x MXM RDMA PORTS cardName portNum or x MXM IB PORTS cardName portNum For example x MXM IB PORTS mlx5 0 1 It is also possible to use several HCAs and ports during the run separated by a comma x MXM IB PORTS mlx5 0 1 mlx5 1 1 MXM will bind a process to one of the HCA ports from the given ports list according to the MXM IB MAP MODE parameter for load balancing Possible values for MXM IB MAP MODE are first Default Maps the first suitable HCA port to all processes affinity Distributes the HCA ports evenly among processes based on CPU affinity nearest Tries to find the nearest HCA port based on CPU affinity 8 Mellanox Technologies Rev 3 0 You may also use an asterisk and a question mark to choose the HCA and the port you would like to use e use all active cards ports that are available e use the first active card port that is available For example x MXM IB PORTS will take all the active HCAs and the first active port on each of them 25 Configuring MXM over the Ethernet Fabric To configure
8. on page 8 e Section 2 4 Configuring Multi Rail Support on page 8 Section 2 5 Configuring MXM over the Ethernet Fabric on page 9 Added the following section Section 2 6 Configuring MXM over Different Transports on page 9 1 5 December 2012 Updated the following sections Section 2 1 Compiling Open MPI with MXM on page 7 Section 2 3 Tuning MXM Settings on page 8 Added the following section Section 24 Configuring Multi Rail Support on page 8 Section 2 5 Configuring MXM over the Ethernet Fabric on page 9 1 1 July 2012 Initial release Mellanox Technologies 5 J Rev 3 0 MellanoX Messaging Library 1 MellanoX Messaging Library 1 1 Overview MellanoX Messaging MXM library provides enhancements to parallel communication libraries by fully utilizing the underlying networking infrastructure provided by Mellanox HCA switch hardware This includes a variety of enhancements that take advantage of Mellanox networking hardware including e e e Multiple transport support including RC DC and UD Proper management of HCA resources and memory structures Efficient memory registration One sided communication semantics Connection management Receive side tag matching Intra node shared memory communication These enhancements significantly increase the scalability and performance of message commu nications in the network alleviating bottlenecks wit
9. rameters you can specify the test you would like to run the message size and the number of iterations to run Mellanox Technologies 11
10. tings ora a 8 2 4 Configuring Multi Rail Support 0 0 00 cee ee 8 2 5 Configuring MXM over the Ethernet FabriC o o o ooooo 9 2 6 Configuring MXM over Different Transports lille 9 2 7 Configuring Service Level Support ssseeeeseeees ss rss ses ss sera 10 Chapter 3 MXM Utilities 555 oxi re UR RE A AA RE 11 3 1 mxm dump config 2 002 ce lees 11 IZ mxm perftest eet xe CXV CE Eee aa t RR davon 11 Mellanox Technologies 3 J Rev 3 0 List of Tables Table 1 Document Revision History o ooooooooorororr ro 5 Table 2 MLNX OFED and MXM Versions 00000 e 7 4 Mellanox Technologies Rev 3 0 Document Revision History Table 1 Document Revision History papain Date Description Revision 3 0 April 2014 Added Section 2 7 Configuring Service Level Support on page 10 2 1 February 2014 Updated the following sections Section 2 1 Compiling Open MPI with MXM on page 7 Section 2 2 Enabling MXM in Open MPI on page 8 Section 2 4 Configuring Multi Rail Support on page 8 Section 2 6 Configuring MXM over Different Transports on page 9 Added the following section e Section 3 1 mxm dump config on page 11 e Section 3 2 mxm perftest on page 11 2 0 August 2013 Updated the following sections Section 2 1 Compiling Open MPI with MXM on page 7 Section 2 3 Tuning MXM Settings
Download Pdf Manuals
Related Search