Home

User Manual for PCIeGen3_x16

1. PASS Bose ravenstein PASS NumbenBoRECABPORntSRACETVOR EET T T 0 Port State of Port 41 on CA 80 HCA INIT InfiniBand Port State of Port 42 on CA 80 HCA INIT InfiniBand Error Counter Check on CA 0 HCA PASS Kernelisystog check a aa PASS Node AGUDA ACA OE ee 00202 69 03 00 18 00 40 DONE After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters can be retrieved by running the com a mand etc infiniband info 4 1 4 Installation Results Software e The OFED and MFT packages are installed under the usr directory The kernel modules are installed under InfiniBand subsystem 1ib modules uname r updates kernel drivers infiniband mlx4 core driver lib modules lt kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx4 mlx4 core ko mlx4 ib driver lib modules lt kernel version extra mlnx ofa kernel drivers infiniband hw mlx4 mlx4 ib ko e mlx core driver lib modules lt kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx5 core mlx5 core ko e mlx ib driver lib modules lt kernel version extra mlnx ofa kernel drivers infiniband hw mlx5 mlx5 ib ko e PoIB lib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko e SER lib modules uname r updates kernel drivers infi
2. 8 Strahlenkontak Mellanox Technologies 53 Rev 1 3 Appendix F Advertencias de seguridad para la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la A m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante periodos de actividad de M rayos 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec AA tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales An Este equipo se debe instalar conforme a los c digos el ctricos
3. Troubleshooting on page 36 January 2013 1 0 First Release Mellanox Technologies 7 Rev 1 3 About this Manual This User Manual describes Mellanox Technologies Connect IB amp Single and Dual QSFP port PCI Express 16 lanes x16 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documenta tion Intended Audience This manual is intended for the installer and user of these cards and assumes basic familiarity with InfiniBand network and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User User Manual describing the set of MFT firmware management Manual tools for a single node Document no 2204UG See http www mellanox com gt Support gt Download Firm ware Tools Mellanox OFED for Linux User Manual describing OFED features performance InfiniBand User Manual diagnostic tools content and configuration See http www mella Document no 2877 nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED IBTA Specification Release 1 2 1 InfiniBand Architecture Specification PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memo
4. EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HH EEEE HEHE HEHEHEHE HEHEHE 192182111211 11 EEEE HEHE HEHEHEHE HEHE E HEETE ETE EE H EEHEHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHE HEHEHEHE HE HEE EE EEEE EEEE HEHEHEHE HEHEHE 111182111211 11 EEEE HEHE HEHEHEHE HEHEHE 118218112111 EEEE HEHEHEHEHE HE HEHEHE EE EEEE EEEE HEHEHEHE HE HEE EE EEEE HEHE HEHEHEHE HE HEE EE EEEE HEHE HEHEHEHE HE HEE HEETE ETETE HEEE HH HHHHHHHHHHHHHHHHHHHHHHHH The FCA Manager will not be started automatically To start FCA Manager now type etc init d fca managerd start There should be single process of FCA Manager running per fabric To start FCA Manager automatically after boot type etc init d fca managerd install service 26 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Check opt mellanox fca share doc fca README txt for quick start instructions Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE dapl THHHRHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE dapl THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE dapl devel THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHH
5. libibem devel Preparing libibem devel Preparing libibumad Preparing libibumad Preparing libibumad devel Preparing libibumad devel Preparing libibumad static Preparing libibumad static Preparing libibmad Preparing libibmad Preparing libibmad devel Preparing libibmad devel Preparing libibmad static Mellanox Technologies 25 EEEE HEHEHEHEHE HEHE HEE HEETE ETETE 11 AAA EEEE HEHEHEHEHE HE HE HEE EE EEEE HEHE HEHEHEHE HE HEE Ea 272 272 272 272 272 272 272 SEE EE an AE AE a 272 272 272 272 272 270 272 EEEE HEHE a HE HEE EEE EEEE HEHE HEHEHEHE HEHEHE 121821112111 EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HE HEE 112182111211 11 EEEE HEHEHEHEHE HE HE HEE 121821112111 EE EEHEHE HEHEHEHE HEHEHE 112182111211 11 EEEE HEHE HEHEHEHE HEHEHE 112182111111 EEEE HEHEHEHE HEHE HE HEE HEETE EE 272 272 272 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HEHE E HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEE HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HE HEE HEETE EEE H EE EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HE EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE EE a 272 272 272 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHEHEHEHE HE HEHEHE HEETE ETE EE HEEE EHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHEHEHEHE HEHEHEHE 121821112111 EEEE HEHE HEHEHEHEHE HEE EE EEEE HEHE HEHEHEHE HEHEHE 121821118211
6. 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der LA maximal empfohlenen Temperatur von 55 C 131 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich AuBerdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr An Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlieBen Sie keine Kabel an oder ab 4 Anschlie en Trennen von Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die LA Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden This equipment should be installed in compliance with local and national elec hb trical codes 52 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3
7. opensm opensmd Preparing opensm devel Preparing opensm devel Preparing opensm static Preparing opensm static Preparing infiniband diags Preparing fca INFO updating IMPORTANT NOTE The FCA Manager and FCA MPI Runtime library are installed in opt mellanox fca directory ce Rev 1 3 Driver Installation EEEE HEHE HEHEHEHE HEHEHE AAA EEEE HEHE HEHEHEHE HE HEE HEETE EEE HEEE HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HE HEE HEETE EEEE EE EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EHHH HHHHHHHHHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHE HE HEE HEETE ETETE HEHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HE AEREA IEA ACACIA IE IE IEA AER E EEEE HEHE HEHEHEHE HE HEE EE EEEE HEHE HEHEHEHE HE HEE 121821112181 11 EEEE HEHE HEHEHEHE HEHEHE 121821112111 EEEE EEEE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HEHEHE 2182111211 EEEE HEHE HEHEHEHEHE HEE HEETE EE EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE A a 11 EEEE HEHEHEHEHE HE HE HEE HEETE EEE H EE EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HE EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE EE a 272 272 272 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHEHEHEHE HE HEHEHE HEETE ETE EE HEEE EHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HEHEHE EE EEEE HEHEHEHEHE HEHEHEHE HEETE 1118211 121111 EEEE HEHE HEHEHEHE HEHEHE a AER E somt sol Asmi Segue Lout Hower Geene
8. 17 3 6 Cables and Modules u saviano senses e retta P OR ea nat 17 3 6 T Gable Installation sr Renten beenve tub eU eR S CREE 17 3 7 Identify the Card in Your System 18 33141 A oett pee ute ICI EUCH 18 Chapter 4 Driver Installation aesseersendsndstestsenkenwerdeidendsnast 19 add PAIK tt edicta de iet ar e D Ed etes ee Rt erts o fad aot 19 4 1 1 Hardware and Software Requirements 19 4 1 2 Downloading Mellanox OFED 19 4 1 3 Installing Mellanox OFED 19 Mellanox Technologies 3 Rev 1 3 4 1 3 1 Pre installation Notes 20 4 1 3 2 Installation Script 21 4 1 3 3 minxofedinstall Return Codes 22 4 1 3 4 Installation Procedure 22 4 1 4 Installation Results 30 4 1 5 Post installation Notes 32 4 1 6 Updating Firmware After Installation 32 Chapter gt Firmware nasse d ones reelle l 34 5 1 Updating Adapter Card Firmware 34 Chapter 6 Troubleshooting 36 6 1 General th set nde ERR ee rin Monte pue ne a dat
9. HE HEE EE 11 EEEE HEHE HEHEHEHE HE HEE a EE E EEEE HEHE HEHEHEHE HEHEHE HH EEEE HEHE HEHEHEHEHE HEE 119218211121 111 EEEE HEHE HEHEHEHE HEHEHE 1821112111 EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE EEEE HE RACER AEREA IE AE IE AER E EEEE HEHE HEHEHEHE HEHEHE 118218118211 1 EEEE EEEE HEHEHEHE HE HEE 111821111111 EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE a 11 EEEE HEHE HEHEHEHE HE HEE HEETE EEEH EE EEEE HEHE HEHEHEHE HEHE E HEETE EE 272 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HE HEE HEETE ETETE EE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE a EE EEEE HEHE HEHEHEHE HEHEHE 181821112111 EEEE HEHEHEHEHE HEHE HEE EE EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HE HEE 121821118211 11 EEEE HEHE HEHEHEHE HEHEHE 118211121111 EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE EE EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHEHEHEHE HE HE HEE HEETE ETETE HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH 24 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 libcxgb4 devel Preparing libcxgb4 devel Preparing libnes Preparing libnes Preparing libnes devel static Preparing libnes devel static Preparing libipathverbs Preparing libipathverbs Preparing libipathverbs devel Preparing libipathverbs devel Preparing libibem Preparing libibem Preparing
10. off power and unplug the cord Te opa D Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged 3 4 Bracket Installation The card is usually shipped with a tall bracket installed If this form factor is suitable for your requirements you can skip the remainder of this section and move to Section 3 5 Card Installa tion Instructions on page 17 If you need to replace it with the short bracket that is included in the shipping box please follow the instructions in this section Note Due to risk of damaging the EMI gasket it is not recommended to replace the bracket more than three times To replace the bracket you will need the following parts Mellanox Technologies 15 Rev 1 3 Hardware Installation The new bracket of the proper height e The 2 screws saved from the removal of the bracket The 2 fiber washers saved from the removal of the bracket 3 4 1 Removing the Existing Bracket 1 Remove the two screws holding the bracket in place The bracket comes loose from the card AA Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers 3 4 2 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up See Figure 2 Do not force the bracket onto the card You may have to gently push the LEDs using a AA sm
11. over any converged network with the same software infrastructure PCI Express 3 0 1 1 and Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x16 edge connector up to 2 0 compatible 8GT s InfiniBand FDR A pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s Memory PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 32Mb SPI Flash device Winbond W25Q32BVSSIG EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb Greater than 100Gb s over InfiniBand Greater than 130M messages sec lus MPI ping latency CPU offload of transport operations Application offload GPU communication acceleration End to end internal data protection End to end QoS and congestion control Hardware based I O virtualization RoHS R6 compliant Benefits World class cluster network and storage performance Guaranteed bandwidth and low latency services VO consolidation Virtualization acceleration Power efficient Scales to tens of thousands of nodes Mellanox Technologies 11 Rev 1 3 Introduction 1 3 Operating Systems and Distributions RHEL and other Linux distributions Microsoft Windows Server 2008 CCS 2003 HPC Server 2008 VMware ESX 5 1 e OpenFabrics Enterprise Distribution OFED 1 4 Connectivity In
12. reite wenen denen DEREN CR MT DEN dre Ys 8 Firmware and Software Updates 9 Chapter L Introduction ever a a en 10 l l Product Overview ennn aa 10 1 2 Features and Benefits 11 1 3 Operating Systems and Distributions 12 TA Connectivity use setes a LER ANGE RAE MS 12 Chapter 2 Interfaces es tace ACE Er eie LAC WAS 13 2 1 InfiniBand Interface rar ern eenen dele date ales Bull kan 13 22 PCLExpress Interface naren ei RR ek M REDIMI OE nad 13 2 3 LED Assienmient seres eie nee ERR MER ares a ne 14 2 4 I2C compatible Interface 14 Chapter 3 Hardware Installation 15 3 1 System Requirements 15 3 141 Hardware smet RUD METER EU EU ud bats 15 3 1 2 Operating Systems Distributions 15 3 13 Software Stacks cc or il di nie la RE 15 3 2 Safety Precautions ati A A eeh Ee 15 3 3 Pre installation Checklist 15 3 4 Bracket Installation iV Anura dann id alan CERES CdR M 15 3 4 1 Removing the Existing Bracket 16 3 4 2 Installing the New Bracket 16 3 5 Card Installation Instructions
13. the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file is used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc e If IPoIB configuration files are included ifcfg ib n files will be installed under etc sysconfig network scripts ona RedHat machine The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled 1 You run the installation script in default mode that is without the option without fw update Mellanox Technologies 31 Rev 1 3 Driver Installation 2 The firmware version of the adapter device is older than the firmware version included with the Mellanox OFED ISO image If an adapter s Flash was originially programmed with an Expansion ROM image the a automatic firmware update will also burn an Expansion ROM image p ncase your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Query
14. to reboot your machine Step 5 The script adds the following lines to etc security limits cont for the userspace com ponents such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinned by a user space application If desired tune the value unlimited to a specific amount of RAM Step 6 For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM refer to the OFED User Manual See Table 2 Documents List on page 8 Step 7 InfiniBand only Run the hca self test ofed utility to verify whether or not the Infini Band link is up The utility also checks for and displays additional information such as HCA firmware version Kernel architecture Driver version Number of active HCA ports along with their states Node GUID Mellanox Technologies 29 Rev 1 3 Driver Installation Note For more details on hca_self_test ofed see the file hea self test readme under does NunbersomiCAssDetected sense TE E il pet Devicercheck ar ee ee EIS PASS KEERDEN x86 64 HOS DAY NC RR NER ETT T MLNX OFED LINUX 2 1 0 0 4 20131125 1547 OFED 2 1 0 0 4 20131125 1547 3 0 13 0 27 default Host eEBMRGhockg TP PASS Havane O CN FORCE v10 10 1000 Firmware Check on CA 0 HCA
15. 1 EEEE HEHE HEHEHEHE HE HEE HEETE ETETE HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHEHEHEHE HE HE HEE HEETE ETE EE HE EEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHE HEHE E HEETE EE EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHE HEHEHEHE HE HEE EE EEEE EEEE HEHEHEHE HEHEHE 111182111211 11 EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HEHEHE 112182111211 11 EEEE HEHE HEHEHEHE HEHEHE 181821112111 EEEE HEHEHEHEHE HE HEHE E 111821111111 EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE a 11 EEEE HEHE HEHEHEHE HE HEE HEETE EEEH EE EEEE HEHE HEHEHEHE HEHE E HEETE EE 272 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HE HEE HEETE ETETE EE EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE a EE EEEE HEHE HEHEHEHE HEHEHE 181821112111 EEEE HEHEHEHEHE HEHE HEE EE EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HE HEE 121821118211 11 EEEE HEHE HEHEHEHE HEHEHE 118211121111 EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE EE EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHEHEHEHE HE HE HEE a 272 272 272 272 272 272 272 272 272 272 Preparing libibmad static Preparing ibsim Preparing ibacm Preparing librdmacm Preparing librdmacm Preparing librdmacm utils Preparing librdmacm devel Preparing librdmacm devel Preparing opensm libs Preparing opensm libs Preparing
16. A Mellanox Connect Accelerate Outperform Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual P N MCB193A FCAT MCB194A FCAT Rev 1 3 www mellanox com Rev 1 3 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT NCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND REL
17. A FCAT Specifications Table Size 2 71in x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical 38 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Table 8 MCB194A FCAT Specifications Table InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Protocol Support Data Rate Up to 56Gb s FDR InfiniBand PCI Express Gen3 SERDES 8 0GT s 16 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 13 15W Power and Max Power Passive Cables 17 84W Environmental Temperature 0 C to 55 C Air Flow 300LFM Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 Regulatory IEC 60068 2 64 29 32 RoHS RoHS R6 a Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 3 LED Operation There are two I O LEDs per port in dual port designs and one bicolor LED in single port designs green and yellow located on the I O panel See Table 9 for different LED functions Table 9 LED Assignment LED Function Yellow Physical link Constant on indicates a good physical link Blinking indicates a problem with the physical link If neither LED is lit then the physical link has not been es
18. AEREA IEA ACACIA IE IE IEA AER E infinipath psm Preparing infinipath psm devel EEEE HEHE HEHEHEHE HE HEE EE EEEE HEHE HEHEHEHE HE HEE 121821112181 11 EEEE HEHE HEHEHEHE HEHEHE 121821112111 Preparing EEEE EEEE HEHEHEHE HEHEHE EE mvapich2 EEEE HEHE HEHEHEHE HEHEHE 2182111211 Preparing THEHHHHHHHHBRHRHHHHHRHRHHHHHHHHBHHHHHHHHRHHHHH HERE openshmem EEEE HEHE HEHEHEHE HE HEE HEETE ETETE 272 272 27 272 272 272 272 Preparing EEEE HEHE HEHEHEHE HE HEE A a 11 hcoll THEHHHHHHHHBHRHHRHHHRHRHHHHHHHHHHHHHHHHHRHHHHHHEHRHEE Preparing EEEE HEHE HEHEHEHE HE HEE HEETE ETE TEE HERR libibprof EEEE HEHE HEHEHEHE HEHE E HEETE ETETE H EEHEHE HHHHHHHHHHHHHHHHHHHHHHHH Preparing EEEE HEHE HEHEHEHE HEHEHE HEETE HH libvma EEEE HEHEHEHEHE HE HEHEHE EE 11 Changing max locked memory to unlimited in etc security limits conf Please log out from the shell and login again in order to update this change Read more about this topic in the VMA s User Manual VMA README txt is installed at usr share doc packages libvma 6 5 3 0 README txt Please refer to VMA journal for the latest changes usr share doc packages libvma 6 5 3 0 journal txt THEHHHHHEHHBHRHHHHHRHHHHHHHHHHBHHRHHHHHHRHHHHH HERR EEEE HEHE HEHEHEHE HE HEE HEETE HEEE HHEH HHHHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHE E 118211182111 EEEE HEHE HEHEHEHE HE HEE HEETE EE EEEE HEHE HEHEHEHE HEHEHE EE EEHEHE HEHEHEHEHE HEHEHE 111218211111 11 EEEE EEEE HEHEHEHE HEHEHE a EE EEEE HEHE
19. ATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale CA 94085 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendXTM FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDXTM TestXTM Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3932 Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Table of Contents Table or Contents ee iles dha adden see aim ET EET 3 Dist OF Tables oars ta A u ri I a od i aO ed BE haa es 5 ASCOT BI GUESS sr ASA ARANA IU b we e pO EE ede 6 R vision History sa co seb eee A enn AA RAS ne ee oes 7 About this Manual ios sina hen pa PUR Y ek EY DRR hek PRX he ae 8 Intended Audience 8 Related Documentation 8 Document Conventions 8 Technical Support
20. Figures Figure 1 MCB194A FCAT Card eee ee hs 13 Figure 2 Place the Bracket on the Card Example 16 Figure 3 Support Download Assistant 35 Figure 4 Mechanical Drawing of the Dual port MCB194A FCAT Adapter Card 40 Figure 5 Dual port Bracket 1444 b te Dene hh b bene ne een en 41 Figure 6 Single port Bracket 41 Figure 7 Connector and Cage Views 42 Figure 8 PCIe Connector Pinout 45 Figure 9 Compatible Connector Plug and Pinout 46 Figure 10 Board Labels ccc oiii ie est Uu ett dto A ad ER RENE Peta au AH Ft e eon 47 6 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Revision History This document was printed on August 19 2014 Table 1 Revision History Table Date Rev Comments Changes August 2014 1 3 Updated power numbers See Figure 8 MCB194A FCAT Spec ifications Table on page 38 June 2013 1 2 Added Section 3 4 Bracket Installation on page 15 December 2013 1 1 Made restructuring changes to the UM Added Chapter 4 Driver Installation on page 19 Added Chapter 5 Firmware on page 34 Added Chapter 6
21. HEHEHE HEHE HEE 111821118211 11 Preparing mlnxofed docs Preparing mpitests mvapich2 1 9 Preparing mpitests openmpi 1 6 5 Preparing mpitests openmpi 1 7 4 Device 0a 00 0 0a 00 0 InfiniBand Mellanox Technologies Device 1011 Link Width is not 8x PCI Link Speed 2 5Gb s Installation finished successfully The firmware version on dev mst mt4113 pciconf0 10 10 1000 is up to date Note To force firmware update use force fw update flag 28 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 er In case your machine has the latest firmware no firmware update will occur and the 4 installation script will print at the end of installation a message similar to the following a The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up LORdater Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci cr0 2 11 500 is up Cate Note To force firmware update use force fw update flag In case your machine has an unsupported network adapter device no firmware update d will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file Step 4 In case the installation script performed firmware updates to your network adapter hardware it will ask you
22. HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE knem mlnx THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Installing knem mlnx kmp default 1 1 90mlnx2 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE knem mlnx kmp default THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHE Installing knem mlnx kmp trace 1 1 90mlnx2 RPM Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE knem mlnx kmp trace THHHRHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installing knem mlnx kmp xen 1 1 90mlnx2 RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE knem mlnx kmp xen THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Installing ummunotify mlnx RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE ummunotify mlnx THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Installing ummunotify mlnx kmp default 1 0 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE ummunotify mlnx kmp default IHHHHHHHHHHHHHHHHHHHHHHHHHHHHBHHBHHHHBHBHRBHRHRHHRHHHHHHE Mellanox Technologies 23 Preparing mpi selector Installing user level RPMs Preparing ofed scripts Preparing libibverbs Preparing libibverbs Preparing libibverbs devel Preparing libibverbs devel Preparing libibverbs devel static Preparing libibverbs devel static Preparing libibverbs utils Preparing libmlx4 Preparing libmlx4 Preparing libmlx4 devel Preparing libm
23. HHHHHHHHHHHHHHHRHHHHRHHE dapl devel THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE dapl devel static THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHRHHE Preparing THHHRHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE dapl devel static THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE dapl utils THHHHHH HHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHRHHHHHHHHHHE Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE perftest THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE mstflint THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE mft THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Preparing THHHRHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE srptools THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE srpd Osei iso Seca on doit Sion Gs Preparing THHHRHHH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE rds tools THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHRHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE rds devel THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHRHHHHHHE ibutils2 THHHRHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHRBHHHHHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
24. HHHHHHHHHHHHHHRHHE ibutils THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE cc mgr THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE dump pr THHHHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHE ar mgr THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE ibdump THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHRHHHHRHHE Preparing THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE infiniband diags compat THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHRHHE Preparing THHHHHHUHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE qperf THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Preparing THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHRHHHHHHE mxm THHHHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHRHHE Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHE Mellanox Technologies 27 Rev 1 3 Driver Installation openmpi EEEE HEHE HEHEHEHE HEHEHE HEETE ETE E HEEE HHHHHHHHHHHHHHHHHHHHHHHH Preparing EEEE HEHE HEHEHEHE HE HEE HEETE EEE HERR openmpi EEEE HEHEHEHEHE HE HE HEE HEETE EEEE EE Preparing EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EHHH HHHHHHHHHHHHHHHHHHHHHHHH bupc EEEE EEEE HEHEHEHE HE HEE HEETE ETETE HERE Preparing EEEE HEHE HEHEHEHE HEHEHE HEETE ETE EE HEEE HE
25. L ONO dixi ULXL QN9 oa223o08323259 NEER DNS uU S239 Wo 2229229885g0E9 222 ze 633698 SSR ERE SCE ZZ p _ TOP 18 35 View into Front of Cage 8 50 Wi Table 10 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN 2 Tx2n 3 TXP 2 Tx2p 4 GND GND 5 TXN 4 Tx4n 6 TXP_4 Tx4p 7 GND GND 8 ModSelL Port0 ModSelL 9 ResetL_Port0 ResetL 42 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Table 10 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 10 VeeRx 11 SCL SCL 12 SDA SDA 13 GND GND 14 RXP_3 Rx3p 15 RXN_3 Rx3n 16 GND GND 17 RXP 1 Rxlp 18 RXN_1 Rxin 19 GND GND 20 GND GND 21 RXN 2 Rx2n 22 RXP 2 Rx2p 23 GND GND 24 RXN 4 Rx4n 25 RXP 4 Rx4p 26 GND GND 27 ModPrsl PortO Mod PrsL 28 IntL IntL 29 VecTx 30 Veel 31 LPMode Port LPMode 32 GND GND 33 TXP 3 Tx3p 34 TXN 3 Tx3n 35 GND GND 36 TXP 1 Txlp 37 TXN_1 Txin 38 GND GND Mellanox Technologies 43 Rev 1 3 A 2 PCI Express x16 Connector Pinout The adapter cards use a standard PCI Express x16 edge connector and the PCI Express x16 stan dard pinout according to the PCI Express 3 0 s
26. Rev 1 3 Figure 3 Support Download Assistant Mellanox TECHNOLOGIES Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR MT 1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information an item Adapter Cards S revious column Switches Gateways 3 If a newer firmware version exists for your adapter card update the firmware as follows Download the firmware image zip file from the Download Center see Step 2a above Unzip the firmware image c Burm the firmware image Please note that the following are examples Enter gt flint d dev mst mt4099 pci cr0 i binary image burn Reboot the computer Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4099 pci cr0 q Image type ConnectX FW Version 2 9 4100 Device ID 4099 Mellanox Technologies 35 Rev 1 3 Troubleshooting 6 6 1 36 Mellanox Technologies Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS
27. Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Ensure that adapter driver s is loaded Try another port on the switch Link indicator light is Make sure the cable is securely attached off Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but Check that the latest driver is loaded with no Check that both the adapter and its link are set to the same speed communication and duplex settings established Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed_info head 1 Information ifconfig a ethtool lt interface gt ethtool i interface of Mellanox port num ibdev2netdev Card Detection Ispci grep i Mellanox Download and install MFT http www mellanox com content pages php pg management tools amp menu section 34 Re
28. ake sure that the latches engage p Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack should be used 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated a Pullon the latch release mechanism thereby unlatching the connector and pull the connector out of the cage b Do not apply torque to the connector when removing it from the adapter card c Remove any cable supports that were used to support the cable s weight Cables especially long copper cables can weigh a substantial amount Make sure that LA the weight of the cable is supported on its own and is not hanging from the adapter card 3 7 Identify the Card in Your System 3 7 1 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt lspci grep i Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27600 Connect IB 18 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 4 Driver Installation 4 1 Linux For Linux download and ins
29. all screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal procedure above Figure 2 Place the Bracket on the Card Example 16 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 3 5 Card Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot A lesser width adapter can be seated into a greater width slot x4 in a x8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter n connector edge with the PCI Express connector slot 4 Applying even pressure at both corners of the card insert the adapter card into the slot until it is firmly seated When the adapter is properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis Do not use excessive force when seating the card as this may damage the system or the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to Mellanox Products Appro
30. binary for your adapter card The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available Note The following steps are an example and do not correspond to this device 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Firmware Tools Make sure to download the package corresponding to your computer s operating system Enter mst start Get the Mellanox mst device name using the command mst status Get the PSID firmware identification and programmed firmware version using the command gt flint d dev mst mt4099 pci cr0 q Image type ConnectX FW Version 2 9 4000 Device ID 4099 Chip Revision 0 Description Node Porti Port2 Sys image GUIDs 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT_1020110019 VSD PSID MT_1020110019 2 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 3 b Enter your card PSID to display the latest firmware 34 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual
31. configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 dev mst mt25418 pci cr0 gt Bei chrace ALU bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is A0 dev mst mt25418 pci msix0 POE chimie suele bus dev fn 02 00 0 bar 0xdeefe000 Size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix pci cr0 In the example listed above this will be dev mst mt25418 pci er0 does not apply Step 3 Burn firmware 1 Burning a firmware binary image using mst flint that is already installed on your machine Please refer to MSTFLINT README txt under docs 2 Burning a firmware image from a mlx file using the mixburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408 fw 25408 ryyel mlx ls mnt firmware fw 4113 10 10 1000 BUILD ID fw ConnectIB mlx MCB191A FCA Al ini MCB193A FBA Al ini MCB194A FCA Al ini syndrome list log exp rom fw strings db MCB192A FCA Al ini MCB193A FCA Al ini mlnx ofed latest txt reg l vrt 212 tmp mlxburn dev dev mst mt4113 pciconf0 fw mnt firmware fw 4113 10 10 1000 fw ConnectIB mlx I Querying device I Using auto detected configuration file mnt firm
32. e ques 36 06 2 Ense EM Rs ee Busen ERE TUE cessas 37 Chapter 7 Specifications secure lose 38 7 1 MCB193A FCAT Specifications 38 7 2 MCB194A FCAT Specifications 38 7 3 LED Operation 4 ees sanat Bee Antha tka de Barata bo 39 7 4 Board Mechanical Drawing and Dimensions 40 7 5 Bracket Mechanical Drawings 41 4 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 List of Tables Table 1 Revision History Table 7 Table2 Documents List aar ar 28 san A a awe ee yee 8 Table 3 Single and Dual port Connect IB Adapter Cards 10 Table 4 Features and Benefits 11 Table 5 Software and Hardware Requirements 19 Table 6 minxofedinstall Return Codes 22 Table 7 MCB193A FCAT Specifications Table 38 Table 8 MCB194A FCAT Specifications Table 38 Table 9 EED Assienmient A Meet neee e UNO 39 Table 10 I2C compatible Connector Pinout 46 Mellanox Technologies 5 Rev 1 3 List of
33. fer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d mst device q ibstat Ports Information bv devinks Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Los File var log messages 8 dmesg gt system logF Mellanox Technologies 37 Rev 1 3 Specifications 7 Specifications 7 1 MCB193A FCAT Specifications Table 7 MCB193A FCAT Specifications Table Size 2 71in x5 6 in 68 9mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Protocol Support Data Rate Up to 56Gb s FDR InfiniBand PCI Express Gen3 SERDES 8 0GT s 16 lanes 2 0 and 1 1 compatible Voltage 12V 3 3 V Typ Power Passive Cables 11 67W Power and Max Power Passive Cables 13 25W Environmental Temperature 0 C to 55 C Air Flow 300LFM Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 Regulator Ed IEC 60068 2 64 29 32 RoHS RoHS R6 a Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 2 MCB194A FCAT Specifications Table 8 MCB194
34. ifferent LED functions 2 4 12C compatible Interface A three pin header on the adapter cards is provided as the I C compatible interface See Figure 4 Mechanical Drawing of the Dual port MCB194A FCAT Adapter Card on page 40 for the location on the board 14 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 3 Hardware Installation 3 1 System Requirements 3 1 1 Hardware A system with a PCI Express x16 slot is required for installing the card 3 1 2 Operating Systems Distributions Please refer to Section 1 3 Operating Systems and Distributions on page 12 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX OFED for Linux See Chapter 4 Driver Installation 3 2 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and wrists Make sure to use only insulated tools Verify that the system is powered off and is unplugged ecce oS It is strongly recommended to use an ESD strap or other antistatic devices 3 3 Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn
35. ing device E Can t auto detect fw configuration file 4 1 5 Post installation Notes Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file 4 1 6 Updating Firmware After Installation In case you ran the mInxofedinstall script with the without fw update option and now you wish to manually update firmware on your adapter card s you need to perform the following steps d If you need to burn an Expansion ROM image please refer to OFED User Manual See Table 2 Documents List on page 8 The following steps are also appropriate in case you wish to burn newer firmware that gt you have downloaded from Mellanox Technologies Web site gt Support gt Firmware a Download Step 1 Start mst host1f mst start Step 2 Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine host1 mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre I2C module is not loaded 32 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 MST devices dev mst mt25418 pciconf0 PCI
36. locales y nacionales 54 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 55
37. lx4 devel Preparing libmlx5 Preparing libmlx5 Preparing libmlx5 devel Preparing libmlx5 devel Preparing libcxgb3 Preparing libcxgb3 Preparing libcxgb3 devel Preparing libcxgb3 devel Preparing libcxgb4 Preparing libcxgb4 Preparing Rev 1 3 Driver Installation Installing mpi selector RPM THEHHHHHHHHBHRHHRHHHRHHHHHHHHHHHHHHHHBHHRHHHHHHHHRHER EEEE HEHEHEHEHE HE HE HEE HEETE ETE EE 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HEHEHE EE EEEE HEHE HEHEHEHE HEHEHE 11218211 1218111 EEEE HEHE HEHEHEHE HEHEHE 118218112111 EEEE HEHEHEHEHE HE HE HEE 112182111211 11 EEEE HEHE HEHEHEHE HEHEHE 121821112111 EEEE EEEE HEHEHEHE HE HEE EE EEHEHE HEHEHEHEHE HEHEHE 118182111211 EEEE HEHE HEHEHEHE HEHE E EE EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE HEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETETE H EEEH HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE HEETE HEEE EE EEEE HEHE HEHEHEHE HEHEHE A 272 272 272 272 272 272 272 EEEE HEHE HEHEHEHE HE HEE HEETE EEE HEHEHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE EEEE HEHEHEHEHE HEE HEETE ETE TE E HE EEEH HHHHEHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HEHEHE 112182111211 11 EEEE HEHE HEHEHEHE HEHEHE 118218112111 SEE EE AEE AE AE HERR EEEE HEHE HEHEHEHE HEHEHE a AAH EEEE HEHE HEHEHEHE HE HEE EE EEEE EEEE HEHEHEHE HE HEE HEETE HEEE HEHEHHE HHHHHHHHHHHHHHHHHHHHHHHH EEEE HEHE HEHEHEHE HE HEE HEETE ETE EE EEEE HEHEHEHEHE HE HE HEE a 11 EEEE HEHE HEHEHEHE
38. n copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel support sh script located under the docs directory Usage mlnx add kernel support sh m mlnx ofed path to MLNX OFED gt directory gt make iso make tgz gt gt make iso Create MLNX OFED ISO image gt make tgz Create MLNX OFED tarball Default t tmpdir local work dir gt kmp k kernel kernel version s kernel sources path to the kernel sources v verbose n name y ves Example The following command will create a MLNX OFED LINUX ISO image for RedHat 6 3 under the tmp directory MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 mlnx add kernel support sh m lt path gt MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 make tgz Note This program will create MLNX OFED LINUX TGZ for rhel6 3 under tmp directory 1 The firmware will not be updated if you run the install script with the without fw update option 20 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y See log file tmp mlnx ofed iso 1380 10g Building OFED RPM
39. necting gt 5 Equipment Installation gt 6 Equipment Disposal gt 7 Local and National Electrical Codes gt 48 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 A1 1996 A2 200 Mellanox Technologies 49 Rev 1 3 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c ble
40. niband ulp iser ib iser ko 30 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 e elPolB lib modules uname r updates kernel drivers net eipoib eth ipoib ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko lib modules uname r updates kernel net rds rds tcp ko Kernel s modules location may vary depending on the kernel s configuration For example lib modules uname r extra kernel drivers net ethernet mellanox mlx4 mlx4 core The package kernel ib devel include files are placed under usr src ofa kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel Theraw package un backported source files are placed under usr src ofa kernel lt ver gt e The script openibd is installed under etc init d This script can be used to load and unload the software stack The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED instal lation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves
41. pecification 44 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Figure 8 PCle Connector Pinout Mellanox Technologies 45 Rev 1 3 A 3 12C compatible Connector Pinout Figure 9 Compatible Connector Plug and Pinout Table 11 12C compatible Connector Connector Pin Number Signal Name 1 SDA O NO St 2 SCL 3 GND 4 NC 5 NC 46 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Appendix B Finding the GUID and Serial Number on the Adapter Card Each Mellanox adapter card has a label on the print side that shows the card serial number and the card GUID for the InfiniBand protocol Figure 10 Board Labels Model No CB194A Connect IB FDR InfiniBand Made in Israel li PIN MCB194A FCAT TT Pr me SIN MT0806X01504 EV um GUID 00 02 C9 03 00 27 05 10 1 Port 1 uses the GUID described on the label To obtain the GUID for Port 2 add 1 to d that of Port 1 Mellanox Technologies 47 Rev 1 3 Appendix C Safety Warnings Below is a list of safety warnings in English For safety warnings in other languages please refer to the appendices in this user manual 1 Installation Instructions EN 2 Over temperature 3 During Lightning Electrical Hazard gt 4 Copper Cable Connecting Discon
42. rforming and most scalable interconnect solution for PCI Express Gen2 and Gen3 servers used in High Performance Computing Enterprise Data Centers and Storage environments Parallel processing transactional services clustered data bases and high performance storage applications will achieve significant performance improvements resulting in reduced completion time and lower cost per operation This chapter covers the following topics 1 1 Product Overview The following tables provide the ordering part number port speed number of ports and PCI Express speed Each adapter comes with two bracket heights short and tall Table 3 Single and Dual port Connect IB Adapter Cards Ordering Part Number OPN Data Transmission Rate Network Connector Types PCI Express PCIe SerDes Speed RoHS Adapter IC Part Number Device ID decimal MCB193A FCAT single port card MCB194A FCAT dual port card InfiniBand FDR 56 Gb s single or dual port QSFP PCIe 3 0 x8 8GT s R6 MT27608A0 FCCR FI 4114 for Physical Function 4113 for Virtual Function 10 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 1 2 Features and Benefits Table 4 Features and Benefits Features InfiniBand Architecture Specification v1 2 1 com pliant VPI enabled adapters facilitate any standard networking clustering storage and management protocol to seamlessly operate
43. rmat MLNX_OFED_LINUX lt ver gt lt OS label gt lt CPU arch gt iso You can download it from http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers Step 3 Use the md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result to the value provided on the download page host1 md5sumMLNX OFED LINUX ver OS label gt iso 4 1 3 Installing Mellanox OFED Mellanox OFED includes an installation script called minxofedinsta11 which performs the fol lowing Mellanox Technologies 19 Rev 1 3 Driver Installation Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX OFED LINUX binary RPMs if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automat ically upgrades the firmware 4 1 3 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Pre existing configuration files will be saved with the extension conf rpmsave P Ifyou need to install Mellanox OFED on an entire homogeneous cluster a common strategy is to mount the ISO image on one of the cluster nodes and the
44. rnel THHHHHHHHHHHHHHHHHHHHRBHRRHHHHHHHHRHHHHHHRHHRHHRHUHRHHE Installing mlnx ofa kernel kmp default 2 1 RPM Preparing THHHHHHHHHHHHHHHHHHHHHHRHHHUHHHHHRHHHRHHHRHHRHHRHUHRHNE mlnx ofa kernel kmp default HHHH HHH HHH HHH HHHH HHH HEHE EHEHE E EE E HEEE HEHHEHE Installing mlnx ofa_kernel kmp trace 2 1 RPM Preparing THHHHHHHHHHHHHHHHHHHHBHRHHHHHHHHHRHHHRHHHHRHHRHUHRHHE mlnx ofa kernel kmp trace HHP HHHH HEHEHE HEHEHEHE HA HAHA TE ETE HE HEEE EH ER A Installing mlnx ofa_kernel kmp xen 2 1 RPM Preparing RFA RERO RAP RHR HEHEHE ER EER ER ER A EEEE HE mlnx ofa kernel kmp xen THHHHHHHHHHHHHHHHHHUHRBHRHHHHHHHHHRHHHRHHHHRRHRURRHHE Installing mlnx ofa kernel devel RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE mlnx ofa kernel devel THHHRHHHUHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE Installing kernel mft mlnx kmp default any RPM Preparing THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHRHHHHHHHHE kernel mft mlnx kmp default JHHHHHHHHHHHHHHHHHBHHHHBHHHHBHHHHHHHHHHHHHHHHHHHHHE Installing kernel mft mlnx kmp trace any RPM Preparing THHHHHHHHHHHHHHHHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE kernel mft mlnx kmp trace IHHHHHHHHHHHHBHHHHBHRHHBHHHHBHHHHHHHHHHHHHHHHHHHHHE Installing kernel mft mlnx kmp xen any RPM Preparing THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE kernel mft mlnx kmp xen THHHRHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHRHHHHHHE Installing knem mlnx RPM Preparing THHHRHHHHHH
45. ry sizes MB and MBytes are used in this document to mean size in Mega Bytes The use of Mb or Mbits small b indicates size in mega bits IB is used in this document to mean InfiniBand In this document PCIe is used to mean PCI Express Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller 8 Mellanox Technologies J Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the data base from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Rev 1 3 Introduction 1 Introduction This is the User Guide for Mellanox Technologies Connect IB 56Gb S InfiniBand adapter cards Connect IB provide the highest pe
46. s Please wait Removing OFED RPMs Created tmp MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 tgz Install newly created MLNX OFED package cd tmp tar xzf MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 tgz MLNX OFED LINUX 2 0 3 0 1 rhel6 3 x86 64 mlnxofedinstall 4 1 3 2 Installation Script The usage of the installation script is described below You will use it during the installation pro cedure described in Section 4 1 3 4 Installation Procedure on page 22 Usage mnt mlnxofedinstall OPTIONS Options c config packages config file Example of the configuration file can be found under docs n net network config file Example of the network configuration file can be found under docs k kernel version kernel version Use provided kernel version instead of uname r p print available Print availablepackages for the current plat form and create a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update fw update only Update firmware Skip driver installation force fw update Force firmware update force Force installation without querying the user all hpc basic msm Install all hpc basic or Mellanox Subnet manager packages correspondingly vma vma vpi Install packages required by VMA to support VPI vma ib In
47. s en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et AN qualifie 6 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 50 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 51 Rev 1 3 Appendix E Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlieBen
48. stall packages required by VMA to work over InfiniBand with vma Set configuration for VMA use to be used with any installation parameter guest Install packages required by guest os hypervisor Install packages required by hypervisor os v vv vvv Set verbosity level umad dev rw Grant non root users read write permission for umad devices instead of default enable affinity Run mlnx affinity script upon boot disable affinity Disable mlnx affinity script Default Mellanox Technologies 21 Rev 1 3 Driver Installation enable sriov Burn SR IOV enabled firmware add kernel support Add kernel support Run mlnx add kernel support sh skip distro check Do not check MLNX OFED vs Distro matching total vfs 0 63 Maximum number of Virtual Functions in SR IOV mode Default 16 Implies enable sriov hugepages overcommit Set 80 of MAX MEMORY as overcommit for a huge page allocation Per priority bit mask uint Default 0 q Set quiet no messages will be printed with fabric collector Install fabric collector package 4 1 3 3 mlnxofedinstall Return Codes Table 6 lists the mInxofedinstall script return codes and their meanings Table 6 minxofedinstall Return Codes Return Code Meaning 0 The Installation ended successfully 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicable to this sy
49. stem configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start the mst driver 4 1 3 4 Installation Procedure Step 1 Login to the installation machine as root Step 2 Mount the ISO image on your machine host1 mount o ro loop MLNX OFED LINUX lt ver gt lt 0S label gt lt CPU arch gt iso mnt Step 3 Run the installation script mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX root sw1014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 64 root swl014 MLNX OFED LINUX 2 0 2 0 mo0 rhel6 3 x86 64 mlnxofedinstall 22 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX Starting MLNX OFED LINUX 2 0 2 0 0 installation Installing mlnx ofa kernel RPM Preparing THHHHHHHHHHHHHHHHHHHHRBHRRHHHHHHHHHHHRHHHHRHHRURHRHE mlnx ofa ke
50. tablished e When logical link is established yellow LED turns off Green Aconstant green indicates a valid logical data activity link without data transfer Logical data activity link A blinking green indicates a valid logical link with data transfer _ If only the yellow LED is lit and the green LED is off then the logical link has not been established Mellanox Technologies 39 Rev 1 3 Specifications 7 4 Board Mechanical Drawing and Dimensions d All dimensions are in millimeters All the mechanical tolerances are 0 1mm Figure 4 Mechanical Drawing of the Dual port MCB194A FCAT Adapter Card I2C compatible Flash Jumper interface HTA S LIN ET g me om A do m Mv fa El B E Bun gt el UI m 5615 er ek CRE H tio va Mad i 4 c E a i BE al n 112 75 12 93 4 129 31 E ee lt lt 40 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 7 5 Bracket Mechanical Drawings Figure 5 Dual port Bracket Port 1 Gasket LEDs Port 2 Figure 6 Single port Bracket Gasket Bicolor LED Mellanox Technologies 41 Rev 1 3 Appendix A Interface Connectors Pinout A 1 QSFP Connector Pinout Figure 7 Connector and Cage Views To P 18 35 View into Rear 7 N TSI4PON X199A 129A apondT QN9 dexi VEX
51. tall the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox web site at http www mellanox com gt Products gt Soft ware gt InfiniBand VPI Drivers gt Linux SW Drivers gt Download This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mel lanox Connect IB adapter hardware installed 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for 1GB Installation Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to Mellanox OFED Release Notes file Installer Privileges The installation requires administrator privileges on the target machine 4 1 2 Downloading Mellanox OFED Step 1 Verify that the system has a Mellanox HCA installed by ensuring that you can see Connect IB entries in the display The following example shows a system with an installed Mellanox HCA host1 lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27600 Connect 1B Step 2 Download the ISO image to your host The image s name has the fo
52. teroperable with InfiniBand switches Passive copper cable with ESD protection Powered connectors for optical and active cable support 12 Mellanox Technologies Connect IB Single and Dual QSFP Port PCI Express Gen3 x16 Adapter Card User Manual Rev 1 3 2 Interfaces Each adapter card includes the following interfaces e InfiniBand Interface PCI Express Interface LED Assignment e 2C compatible Interface Figure 1 MCB194A FCAT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables 2 1 InfiniBand Interface The network ports of the Connect IB adapter cards are compliant with the InfiniBand Architecture Specification Release 1 2 1 InfiniBand traffic is transmitted through the cards QSFP connectors 2 2 PCI Express Interface The Connect IB adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through x16 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists the PCIe interface features PCI Express 2 0 or 3 0 compliant e Auto negotiates tox16 x8 x4 or x1 Mellanox Technologies 13 Rev 1 3 Interfaces Support for MSI X mechanisms 2 3 LED Assignment There are two I O LEDs per port in dual port designs and one bicolor LED located on the I O panel See Section 7 3 LED Operation on page 39 for d
53. ved Cable Lists at www mellanox com related docs user manuals Mellanox approved cables pdf 3 6 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated a Support the weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack b Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card c Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card d Make sure that the connector locks in place Mellanox Technologies 17 Rev 1 3 Hardware Installation 3 After inserting a cable into a port the Yellow LED indicator will light when the physical con nection 1s established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the Green LED will come on When data is being transferred the green led will blink d When installing cables m
54. ware fw 4113 10 10 1000 MCB192A FCA A1 ini PSID MT 1240110019 I Generating image Current FW version on flash 10 10 1000 New FW version 10 10 1000 Note The new FW version is not newer than the current FW version on flash Do you want to continue y n n y Burning FS3 FW image without signatures OK Restoring signature OK I Image burn completed successfully Mellanox Technologies 33 Rev 1 3 Firmware 5 5 1 Firmware Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com gt Support gt Download Firmware Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mel lanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm ware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Support gt Down load Firmware Tools A firmware binaries table lists a binary file per adapter card The file name of each such binary is composed by combining the firmware name the firmware release version and the card part num ber Please contact Mellanox System Support if you cannot find the firmware

User Manual for PCIeGen3_x16

Contents

Download Pdf Manuals

Related Search

Related Contents