Home

3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual

1. Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4 GND GND 5 TXN_4 Tx4n 6 TXP_4 Tx4p 7 GND GND 8 ModSelL_Port0 ModSelL 9 ResetL_Port0 ResetL 10 VecRx 11 SCL SCL 12 SDA SDA 13 GND GND 14 RXP_3 Rx3p 15 RXN_3 Rx3n 16 GND GND 17 RXP_1 Rxlp 18 RXN_1 RxIn 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP_2 Rx2p 23 GND GND 24 RXN_4 Rx4n 25 RXP_4 Rx4p 26 GND GND 27 ModPrsl PortO Mod PrsL 28 IntL IntL 29 VecTx 30 Vecl 31 LPMode PortO LPMode 32 GND GND 33 TXP 3 Tx3p 34 TXN 3 Tx3n 35 GND GND 36 TXP 1 37 1 1 38 GND GND The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard 54 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 pinout according to the PCI Express 3 0 specification Figure 14 PCle Connector Pinout NA 81 14 31 __ 2 N 4124 LA2 _ B m 12 PAB 84 GNO BB auci SND 88 SMDAT TDI 27 22 TASTE HAI WAKEWOBFF All 813 REFCLK H14 REFCLK B15 AlE 816 GND 15 81124 penno 17 2838 A _B19 PETp1 Rsvp Aly
2. 34 4 2 8 Uninstalling MLNX WinOF 35 4 2 8 1 Attended Uninstall 35 4 2 8 2 Unattended Uninstall 35 4 2 0 Assigning Port IP After Installation 35 4 2 10 Port Type Management on Windows 37 43 VMware Diver ee ee RE AR ex EES 39 4 3 1 Installing and Running VBI Driver on 1 5 39 4 3 2 Installing and Running the offline bundle Driver on ESXi 5 x 40 4 4 Removing the VIB offline bundle Driver 40 4 5 gt FlexBoot I sw aa e Hin Der 40 Chapter 5 Updating Adapter Card Firmware 41 Chapter 6 43 6 1 General ues m ue Rae Se tm rere me dee re ses 43 6 2 io ae te epe Re VR daca lents 44 6 3 sil eec sett sei eet Eo Eo tus 45 Chapter 7 49 7 1 MCX313A BCBT Specifications 46 7 2 MCX314A BCBT Specifications 47 7 3 Adapter LED Operation 5 een ne ts een EL Ed E IP eut 48 7 4 Board Mechanical Drawing and Dimensions
3. 21 cup PERp1 L_A21 25 PERp2 325 8254 pene 827 2 Ea Petra 422 GND PERpa A23 lt 831 PRSNT Aai BA PETp4 Rsvp PAR 835 1 PERp4 335 2 PERn4 _827 PETp5 GND LA3Z BBA GND PEApS Azo _ B41 Ati 843 ann PERS LA43 _ 845 PETp7 46 FAME EL ra EN a 849 GND GND 12C compatible Connector Pinout Figure 15 Compatible Connector Plug and Pinout Mellanox Technologies 55 Rev 2 0 Connector Pin Number Signal Name 1 SPSDA 2 SPSCL 3 GND 4 NC 5 56 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Appendix B Finding the MAC and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number and the card MAC for the Ethernet protocol Figure 16 Card Product Label Example Model No CX354A ConnectX 3 FDR InfiniBand 40GigE P N MCX354A FCAT Rev A1 NIMM IN IN S N MT1106X01504 GUID 00 02 C9 03 00 27 05 1C 00 02 C9 27 05 1 Made in Israel Sm AN Mellanox Technologies 57 Rev 2 0 Appendix Safety Warnings 1 Installation Instructions LA Read all installation instructions before connecting the equipment to
4. Section 1 3 Operating Systems Distributions on page 11 Section 1 4 Connectivity on page 12 1 1 Product Overview The following tables provide the ordering part number port speed number of ports and PCI Express speed Each adapter comes with two bracket heights short and tall Table 3 Single and Dual port 40 Gigabit Ethernet Adapter Cards Ordering Part Number OPN SEDET 40GigE Data Transmission Rate MCX314A BCBT dual port QSFP ape o Aponte MCX313A BCBT single port QSFP PCI Express SERDES Speed PCIe 3 0 x8 8GT s RoHS R6 Adapter IC Part Number MT27518A1 FCCR BE 4099 for Physical Function Device AD 4100 for Virtual Function 10 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 1 2 1 3 Mellanox Technologies 11 Features and Benefits Table 4 Features PCI Express PCIe 10 40 56 Gigabit Ethernet Memory RDMA over Converged Ethernet RoCE CPU offload GPUDirect RDMA Sockets Acceleration Quality of Service QoS Hardware based I O virtualization SR IOV Storage Acceleration Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ba 40 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover P
5. 49 7 5 Bracket Mechanical Drawing 51 7 6 Regulatory Statements 52 Appendix A Interface Connectors Pinout 53 AT QSFP Connector Pinout 53 A 2 PCI Express x8 Connector Pinout 54 I2C compatible Connector Pinout 55 Appendix B Finding the MAC and Serial Number on the Adapter Card 57 Appendix Safety Warnings ssssesssssssssssse 99 Appendix Avertissements de s curit d installation Warnings in French 60 Appendix E Sicherheitshinweise Warnings in German 62 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 64 4 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 List of Tables Table 1 Revision History Table 7 Tabl 2 Documents List 5554 era ea d s OPE 8 Table3 Single and Dual port 40 Gigabit Ethernet Adapter Cards 10 Table 4 Features 24 2 De She 11 Table 5 Software and Hardware Requirements 22 Table 6 Software and Hardware Requirements 24 Table 7
6. Installing the incorrect exe file is prohibited If you do so an error message will be dis played 3 For example if you try to install a 64 bit exe on a 32 bit machine the wizard will dis play the following or a similar error message Windows Installer x A This installation package is not supported by this processor type Contact your product vendor OK 4 2 3 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start gt Run and enter CMD 24 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 2 Enter the following command MLNX VPI WinOF version All OS lt arch gt exe a Step3 Click Next to create a server image Welcome to the InstallShield Wizard for MLNX_VPI The InstallShield R Wizard will install MLNX_VPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Step4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location for the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX at the specified network location or click Cancel to exit the wizard Network location
7. Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux Operating Systems RedHat EL5 8 RedHat EL5 9 RedHat EL6 2 RedHat EL6 3 OEL62 2 6 32 279 19 1 e OEL6 3 2 6 32 279 19 1 SLESII SPI SLES11 SP2 Software Dependencies To install the driver software kernel sources must be installed on the machine MLNX_EN driver cannot coexist with OFED software on the same machine Hence when installing MLNX_EN all OFED packages should be removed done by the mInx_en install script 4 1 2 Installing the Driver Step 1 Download Driver Package Please download the current driver package from http www mellanox com gt Products gt Software gt Ethernet Driver gt Linux Driver gt Download Step 2 Install Driver Run the following commands to install the driver gt tar xzvf mlnx en 1 5 10 tgz file gt GC 5 110 f install sh The package consists of several source RPMs The install script rebuilds the source RPMs and then installs the created binary RPMs The created kernel module binaries are placed under lib 22 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 modules lt kernel ver gt updates kernel drivers net mlx4 mlnx en installer supports 2 modes of installation The install scripts selects the mode of driver installation depending of the ru
8. a InstallShield Mellanox Technologies 25 Rev 2 0 Driver Installation Step5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location for the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of MLNX_VPI at the specified network location or click Cancel to exit the wizard Network location ee _ __________ _ _ _ _ Change InstallShield Step 6 To complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX_ PI Click Finish to exit the wizard 4 2 4 Installing MLNX WinOF This section provides instructions for two types of installation procedures Attended Installation An installation procedure that requires frequent user intervention e Unattended Installation 26 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 An automated installation procedure that requires user intervention Both Attended and Unattended installations require administrator privileges 4 2 4 1 Attended Installation The following is an example of aMLNX_WinOF_win8 x64 installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF To configure your setup to contai
9. 64 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 65
10. Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die LA Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden This equipment should be installed in compliance with local and national elec LA trical codes 62 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 63 Rev 2 0 Appendix F Advertencias de seguridad para la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las
11. Distributions on page 11 3 1 3 Software Stacks Mellanox OpenFabric software package MLNX_EN for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation 3 2 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components 1 Remove any metallic objects from your hands and wrists 2 Make sure to use only insulated tools 3 Verify that the system is powered off and is unplugged 4 Itis strongly recommended to use an ESD strap or other antistatic devices 3 3 Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord Pp Ni Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged 3 4 Bracket Installation Instructions The card is usually shipped with a tall bracket installed If this form factor is suitable for your requirements you can skip the remainder of this section and move to Section 3 5 Card Installa tion Instructions on page 18 If you need to replace it with the short bracket that is included in the shipp
12. Installation i xou 24 2 2 astra er una SEES 19 3 7 Identify the Card in Your System 20 335 T C On Windows sec Se 20 3 7 2 On Linux RR NR AP ate ea A 21 Chapter 4 Driver Installation ses 22 4 1 Linux Driver mia Sonate rhone dunes ae eats 22 4 1 1 Hardware and Software Requirements 22 Mellanox Technologies 3 Rev 2 0 4 1 2 Installing the Driver 2 22 41 3 Eo ding th Driver 5 4 Biken RER eh 23 4 1 4 Unloading the Driver 23 4 1 5 Uninstalling the Driver 23 4 2 Windows Driver 5 nn en aa ta 23 4 2 1 Hardware and Software Requirements 24 4 2 2 Downloading MLNX WinOF 24 4 2 3 Extracting Files Without Running Installation 24 4 2 4 Installing MLNX_WinOF 26 4 2 4 1 Attended Installation 27 4 2 4 2 Unattended Installation 32 4 2 5 Upgrading MLNX_WinOF 33 4 2 6 Installation Results 33 4 2 7 OpenSM Activation
13. MCX313A BCBT Specifications Table 46 Table 8 MCX314A BCBT Specifications Table 47 Table 9 Physical and Logical Link Indication 48 Mellanox Technologies 5 Rev 2 0 List of Figures Figure 1 MCX314A BCBT Card 13 Figure 2 I2C compatible Connector 14 Figure 3 Bracket SCreWS ooi ue eee ie ea leid lan 16 Figure 4 Gasket Location on Adapter Card 17 Figure 5 Gasket Installation 17 Figure 6 Placing the Bracket on the Card 18 Figure 7 a ana a eed Ow qued ea anne 20 Figure 8 Support Download Assistant 42 Figure 9 Mechanical Drawing of the Single port MCX313A BCBT Adapter Card 49 Figure 10 Mechanical Drawing of the Dual port MCX314A BCBT Adapter Card 50 Figure 11 Dual port Bracket viii ea te ta 51 Figure 12 Single port Bracket 51 Figure 13 Connector and Cage Views 53 Figure 14 PCIe Connector Pinout 55 Figure 15 Co
14. instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la LA m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como minimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga electrica No utilizar el equipo ni conectar o desconectar cables durante periodos de actividad de La rayos 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec LA tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos
15. lt Back Cancel 4 2 4 2 Unattended Installation The following is an example of aMLNX_WinOF_win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter cmd Step 2 Install the driver Run gt MLNX WinOF 4 40 0 All win8 x64 exe S v qn 32 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Step3 Optional To configure your setup to contain the logs option please run the following com mand gt MLNX WinOF 4 40 0 All win8 x64 exe S v qn v l vx LogFile Step 4 Optional If you do not want to upgrade your firmware version run the following command gt MLNX WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUPGRD 1 For further help please run gt MLNX WinOF 4 40 0 All win8 x64 exe v h 4 2 5 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version e In Windows 2012 MLNX WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade 4 2 6 Installation Results Upon installation completion you can verify
16. the latest firmware version using the PSID board ID Firmware Version E http www mellanox com supportdownloader Upgrade flint d lt mst_device gt i lt firmware_bin_file gt b Event log viewer MST device logs mst start Collect log file mst status C Users Administrator gt flint d lt mst_device gt de gt dump_configuration log C Users Administrator gt mstdump lt mst_device gt dc gt mstdump log Mellanox Technologies 45 Rev 2 0 Specifications 7 Specifications 7 4 MCX313A BCBT Specifications Table 7 MCX313A BCBT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP Ethernet 40GBASE CRA SR4 LR4 also supports the following when used with a QSA 10GBASE CR SR LR 1000BASE T SR LR Protocol Support Data Rate 1 10 40Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 54W Max Power Passive Cables 8 19W Power and Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFMP EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Regulatory Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to M
17. the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed LA ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting Some copper cables are heavy and not flexible as such they should be carefully LA attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes h This equipment should be installed in compliance with local and national electrical codes 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure 58 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 CLASS 1 LASER PRODUCT and reference to the most recent laser standards TEC 60 825 1 1993 A1 1997 A2
18. the successful addition of the network card s through the Device Manager To see the Mellanox network adapter device and the Ethernet or Mellanox Technologies 33 Rev 2 0 Driver Installation IPoIB network device depending on the used card for each port display the Device Manager and expand System devices or Network adapters File Action View Help en Br amp Mellanox ConnectX 3 Ethernet Adapter 2 Microsoft Kernel Debug Network Adapter 4 Other devices a Unknown device Ports COM amp LPT dmh Print queues gt Processors X7 Storage controllers j System devices 188 ACPI Fixed Feature Button Composite Bus Enumerator 188 Direct memory access controller High precision event timer Intel R 5520 5500 X58 Hub Control Status and RAS Registers 3423 Intel R 5520 5500 X58 Hub GPIO and Scratch Pad Registers 3422 Intel R 5520 5500 X58 Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 1 0 Hub PCI Express Root Port 3 340A Intel R 5520 5500 X58 Hub PCI Express Root Port 7 340E Ml Intel R 5520 5500 X58 I O Hub System Management Registers 342E 1i Intel R 5520 5500 X58 Hub to ESI Port 3406 Ml Intel R 82801 PCI Bridge 244E 2 Intel R ICH10 Family PCI Express Root Port 1 3440 ntel R ICH10 Family PCI Express Root Port 5 3448 1 Intel R ICH10R LPC Interface Controller 3416 Microsoft ACPI Complia
19. 2001 and EN 60825 1 1994 A1 1996 A2 20 Mellanox Technologies 59 Rev 2 0 Appendix 0 Avertissements de s curit d installation Warnings French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les cables 4 Branchement debranchement des c bles en cuivre Les cables en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L elimination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en v
20. CI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 16MB SPI Flash device M25PX16 VMN6P device by ST Microelectronics EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb Leveraging Data Center Bridging capabilities RoCE provides efficient low latency RDMA services over Layer 2 Ethernet Adapter functionality enabling reduced CPU overhead allowing more available CPU Using GPUDirect RDMA adapters can directly read and write CUDA host and device memory eliminating unnecessary system memory copies and CPU over head resulting in significant performance improvements Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10 or A0GbE The hardware based stateless offload engines in ConnectX 3 reduce the CPU overhead of IP packet transport Sockets acceleration software further increases performance for latency sensitive applica tions Support for port based Quality of Service enabling various application require ments for latency and SLA ConnectX 3 provides dedicated adapter resourcesand guaranteed isolation and protection for virtual machines within the server ConnectX 3 Pro SR IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines VM within the server VO virtualization with ConnectX 3 Pro gives data center managers be
21. Mellanox TECHNOLOGIES ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual P N MCX313A BCBT MCX314A BCBT Rev 2 0 www mellanox com Rev 1 8 THIS HARDWARE SOFTWARE TEST SUITE PRODUCT 5 AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVE
22. N IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2013 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale MLNX OS PhyX SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroXTM MetroDXTM ScalableHPC Unbreakable LinkTM are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3688 ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Table of Contents Table of Contents s 4 eR e e e eme er ee Eist of Tables 222 2 FA er nee Revision History ss a eek ee na U Y About this Manual na een nes Technical Suppott aan ch Nea RE EE 8 Firmware and Software Updates 9 Chapter 1 Introduction sa ars u ann 10 11 Product Overview t
23. P Port Adapter Card User Manual Rev 2 0 Do not use excessive force when seating the card as this may damage the system the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user_manuals Mellanox_approved_cables pdf 3 6 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated When installing cables make sure that the latches engage Ro Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 3 After inserting a cable into a port the Green LED indicator will light when the physical con nection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See Section 7 3 Adapter LED Operation on page 48 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the Yellow LED will light When data is being transferred the yellow led will blink See Section 7 3 Adapter LED Operation on page 48 5 Care should be ta
24. ant with the IEEE 802 3 Ether net standards listed in Table 4 Features on page 11 Ethernet traffic is transmitted through the cards QSFP connectors 2 2 PCI Express Interface The ConnectX 3 adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists the PCle interface features e PCIe Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 8 0GT s link rate x8 e Auto negotiates to x8 x4 x2 or x1 Mellanox Technologies 13 Rev 2 0 Interfaces Support for MSI MSI X mechanisms 2 3 I C compatible Interface A three pin header on the adapter cards is provided as the PC compatible interface See Figure 10 Mechanical Drawing of the Dual port MCX314A BCBT Adapter Card on page 50 for the location on the board Figure 2 Connector 2 4 LED Interface There are two LEDs per port For LED specifications please refer to Section 7 3 Adapter LED Operation on page 48 14 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 3 Hardware Installation 3 1 System Requirements 3 1 1 Hardware A system with a PCI Express x8 slot is required for installing the card 3 1 2 Operating Systems Distributions Please refer to Section 1 3 Operating Systems
25. card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the MFT package The package is available at http www mellanox com gt Products gt Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci 0 d Get the PSID firmware identification and programmed firmware version using the command gt ilime C ekew musiE mwEAQ9S pei E ae ENDS E ConnectX 3 FW Version 2 9 4000 Device ID 4099 Chip Revision 0 Description Node Port Pont Sys image GUIDs 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MESO 0110019 281003 MEEO 0 1L 1 2 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloade
26. d Mellanox Technologies 29 Rev 2 0 Driver Installation P This step requires rebooting your machine at the end of the installation Step 7 Select a Complete or Custom installation follow Step a and on on page 30 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete 3 All program features will be installed Requires the most disk space Custom Choose which program Features you want installed and where they will be installed Recommended for advanced users InstallShield lt Back Cancel a Select the desired feature to install e OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically e Performances tools install the performance tools that are used to measure the InfiniBand performance in user environment e Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment e SDK contains the libraries and DLLs for developing InfiniBand application over IBAL e Documentation contains the User Manual and Installation Guide ND FLTR DLLs contains the files for standalone installation of the mlx4nd provider 30 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Custom Setup Select the program features you want installed Click on a
27. e Removed Blinking indicates a problem with the physical link from Table 9 Physical and Logical Link Indication on page 48 April 2012 1 3 Updated power numbers in Table 7 MCX313A BCBT Specifi cations Table on page 46 and Table 8 MCX314A BCBT Specifications Table on page 47 January 2012 1 2 Minor edits e Updated LED functions in Section 2 3 I2C compatible Inter face on page 14 e Formatted specification tables in Appendix 7 Specifications on page 46 October 2011 1 1 Added new OPNs based on ConnectX 3 Step Al devices July 2011 1 0 Minor edits July 2011 0 10 First Release Mellanox Technologies 7 Rev 2 0 About this Manual This User Manual describes Mellanox Technologies ConnectX 3 40 Gigabit Ethernet Single and Dual QSFP port PCI Express x8 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant docu mentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet networks and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User User Manual describing the set of MFT firmware management Manual tools for a single node Document no 2204UG See http www mellanox com gt Products gt Software gt Fir
28. e at http www mellanox com Download this package Mellanox Technologies 23 Rev 2 0 Driver Installation from the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt Windows SW Driver gt Download Follow the installation instructions included in the download package also available from the download page 4 2 4 Hardware and Software Requirements Table 6 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Windows Server 2008 R2 64 bit only Installer Privileges The installation requires administrator privileges on the target machine 4 2 2 Downloading MLNX WinOF Follow these steps to download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click start gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURES On an x64 64 bit machine the output will be AMD64 Step2 Goto the MLNX WinOF for Windows Web page at http www mellanox com gt Products gt Software gt Ethernet Drivers gt Windows SW Drivers Step3 Download the exe image according to the architecture of your machine see Step 1 The name of the exe is in the following format MLNX_VPI_WinOF lt version gt _All_ lt OS gt _ lt arch gt exe
29. elect the Port Protocol tab from the Properties sheet Note The Port Protocol tab is displayed only if the NIC is a VPI IB and ETH If the NIC is either only IB or ETH the tab will not be shown General Port Protocol Driver Details Events Resources Current Setting Port2 Eth Mellanox Eb HCA Port Type Configuration HW Defaults Port 1 ETH C AUTO Port 2 C B ETH C AUTO Port Protocol Configuration This menu displays the adapter s port type and enables you to set the network protocols for the network adapter ports network protocol is determined according to the NIC s Hardware Defaults port type You can choose the protocol explicitly by selecting the port type to InfiniBand IB or Ethernet Eth enahle Aita Sensina nlease rhonse Al ITN If the v Cancel 38 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Step 3 In this step you can perform two different functions a Choose the desired port protocol for the available port s and b activate or deactivate the WSD ND and or SDP ULPs Note IB must be always the first port in Port 1 If you choose ETH as your first port in Port 1 then the second port in Port 2 can be only ETH Note WSD is not supported in Windows 7 Consequently on this OS the WSD checkbox is grayed out and cannot be selec
30. ellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user manuals Mellanox approved cables pdf a For both operational and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 46 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 72 MCX314A BCBT Specifications Table 8 MCX314A BCBT Specifications Table Size 2 71n x5 6 in 68 90mm x 142 25 mm Physical Connector QSFP Ethernet 40GBASE CRA SR4 LR4 also supports the following when used with a QSA 10GBASE CR SR LR 1000BASE T SR LR Protocol Support Mamm PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 7 69W Max Power Passive Cables 9 35W Power and Environmental Temperature Operational 0 C to 55 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 200LFMP EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Regulatory Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RoHS R6 Please refer to Mellanox Products Approved Cable Lists at Cable Support www mellanox com related docs user_manuals Mellanox_approved_cables pdf a For both operatio
31. es 43 Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Ensure that adapter driver s is loaded Try another port on the switch Make sure the cable is securely attached Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but with no communication established Check that the latest driver is loaded Check that both the adapter and its link are set to the same speed and duplex settings Rev 2 0 Troubleshooting 6 2 Linux cat etc issue uname cat proc cupinfo grep model name uniq Environment ofed_
32. et 1 Place the bracket onto the card until the screw holes line up See Figure 6 Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal proce dure above Mellanox Technologies 17 Rev 2 0 Hardware Installation Figure 6 Placing the Bracket on the Card 3 Make sure that the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 Ibs in torque on the screws 3 5 Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot A lesser width adapter can be seated into a greater width slot x4 in a x8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter connector edge with the PCI Express connector slot 4 Applying even pressure at both corners of the card insert the adapter card into the slot until it is firmly seated When the adapter is properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis 18 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSF
33. her information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 8 Mellanox Technologies 39 Rev 2 0 Driver Installation 4 3 2 4 4 4 5 Installing and Running the offline_bundle Driver on ESXi 5 x 1 Copy the offline bundle zip file to ESXi 5 0 machine and extract its contents 2 You can install the driver in one of the following ways a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4_en driver offline bundle package Run gt esxcli software vib install d lt path gt mlx4_en mlnx 1 6 1 2 offline_bundle 471530 zip c Reboot ESXi server The driver will be loaded automatically OR a Update the driver Run gt esxcli software vib update n net mlx4 en d lt path gt mlx4_en mlnx 1 6 1 2 offline bundle 471530 zip b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload_mod 1 grep mlx4_en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by MLX4_EN driver should be displayed zu In Non Multifunction Mode port 2 is ident
34. ified as a pseudo device Therefore devices are not seen by vSphere when added as uplink 3 For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 8 Removing the VIB offline bundle Driver To remove the VIB offline bundle driver package from the ES Xi server machine run gt esxcli software vib remove n net mlx4 en FlexBoot FlexBoot supports remote Boot over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot software is based on the open source 1 project see www ipxe org For more information go to http www mellanox com gt Products gt Software gt Ethernet Drivers gt Download 40 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 5 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally and the most recent firmware can be obtained from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single
35. igueur 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 60 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 61 Rev 2 0 Appendix Sicherheitshinweise Warnings in German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der 7 maximal empfohlenen Temperatur von 55 131 F betrieben werden Es ist ein Luft strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschlie en Trennen von
36. ination Folder Click Next to install to this Folder or click Change to install to a differen Install MLNX VPI to C Program _ InstallShield Step 5 The firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card in this case the firmware will not be updated 28 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 e Ifthe user has a standard card and the firmware version is older than the one specified in WinOF Installation Guide 4 40 the firmware will be updated accordingly However if the user has both OEM card and Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the HCA s firmware version Recommended Upgrading the firmware version will reboot your machine Note One or more of your HCA adapters has an old firmware version We recommend upgrading to a newer firmware version to enable improved functionality and support driver s capabilities InstallShield Step 6 Configure your system for maximum performance by checking the maximum performance box Maximum Performance Check this box to configure your system for maximum performance Y Check this box to configure your system For maximum performance Recommended Note This step requires you to reboot the machine at the end of the installation process InstallShiel
37. info head 1 Information ifconfig a ethtool lt interface gt ethtool i lt interface_of_Mellanox_port_num gt ibdev2netdev Card Detection Ispci grep i Mellanox Download and install MFT http www mellanox com content pages php pg management tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d lt mst_device gt q ibstat Ports Information Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF 44 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 6 3 Windows From the Windows desktop choose the Start menu and run msinfo32 Environment To export system information to a text file choose the Export option Information from the File menu Assign a file name and save Download and install MFT http www mellanox com content pages php pg management tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmwar ne Once installed open aCMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d lt mst_device gt q Ports Information vstat Download
38. ing box please follow the instructions in this section To replace the bracket you will need the following parts The new bracket of the proper height One new gasket Mellanox Technologies 15 Rev 2 0 Hardware Installation e The 2 screws saved from the removal of the bracket The 2 fiber washers saved from the removal of the bracket 3 4 1 Removing the Existing Bracket Figure 3 Bracket Screws Screws 1 Remove the two screws holding the bracket in place The bracket comes loose from the card LA Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers 3 4 2 Installing the New Bracket 3 4 2 1 Gasket Installation Note The following instructions relate to the installation of the gasket for adapter cards that are not shipped with a gasket already installed Mellanox Technologies now provides all ConnectX 3 adapter cards with the gasket already installed unto the bracket 1 Remove the paper to expose the adhesive on the gasket 2 Place the gasket onto the new bracket Make sure to correctly align the gasket with the hole in the bracket 3 If the old gaskets are still on the card remove them before installing the new bracket 16 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Figure 4 Gasket Location on Adapter Card Gasket Figure 5 Gasket Installation E BR A 3 4 2 2 Installing the Brack
39. ken as not to impede the air exhaust flow through the ventilation holes Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated a Mellanox Technologies 19 Rev 2 0 Hardware Installation 3 7 Identify the Card in Your System 3 7 1 On Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 adapter card 3 Right click the mouse on your adapter s row and select properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or 5 Hardware Ids Windows 2008 R2 from the Properties pull down menu Figure 7 PCI Device General Port Protocol Driver Details Events Resources l E A Mellanox Connect lt 3 4088 Network Adapter E Property Hardware Ids v Value PCINVEN 1583 amp DEV 100355UBSYS O05015B3 amp REV 00 PCIVEN 15B3 amp DEV 100355UBSYS 00501583 PCINVEN 15B3 amp DEV 1003 amp CC 028000 PCIVEN 15B38DEV 1003 amp CC 0280 6 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above no
40. m ware Tools MLNX_EN for Linux README Driver Kit This document provides information on the MLNX_EN Linux for Mellanox Adapter Cards with 1040GigE driver and instructions for installing the driver on Mellanox Con Support nectX adapter cards supporting 10 40Gb s Ethernet Document no 2950 WinOF VPI for Windows User Manual describing WinOF features performance InfiniBand User Manual diagnostic tools content and configuration See http www mella Document no 3280 nox com gt Products gt Software gt Windows SW Drivers gt Mellanox OFED for Windows WinOF Mellanox MLX4_EN Driver for VMware User Manual describing MLX4_EN driver for WMware features README performance diagnostic tools content and configuration See Document no 3527 http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers IEEE Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega Bytes The use of Mb or Mbits small b indicates size in mega bits In this document PCIe is used to mean PCI Express Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us
41. mpatible Connector Plug and Pinout 55 Figure 16 Card Product Label Example 57 6 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Revision History This document was printed on June 27 2013 Table 1 Revision History Table Date Rev Comments Changes June 2013 2 0 Updated User Manual structure February 2013 1 8 e Added Section 4 3 VMware Driver on page 39 e Updated Figure 7 PCI Device on page 20 and Figure 8 Device Manager on page 58 Updated all Web links in the User Manua Figure 10 Mechanical Drawing of the Dual port MCX314A BCBT Adapter Card on page 50 November 2012 1 77 Updated Web links in the following locations Section 6 5 1 Software Requirements on page 59 Table 7 MCX313A BCBT Specifications Table on page 46 e Table 8 MCX314A BCBT Specifications Table on page 47 October 2012 1 6 Fixed Table 5 Jumper Configuration on page 17 e Updated figures in Section 4 2 Windows Driver on page 23 by providing ConnectX 3 examples August 2012 1 5 e Added operational and non operational temperature and humid ity level to the following tables Table 7 MCX313A BCBT Specifications Table on page 46 Table 8 MCX314A BCBT Specifications Table on page 47 May 2012 1 4
42. n icon in the list below to change how a Feature is installed Feature Description Performace Tools Subnet manager 5M Analyze Tools SDK Documentation ND FLTR DLLs This feature requires 1164KB on your hard drive Install to C Program Files Mellanox MLNX_VPI IB Tools InstallShield b Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Mellanox Technologies 31 Rev 2 0 Driver Installation Step8 Click Finish to complete the installation InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin 9 109 Cl Show release notes Cancel InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard You chose to run performance tuning The log file can be found at windows System321LogFiles PerformanceTunin 9 69 Firmware upgrade Failed with error 1003 Error Description We could not find a new version for the chosen device Please refer to the UM for Cl Show release notes
43. n the logs option please run the following command after opening aCMD console MLNX_VPI_Win0F 4_40_0_A11 win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run the following command MLNX WinOF 4 40 0 All win8 x64 exe v MT_SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Step2 Click Next in the Welcome screen qui kr Welcome to the InstallShield Wizard for MLNX_YPI The InstallShield R Wizard will install MLNX_YPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Mellanox Technologies 27 Rev 2 0 Driver Installation Step 3 Read then accept the license agreement and click Next License Agreement Please read the Following license agreement carefully Copyright 2005 2013 Mellanox Technologies All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer 2 NTadintribhntrienn in binare Farm mint mm Anun tha share 8 I accept the terms in the license agreement O I do not accept the terms in the license agreement InstallShield Step 4 Select the target folder for the installation Dest
44. nal and non operational states b Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet Mellanox Technologies 47 Rev 2 0 Specifications 7 3 Adapter LED Operation There are two I O LEDs per port See Table 9 for different LED functions Table 9 Physical and Logical Link Indication LED Function Green physical link Constant on indicates a good physical link If neither LED is lit then the physical link has not been established Yellow logical data blinking yellow indicates activity data transfer activity link e Stays off when there is no activity The short bracket has the same port and LED footprints as the tall bracket 48 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 7 4 Board Mechanical Drawing and Dimensions All dimensions are in millimeters All the mechanical tolerances are 0 1mm ha Figure 9 Mechanical Drawing of the Single port MCX313A BCBT Adapter Card J1 PC Connector Flash Jumper 142 24 Fe a EE y E JE aa 3 E AB a Mellanox Technologies 49 Rev 2 0 Specifications Figure 10 Mechanical Drawing of the Dual port MCX314A BCBT Adapter Card PC Connector Flash Jumper 50 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card Use
45. nning OS kernel version 1 Kernel Module Packaging KMP mode where the source rpm is rebuilt for each installed flavor of the kernel This mode is used for RedHat and SUSE distributions 2 Non KMP installation mode where the sources are rebuilt with the running kernel This mode 1s used for vanilla kernels Note If the Vanilla kernel is installed as rpm please use the disable kmp flag when installing the driver The kernel module sources are placed under usr src mellanox mlnx en 1 5 10 Run the follow ing commands to recompile the driver gt cd usr src mellanox mlnx en 1 5 10 gt scripts mlnx_en_patch sh gt make gt make install The uninstall and performance tuning scripts are installed Note If the driver was installed without kmp support the sources would be located under usr srs minx_en 1 5 10 4 1 3 Loading the Driver Step 1 Make sure no previous driver version is currently loaded Run gt modprobe r mlx4_en Step 2 Load the new driver version Run gt modprobe mlx4_en The result is a new net device appearing in ifconfig a output 4 1 4 Unloading the Driver To unload the Ethernet driver run gt modprobe mlx4_en 4 1 5 Uninstalling the Driver To uninstall the mlnx_en driver run gt sbin mlnx_en_uninstall sh 4 2 Windows Driver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web sit
46. nt System Microsoft Generic IPMI Compliant Device Ml Microsoft System Management BIOS Driver Microsoft Virtual Drive Enumerator Numeric data processor PCI Express Root Complex 4 2 7 OpenSM Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as it requires the GUID parameter to decide on which port to work Setting OpenSM upon setup results in 1t working only for the first port and not for the others gt To run OpenSM as a service assuming the package was installed in the default path use sc create 5 1 binPath c Program Files Mellanox MLNX_VPI IB Tools opensm exe service start auto gt To start the service run SIG For further information please refer to the OpenSM Subnet Manager chapter in the User Manual 34 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 4 2 8 Uninstalling MLNX_WinOF 4 2 8 1 Attended Uninstall gt To uninstall MLNX_WinOF on a single node perform one of the following options 1 Click Start gt C
47. ontrol Panel gt Programs and Features gt MLNX_VPI gt Uninstall NOTE This requires elevated administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start gt All Programs gt Mellanox Technologies gt MLNX_WinOF gt Uninstall MLNX_WinOF 4 2 8 2 Unattended Uninstall gt To uninstall MLNX_WinOF in unattended mode perform the following Step 1 Open a CMD console Step 2 Uninstall the driver Run MLNX_VPI_WinoF 4_40_0_All_win8_x64 exe S x v qn 4 2 9 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine gt To obtain the MAC address Step 1 Open a CMD console Step 2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters gt To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices T e Net Networ gt v Search Network Connections 2 Organize He Name Device Name Y Ethernet Broadcom BCM5709C NetXtreme Il GigE NDIS XJ Ethernet 2 m 5709 Net Ethernet 3 Ethernet 4 Local Area Connection IBM USB Rernote NDIS Network Device Mellano
48. r See Figure 8 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the firmware release version and the card part number Note Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card Mellanox Technologies 41 Rev 2 0 Updating Adapter Card Firmware Figure 8 Support Download Assistant AM Mellanox Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR MT_1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards 1 If a newer firmware version exists for your adapter card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above b Unzip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci cr0 i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4099 pci cr0 q IMACS VEER ConnectX FW Version 22 400 Device ID 4099 42 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 6 6 1 Mellanox Technologi
49. r Manual Rev 2 0 7 5 Bracket Mechanical Drawing Figure 11 Dual port Bracket Port 1 LEDs Port 2 Figure 12 Single port Bracket OO LEDs Gasket Port 1 Mellanox Technologies 51 Rev 2 0 Specifications 76 Regulatory Statements For regulatory statements for all ConnectX 3 cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf 52 Mellanox Technologies Rev 2 0 lews Connector and Cage Vi 18 35 Top Figure 13 View into Rear of Module Interface Connectors Pinout ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual QSFP Connector Pinout Appendix A A 1 53 GND Fa QNO of GND QNO TX1n wx TXtp diei 2 Be GND GND QN9 pu TX3n i 4 i UEXE TX3 d TX4p l dpxy exa GND i GND i UND AND LPMode ModSelL Sid pow vas 1 ResetL p ul 974 128 VccTx 5 XYI VccRx 88 IntL 19S9y SCL ModPrsL SDA 8 epowan GND AND GND R3 RX4p O i dpXL RXSp 5 dex 3 n RX3n 5 uex LE upXL GND u RX2 per ano d XL i M RX2n gt x UZXL RX1n ux GND 5 AND ES GND ki Mellanox Technologies Rev 2 0 A 2 PCI Express 8 Connector Pinout Table 10 Connector Pin Number and Name to Signal Name Map
50. s san er BR HIHI ER ee 10 1 2 Features and Benefits en a ET EI RE nl 11 1 3 Operating Systems Distributions 11 14 coser GEE en Ghee ew Ge Rew GERS m Re i 12 Chapter 2 Interface cts 2 1 Ethernet Interface 055 a Ra EN RENE es 13 222 PCI Express Interface 58080 la nn 13 2 3 I2C compatible Interface 14 2 4 LED Interface coi ias EHE LG 14 Chapter 3 Hardware Installation 15 3 1 System Requirements ie ta 15 Soll Hardware te sk tet dns baie en 15 3 1 2 Operating Systems Distributions 15 3 13 Software Stacks sce as 15 3 2 Safety Precautions unten nn nl 15 3 3 Pre installation Checklist 15 3 4 Bracket Installation Instructions 15 3 4 1 Removing the Existing Bracket 16 3 4 2 Installing the New Bracket 16 342 1 Gasket Installation u un dar 16 3 4 2 2 Installing the Bracket 17 3 5 Card Installation Instructions 18 3 6 Cables and Modules 19 3 6 1 Cable
51. ss Subnet mask Default gateway Obtain DNS server address automatically 0 Use the following DNS server addresses Preferred DNS server Alternate DNS server _ validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from a CMD console gt last ale Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix IP Address Subnet Mask Default Gateway Ud 46412663 8 255 5255 0 5 0 4 2 10 Port Type Management on Windows After installing Mellanox WinOF VPI for Windows on your machine you can change a port s pro tocol configuration The following steps describe how to configure the port type Step 1 Display the Device Manager and expand System Devices Mellanox Technologies 37 Rev 2 0 Driver Installation File Action es nO 686 4 amp Network adapters View Help Broadcom BCM5709C NetXtreme 11 GigE NDIS VBD Client 48 Ca Broadcom BCM5709C NetXtreme Il GigE NDIS VBD Client 49 Ca IBM USB Remote NDIS Network Device E Microsoft Kernel Debug Network Adapter Y Ports COM amp LPT deh Print queues b Processors DP Security devices lt Storage controllers System devices Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties S
52. ted 4 3 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ESXi Server software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers gt Download Follow the installation instructions included in the download package also available from the download page 4 3 1 Installing and Running the VBI Driver on ESXi 5 x 1 Log into the VMware ESXi server machine as root 2 You can either a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4_en driver VIB package Run gt esxcli software vib install v lt vib_url gt Reboot ESXi server The driver will be loaded automatically OR a Update the driver Run gt esxcli software vib update v lt vib_url gt b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload_mod 1 grep mlx4_en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by MLX4_EN driver should be displayed In Non Multifunction Mode port 2 is identified as pseudo device Therefore devices 2 are not seen by vSphere when added as uplink 2 For furt
53. through the following methods 8 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the database from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader Mellanox Technologies 9 Rev 2 0 Introduction 1 Introduction This is the User Guide for Mellanox Technologies Ethernet adapter cards based on the Con nectX 3 EN integrated circuit device These adapters connectivity provide the highest perform ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics e Section 1 1 Product Overview on page 10 Section 1 2 Features and Benefits on page 11
54. tice the sub string PCNVEN_15B38 DEV_1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID If the PCI device does not have a Mellanox adapter ID return to Step 2 to check another device N The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 t 20 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 3 7 2 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt Ispci grep i Mellanox 27 00 0 Network controller Mellanox Technologies MT27500 Family ConnectX 3 Mellanox Technologies 21 Rev 2 0 Driver Installation 4 Driver Installation 4 1 Linux Driver For Linux download and install the latest MLNX_EN driver software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt ConnectX 3 EN 10 40GigE Linux Driver gt Download Follow the installation instructions included in the download package also available from the download page 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms CPU architectures e 86 64 x86 e power pc
55. tter server utilization while reducing cost power and cable complexity A consolidated compute and storage network achieves significant cost perfor mance advantages over multi fabric networks Standard block and file access protocols can leverage InfiniBand RDMA for high performance stor age access Operating Systems Distributions Citrix XenServer 6 1 Rev 2 0 Introduction Novell SLES Red Hat Enterprise Linux RHEL and other Linux distributions Microsoft Windows Server 2008 2012 e OpenFabrics Enterprise Distribution OFED e OpenFabrics Windows Distribution WinOF Ubuntu 12 04 VMware ESXi 4 x and 5 x 1 4 Connectivity e Interoperable with InfiniBand or 10 40 Gb Ethernet switches Interoperable with 56GbE Mellanox Switches Passive copper cable with ESD protection Powered connectors for optical and active cable support to SFP connectivity through QSA module 12 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Adapter Card User Manual Rev 2 0 2 Interfaces Each adapter card includes the following interfaces Ethernet Interface PCI Express Interface e 2C compatible Interface LED Interface Figure 1 MCX314A BCBT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables 2 1 Ethernet Interface The network ports of the ConnectX 3 adapter cards are compli
56. x Technologies 35 Rev 2 0 Driver Installation Step2 Right click a Mellanox Local Area Connection and left click Properties Networking Sharing Connect using Mellanox IPolB Adapter This connection uses the following items A Client for Microsoft Networks 5 QoS Packet Scheduler a File and Printer Sharing for Microsoft Networks Microsoft Network Adapter Multiplexor Protocol Link Layer Topology Discovery Mapper 1 0 Driver de Link Layer Topology Discovery Responder Internet Protocol Version 6 TCP IPvB E Internet Protocol Version 4 TEP 1Pw4 Uninstall Properties Transmission Control Protocol Intemet Protocol The default wide area network protocol that provides communication across diverse interconnected networks Description j Step 3 Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties 36 Mellanox Technologies ConnectX 3 Ethernet Single and Dual QSFP Port Adapter Card User Manual Rev 2 0 Step 4 Select the Use the following IP address radio button and enter the desired IP information General You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator for the appropriate IP settings Obtain an IP address automatically Use the following IP address IP addre

3 Ethernet Single and Dual QSFP+ Port Adapter Card User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents