Home

User Manual - Boston Server & Storage Solutions

1. Preparing HH A AE EEE HE AE compat dapl devel BEA AAA HU HU AH HE HE AA ME ME A HU Prepar ng HH CHEERY compat dapl devel Prepar ng HH HEH dapl HH 4 Ht HH H HH H H i H H HH H H HH HH Prepar ng HH HHH dapl HH H HH HHHH H HH H i H HHH H H HH H H HH HH Prepar ng HH A AE EEE HAE AE A HE HH HA EH dapl deve HH HH HH i H H HHH H HH HH Prepar ng HH HAE AA EEE HAE AE A HAE HE AE HA EH dapl deve L THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing HH HEHEHE dapl devel static Preparing a HH HH He HH HH H HH HH H HH H HHHH HH H HH H HH HH dapl deve l static ng HH HEH dapl utils 4 Ht HH HH i H H HH H H 4 HH Preparing A HH HH HH HH HH H H HH H HH H HH HH perftest HH HAE A HE EEE AE EE A HAE EE HAE HEHE HH Preparing HH A AE EEE HAE EE HAE EE HAE EH mstflint A E R AHRR HEREHEREA HER HE HEHHEH Preparing HH AA EEE AE EH HE HH EH HA EH mft HH HH HHH H H HH H H HH HH Preparing
2. libmlx4 deve HH HE EEE AE EEE HE HH Preparing 4 HH NE libmlx4 devel HH A AE EEE AE AE A EH AE HA EH Preparing HH HEHEHE libmlx5 THHHHHHHBHHHHHBHHHHHHHHHBHBHHHHHHHBHBHHHHHHHBHBHHEHNE Preparing HH HAE A HE EEE AE EE A HAE EE HAE HEHE HE HE libmlx5 HH HH HH HH HH HH HH Preparing libmlx5 devel HH A AE EEE AE AE A HAE HE HH HA EH Preparing 00 HH HE EEE HE AE AE A libmlx5 devel HH A AE EEE HAE AE A EEE AE HEHE HEHEHE HH EH Preparing HH HEHE A A R AHRR HERRE EHER EEHEEHE libmverbs devel THHHHHHHBHHHHHBHHHHHHHHBHHHHHHHBHBHHHHHHHBHBHHEHHE Preparing THHHBHHHHHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE libmverbs devel THHHBHHHBHHHBHHBHHHHHHHHBHHHBHHHHBHHHHHHBHBHHHBRHRHHI Preparing HH A AE EEE HAE AE A libmge HH 4 HH HH HH 4 4 HH HH Preparing HH A HE EEE EE A libmge HH HH Het HH HH HH HH HH Preparing HH AA EEE HAE AE A HAE HE EE EH HAE EH libmge devel HH AA EEE HAE EEE HE EE HA EH Preparing libmge devel HH HE EEE HE AE AE A HAE HE AE EH LEA EH Preparing HH A A A ARARA REER REEERE HEERE libibcm THHHHHHHBHHHHHBHBHHHHHHBHHHH
3. libibumad static Preparing HH HEH libibmad HH AA EEE HAE AE A HAE HE EE HAE EH Preparing HH HHH libibmad HH A AE EEE HAE AE A HAE HE AE HEHE HEHEHEHEH EH Preparing HH A AE EEE HAE AE A HE HH HA EH libibmad devel HH HHH Preparing HH HAE AA EEE HAE AE A HAE HE AE HA EH libibmad devel THHHBHHHBHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing HH HEHEHE libibmad static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHRHBHHRHRI Preparing LA HH HH HH HH HH HH HH HH HH HH libibmad static Preparing HH HEH ibsim 4 Ht i HH H H 4 HH Preparing HH EHR ibacm HH HH HH HH Preparing HH A AE EEE HAE EE HAE EE HAE EH librdmacm A E R AHRR HEREHEREA HER HE HEHHEH Preparing THHHHHHHBHHHHHBHHHHHHHHHHHHHHHHHBHHHHHHHHBHHHBSHBRHHI librdmacm HH AA EEE HAE AE A HAE HE AE HA Preparing HH A AE EEE HAE EE HAE AE HE HEHEHEHEH EH librdmacm utils HH AA EEE HAE EE HAE Preparing 4 THHHBHHHHBHHHHHBHHHHHHHHBHHHBHHHHBHHHHHHHHBHHHBSHRHHI librdmacm devel HH Ht HH HH HH Preparing HH AE AE A HAE HE HH HA EH librdmacm dev
4. HH HH HH He dt dy HH HH HH HH Preparing HHH HH HH HH mvapich2 gcc HH HH HH HH HH HH Preparing HHH HH HHH HHH HH openmpi gcc T1HBHBHUHHBWHHSHHHUUE BSHHBHHU H H AE HEH HE E AE HBHHBHE B HE Preparing HHH HH HHH HH HH mpitests mvapich2 gcc HH HH HH HH HHH HH HH HH Preparing HHH HH 4 mpitests openmpi gcc HH HH He 41 4E 4L HH HH HH HH Preparing HH Het HH HH mlnxofed docs i HH HH HH HH HH HH HH Mellanox Technologies 31 1 3 Driver Installation Step 4 Step 5 Step 6 Step 7 Device 05 00 0 05 00 0 Ethernet controller Mellanox Technologies MT26448 ConnectX EN 10GigE PCIe 2 0 5GT s rev b0 Link Width is not 8x PCI Link Speed 5Gb s Device 07 00 0 07 00 0 Ethernet controller Mellanox Technologies MT27500 Famil
5. High precision event timer Intel R 5520 5500 X58 Hub Control Status and RAS Registers 3423 Intel R 5520 5500 X58 1 0 Hub GPIO and Scratch Pad Registers 3422 Intel R 5520 5500 X58 Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 Hub PCI Express Root Port 3 340A pM Intel R 5520 5500 X58 Hub PCI Express Root Port 7 340E Intel R 5520 5500 X58 1 0 Hub System Management Registers 342E Intel R 5520 5500 58 Hub to ESI Port 3406 Ml Intel R 82801 PCI Bridge 244E 2 Intel R ICH10 Family PCI Express Root Port 1 3440 2 Intel R ICH10 Family PCI Express Root Port 5 3448 E Intel R ICH10R LPC Interface Controller 3416 1 Microsoft ACPI Compliant System Microsoft Generic Compliant Device Microsoft System Management BIOS Driver 188 Microsoft Virtual Drive Enumerator Numeric data processor PCI Express Root Complex 4 2 6 OpenSM Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as it requires the GUID parameter to decide on which port to wor
6. 70 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Appendix B Finding the GUID and Serial Number on the Adapter Card Each Mellanox adapter card has a different identifier printed on the label serial number the card MAC for the Ethernet protocol and the card GUID for the InfiniBand protocol VPI cards have both a GUID and a MAC derived from the GUID Figure 13 Card Product Label Model No CX354A ConnectX 3 FDR InfiniBand 40GigE P N MCX354A FCAT Rev A1 S N MT1106X01504 GUID 00 02 C9 03 00 27 05 1C ARI MAC 00 02 C9 27 05 1C Made in Israel Port 1 uses the GUID or MAC described on the label To obtain the GUID or MAC for Port 2 on dual port cards add 1 to the GUID or MAC of Port 1 N Mellanox Technologies 71 Rev 1 3 Appendix C Safety Warnings 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed LA ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Copper Cable Connecting Disconnecting
7. 1 tees L Step 3 Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties 50 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step 4 Select the Use the following IP address radio button and enter the desired IP information General You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator For the appropriate IP settings Obtain an IP address automatically Use the following IP address IP address Subnet mask Default gateway Obtain DNS server address automatically Use the following DNS server addresses Preferred DNS server Alternate DNS server _ validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from a CMD console gt ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix IP Addresse 2 1A 12563 Subnet Mask 6 8 255 25500 Default Gateway 4 2 9 Port Type Management on Windows After installing Mellanox WinOF VPI for Windows on your machine you can change a port s pro tocol configuration The following steps describe how to configure the port type Step 1 Displ
8. 17 3 4 2 Installing the New Bracket 17 3 5 Card Installation Instructions 18 3 6 Cables and Modules 19 3 6 1 CableInstallation 4 44 448 REESE sn 19 3 7 Identify the Card in Your System 20 3 7 1 On WindOWS UN SCR RU RR RR REOR AC RU 20 3 52 On LINUR ee ENS 21 Chapter 4 Driver Installation 22 4 1 Driver A ei sel 22 4 1 1 Hardware and Software Requirements 22 4 1 2 Downloading Mellanox 22 Mellanox Technologies 3 Rev 1 3 4 1 3 Installing OFED 23 4 1 3 1 Pre installation Notes 23 4 13 2 Installation SCtipt uo a IE Pete um debe 24 4 1 3 3 mInxofedinstall Return Codes 25 4 1 3 4 Installation Procedure 26 4 1 4 Installation Results 33 4 1 5 Post installation Notes 35 4 1 6 Updating Firmware After Installation 35 4 1
9. Industry Standard PCI Express 3 0 Base and Card Electromechan ical Specifications Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega Bytes The use of Mb or Mbits small b indicates size in mega bits IB is used in this document to mean InfiniBand In this document PCIe is used to mean PCI Express Mellanox Technologies 9 Rev 1 3 Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database infor mation for Mellanox products Access the database from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader 10 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 1 Introduction This is
10. gt flint d dev mst mt4103 pci q Image type ConnectX 3 Pro FW Version 2 9 4000 Device ID 4103 Chip Revision 0 Description Node Porti Port2 Sys image 60105 000002c900000200 000002c900000201 000002c900000202 000002c900000203 MACs 000002c90200 000002c90201 Board ID MT 1020110019 VSD PSID MT 1020110019 1 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 5 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the firmware release version and the card part number Note Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card Mellanox Technologies 55 Rev 1 3 Updating Adapter Card Firmware Figure 5 Support Download Assistant AM Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Select a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards 1 If a newer firmware version exists for your adapter card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above b Un
11. HHH H HH H HH H HH HH dump pr HA HAL A Preparing HH HHH ar mgr HH HH HH HH Preparing HH CHEERY ibdump HH HH HHH HH HH HH Preparing HH HHH infiniband diags HH A AE EEE R AHRR EHER REEERE EH Preparing HH HHH qperf HH He HH HHH HH HH Preparing HH A AE EEE HAE AE A EH HE EE HA EH fca HH HHHH HH 4 HH H H HH 4 INFO updating IMPORTANT NOTE The FCA Manager and FCA MPI Runtime library are installed in opt mellanox fca directory The FCA Manager will not be started automatically To start FCA Manager now type etc init d fca managerd start There should be single process of FCA Manager running per fabric To start FCA Manager automatically after boot type etc init d fca managerd install service Check opt mellanox fca share doc fca README txt for quick start instructions Preparing HH HHH HHH HH mxm HH HH HH HH HH HH HH Preparing HH HHH HH HH openshmem HH HH HH HH HH HH Preparing HHH HH HHH
12. Mellanox Firmware Tools MFT User Manual Document 2204UG User Manual describing the set of MFT firmware management tools for a single node See http www mellanox com gt Products gt Software gt Firm ware Tools Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED WinOF VPI for Windows User Manual Document no 3280 User Manual describing WinOF features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Windows SW Drivers Performance Tuning Guidelines for Mellanox Network Adapters Document no 3368 Manual describes important tuning parameters and settings that can improve performance for Mellanox drivers Mellanox InfiniBand OFED Driver for VMware vSphere 5 x User Manual Document no 3464 User Manual describing OFED driver for VMware features per formance InfiniBand diagnostic tools content and configuratio IBTA Specification Release 1 2 1 InfiniBand Architecture Specification IEEE Std 802 5 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 PCI Express 3 0 Specifications
13. OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically Performances tools install the performance tools that are used to measure the InfiniBand performance in user environment Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment SDK contains the libraries and DLLs for developing InfiniBand application over IBAL Documentation contains the User Manual and Installation Guide NDFLTR DLLs contains the files for standalone installation of the mlx4nd provider Custom Setup Select the program features you want installed Click on an icon in the list below to change how a feature is installed Feature Description Subnet manager SM Performace Tools Analyze Tools SDK Documentation ND FLTR DLLs This Feature requires 1164KB on your hard drive Install to C Program Files Mellanox MLNX_YPIYIB Tools InstallShield Mellanox Technologies 45 Rev 1 3 Driver Installation Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation If you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield Step 7 Click Finish to complete the installation InstallShield Wiza
14. HH A AE EEE HAE AE A EEE AE HEHE HEHEHE HH EH srptools HH HH HH H H HH H H HH 4 Preparing 47 HH HH HH HH HH H H HH H HH H HH HH rds tools THHHBHHHHHHHHHBHHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHNE Preparing A HH HH H HH H HH H H H HH H HH H HH HH rds devel HH A AE EEE HAE AE A Preparing a HH HH HH HH H H 4 H HH H HH HH ibutils2 HH HAE AE A A EE EEE EEE HAE EE Preparing 2 HH H HH HHHH H HH H HH H HHH H H HH H HH H HH HH ibutils HH AA EEE HE AE AE A HAE HE EE HA EH Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Preparing 4 HH A AE EEE HAE AE A HAE HH HA EH mgr HH HHHH HH 4 HH H H HH HH Preparing Ad HH HH HH HH
15. Some copper cables are heavy and not flexible as such they should be carefully LA attached to or detached from the connectors Refer to the cable manufacturer for spe cial warnings and instructions 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula tions 7 Local and National Electrical Codes k This equipment should be installed in compliance with local and national electrical codes 8 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure 72 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 CLASS 1 LASER PRODUCT and reference to the most recent laser standards 60 825 1 1993 A1 1997 A2 2001 and EN 60825 1 1994 1 1996 A2 20 Mellanox Technologies 73 Rev 1 3 Appendix D Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante LA d passant le maximum recommand de 55
16. C 131 F Un flux d air de 200LFM cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Branchement d branchement des c bles en cuivre Les c bles en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention LA en les branchant et en les d branchant des connecteurs Consultez le fabricant des c bles pour conna tre les mises en garde et les instructions sp ciales 5 Installation du mat riel k Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L elimination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 7 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 74 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 8 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition a
17. Port Adapter User Manual Rev 1 3 Step 6 To complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX_YPI Click Finish to exit the wizard lt Back Cancel 4 2 3 Installing MLNX WinOF This section provides instructions for two types of installation procedures Attended Installation An installation procedure that requires frequent user intervention e Unattended Installation An automated installation procedure that requires no user intervention Both Attended and Unattended installations require administrator privileges kall 4 2 3 1 Attended Installation The following is an example of a MLNX WinOF x64 installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF To configure your setup to contain the logs option please run the following command after opening a CMD console MLNX VPI WinOF 4 40 0 All win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run the following command MLNX VPI WinOF 4 40 0 All win8 x64 exe v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Mellanox Technologies 41 Rev 1 3 Driver Installation Click Next in the Welcome screen Welcome to the InstallShield Wizard for MLNX The InstallShield R Wizard will install MLNX_YPI on your computer To
18. 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 y AA 3 7 2 On Linux Get the device location on the PCI bus by running lspci and locating lines with the string Mella nox Technologies Mellanox Technologies 21 1 3 Driver Installation 4 Driver Installation The below sections are examples using ConnectX 3 device and will be updated to include ConnectX 3 Pro in future releases 4 1 Linux Driver For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox web site at http www mellanox com gt Products gt Soft ware gt InfiniBand VPI Drivers gt Download This chapter describes how to install and test the Mellanox OFED for Linux package on a single host machine with Mellanox ConnectX 3 Pro adapter hardware installed 4 1 1 Hardware and Software Requirements Table 5 Software and Hardware Requirements Requirements Description Platforms For the list of supported architecture platforms please refer to the Mellanox OFED Release Notes file Required Disk Space for 1GB Installation Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux operating system For the list of supported operating system distributions and kernels please refer to the http www mellanox com gt Products gt Software gt Infini Band V
19. 7 Uninstalling Mellanox 37 4 1 8 Port Type Management on Linux 37 4 1 9 37 4 1 9 1 Enabling Auto Sensing 38 42 Windows DEIVeE s ueterem ore Wu cR mca wea te e os 38 4 2 1 Downloading MLNX_WinOF 38 4 2 2 Extracting Files Without Running Installation 39 4 2 3 Installing MLNX 41 4 2 3 1 Attended Installation 41 4 2 3 2 Unattended Installation 47 4 2 44 Upgrading MLNX_WinOF 47 4 2 5 Installation Results ass a ec ere En e Eo e 48 4 2 6 5 Activation 22 232 does RR Eh 48 427 Uninstalling MLNX 49 4 2 7 1 Attended Uninstall 49 42 1 2 Unattended Uninstall a Re Eh ette quB 49 4 2 8 Assigning Port IP After Installation 49 4 2 9 Port Type Management on Windows 51 4 3 P rtormance Tuning i ele vA ea ala 53 44 VMware Driver 53 4 4 1 Installing Mellanox InfiniBand OFED Driver for VMware
20. ULXL QN9 oa o z 2 5 6 QN9 92a9gaa 800 szs abo9muu9219 2 gt O L ONU GE 6226808 SPREE RS EE SEES _ TOP 18 35 View into Front of Cage B 50 Table 13 Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN 2 Tx2n 3 TXP 2 Tx2p 4 GND GND 5 TXN 4 Tx4n 6 TXP 4 Tx4p 7 GND 8 ModSelL_Port0 ModSelL 9 ResetL_PortO ResetL 10 11 SCL SCL 68 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Table 13 Connector Number and Name to Signal Name Connector Pin Number Connector Pin Name Port A Signal Name 12 SDA SDA 13 GND GND 14 RXP 3 Rx3p 15 RXN 3 Rx3n 16 GND GND 17 RXP 1 1 18 RXN 1 RxIn 19 GND GND 20 GND GND 21 RXN_2 Rx2n 22 RXP 2 Rx2p 23 GND GND 24 RXN 4 Rx4n 25 RXP 4 Rx4p 26 GND GND 2T ModPrsl PortO Mod PrsL 28 IntL IntL 29 VecTx 30 1 31 LPMode 0 LPMode 32 GND GND 33 TXP 3 Tx3p 34 TXN 3 Tx3n 35 GND GND 36 TXP 1 37 TXN 1 38 GND GND A 2 PCI Express x8 Connector Pinout The adapter cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Mellanox Tec
21. gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h 4 2 4 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version In Windows 2012 MLNX WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade Mellanox Technologies 47 Rev 1 3 Driver Installation 4 25 Installation Results Upon installation completion you can verify the successful addition of the network card s through the Device Manager To see the Mellanox network adapter device and the Ethernet or IPoIB network device depending on the used card for each port display the Device Manager and expand System devices Network adapters File Action View Help gt As amp Mellanox ConnectX 3 Ethernet Adapter 2 Microsoft Kernel Debug Network Adapter 4 Other devices Unknown device Ports COM amp LPT p Print queues gt D Processors b lt gt Storage controllers 4 M System devices ACPI Fixed Feature Button 1 Composite Bus Enumerator li Direct memory access controller
22. of the network adapter installed in your machine To obtain the MAC address Step 1 Open a CMD console Step 2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices T E Net Networ gt v Search Network Connections 2 Organize v Device Y Ethernet Broadcom BCMS5709C NetXtre x Ethernet 2 Ethernet 3 Y Ethernet 4 M U Local Area Connection IBM USB Remote Mellanox Technologies 49 Rev 1 3 Driver Installation Step2 Right click a Mellanox Local Area Connection and left click Properties Networking Sharing Connect using Mellanox PolB Adapter This connection uses the following items QoS Packet Scheduler File and Printer Sharing for Microsoft Networks Microsoft Network Adapter Multiplexor Protocol Link Layer Topology Discovery Mapper 1 0 Driver Link Layer Topology Discovery Responder A Internet Protocol Version 6 TCP IP v6 Install Uninstall Properties Description Transmission Control Protocol Intemet Protocol The default wide area network protocol that provides communication across diverse interconnected networks v v v das for Microsoft Networks
23. off Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but Check that the latest driver is loaded with no Check that both the adapter and its link are set to the same speed communication and duplex settings established Rev 1 3 Troubleshooting 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed info head 1 Information ifconfig a ethtool lt interface gt ethtool i interface of Mellanox port num ibdev2netdev Card Detection lspci grep 1 Mellanox Download and install http www mellanox com con tent pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d lt mst_device gt q ibstat Ports Information Ibv_devinfo Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF 58 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 6 3 Windows From the Windows desktop choose the Start menu and run msinfo32 To export system information to a text file choose
24. the User Guide for Mellanox Technologies VPI adapter cards based on the ConnectX 9 3 Pro VPI integrated circuit device These adapters connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics Section 1 1 Product Overview on page 11 Section 1 2 Features and Benefits on page 12 Section 1 3 Operating Systems Distributions on page 13 Section 1 4 Connectivity on page 13 1 1 Product Overview The following section provides the ordering part number port speed number of ports and PCI Express speed Each adapter comes with two bracket heights short and tall Table 3 Single and Dual port FDR Adapter Cards MCX353A FCCT single port card ing P P MCX354A FCCT dual port InfiniBand FDR 56 Gb s Data Transmission Rate Ethernet 40 56Gb s single or dual port QSFP Network Connector Types PCI Express PCIe SerDes Speed PCIe 3 0 x8 8GT s RoHS R6 Adapter IC Part Number MT27528A0 FCCR FV 4099 for Physical Function Deytce 4100 for Virtual Function Mellanox Technologies 11 Rev 1 3 Introduction 1 2 Table 4 Features Virtual Protocol Interconnect InfiniBand Architecture Specification v1 2 1 compliant PCI Express PCIe 10 40 56 Gigabit
25. vSphere 53 4 4 2 Uninstalling Mellanox InfiniBand OFED 54 4 5 FlexBo6t sue tons A eMe bee Pow etd ees 54 Chapter 5 Updating Adapter Card Firmware 55 Chapter 6 Troubleshooting 57 6 1 General nn ne EF HERE INE 57 6 2 nur jp yon el RR EI ERE PEEL pU 58 6 3 Windows owe Er EE SES E Se e Sa 59 Chapter 7 60 7 1 MCX353A FCCT Specifications 60 7 2 MCX354A FCCT Specifications 62 7 3 Adapter LED Operation 63 7 4 Board Mechanical Drawing and Dimensions 64 7 5 Bracket Mechanical Drawing 66 7 6 Regulatory Statements 67 Appendix A Interface Connectors Pinout 68 QSFP Connector 68 4 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 A 2 PCI Express x8 Connector Pinout 69 12 Connector Pinout 70 Appendix Finding the GUID MAC and Serial Number o
26. wrists 2 Make sure to use only insulated tools 3 Verify that the system is powered off and is unplugged 4 It is strongly recommended to use an ESD strap or other antistatic devices Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged Bracket Installation Instructions The card is usually shipped with a tall bracket installed If this form factor is suitable for your requirements you can skip the remainder of this section and move to Section 3 5 Installa tion Instructions on page 18 If you need to replace it with the short bracket that is included in the shipping box please follow the instructions in this section Note Due to risk of damaging the EMI gasket it is not recommended to replace the bracket more than three times To replace the bracket you will need the following parts 16 Mellanox Technologies ConnectX 3 Pro VPI Single Dual QSFP Port Adapter Card User Manual Rev 1 3 The new bracket of the proper height The 2 screws saved from the removal of the bracket The 2 fiber washers saved from the removal of t
27. 0 2 0 0 installation Installing mlnx ofa kernel RP Preparing 27 HH H HH HH HH H HH HH mlnx ofa kernel HH H Het HH HH H HH H HH HH H HH HHHH HH HH H HH HH Installing kmod minx ofa kernel RPM Preparing 22 HHH HH HH H HH H HHHH HH H HH HH kmod mlnx ofa kernel HHH HH idt HH 4 H H HH H HH HH Installing mlnx ofa kernel devel RPM Preparing HHH HHH HH HHH HH HHH 1 HH HH kernel devel H HH HH H HH HHH Het HH HH Installing kernel mft RPM Preparing HERT T T T T mft HH HHHHH HH HH H HH HH HH HH H H Installing knem RPM Preparing box HH H 4H HHH HH HH H H 4 HH HH knem HH H Het HH HH H HH H HH HHHH HH H HH H HH HH Installing mpi selector RPM Preparing Jis H HH HH H HHH H HH HHHH H HH HH mpi selector HH H 4H HHH HHH HH HHH H H HHH HH HH Mellanox Technologies 27 Rev 1 3 Driver Installation Installing user l
28. AA Mellanox TECHNOLOGIES ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual P N MCX353A FCCT MCX354A FCCT Rev 1 3 www mellanox com Rev 1 1 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADV
29. ED Driver To uninstall the driver 1 Log into the ESXi5 0 server with root permissions 2 List the existing InfiniBand OFED driver modules i esxcli software vib list grep Mellanox net ib core 1 8 1 0EM 500 0 0 472560 Mellanox Partner Supported 2013 02 09 net ib ipoib 1 8 1 0EM 500 0 0 472560 Mellanox Partner Supported 2013 02 09 3 Remove each module using the esxcli software vib remove command gt esxcli software vib remove n net ib ipoib gt esxcli software vib remove n net mlx4 ib gt esxcli software vib remove n net ib umad gt esxcli software vib remove n net ib sa gt esxcli software vib remove n net ib mad gt esxcli software vib remove n net ib core gt esxcli software vib remove n net mlx4 core Reboot the server FlexBoot FlexBoot supports remote Boot over InfiniBand BoIB and over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot soft ware is based on the open source iPXE project see www ipxe org For more information go to http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Download 54 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 5 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates
30. Ethernet InfiniBand FDR InfiniBand FDR10 InfiniBand QDR Memory Virtualized Overlay Networks RDMA over Converged Ethernet RoCE CPU offload GPUDirect RDMA Features and Benefits VPL enabled adapters facilitate any standard networking clustering storage and management protocol to seamlessly operate over any converged network with the same software infrastructure ConnectX 3 Pro delivers low latency high bandwidth and computing efficiency for performance driven server and storage clustering applications ConnectX 3 Pro is InfiniBand Architecture Specification v1 2 1 compliant Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ba 40 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover A pre standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 14 0625Gb s with a 64b 66b encoding resulting in an effective bandwidth of 54 54Gb s An InfiniBand transport and link layer running over a physical layer using the PCS encoding of 64b 66b and a bit rate of 10 3125Gb s per lane A standard InfiniBand data rate where each lane of a 4X port runs a bit rate of 10Gb s with an 8b 10b encoding PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI
31. HHHBHBHHHHHHHBHBHHHEHE Preparing THHHBHHHBHHHHHBHBHHHHHHBHHHHHHHBHBHHHHHHHBHBHHN NE libibem AREREHEHE AAA EEE HAE A AE EEE HE AE AE A HAE HE EE HA EH 28 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Preparing HH A AE EEE A EE HE HE HH HA EH libibcn devel HHH HH H HH H H HH HH Preparing HH HEHEHE libibcn devel HH HH HH HH Preparing HH HHH libibumad HH Y Preparing A EREHEHE AAA EEE HAE AA EEE HAE AE A HAE HE EE EH HAE EH libibumad HH HE A A HH Preparing HH A AE EEE AE EE EEE HE AE HEHE HEHEHEHEH EH libibumad devel HH HB A E R AHRR EHER REEERE HER HEHEHHEH Preparing HH A AE EEE AE EE EH HE HH HE EEHEEHE libibumad devel THHHBHHHHHHHHHBHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHHE Preparing AREA AAA EEE HAE A AE EEE A libibumad static THHHHHHHHHHHHHHHHHHRHBHHHHHHHHHHHHHHHHHHHRHBHHRHRI Preparing
32. ISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 4972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectXG Connect IB CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect amp and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number MLNX 15 732 ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Table of Contents Table of Contents su a s u q l e eg in nm m S d dass n np nc ss te Eist OF Tables 2 8 a rre apre tae re EE aee aen Lastot Figures oor RP D D DE ARS Revision SION s oer AR RADI XE OR SUE AAA rer A 5 na About this Manual aise ate Technical Suppott is slm b
33. PI Drivers gt Mellanox OFED Release Notes file Installer Privileges The installation requires administrator privileges on the target machine 4 1 2 Downloading Mellanox OFED Step 1 Verify that the system has a Mellanox network adapter HCA NIC installed by ensuring that you can see ConnectX entries in the display The following example shows a system with an installed Mellanox HCA host1f lspci v grep Mellanox 02 00 0 InfiniBand Mellanox Technologies MT27520 ConnectX 3 Pro Step 2 Download the ISO image to your host The image s name has the format MLNX_OFED_LINUX lt ver gt lt 0S label gt lt CPU arch gt iso You can download it from http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers 22 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step 3 Use the md5sum utility to confirm the file integrity of your ISO image Run the following com mand and compare the result to the value provided on the download page host1 md5sumMLNX OFED LINUX lt ver gt lt 0S label gt iso 4 1 3 Installing Mellanox OFED OFED includes an installation script called m1nxofedinstall which performs the fol lowing Discovers the currently installed kernel Uninstalls any software stacks that are part of the standard operating system distribution or another vendor s commercial stack Installs the MLNX OFED LIN
34. UX binary RPMs if they are available for the current kernel Identifies the currently installed InfiniBand and Ethernet network adapters and automat ically upgrades the firmware 4 1 3 1 Pre installation Notes The installation script removes all previously installed Mellanox OFED packages and re installs from scratch You will be prompted to acknowledge the deletion of the old packages Pre existing configuration files will be saved with the extension conf rpmsave If you need to install OFED on entire homogeneous cluster common strategy is to mount the ISO image on one of the cluster nodes and then copy it to a shared file system such as NFS To install on all the cluster nodes use cluster aware tools such as pdsh If your kernel version does not match with any of the offered pre built RPMs you can add your kernel version by using the mlnx add kernel support sh script located under the docs directory Usage mlnx add kernel support sh m mlnx ofed path to MLNX OFED directory make iso make tgz gt gt make iso Create MLNX OFED ISO image gt make tgz Create MLNX OFED tarball Default t tmpdir local work dir kmp v verbose 1 The firmware will not be updated if you run the install script with the without fw update option Mellanox Technologies 23 Rev 1 3 Driver Installation Example The followin
35. VE TEE 4 Port State of Port 1 on 0 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 0 NIC UP 1X QDR Ethernet Port State of Port 1 on CA 1 NIC UP 1X QDR Ethernet Port State of Port 2 on CA 1 NIC UP 1X QDR Ethernet Error Counter Check on CA 0 NIC NA Eth ports Error Counter Check on CA 1 NIC NA Eth ports Check daa PASS NOS 9700 CA 0 22205050005056 00 02 c9 03 00 07 4f f8 GUL oa CA ONT maaa 00 02 9 03 00 35 4 0 Te DONE After the installer completes information about the Mellanox OFED installation such as prefix kernel version and installation parameters can be retrieved by running the com P mand etc infiniband info 4 1 4 Installation Results Software The OFED and packages are installed under the usr directory The kernel modules are installed under InfiniBand subsystem lib modules uname r updates kernel drivers infiniband milx4 core driver lib modules lt kernel version gt extra mlnx ofa kernel drivers net ethernet mellanox mlx4 mlx4 core ko Mellanox Technologies 33 1 3 Driver Installation mlx4 ib driver lib modules kernel version extra mlnx ofa kernel drivers infiniband hw mlx4 mlx4 ib ko mlx5 core driver lib modules kernel version extra mlnx ofa kernel drivers net ethernet mellanox mlx5 core m
36. achine see Step 1 The name of the exe is in the following format MLNX_VPI_WinOF lt ver sion All OS arch exe 1 Inchal ows Installer 4 2 2 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start gt Run and enter CMD Step 2 Enter the following command Step 3 Click Next to create a server image WARNING This program is protected by copyright law and international treaties Mellanox Technologies 39 Rev 1 3 Driver Installation Step 4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of at the specified network location or click Cancel to exit the wizard Network location rM InstallShield Step 5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location for the server image of the product Enter the network location or cick Change to browse to a location Click Install to create server image of MLNX VPT at the specified network location or click Cancel to exit the wizard InstaliShield 40 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP
37. acket Installation Instructions on page 16 April 2014 1 2 Added Section 4 3 Performance Tuning on page 53 Added Performance Tuning Guidelines to Related Documenta tion on page 9 December 2013 1 1 Added the following note to Chapter 5 Updating Adapter Card Firmware on page 55 Note The shown versions and or parameter values in the exam ple below may not reflect the latest or actual values for this prod uct and are included here for illustration purposes only Specified max power available through QSFP ports in all speci fication tables See Chapter 7 Specifications on page 60 Updated link to approved cables list in all specification tables See Chapter 7 Specifications on page 60 June 2013 1 0 First Release 8 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 About this Manual This User Manual describes Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP port PCI Expressx8 adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documenta tion Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture specifications Related Documentation Table 2 Documents List
38. ards to Ethernet and or InfiniBand For details on this script please see Section 4 1 8 Port Type Management on Linux The directory etc infiniband is created with the files info and openib conf and connectx conf The info script can be used to retrieve Mellanox OFED instal lation information The openib conf file contains the list of modules that are loaded when the openibd script is used The connectx conf file saves the ConnectX adapter card s ports configuration to Ethernet and or InfiniBand This file is used at driver start restart etc init d openibd start The file 90 ib rules is installed under etc udev rules d 34 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 If OpenSM is installed the daemon opensmd is installed under etc init d and opensm conf is installed under etc If IPoIB configuration files are included ifcfg ib n files will be installed under etc sysconfig network scripts ona RedHat machine The installation process unlimits the amount of memory that can be pinned by a user space application See Step 5 Man pages will be installed under usr share man Firmware The firmware of existing network adapter devices will be updated if the following two conditions are fulfilled 1 You run the installation script in default mode that is without the option without fw update 2 The firmware version of the adapter devi
39. ay the Device Manager and expand System Devices Mellanox Technologies 51 Rev 1 3 Driver Installation File Action View Help PRE Network adapters Broadcom 5709 NetXtreme Il GigE NDIS Client 48 Broadcom BCM5709C NetXtreme GigE NDIS VBD Client 49 IBM USB Remote NDIS Network Device E Microsoft Kernel Debug Network Adapter Ports amp LPT Print queues Processors Security devices lt Storage controllers System devices K m 4 Step 2 Right click on the Mellanox ConnectX VPI network adapter and left click Properties Select the Port Protocol tab from the Properties sheet Note The Port Protocol tab is displayed only if the NIC is a VPI IB and ETH If the NIC is either only IB or ETH the tab will not be shown General Port Protocol Driver Details Events Resources Current Setting Eth Port2 Eth Port Type Configuration HW Defaults Port 1 ETH C AUTO Port 2 C IB ETH C AUTO Port Protocol Configuration This menu displays the adapter s port type and enables youto _ set the network protocols for the network adapter ports network protocol is determined according to the NIC s Hardware Defaults port type Y ou can choose the protocol explicitly by selecting the port type to InfiniBand IB or Ethernet Et
40. ce is older than the firmware version included with the Mellanox OFED ISO image gt If an adapter s Flash was originally programmed with an Expansion ROM image the automatic firmware update will also burn an Expansion ROM image In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file 4 1 5 Post installation Notes Most of the Mellanox OFED components can be configured or reconfigured after the installation by modifying the relevant configuration files See the relevant chapters in this manual for details The list of the modules that will be loaded automatically upon boot can be found in the etc infiniband openib conf file 4 1 6 Updating Firmware After Installation In case you ran the m1nxofedinstall script with the without fw update option and now you wish to manually update firmware on your adapter card s you need to perform the following steps If you need to burn an Expansion ROM image please refer to OFED User Manual See Table 2 Documents List on page 9 Mellanox Technologies 35 1 3 Driver Installation The following steps also appropriate in case you wish to burn newer firmware that you have downloaded from Mellanox Technol
41. continue click Next WARNING This program is protected by copyright law and international treaties Step 2 Read then accept the license agreement and click Next License Agreement Please read the Following license agreement carefully Copyright c 2005 2013 Mellanox Technologies 11 rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer inana in hinart farm mint wanwa Anun tha I accept the terms in the license agreement I do not accept the terms in the license agreement InstallShield 42 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step 3 Select the target folder for the installation Destination Folder Click Next to install to this Folder or click Change to install to a different Install MLNX VPI to C Program Files Mellanox MLNX_ PT InstallShield Step 4 The firmware upgrade screen will be displayed in the following cases Ifthe user has an OEM card in this case the firmware will not be updated Ifthe user has a standard Mellanox card and the firmware version is older than the one specified in WinOF Installation Guide 4 40 the firmware
42. d The port link type can be configured for each device in the system at run time using the sbin connectx port config script This utility will prompt for the PCI device to be modified if there is only one it will be selected automatically In the next stage the user will be prompted for the desired mode for each port The desired port configuration will then be set for the selected device This utility also has a non interactive mode sbin connectx port config d device PCI device ID gt c conf lt portl port2 gt 4 1 9 Auto Sensing Auto Sensing enables the NIC to automatically sense the link type InfiniBand or Ethernet based on the link partner and load the appropriate driver stack InfiniBand or Ethernet For example if the first port is connected to an InfiniBand switch and the second to Ethernet switch the NIC will automatically load the first switch as InfiniBand and the second as Ethernet Mellanox Technologies 37 1 3 Driver Installation 4 1 9 1 Enabling Auto Sensing 4 2 4 2 1 Upon driver start up 1 Sense the adapter card s port type If a valid cable or module is connected QSFP SFP or SFP with EEPROM in cable module Set the port type to the sensed link type IB Ethernet Otherwise Set the port type as default Ethernet During driver run time Sense a link every 3 seconds if no link is sensed detected If sensed set the port type as sensed Windows Dri
43. d by a user space application If desired tune the value unlimited to a specific amount of RAM For your machine to be part of the InfiniBand VPI fabric a Subnet Manager must be running on one of the fabric nodes At this point Mellanox OFED for Linux has already installed the OpenSM Subnet Manager on your machine For details on starting OpenSM refer to the OFED User Manual See Table 2 Documents List on page 9 InfiniBand only Run the hca self test ofed utility to verify whether or not the Infini Band link is up The utility also checks for and displays additional information such as HCA firmware version Kernel architecture 32 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Driver version Number of active HCA ports along with their states Node GUID Note For more details on hca self test ofed seethefilehca self test readme under docs hca self test ofed Performing Adapter Device Self Test Number of CAs Detected d n n 2 Check Puman anan asas PASS A ans nn x86 64 HOS CRD IV CE MLNX OFED LINUX 2 0 2 0 0 OFED 2 0 2 0 0 2 6 32 279 e16 x86 64 Hoc Wier WN CEES nmana PASS Imire on CA 0 MEC v2 9 1000 Firmware Check on CA 0 NIC PASS ques o CA 1 NE oo 3922 dil HOC Firmware Check on CA 1 NIC PASS HOSP PASS Number of CA POrtS ACECI
44. dmacm mft mstflint diagnostic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector perftest sdpnetstat and libsdp srptools rds tools static and dynamic libraries hpc Install all kernel modules libibverbs libibumad librdmacm mft mstflint diagnos tic tools OpenSM ib bonding MVAPICH Open MPI MPI tests MPI selector dynamic librar ies basic Install all kernel modules libibverbs libibu mad mft mstflint dynamic libraries msm Install all kernel modules libibverbs libibumad mft mstflint diagnostic tools OpenSM ib bonding dynamic libraries NOTE With msm flag the OpenSM daemon is configured to run upon boot vma Install packages required by VMA to support both IB and Ethernet vma ib Install packages required by VMA to work over InfiniBand vma eth Install packages required by VMA to work over Ethernet v vv vvv Set verbosity level q Set quiet no messages will be printed umad dev rw Grant non root users read write permission for umad devices instead of default hugepages overcommit Set 80 of MAX MEMORY as overcommit for a huge page allocation 4 1 3 3 minxofedinstall Return Codes Mellanox Technologies 25 1 3 Driver Installation Table 6 lists the mInxofedinstall script return codes and their meanings Table 6 minxofedinstall Return Codes Return Code Meaning 0 The Installation ended successful
45. ee hea DAI hee RR 10 Firmware and Software 10 Chapter 1 Introduction 11 1 1 Product Overview sau 2 BE ee Ee E 11 1 2 Features and Bereits 2 uma RULES SR Wee E delere 12 1 3 Operating Systems Distributions 13 14 Connectivity Ebene LEE Mn 13 Chapter 2 Interface acc ea 14 2 1 InfiniBand Interface 14 2 2 Ethernet Interlace seu a LE E ee y PS ea 14 2 3 PCLBxpress Interface eo eo ee a IUE 15 2 4 I2C compatible Interface 15 225 EBD Interface u aya mapasa VE ee BA des 15 Chapter 3 Hardware 16 3 1 System Requirements 16 Xd Hardware qaku Rete bec erae aquy 16 3 1 2 Operating Systems Distributions 16 3 13 SoftWare Stacks iu ox Aa S RR 16 3 2 Safety Precautions up epe DOO LP ee TE RE 16 3 3 Pre installation Checklist 16 3 4 Bracket Installation Instructions 16 3 4 1 Removing the Existing Bracket
46. el HH HHHH HH 4 HH H H HH HH Prepar ng HH HEH opensm libs Preparing HH A AE EEE HE AE A EEE HE HH HA EH opensm libs Preparing HH CHEERY opensm H HH HH HHH H H HH H H HH HH Mellanox Technologies 29 Driver Installation Preparing HH A AE EEE AE EE EH HE HH HA EH open sm devel HHH HH HH HH H H H HH H H HH HH Preparing A A HAE EE HAE HE AE HA EH opensm devel HH A A ARAARA Preparing A REREHEHE AAA EEE HAE AA EEE HAE EEE HE HE AE HA EH opensm static A E HH HEHEHEHEHE HEHE HEHE HEHEHEHEH HER HEHEHHEH Preparing HH AA EEE A AE A HAE HE HH HAE EH opensm static Preparing HH A AE EEE AE EE HAE HE HH HA EH con pat dapl HH HH Ht HH HH H Het H H HH H H HH HH Preparing HH A AE EEE HE AE EH HE HH HA EEHEEHE compat dapl
47. erations or a slave responding to PCI bus operations The following lists PCIe interface features e PCIe Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 or 8 0GT s link rate x8 Auto negotiates to x8 x4 x2 or x1 Support for MSI MSI X mechanisms 2 4 12C compatible Interface A three pin header on the adapter cards is provided as the I C compatible interface See Figure 7 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card on page 65 for the location on the board 2 5 LED Interface There are two LEDs per port For LED specifications please refer to Section 7 3 Adapter LED Operation on page 63 Mellanox Technologies 15 Rev 1 3 Hardware Installation 3 1 3 1 1 3 2 3 3 3 4 Hardware Installation System Requirements Hardware A system with a PCI Express x8 slot is required for installing the card Operating Systems Distributions Please refer to Section 1 3 Operating Systems Distributions on page 13 Software Stacks Mellanox OpenFabric software package MLNX OFED for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation on page 22 Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components Remove any metallic objects from your hands and
48. estim mungen erfolgen 7 Regionale und nationale elektrische Bestimmungen t Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden This equipment should be installed in compliance with local and national elec LA trical codes 76 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 8 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gef hrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 77 Rev 1 3 Appendix F Advertencias de seguridad la insta laci n Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la LA m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odo
49. evel RPMs Preparing 4 THHHBHHHHHHHBHBHHHHHHHHBHHHBRHHHHBHHHHHHHHBHHHBHRHHI ofed scripts Preparing HH HAE A HE EEE AE EE A HAE AE EE HAE EE libibverbs HH HEHE A A HH Preparing HH Y libibverbs A EREHEHE AAA EEE HAE AA EEE HAE AE A HAE HE EE EH HAE EH Preparing oo HH AA EEE HAE AE libibverbs devel HH HE A A HH Preparing HH A AE EEE AE AE A HAE HE AE HE EH libibverbs devel A A RAER HERRE EHER EEHEEHE Preparing THHHBHHHHHHHHHBHBHHHHHHBHHHHHHHHBHBHHHHHHHBHBHHHHE libibverbs devel static THHHHHHHHHHHHHHHHHHRHHBHHHHHHHHHHHHHHHHHHHRHHBHHRHRI Preparing HH A AE EEE HAE AE A HAE AE HAE EH libibverbs devel static THHHHHHHHHHHHHHHHHHRHHBHHHHHHHHHHHHHHHHHHHRHBHHRHRI Preparing HH A EEE AE EE A HAE EE EH HAE EE libibverbs utils HH HHH Preparing HH A AE EEE HAE AE A HAE EE HAE EH libmverbs THHHHHHHBHHHHHBHHHHHHHHBHHHHHHHHBHHHHHHHHBHHHBSHRHHI Prepar 1 ng HH uud HH i 4 4 H H HH H H HH HH libmverbs A EEE HE AE AE A EH HE AE HE HEHEHEHEH EH Preparing HHH HH HHH HHH HH libmlx4 HH HH HH HH HH HH Preparing libmlx4 HH HH HH i HH 4 HH Preparing
50. fw 25408 rel mlx Step 4 Reboot your machine after the firmware burning is completed 36 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 4 1 7 Uninstalling Mellanox OFED Use the script usr sbin ofed uninstall sh to uninstall the Mellanox OFED package The script is part of the ofed scripts RPM 4 1 8 Port Type Management on Linux ConnectX ports can be individually configured to work as InfiniBand or Ethernet ports By default both ConnectX ports are initialized as InfiniBand ports If you wish to change the port type use the connectx port config script after the driver is loaded Running sbin connectx port config s will show current port configuration for all ConnectX devices Port configuration is saved in the file etc infiniband connectx conf This saved configuration is restored at driver restart only if restarting via etc init d openibd restart Possible port types are eth Ethernet 16 auto Link sensing mode Detect port type based on the attached network type If link is detected the driver retries link sensing every few seconds Table 7 lists the ConnectX port configurations supported by VPI Table 7 Supported ConnectX Port Configurations Port 1 Configuration Port 2 Configuration ib ib ib eth eth eth Note that the configuration Port eth and Port2 ib is not supporte
51. g command will create a MLNX OFED LINUX ISO image for RedHat 5 6 under the tmp directory MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 docs mlnx add kernel support sh i mnt MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 iso All Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Removing OFED RPMs Running mkisofs Created tmp MLNX OFED LINUX 1 5 3 rhel5 6 x86 64 iso 4 1 3 2 Installation Script The usage of the installation script is described below You will use it during the installation pro cedure described in Section 4 1 3 4 Installation Procedure on page 26 Usage mnt mlnxofedinstall OPTIONS 24 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Options c config packages config file Example of the configuration file can be found under docs n net network config file Example of the network configuration file can be found under docs p print available Print availablepackagesforthecurrent plat form and create a corresponding ofed conf file The installation script exits after creating ofed conf without 32bit Skip 32 bit libraries installation without depcheck Skip Distro s libraries check without fw update Skip firmware update force fw update Force firmware update force Force installation without querying the user all Install all kernel modules libibverbs libibu mad libr
52. h Aita Sensina nlease rhonse AITA If the Cancel 52 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step 3 In this step you can perform two different functions a Choose the desired port protocol for the available port s and b activate or deactivate the WSD ND and or SDP ULPs Note IB must be always the first port in Port 1 If you choose ETH as your first port in Port 1 then the second port in Port 2 can be only ETH Note WSD is not supported in Windows 7 Consequently on this OS the WSD checkbox is grayed out and cannot be selected 4 3 Performance Tuning For guidelines on improving network adapter performance please refer to the performance tun ing guidelines for Linux Windows environment in the document http www mellanox com related docs prod software Performance Tuning Guide for Mellanox Network Adapters pdf 4 4 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ES Xi Server software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt VMware Drivers gt Download Follow the installa tion instructions included in the download package also available from the download page 4 4 1 Installing Mellanox InfiniBand OFED Driver for VMware vSphere This chapter describes h
53. he bracket 3 4 1 Removing the Existing Bracket Figure 2 Bracket Screws Screws 1 Remove the two screws holding the bracket in place The bracket comes loose from the card LA Be careful not to put stress on the LEDs 2 Save the two screws and the two fiber washers 3 4 2 Installing the New Bracket 1 Place the bracket onto the card until the screw holes line up See Figure 3 Do not force the bracket onto the card You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket 2 Screw on the bracket using the screws and washers saved from the bracket removal proce dure above Mellanox Technologies 17 Rev 1 3 Hardware Installation Figure 3 Placing the Bracket on the Card 3 Make sure that the LEDs are aligned onto the bracket holes 4 Use a torque driver to apply up to 2 Ibs in torque on the screws 3 5 Installation Instructions 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 2 Open the system case 3 Place the adapter in an available PCI Express slot A lesser width adapter can be seated into a greater width slot x4 in a x8 but a greater width adapter cannot be seated into a lesser width slot x8 in a x4 Align the adapter connector edge with the PCI Express connector slot 4 Applying even pressure at both co
54. hnologies 69 Rev 1 3 Figure 11 PCle Connector Pinout 5 PRSNT14 1 __ 2 412 12V LA2 B m A3 H4 As B5 sucuk TCK A5 gt SMDAT PAB BZ 100 RR Tus LAS 2 TRST A8 R10 3 3Vaux AN 10 11 wAKES OBFF 5 B13 REFOLK AT 814 REFCLK Als ms GNO LA15 815 GND Perpo 236 e B1Z A12 B18 GND A18 19 PETp1 rsvp PA S 21 cup 1 21 22 GND PERn1 LA22 B23 2 HAM 25 25 25 GND pene 2 2 A27 B28 A28 829 PERpa A2 lt 831 1 BA Rsvp PAR 835 PERp4 835 2 GND PERn4 A35 B3Z A22 RB39 nn 29 _ RB41 41 B42 A42 B43 ann 43 _ B45 PETp7 GND LA45 46 FAME BAL PERp7 41 PRSNT2 PERNT Ada _849 GND GND 49 12C compatible Connector Pinout Figure 12 Compatible Connector Plug and Pinout Connector Pin Number Signal Name 1 SPSDA 2 SPSCL u 3 GND 4 NC 5 NC
55. ies 63 1 3 Specifications 7 4 Board Mechanical Drawing and Dimensions dimensions in millimeters the mechanical tolerances are 0 1mm Figure 6 Mechanical Drawing of the Single portMCX353A FCCT Adapter Card J1 PC Connector Flash Jumper 142 24 1 E e EA 3 E AB A M EJES 8 87 104 A gt Er ees CT va xa 64 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Figure 7 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card J1 PC Connector Mellanox Technologies 65 1 3 Specifications 7 5 Bracket Mechanical Drawing Figure 8 Dual port Bracket LEDs Port 2 66 Mellanox Technologies ConnectX 3 Pro Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Figure 9 Single port Bracket OO LEDs Gasket Port 1 7 6 Regulatory Statements For regulatory statements for all ConnectX 3 Pro cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Mellanox Technologies 67 Rev 1 3 Appendix A Interface Connectors Pinout QSFP Connector Pinout Figure 10 Connector and Cage Views Top 18 35 View into Rear Le 77777 Tid POW qu X129A 129A apod QN9 dexi vex QN9 dixi
56. in 68 90mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5 per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000BASE R 40GBASE R4 Protocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40 56Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 8 24W Max Power Passive Cables 8 99W Power and Max power available through QSFP port 1 5W Environmental Temperature Operational 0 C to 55 C Non operational 0 to 70 Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com Products Cables and Transceivers a The ConnectX 3 Pro adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mella nox InfiniBand product b For both operational and non operational states 62 Mellanox Technol
57. includes one 4MB SPI Flash device M25PX16 VMN6P device by ST Microelectronics EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb ConnectX 3 Pro effectively addresses the increasing demand for an overlay net work enabling superior performance by introducing advanced NVGRE and VXLAN hardware offload engines that enable the traditional offloads to be per formed on the encapsulated traffic With ConnectX 3 Pro data center operators can decouple the overlay network layer from the physical NIC performance thus achieving native performance in the new network architecture Leveraging Data Center Bridging capabilities provides efficient low latency RDMA services over Layer 2 Ethernet Adapter functionality enabling reduced CPU overhead allowing more available CPU Using GPUDirect RDMA adapters can directly read and write CUDA host and device memory eliminating unnecessary system memory copies and CPU over head resulting in significant performance improvements 12 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Table 4 Features Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10 40 56GbE The hardware based stateless Sockets Acceleration offload engines in ConnectX 3 Pro reduce the CPU overhead of IP packet trans port Sockets acceleration software further increases perfo
58. k Setting OpenSM upon setup results in it working only for the first port and not for the others Torun OpenSM as a service assuming the package was installed in the default path use sc create OpenSM1 binPath c Program Files Mellanox MLNX VPI IB Tools opensm exe service start auto To start the service run Sc start opensm 48 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 For further information please refer to the OpenSM Subnet Manager chapter in the User Manual 4 2 7 Uninstalling MLNX WinOF 4 2 7 1 Attended Uninstall gt To uninstall MLNX WinOF on a single node perform one of the following options 1 Click Start gt Control Panel gt Programs and Features gt MLNX VPI gt Uninstall NOTE This requires elevated administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start gt Programs gt Mellanox Technologies gt MLNX_WinOF gt Uninstall MLNX WinOF 4 2 7 2 Unattended Uninstall gt To uninstall MLNX WinOF in unattended mode perform the following Step 1 Open a CMD console Step 2 Uninstall the driver Run MLNX VPI WinOF 4 40 0 All wing x64 exe S x v qn 4 2 8 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address
59. lx5 core ko ib driver lib modules kernel version extra mlnx ofa kernel drivers infiniband hw mlx5 mlx5 ib ko IPoIB lib modules uname r updates kernel drivers infiniband ulp ipoib ib ipoib ko iSER lib modules uname r updates kernel drivers infiniband ulp iser ib iser ko elPoIB lib modules uname r updates kernel drivers net eipoib eth ipoib ko SRP lib modules uname r updates kernel drivers infiniband ulp srp ib srp ko RDS lib modules uname r updates kernel net rds rds ko lib modules uname r updates kernel net rds rds rdma ko lib modules uname r updates kernel net rds rds tcp ko Kernel s modules location may vary depending on the kernel s configuration For example lib modules uname r extra kernel drivers net ethernet mellanox m1x4 mlx4 core The package kernel ib devel include files are placed under usr src ofa kernel include These include files should be used when building kernel modules that use the stack Note that the include files if needed are backported to your kernel Theraw package un backported source files are placed under usr src ofa kernel lt ver gt The script openibd is installed under etc init d This script can be used to load and unload the software stack The script connectx port config is installed under sbin This script can be used to configure the ports of ConnectX network adapter c
60. ly 1 The installation failed 2 No firmware was found for the adapter device 22 Invalid parameter 28 Not enough free space 171 Not applicable to this system configuration This can occur when the required hardware is not present on the system 172 Prerequisites are not met For example missing the required software installed or the hardware is not configured correctly 173 Failed to start the mst driver 4 1 3 4 Installation Procedure Step 1 Login to the installation machine as root Step2 Mount the ISO image on your machine host1 mount o ro loop MLNX OFED LINUX ver 0S label gt lt CPU arch gt iso mnt 26 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step3 Run the installation script mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 641 root swl014 MLNX OFED LINUX 2 0 2 0 0 rhel6 3 x86 641 mlnxofedinstall This program will install the MLNX OFED LINUX package on your machine Note that all other Mellanox OEM OFED or Distribution IB packages will be removed Do you want to continue y N y Uninstalling the previous version of MLNX OFED LINUX Starting MLNX OFED LINUX 2
61. n the Adapter Card 71 Appendix C Safety Warnings 72 Appendix D Avertissements de s curit d installation Warnings in French 74 Appendix E Sicherheitshinweise Warnings in German 76 Appendix F Advertencias de seguridad para la instalaci n Warnings in Spanish 78 Mellanox Technologies 5 1 3 List of Tables Table 1 Revision History Table 8 Tabl 2 Documents List u a ds da ee denn 9 Table 3 Single and Dual port FDR Adapter Cards 11 Tabl 4 F atur s Se de dde die ds need 12 Table 5 Software and Hardware Requirements 22 Table 6 minxofedinstall Return 26 Table 7 Supported ConnectX Port Configurations 37 Table 8 Software and Hardware Requirements 38 Table 9 354 Specifications Table 60 Table 10 MCX354A FCCT Specifications Table 62 Table 11 Physical and Logical Link Indications InfiniBand Mode 63 Table 12 Physical and Logical Link Indication Ethernet Mode 63 6 Mellanox Technologie
62. occasionally and the most recent firmware can be obtained from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the package The package is available at http www mellanox com gt Products gt Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci cr0 d Get the PSID firmware identification and programmed firmware version using the command Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for this product and are included here for illustration purposes only
63. ogies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 c Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet 7 3 Adapter LED Operation There are two I O LEDs per port See Table 12 for different LED functions in InfiniBand mode and Table 12 for different LED functions in Ethernet mode Table 11 Physical and Logical Link Indications InfiniBand Mode LED Function Green physicallink Constant on indicates a good physical link Blinking indicates a problem with the physical link If neither LED is lit then the physical link has not been established Yellow logical data constant yellow indicates a valid logical data activity link without data transfer activity link A blinking yellow indicates a valid logical link with data transfer If only the green LED is lit and the yellow LED is off then the logical link has not been established Table 12 Physical and Logical Link Indication Ethernet Mode LED Function Green physical link Constant on indicates a good physical link If neither LED is lit then the physical link has not been established Yellow logical data blinking yellow indicates activity data transfer activity link Stays off when there is no activity The short bracket has the same port and LED footprints as the tall bracket Mellanox Technolog
64. ogies Web site http www mella ae nox com gt Support gt Firmware Download Step 1 Start mst hostl mst start Step 2 Identify your target InfiniBand device for firmware update 1 Get the list of InfiniBand device names on your machine host1 mst status MST modules MST PCI module loaded MST PCI configuration module loaded MST Calibre 120 module is not loaded MST devices dev mst mt25418 pciconf0 PCI configuration cycles access bus dev fn 02 00 0 addr reg 88 data reg 92 Chip revision is A0 dev mst mt25418 pci PCI direct access bus dev fn 02 00 0 bar 0xdef00000 size 0x100000 Chip revision is dev mst mt25418 pci msix0 PCI direct access bus dev fn 02 00 0 bar 0xdeefe000 size 0x2000 dev mst mt25418 pci uar0 PCI direct access bus dev fn 02 00 0 bar 0xdc800000 size 0x800000 2 Your InfiniBand device is the one with the postfix _pci_cr0 In the example listed above this will be dev mst mt25418 pci Step 3 Burn firmware 1 Burning a firmware binary image using mst flint that is already installed on your machine Please refer to MSTFLINT README txt under docs 2 Burning a firmware image from a mlx file using the mlxburn utility that is already installed on your machine The following command burns firmware onto the ConnectX device with the device name obtained in the example of Step 2 host1 mlxburn dev dev mst mt25418 pci cr0 fw mnt firmware fw 25408
65. onnectivity through QSA module Interoperable with 40 56 Gb s InfiniBand Passive copper cable with ESD protection Mellanox Technologies 13 Rev 1 3 Interfaces 2 Interfaces Each adapter card includes the following interfaces InfiniBand Interface Ethernet Interface PCI Express Interface 2C compatible Interface LED Interface Figure 1 MCX354A FCCT Card The adapter cards include special circuits to protect from ESD shocks to the card server when plugging copper cables 2 1 InfiniBand Interface The network ports of the ConnectX 3 Pro adapter cards are compliant with the InfiniBand Archi tecture Specification Release 1 2 1 InfiniBand traffic is transmitted through the cards QSFP nectors 2 2 Ethernet Interface The network ports of the ConnectX 3 Pro adapter cards are compliant with the IEEE 802 3 Ethernet standards listed in Table 4 Features on page 12 Ethernet traffic is transmitted through the cards QSFP connectors For connecting to an SFP interface you use Mella nox QSA QSFP to SFP adapter module to be ordered separately 14 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 2 3 PCI Express Interface The ConnectX 3 Pro adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus op
66. ow to install and test the Mellanox InfiniBand OFED Driver for VMware vSphere package on a single host machine with Mellanox InfiniBand hardware installed The InfiniBand OFED driver installation on VMware ESXi Server 5 0 is done using VMware s VIB bundles Please uninstall any previous Mellanox driver packages prior to installing the new ver 2 sion Refer to Uninstalling Mellanox InfiniBand OFED Driver on page 54 To install the driver 1 Log into the ESXi5 0 server with root permissions 2 Install the driver esxcli software vib install d bundle file gt 5a Reboot the machine B Verify the driver was installed successfully i esxcli software vib list grep Mellanox net ib core 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib ipoib 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib mad 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib sa 1 8 1 OEM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net ib umad 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net mlx4 core 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 net mlx4 ib 1 8 1 0EM 500 0 0 472560 ellanox PartnerSupported 2013 02 09 Mellanox Technologies 53 1 3 Driver Installation 4 4 2 4 5 After the installation process all kernel modules are loaded automatically upon boot Uninstalling Mellanox InfiniBand OF
67. place When installing cables make sure that the latches engage Ah Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card 3 After inserting a cable into a port the Green LED indicator will light when the physical con nection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port See Section 7 3 Adapter LED Operation on page 63 4 After plugging in a cable lock the connector using the latching mechanism particular to the cable vendor When a logical connection is made the Yellow LED will light When data is being transferred the yellow led will blink See Section 7 3 Adapter LED Operation on page 63 Mellanox Technologies 19 1 3 Hardware Installation 5 Care should be taken as not to impede the air exhaust flow through the ventilation holes Use cable lengths which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 6 To remove a cable disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated a Pullon the latch release mechanism thereby unlatching the connector and pull the connector out of the cage b Do not apply torque to the connector when removing it from the adapter card c Remove any cable s
68. rd Completed The InstallShield Wizard has successfully installed MLNX Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin glog Cl Show release notes Cancel 46 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 If the firmware upgrade fails the following message will be displayed InstallShield Wizard Completed The InstallShield Wizard has successfully installed VPI Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin g log Firmware upgrade Failed with error 1003 Error Description We could not Find a new version For the chosen device Please refer to the UM for Show release notes 4 2 3 2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter Step 2 Install the driver Run gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn Step 3 Optional To configure your setup to contain the logs option please run the following com mand gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v l vx LogFile Step 4 Optional If you do not want to upgrade your firmware version run the following command
69. rmance for latency sen sitive applications Support for port based Quality of Service enabling various application require Quahty ments for latency SLA Hardware based I O ConnectX 3 Pro provides dedicated adapter resourcesand guaranteed isolation virtualization and protection for virtual machines within the server ConnectX 3 Pro SR IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines VM within the server I O virtualization with ConnectX 3 Pro gives data center managers better server utilization while reducing cost power and cable complexity SR IOV A consolidated compute and storage network achieves significant cost perfor mance advantages over multi fabric networks Standard block and file access protocols can leverage InfiniBand RDMA for high performance stor age access Storage Acceleration 1 3 Operating Systems Distributions Citrix XenServer 6 1 RHEL CentOS 5 X and 6 X Novell SLES10 SP4 e SLESII SPI SLES 11 SP2 OEL Fedora 14 15 17 Ubuntu 12 04 Windows Server 2008 2012 FreeBSD OpenFabrics Enterprise Distribution OFED OpenFabrics Windows Distribution WinOF VMware ESXi 4 x and 5 x 14 Connectivity Interoperable with 10 40 switches Interoperable with 56GbE Mellanox switches Passive copper cable with ESD protection Powered connectors for optical and active cable support to SFP c
70. rners of the card insert the adapter card into the slot until it is firmly seated When the adapter is properly seated the adapter port connectors are aligned with the slot opening and the adapter faceplate is visible against the system chassis 18 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Do not use excessive force when seating the card as this may damage the system the adapter 5 Secure the adapter with the adapter clip or screw 6 Close the system case 3 6 Cables and Modules To obtain the list of supported cables for your adapter please refer to www mellanox com gt Prod ucts gt Cables and Transceivers 3 6 1 Cable Installation 1 All cables can be inserted or removed with the unit powered on 2 To insert a cable press the connector into the port receptacle until the connector is firmly seated a Support the weight of the cable before connecting the cable to the adapter card Do this by using a cable holder or tying the cable to the rack b Determine the correct orientation of the connector to the card before inserting the connector Do not try and insert the connector upside down This may damage the adapter card c Insert the connector into the adapter card Be careful to insert the connector straight into the cage Do not apply any torque up or down to the connector cage in the adapter card d Make sure that the connector locks in
71. s ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 List of Figures Figure 1 MCX354A FCCT Card 14 Fisure 2 Bracket SCTeWS 2 n las dee i VERI don DEL au Up ett 17 Figure3 Placing the Bracket on the Card 18 Figure 4 PCI Device 222222252225 SER Di hs oe ui 20 Figure 5 Support Download Assistant 56 Figure 6 Mechanical Drawing of the Single portMCX353A FCCT Adapter Card 64 Figure 7 Mechanical Drawing of the Dual port MCX354A FCCT Adapter Card 65 Figure 8 Dual port 66 Figure 9 Single port Bracket 67 Figure 10 Connector and Cage Views 68 Figure 11 PCIe Connector 70 Figure 12 Compatible Connector Plug and Pinout 70 Figure 13 Card Product Label 71 Mellanox Technologies 7 Rev 1 3 Revision History This document was printed on May 28 2014 Table 1 Revision History Table Date Rev Comments Changes 2014 1 3 Added note on bracket replacement See Section 3 4 Br
72. s de actividad de 4 rayos 4 Conexi n y desconexi n del cable Copper Dado que los cables de cobre son pesados y no son flexibles su conexi n a los conec LA tores y su desconexi n se deben efectuar con mucho cuidado Para ver advertencias o instrucciones especiales consultar al fabricante del cable 5 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nica mente de personal capacitado y competente 6 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 7 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme a los c digos el ctricos locales y nacionales 8 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos 78 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 79
73. the Export option from the File menu Assign a file name and save Environment Information Download and install http www mellanox com con tent pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d lt mst_device gt q Ports Information vstat Download the latest firmware version using the PSID board Firmware Version ID Upgrade http www mellanox com supportdownloader flint d mst device firmware bin file b Eventlog viewer MST device logs mst start Collect log file mst status CNUsers Administrator flint d mst device dc gt dump configuration log e CAUsers Administrator mstdump mst device dc gt mstdump log Mellanox Technologies 59 Rev 1 3 Specifications 7 Specifications 71 MCX353A FCCT Specifications Table 9 MCX354A FCCT Specifications Table Size 2 12in x5 6 in 53 59mm x 142 25 mm Physical Connector QSFP InfiniBand Copper and optical InfiniBand IBTA v1 2 1 Auto Negotiation 1X 2X 4X SDR 2 5Gb s per lane DDR 5Gb s per lane QDR 10Gb s per lane FDR10 10 3125Gb s per lane FDR 14 0625Gb s per lane port Ethernet 10GBASE CX4 10GBASE R and 1000BASE R 40GBASE R4 Pro
74. tocol Support Data Rate Up to 56Gb s FDR InfiniBand 1 10 40 56Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 12V 3 3V Typ Power Passive Cables 6 87W Max Power Passive Cables 7 30W Power and Max power available through QSFP port 1 5W Environmental Temperature Operational 0 C to 55 C Non operational 0 to 70 Humidity 90 relative humidity Air Flow 200LFM EMC Refer to the following link www mellanox com related docs user manuals Regulatory and Compliance Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a The ConnectX 3 Pro adapters supplement the IBTA auto negotiation specification to get better bit error rates and longer cable reaches This supplemental feature only initiates when connected to another Mella nox InfiniBand product 60 Mellanox Technologies ConnectX 3 Pro VPI Single Dual QSFP Port Adapter Card User Manual Rev 1 3 b For both operational and non operational states c Air flow is measured 1 from the heat sink between the heat sink and the cooling air inlet Mellanox Technologies 61 Rev 1 3 Specifications 7 2 54 Specifications Table 10 MCX354A FCCT Specifications Table Size 2 711 x5 6
75. u rayonnement grave PRODUIT LASER DE CLASSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 1 1996 A2 2001 Mellanox Technologies 75 Rev 1 3 Appendix E Sicherheitshinweise Warnings German l Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromversorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der 7 maximal empfohlenen Temperatur von 55 131 F betrieben werden Es ist ein Luft strom von 200 bei maximaler Umgebungstemperatur erforderlich Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ftungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten 3 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab 4 Anschliefen Trennen von Kupferkabel Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die LA Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnun gen und Anleitungen des Kabelherstellers 5 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausge tauscht oder gewartet werden 6 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze B
76. upports that were used to support the cable s weight 3 7 Identify the Card in Your System 3 7 1 Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 Pro adapter card 3 Right click the mouse on your adapter s row and select properties to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 or Hardware Ids Windows 2008 R2 from the Properties pull down menu Note Figure 4 is an example using ConnectX 3 device This figure will be updated in future releases Figure 4 PCI Device General Pott Protocol Driver Details Events Resources ee ret gt Mellanox Connect lt 3 VPI MT04099 Network Adapter Property Hardware Ids v Value PEIWEN 15 3 100355UBSYS 005015B3 amp REV 00 PCIMWEN_15B38DEV_1003 SUBSYS_00501583 PCIMVEN_15B3 amp DEV_1003 amp CC_028000 PCINVEN 15B3 amp DEV 1003 amp CC 0280 1 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string _ 15 amp 1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID 20 Mellanox Technologies ConnectX
77. ver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com Download this package from the Mellanox web site at http www mellanox com gt Products gt Software gt Infini Band VPI Drivers gt Windows SW Driver gt Download Follow the installation instructions included in the download package also available from the download page Hardware and Software Requirements Table 8 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Installer Privileges The installation requires administrator privileges on the target machine Downloading MLNX_WinOF Follow these steps to download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click start gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURE On an x64 64 bit machine the output will be AMD64 Step 2 Go to the MLNX_WinOF for Windows Web page at http www mellanox com gt Products gt Software gt InfiniBand VPI Drivers gt Windows SW Drivers 38 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Step 3 Download the exe image according to the architecture of your m
78. will be updated accordingly However if the user has both OEM card and Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the 5 firmware version Recommended Upgrading the Firmware version will reboot your machine Note One or more of your adapters has an old firmware version We recommend upgrading to a newer Firmware version to enable improved functionality and support driver s capabilities InstallShield Mellanox Technologies 43 Rev 1 3 Driver Installation Step 5 Configure your system for maximum performance by checking maximum performance box Maximum Performance Check this box to configure your system for maximum performa Y Check this box to configure your system for maximum performance Recommended Note This step requires you to reboot the machine at the end of the installation process InstallShield Step 6 Select a Complete or Custom installation follow Step a on page 45 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete All program features will be installed Requires the most disk space Choose which program features you want installed and where they will be installed Recommended For advanced users InstallShield 44 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 Select the desired feature to install
79. y ConnectX 3 Link Width 8x PCI Link Speed 5Gb s Installation finished successfully The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci cr0 2 11 500 is up to date Note To force firmware update use force fw update flag In case your machine has the latest firmware no firmware update will occur and the installation script will print at the end of installation a message similar to the following The firmware version on dev mst mt26448 pci cr0 2 9 1000 is up to date Note To force firmware update use force fw update flag The firmware version on dev mst mt4099 pci 2 11 500 is up to date Note To force firmware update use force fw update flag In case your machine has an unsupported network adapter device no firmware update will occur and the error message below will be printed Please contact your hardware vendor for help on firmware updates Error message I Querying device E Can t auto detect fw configuration file In case the installation script performed firmware updates to your network adapter hardware it will ask you to reboot your machine The script adds the following lines to etc security limits conf for the userspace com ponents such as MPI soft memlock unlimited hard memlock unlimited These settings unlimit the amount of memory that can be pinne
80. zip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully gt flint d dev mst mt4103 pci Image type ConnectX 3 Pro FW Version 2 9 4100 Device ID 4103 56 Mellanox Technologies ConnectX 3 Pro VPI Single and Dual QSFP Port Adapter Card User Manual Rev 1 3 6 6 1 Mellanox Technologies 57 Troubleshooting General Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is Ensure that adapter driver s is loaded Try another port on the switch Make sure the cable is securely attached

User Manual - Boston Server & Storage Solutions

Contents

Download Pdf Manuals

Related Search

Related Contents

User Manual - Boston Server &amp; Storage Solutions

Contents

Download Pdf Manuals

Related Search

Related Contents

User Manual - Boston Server & Storage Solutions