Home

ConnectX-3 10GbE User Manual for OCP

1. 47 Figure 10 Myl r on Board 0 oh op REL ee oa ea Sea ad 47 Figure 11 Bezel Assembly Front View 49 Figure 12 Bezel Assembly Back View 50 Figure 13 MCX341A XCCN Board Label Example only see 52 Figure 14 MCX341A XCDN Board Label Example only see Note 53 Figure 15 MCX342A XCCN Board Label Example only see 53 Figure 16 MCX342A XCDN Board Label Example only see Note 54 Figure 17 MCX341A XCEN Board Label Example only see Note 54 Figure 18 MCX342A XCEN Board Label Example only see Note 55 Figtire 19 SEP Cage Pinot note aeter Buca ter Sea a 57 Figure 20 PCIe Connector 58 Figure 21 Compatible Connector Plug and Pinout 59 Figure 22 NC SI Interface 59 Mellanox Technologies 5 Rev 1 8 List of Tables Table 1 Revision History 2 7 Table 2 Documents Listin ne IL 9 Table 3 Single port 10 Gigabit Ethernet Adapter Cards 11 Table 4 Dual port 10 Gigabit Etherne
2. Online Resources Mellanox Technologies web pages http www mellanox com Mellanox Technologies Firmware download web page http www mellanox com gt Support gt Download Center Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b indicates size in mega bits Technical Support Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods URL http www mellanox com gt Support E mail support mellanox com Tel 1 408 916 0055 Customers who purchased Mellanox M 1 Global Support Services please see your contract for details regarding Technical Support Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller Firmware and Software Updates The Mellanox support downloader contains software firmware and knowledge database informa tion for Mellanox products Access the data base from the Mellanox Support web page http www mellanox com gt Support or use the following link to go directly to the Mellanox Support Download Assistant page http www mellanox com supportdownloader 10 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 1 Introduction This is the User Guide for Mellanox Technol
3. 57 D PinoUlc rede ene e PROP Moi Sex AU RON 57 D 2 PCI Express x8 Connector Pinout 57 D 3 I2C compatible Connector 2 2 59 DA NGC SIInterface Pinout dedere e ERA 59 Appendix E Safety Warnings 60 Appendix F Avertissements de s curit d installation Warnings in French 62 Appendix Sicherheitshinweise Warnings in German 64 Appendix H Advertencias de seguridad para la instalaci n Warnings in Spanish 66 4 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 List of Figures Figure 1 Connector on Motherboard 17 Figure 2 Connector on a en OR ate uei 17 Ligure Standolls nr 2er aa E DECRE 18 Lipure4 Card Installed 2 322282 hoo le Ried whe ar I La eee 19 Ergure5 PCI Device timida Senke Pata 20 Figure Support Download Assistant 39 Figure 7 Mechanical Drawing of the Single port MCX341A Mezzanine Card 45 Figure 8 Mechanical Drawing of the Dual port MCX342A Mezzanine Card 46 Figure 9 Mechanical Drawing of Mylar
4. on page 52 Added note to Section 3 4 Installation Instructions on page 16 Added Appendix A Bezel Assembly Optional on page 49 Added Cable Support to Table 8 MCX341A Specifications Table on page 43 September 2012 1 0 First Release 8 Mellanox Technologies 8 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 About this Manual This User Manual describes Mellanox Technologies ConnectX 3 10 Gigabit Ethernet Single and Dual SFP Port PCI Express x8 Mezzanine cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant docu mentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Ethernet networks and architecture specifications Related Documentation Table 2 Documents List Mellanox Firmware Tools MFT User Manual Document no 2204UG User Manual describing the set of MFT firmware management tools for a single node See http www mellanox com gt Products gt Software gt Firm ware Tools MLNX_EN for Linux README Driver Kit for Mellanox Adapter Cards with 10GigE Support Document no 2950 This document provides information on the MLNX_EN Linux driver and instructions for installing the driver on Mellanox Con nectX adapter cards sup
5. AE EE EET OR UOI UO OA Figure 15 MCX342A XCCN Board Label Example only see Note Model No CX342A ConnectX 3 EN 10 Gigabit Ethernet Made in Israe 111 S N MTYYWWSZZZZZ P N MCX342A XCCN Rev A1 HHHHH MM MAC HHHHHHHHHHHH ll Mellanox Technologies 53 Rev 1 8 Figure 16 MCX342A XCDN Board Label Example only see Note Model No CX342A ConnectX 3 EN 10 Gigabit Ethernet Made in Israe AM A vs S N MTYYWWSZZZZZ P N MCX342A XCDN Rev A1 SE HHHHHH TEE DU iii Figure 17 MCX341A XCEN Board Label Example only see Note Model No CX341A ConnectX 3 EN 10 Gigabit Ethernet Made in Israel S N MTYYWWXSSSSS PIN MCX341A XCEN Rev A6 MAC WWWWWWWWWWWW INNEN IN 00 0 0000 IN PO MAC WWWWWWWWWWWW MEAE UO 54 Mellanox Technologies 8 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Figure 18 MCX342A XCEN Board Label Example only see Note Model No CX342A ConnectX 3 EN 10 Gigabit Ethernet Made in Israel LAN NN S N MTYYWWXSSSSS P N MCX342A XCEN Rev A1 MAC HHHHHHHHHHHH HHHHHHHHHHHH P1 C 1 Retrieving Adapter Card GUID On ConnectX 3 Ethernet adapter cards there is a mismatch between the GUID value returned by firmware management tools a
6. MU 16 3 3 Pre installation Checklist 16 3 4 Card Installation Instructions 16 3 5 CablesandModules 19 3 6 Identify the Card in Your System 19 3 6 1 Windows see eee rc pee eas Tee ee ERR ne dette 19 3 6 2 ON LINUX u EE Ute een co d pe WE NEUES 20 Chapter 4 Driver 21 Linux Driver pe au tl rl tp eb es A doa E dl 21 4 1 1 Hardware and Software Requirements 21 4 1 2 Install th Driyer donnant ur RE RE duree dre ew qnd e 21 4 1 3 Loading the Driver 2 22 4 1 4 Unloading the Driver 22 4 1 5 Uninstalling the Driver 22 42 Windows Driver etnias EA ee e ger aa 22 4 2 1 Hardware and Software Requirements 23 4 2 2 Downloading MLNX_WinOF 2 23 4 2 3 Extracting Files Without Running Installation 23 4 2 4 Installing MLNX Wi
7. lt mst_device gt 4 ibstat Ports Information iby done Firmware Version To download the latest firmware version refer to Upgrade http www mellanox com supportdownloader Collect Log File var log messages dmesg gt system logF Mellanox Technologies 41 Rev 1 8 Troubleshooting 6 3 Windows From the Windows desktop choose the Start menu and msinfo32 Environment To export system information to a text file choose the Export option Information from the File menu Assign a file name and save Download and install MFT http www mellanox com content pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmwar Once installed open a CMD window and run cd C Program Files Mellanox WinMFT mst start mst status flint d lt mst_device gt 4 Ports Information vstat Download the latest firmware version using the PSID board ID Firmware Version cyte Mersio http www mellanox com supportdownloader Upgrade flint d lt mst_device gt i firmware bin file b Eventlog viewer MST device logs mst start Collect log file mst status C Users Administrator gt flint d lt mst_device gt dc gt dump configuration log e CAUsers Administrator mstdump mst device dc gt mstdump log 42 Mellanox Technologi
8. D E N 10GigE MCX342A XCCN dual port SFP IPMI disabled MCX342A XCDN dual port SFP IPMI enabled MCX342A XCEN dual port SFP with IPMI and UEFI IPv6 support PCIe 3 0 x8 8GT s R6 MT27518A1 FCCR XE 4099 for Physical Function 4100 for Virtual Function 12 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 1 2 Mellanox Technologies 13 Features and Benefits Table 5 Features PCI Express PCIe 10 Gigabit Ethernet Memory RDMA over Ethernet RoE CPU offload Sockets Acceleration Quality of Service QoS Hardware based I O virtualization SR IOV IPMI support Wake on LAN WoL Reset on LAN RoL NC SI Uses PCIe Gen 3 0 1 1 and 2 0 compatible through an x8 edge connector up to 8GT s Mellanox adapters comply with the following IEEE 802 3 standards IEEE Std 802 3 2008 Ethernet IEEE Std 802 3ae 10 Gigabit Ethernet IEEE Std 802 3ad Link Aggregation and Failover PCI Express stores and accesses InfiniBand and or Ethernet fabric connection information and packet data SPI includes one 16Mb SPI Flash device M25PX16 VMN6P device by ST Microelectronics EEPROM accessible through the I C compatible interface The EEPROM capacity is 4Kb Leveraging Data Center Bridging capabilities RoE provides efficient low latency RDMA services over Layer 2 Ethernet Adapter functionality enabling reduced CPU overhead allowing
9. MCX341A XCCN 1 FW changes to 2 14 1250 IPMI EN B 4 Changes in MCX341A XCEN from MCX341A XCDN 1 FW changes to 2 14 1660 2 Added protective Mylar attached to the card see Section 7 3 1 Mylar Mechanical Drawing and Dimensions on page 47 Mellanox Technologies 51 Rev 1 8 Appendix C Finding the MAC GUID and Serial Number on the Mezzanine Card Each Mellanox Mezzanine card has a label on the print side that shows the card serial number and the card MAC for Ethernet protocol Note The revision indicated on the labels in the following figures do not necessarily represent the latest revision of the card Card revision changes are communicated via Product Change Notification PCN documents that are available via card suppliers Figure 13 MCX341A XCCN Board Label Example only see Note Model No CX341A ConnectX 3 EN 10 Gigabit Ethernet Made in Israel muss SIN MTYYWWXSSSSS P N MCX341A XCCN Rev ME MAC HHHHHHHHHHHH mac MMMMMMMMMM N 52 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Figure 14 MCX341A XCDN Board Label Example only see Note Model No CX341A ConnectX 3 EN 10 Gigabit Ethernet Made in Israel S N MT1210X00011 PIN MCX341A XCDN Rev A3 ME MAC VWWWWWWWWWWWW DAL OCONI ONI UR OI UOL UNI OO OO URL OO OA II UOTI MAC
10. MEZZ TX DP C 2 MEZZ TX DN C 2 GND 40 87 cn 521 L C 3 RX MEZZ RX DN 3 GND 17 GND 18 EX DD 4 MEZZ RX DN 4 cts am 15 CIS ZND 20 EX DD S RX DN S TX DP C S GND 21 74 TX DN C 6 GND 22 CND 48 EX DD S GND 45 RX DN 6 MEZZ TX DP C 7 2 MEZZ TX DN C 7 GND 24 GCND 50 RX DD 7 51 EX DN 7 CND 52 MEZZ DRENT2 N GND 25 PCle Connector 58 Mellanox Technologies Rev 1 8 ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP 0 3 I C compatible Connector Pinout Figure 21 Compatible Connector Plug Pinout Table 11 I2C compatible Connector Pinout Connector Pin Number Signal Name 1 SPSDA 123 2 SPSCL 3 GND D 4 NC SI Interface Pinout Figure 22 NC SI Interface Pinout NCSI_TXD 1 0 Rx 1 0 NCSI_RXD 1 0 4 Po 20 Management Controller ConnectX 3 Pro NCSI TX EN CRS DV NCSI CRS DV TX EN Mellanox Technologies 59 Rev 1 8 Appendix E Safety Warnings For safety warnings in French see Avertissements de s curit d installation Warnings in French on page 34 For safety warnings in German see Sicherheitshinweise Warnings in German on page 36 For safety warnings in Spanish see Advertencias de seguridad para la instalaci n
11. Reinstall the drivers for the network driver files may be damaged or deleted Reboot the server Adapters stopped working after installing another adapter Try removing and re installing all adapters Check that cables are connected properly Make sure your motherboard has the latest BIOS Link indicator light is off Ensure that adapter driver s is loaded Try another port on the switch Make sure the cable is securely attached Check your are using the proper cables that do not exceed the rec ommended lengths Verify that your switch and adapter port are compatible Link light is on but with no communication established Check that the latest driver is loaded Check that both the adapter and its link are set to the same speed and duplex settings ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 6 2 Linux cat etc issue uname a cat proc cupinfo grep model name uniq Environment ofed_info head 1 Information ifconfig ethtool lt interface gt ethtool i interface of Mellanox port num ibdev2netdev Card Detection Ispci grep Mellanox Download and install http www mellanox com content pages php pg management_tools amp menu_section 34 Refer to the User Manual for installation instructions Mellanox Firmware Tool MFT Once installed run mst start mst status flint d
12. SR 10GBASE LR and 1OGBASE CX Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0GT s 8 lanes 2 0 and 1 1 compatible Voltage 5V 3 3V Typ Power Passive Cables 5 09W Max Power Passive Cables 6 38W Power and Max power available through SFP port 1W Environmental Temperature Operational 0 C to 45 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 300LFM EMC Refer to the following link www mellanox com related docs user_manuals Regulatory_and_Compliance_Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 TEC 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from the Mezz from the cooling air inlet side 44 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 Board Mechanical Drawing and Dimensions dimensions are in millimeters All the mechanical tolerances 0 1mm Figure 7 Mechanical Drawing of the Single port MCX341A Mezzanine Card 110 05 ee 42 05 68 I2C connector Mellanox Technologies 45 Rev 1 8 Specifications Figure 8 Mechanical Drawing of the Dual port MCX342A Mezzanine Card 1
13. Warn ings in Spanish on page 38 1 Installation Instructions LA Read all installation instructions before connecting the equipment to the power source 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended 55 C 131 F To guarantee proper air flow allow at least 8cm 3 inches of clearance around the ven tilation openings 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or dis connect cables 4 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regula LA tions 5 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical LA codes 6 Hazardous Radiation Exposure Caution Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation exposure 60 Mellanox Technologies ConnectX 3 Ethernet Single Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 CLASS 1 LASER PRODUCT and reference to the most recent laser standards IEC 60 825 1 1993 A1 1997 2 2001 and EN 60825 1 1994 1 1996 A2 2001 Mellanox Technologies 61 Rev 1 8 Appendix F Avertissements de s curit d installation Warnings in French 1 Instructions d installation Lisez toutes les instruc
14. and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 a Update the driver Run gt esxcli software vib update n net mlx4 en d lt path gt mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4 en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by MLX4_EN driver should be displayed um In Non Multifunction Mode port 2 is identified as a pseudo device Therefore devices gt are not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 9 PIS 4 3 3 Removing the VIB offline bundle Driver remove the VIB offline bundle driver package from the ES Xi server machine run gt esxcli software vib remove n net mlx4 en 4 4 FlexBoot FlexBoot supports remote Boot over Ethernet This technology is based on the Preboot Execution Environment PXE standard specification and FlexBoot software is based on the open source iPXE project see www ipxe org For more information go to http www mellanox com gt Products gt Software gt Ethernet Drivers gt Download Mel
15. front bezel to protect against improper insertions and extractions of cables The bezel design should be in compliance with section Bezel Design for Systems Using SFP Transceivers in the MSA INF 8074i Specification for SFP Small Formfactor Pluggable Transceiver The bezel should be designed with the following features Cutouts should allow proper mounting of the cage assembly Dimensions of bezel cutouts and distance between them should be in compliance with MSA specification Insertion of bezel should compress the EMI ground springs of the cage assembly panel The bezel should not disturb the functioning of the locking latch when inserting cables into the cage P The bezel will cause the SFP cage to be slanted with respect to the board Figure 11 Bezel Assembly Front View locking latch bezel Mellanox Technologies 49 Rev 1 8 Figure 12 Bezel Assembly Back View EMI springs Slant below cage 50 Mellanox Technologies ConnectXG 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Appendix Card Change History B 1 Changes in Rev A2 from Rev A1 MCX341A XCCN 1 Added soldering instructions for 9 TH legs SFP cage 2 Board label change to the latest format 3 Major revision change to A2 B 2 Changes in Rev A3 from Rev A2 MCX341A XCCN 1 FW change to 2 10 3898 IPMI DIS 2 Major revision change to A3 B 3 Changes in MCX341A XCDN from
16. more available CPU Applications utilizing TCP UDP IP transport can achieve industry leading throughput over InfiniBand or 10GbE The hardware based stateless offload engines in ConnectX 3 reduce the CPU overhead of IP packet transport Sockets acceleration software further increases performance for latency sensitive applica tions Support for port based Quality of Service enabling various application require ments for latency and SLA ConnectX 3 provides dedicated adapter resources and guaranteed isolation and protection for virtual machines within the server ConnectX 3 SR IOV technology provides dedicated adapter resources and guar anteed isolation and protection for virtual machines VM within the server I O virtualization with ConnectX 3 gives data center managers better server utiliza tion while reducing cost power and cable complexity Enabled for the following MCX341A XCDN MCX342A XCDN MCX341A XCEN MCX342A XCEN To be updated in future release To be updated in future release The adapter supports a slave Network Controller Sideband Interface NC SI that can be connected to a BMC Enabled for MCX341A XCGN and MCX342A XCGN Rev 1 8 Introduction Table 5 Features Unified Extensible See Section 2 5 Unified Extensible Firmware Interface UEFD on Firmware Interface page 15 UEFI 1 3 Operating Systems Distributions Novell SLES Red Hat Enterprise Linux RHEL Fedora CentOS and other Linux d
17. www mellanox com gt Products gt Software gt Ethernet Driver gt Linux Driver gt Download Step 2 Install Driver Run the following commands to install the driver zv minsen gt cd 1 5 10 gt install sh The package consists of several source RPMs The install script rebuilds the source RPMs and then installs the created binary RPMs The created kernel module binaries are placed under lib Mellanox Technologies 21 Rev 1 8 Driver Installation 4 2 modules lt kernel ver gt updates kernel drivers net mlx4 mlnx en installer supports 2 modes of installation The install scripts selects the mode of driver installation depending of the running OS kernel version 1 Kernel Module Packaging KMP mode where the source rpm is rebuilt for each installed flavor of the kernel This mode is used for RedHat and SUSE distributions 2 Non KMP installation mode where the sources are rebuilt with the running kernel This mode is used for vanilla kernels Note If the Vanilla kernel is installed as rpm please use the disable kmp flag when installing the driver The kernel module sources are placed under usr src mellanox mInx en 1 5 10 Run the follow ing commands to recompile the driver gt cd usr src mellanox mlnx en 1 5 10 4 scripts mlnx en patch sh d make d make install The uninstall and performance tuning scripts are installed Note If the driver was in
18. 16 4 gt E 110 5 gt au zn e 3 SEE E El 42 05 e p ox t 68 e aie a 46 Mellanox Technologies 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 31 Mylar Mechanical Drawing and Dimensions Figure 9 Mechanical Drawing of Mylar 65 17 a 62 20 5 77 17 _ Figure 10 Mylar on Board Mellanox Technologies 47 Rev 1 8 Specifications 7 4 Adapter LED Operation There are two I O LEDs per port See Table 10 for different LED functions Table 10 Physical and Logical Link Indication LED Function Green physical link Constant on indicates a good physical link If neither LED is lit then the physical link has not been established Yellow logical data blinking yellow indicates activity data transfer activity link e Stays off when there is no activity The short bracket has the same port and LED footprints as the tall bracket 7 5 Regulatory Statements For regulatory statements for all ConnectX 3 cards please refer to http www mellanox com related docs user manuals Regulatory and Compliance Guide pdf 48 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Appendix A Bezel Assembly Optional It is recommended to support the cards SFP cages with a
19. ANN Mellanox TECHNOLOGIES ConnectX 3 Ethernet Single and Dual SFP Port Mezzanine Card User Manual for Open Compute Project P N MCX341A XCCN MC342A XCCN MCX341A XCDN MCX342A XCDN MCX341A XCEN MCX342A XCEN Rev 1 8 www mellanox com Rev 1 8 THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWIS
20. E ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2014 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX Connect IB CORE Direct InfiniBridge InfiniHost InfiniScale MetroX MLNX OS PhyX ScalableHPC SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroDX TestX Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 4026 ConnectXG 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Table of Contents Table of Contents rev sr D EXC DW EV nm ate e a es List of Figures corio A eh DEREN dhe eM EX ars UE List of Lables senan e e i u R vision History eese csi nr een eo ere en
21. GRD 1 For further help please run gt MLNX VPI WinOF 4 40 0 All win8 x64 exe v h 4 2 5 Upgrading MLNX WinOF The MLNX WinOF driver upgrades automatically MLNX_WinOF Windows 2008R2 driver by uninstalling the previous version and installs the new driver However MLNX_WinOF driver upgrade in Windows 2012 driver do not completely uninstall the previous version In Windows 2012 MLNX_WinOF Rev 4 2 and above the network configuration is saved upon driver upgrade In Windows 2008 R2 the existing configuration files are not saved upon driver upgrade 4 2 6 Installation Results Upon installation completion you can verify the successful addition of the network card s through the Device Manager To see the Mellanox network adapter device and the Ethernet or IPoIB network device depending on the used card for each port display the Device Manager and expand System devices or Network adapters 32 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 File Action View Help 3 amp Mellanox ConnectX 3 Ethernet Adapter 2 Microsoft Kernel Debug Network Adapter 4 Other devices Unknown device Ports amp LPT p Print queues gt Processors b lt gt Storage controllers 4 j System devices JM ACPI Fixed Feature Button Composite Bus Enumerator Wi Direct memory access controller Hig
22. SM upon setup results in it working only for the first port and not for the others Torun OpenSM as a service assuming the package was installed in the default path use Sc create OpenSM1 binPath c Program Files Mellanox MLNX VPI IB Tools opensm exe service start auto gt To start the service run sc start opensm For further information please refer to the OpenSM Subnet Manager chapter in the User Manual Mellanox Technologies 33 Rev 1 8 Driver Installation 4 2 8 Uninstalling MLNX_WinOF 4 2 8 1 Attended Uninstall gt To uninstall MLNX_WinOF single node perform one of the following options 1 Click Start gt Control Panel gt Programs and Features gt MLNX_VPI gt Uninstall NOTE This requires elevated administrator privileges 2 Double click the exe and follow the instructions of the install wizard 3 Click Start gt All Programs gt Mellanox Technologies gt MLNX_WinOF gt Uninstall MLNX_WinOF 4 2 8 2 Unattended Uninstall gt To uninstall MLNX_WinOF in unattended mode perform the following Step 1 Open a CMD console Step 2 Uninstall the driver Run MLNX VPI WinOF 4 40 0 11 win8 x64 exe S x v qn 4 2 9 Assigning Port IP After Installation By default your machine is configured to obtain an automatic IP address via a DHCP server In some cases the DHCP server may require the MAC address of the network adapter installed in your machine To obtain t
23. SSE 1 et r f rences aux normes laser les plus r centes CEI 60 825 1 1993 A1 1997 A2 2001 et NE 60825 1 1994 1 1996 A2 2001 Mellanox Technologies 63 Rev 1 8 Appendix Sicherheitshinweise Warnings German 1 Installationsanleitungen gt 2 bertemperatur 3 Bei Gewitter Elektrische Gefahr a 4 Ger teinstallation Esama 5 Ger teentsorgung iE 6 Regionale und nationale elektrische Bestimmungen A ssemma 64 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 Strahlenkontak Achtung Nutzung von Steuerungen oder Einstellungen oder Ausf hrung von Prozeduren die hier nicht spezifiziert sind kann zu gefahrlichem Strahlenkon takt f hren Klasse 1 Laserprodukt und Referenzen zu den aktuellsten Lasterstandards ICE 60 825 1 1993 A1 1997 A2 2001 und EN 60825 1 1994 A1 1996 A2 2001 Mellanox Technologies 65 Rev 1 8 Appendix H Advertencias de seguridad para la instalacion Warnings in Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 55 C 131 F Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pu
24. Technologies MT27500 Family ConnectX 3 20 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 4 Driver Installation 4 1 Linux Driver For Linux download and install the latest MLNX_EN driver software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt ConnectX 3 EN 10GigE Linux Driver gt Download Follow the installation instructions included in the download package also available from the download page 4 1 4 Hardware and Software Requirements Table 6 Software and Hardware Requirements Requirements Description Platforms CPU architectures x86_64 x86 e Device ID For the latest list of device IDs please visit http pci ids ucw cz read PC 15b3 Operating System Linux Operating Systems RedHat EL5 8 RedHat EL5 9 RedHat EL6 2 RedHat EL6 3 OEL62 2 6 32 279 19 1 e OEL6 3 2 6 32 279 19 1 SLESII SPI SLES11 SP2 Software Dependencies To install the driver software kernel sources must be installed on the machine MLNX_EN driver cannot coexist with OFED software on the same machine Hence when installing MLNX_EN all OFED packages should be removed done by the mlInx en install script 4 1 2 Installing the Driver Step 1 Download Driver Package Please download the current driver package from http
25. UR ERU where pene n A U About this Manual eher en sehe nee ne Chapter 1 Introduction as een nee unse nee er 11 Product Overview ee DR nee REP RCM er EIE 11 1 2 Features and Benefits dea dure Gece adn een ae ARA ota 13 1 3 Operating Systems Distributions 14 ERA O De e e t e e e ae eoe dee ee gel Bere 14 Chapter 2 Interfaces in RU ax amer I Ea sure eese ee AS 247 Bthernet SEP Interface ocd cus ves dan eere ARMUT eie ae EAE 15 2 2 PCI Express Intetface uo I DROPS DU Hr ORI E RE QR ENDE 15 23 I2C compatible 7 2 2 15 24 ED Interface u see age Shay So Ave peser 15 2 5 Unified Extensible Firmware Interface 15 Chapter 3 Hardware Installation 16 3 1 System Requirements 42 53 2 2 233 21a 1a Me een ET CEDAT br OE DR Ye 16 3 121 Hardware 32 a ee ae Mie Ee DS ed e dee deed e edes een 16 3 1 2 Operating Systems Distributions 16 33173 Software Stacks coe nee dus rere ab NETT 16 32 Safety PIecautions s oos LR dos Lun AE tie Re ee AE
26. del No CX341A BMC MAC us 200 Bl EN 10 Gigabit Eiherne RME L Pp 4 7 PIN MOKB4TA XCON Rev M en e 0026200 89 5 1S a s 4 Made in 0 0026200 8 c 0002080 1F33A im p Push down the card into the connector until it is firmly seated The standoffs must go through the available holes on the card 18 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Figure 4 Card Installed 3 5 Cables and Modules The SFP port can connect to the Ethernet through both direct attach cables and stand alone transceivers with optical cables SFP transceiver modules are available for both SR and LR pro tocols Note In order to support SFP transceiver modules the host must supply 3 3Vaux 3 3V 3 5 or use 3 3V 7 tolerant modules To obtain the list of supported cables for your adapter please refer to Mellanox Products Approved Cable Lists at www mellanox com related docs user_manuals Mellanox_approved_cables pdf 3 6 Identify the Card in Your System 3 6 1 On Windows 1 Open Device Manager on the server Click start gt Run and then enter devmgmt msc 2 Expand System Devices and locate your Mellanox ConnectX 3 adapter card 3 Right click the mouse on your adapter s row and select properti
27. e Added Appendix A Bezel Assembly Optional on page 49 March 28 2014 1 6 e Added Section A 5 on page 46 describing Mylar addition to cards e Updated MX341A XCEN label in Appendix B on page 49 to show Rev A6 November 05 2013 1 5 Updated structure of the User Manual e Added Section 7 2 MCX342A Specifications on page 44 Added Figure 8 Mechanical Drawing of the Dual port MCX342A Mezzanine Card on page 46 e Added Appendix B 4 Changes in MCX341A XCEN from MCX341A XCDN on page 51 Added the following note to Chapter 5 Updating Adapter Card Firmware on page 38 Note The shown versions and or param eter values in the example below may not reflect the latest or actual values for this product and are included here for illustra tion purposes only e Added Chapter 6 Troubleshooting on page 40 e Added Appendix Retrieving Adapter Card GUID on page 55 October 16 2013 1 4 Added MCX342A XCCN MCX342A XCDN MCX341A XCEN and MCX342A XCEN April 17 2013 1 3 Added MCX341A XCDN to the document Added Appendix B 3 Changes in MCX341A XCDN from MCX341A XCCN on page 51 December 11 2012 1 2 Updated Figure 7 Mechanical Drawing of the Single port MCX341A Mezzanine Card on page 45 Mellanox Technologies 7 Rev 1 0 Date Rev Comments Changes December 6 2012 1 1 Updated label in Figure 13 MCX341A XCCN Board Label Example only see Note
28. ect a Family Select a Line Select an OPN Select a PSID Rev Product Support Information Adapter Cards Se Switches 1 If a newer firmware version exists for your adapter card on the Web update the firmware as follows a Download the firmware image zip file from the Support Downloader see Step 2a above b Unzip the firmware image c Burn the firmware image Enter gt flint d dev mst mt4099 pci i binary image burn a Reboot the computer b Enter mst start c Verify that the card firmware was updated successfully flint d dev mst mt4099 pci crO q Image type ConnectX FW Version 2 9 4100 Device ID 4099 Mellanox Technologies 39 Rev 1 8 Troubleshooting 6 6 1 40 Mellanox Technologies Troubleshooting General GUID of ConnectX 3 Ethernet adapter cards Please use the GUID value returned by the fabric driver utilities not Oxfffff See Appendix C 1 Retrieving Adapter Card GUID on page 55 Server unable to find the adapter Ensure that the adapter is placed correctly Make sure the adapter slot and the adapter are compatible Install the adapter in a different PCI Express slot Use the drivers that came with the adapter or download the latest Make sure your motherboard has the latest BIOS Try to reboot the server The adapter no longer works Reseat the adapter in its slot or a different slot if necessary Try using another cable
29. es ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 Specifications 71 MCX341A Specifications Table 8 MCX341A Specifications Table Size 2 68in x 4 3 in 68mm x 110mm Physical Connector SFP 10Gb s Ethernet 1OGBASE SR 10GBASE LR and 10GBASE CX Protocol Support Data Rate 1 10Gb s Ethernet PCI Express Gen3 SERDES 8 0 8lanes 2 0 and 1 1 compatible Voltage 5V 3 3V Typ Power Passive Cables 3 72W Max Power Passive Cables 5 10W Power and Max power available through SFP port 1W Environmental Temperature Operational 0 C to 45 C Non operational 0 C to 70 C Humidity 90 relative humidity Air Flow 3001 EMC Refer to the following link www mellanox com related docs user_manuals Regulatory_and_Compliance_Guide pdf Safety IEC EN 60950 1 2006 ETSI EN 300 019 2 2 60068 2 64 29 32 Regulatory RoHS RoHS R6 Cable Support Please refer to www mellanox com gt Products gt Cables and Transceivers a For both operational and non operational states b Air flow is measured 1 from the Mezz from the cooling air inlet side Mellanox Technologies 43 Rev 1 8 Specifications 7 2 MCX342A Specifications Table 9 MCX342A Specifications Table Size 2 68in x 4 3 in 68mm x 110mm Physical Connector SFP 10Gb s Ethernet 1OGBASE
30. es to display the adapter card properties window 4 Click the Details tab and select Device Instance Id Windows 2003 5 Hardware Ids Windows 2008 R2 from the Properties pull down menu Mellanox Technologies 19 Rev 1 8 Hardware Installation Figure 5 PCI Device General Port Protocol Driver Details Events Resources A Mellanox Connect lt 3 MT 04099 Network Adapter Property Hardware Ids v Value PCINVEN_15B38DEV_1003 amp SUBSYS_005015B3 amp REV_00 PCIVEN 15B3 amp DEV 1003 amp SUBSYS 00501583 PCIVEN 15B3 amp DEV 1003 amp CC 028000 PCINVEN 15B3 amp DEV 1003 amp CC 0280 6 In the Value display box check the fields VEN and DEV fields are separated by amp In the display example above notice the sub string 15B3 amp DEV 1003 VEN is equal to 0x15B3 this is the Vendor ID of Mellanox Technologies and DEV is equal to 1003 this is a valid Mellanox Technologies PCI Device ID If the PCI device does not have a Mellanox adapter ID return to Step 2 to check another device N The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi tory at http pci ids ucw cz read PC 15b3 N 3 6 2 On Linux Get the device location on the PCI bus by running Ispci and locating lines with the string Mella nox Technologies gt Ispci grep i Mellanox 27 00 0 Network controller Mellanox
31. fication and programmed firmware version using the command Note The shown versions and or parameter values in the example below may not reflect the lat est or actual values for this product and are included here for illustration purposes only flint d dev mst mt4099 pci cr0 q Image type ConnectX FW Version 2 30 4260 Device ID 4099 Description Node Porti Port2 Sys image GUIDs ETE EAE YE EAE IE JEJE E YE IE MACs 0002c9305350 010 02 93 0 53 5al VSD PSID MT_1080120023 1 Compare the programmed firmware version with the latest available a Go to Mellanox s web site http www mellanox com supportdownloader See Figure 6 b Enter your card PSID to display the latest firmware file The file name of the binary is composed by combin ing the firmware name the firmware release version and the card part number Note Please contact Mellanox System Support if you cannot find the firmware binary for your adapter card 38 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Figure 6 Support Download Assistant AM Mellanox Mellanox Support Download Assistant Support Index Documentation Login Customer Support Returns Home CLEAR MT_1020110019 PSID or OPN Identifying Adapter Cards PSID Browse for Product Support START OVER Sel
32. h precision event timer Ki Intel R 5520 5500 X58 1 0 Hub Control Status and RAS Registers 3423 Intel R 5520 5500 X58 I O Hub GPIO and Scratch Pad Registers 3422 gl Intel R 5520 5500 X58 I O Hub PCI Express Root Port 1 3408 Intel R 5520 5500 X58 1 0 Hub PCI Express Root Port 3 340A ntel R 5520 5500 X58 1 0 Hub PCI Express Root Port 7 340E Intel R 5520 5500 X58 1 0 Hub System Management Registers 342E Intel R 5520 5500 58 I O Hub to ESI Port 3406 Ml Intel R 82801 PCI Bridge 244E Intel R ICH10 Family PCI Express Root Port 1 3440 Ml Intel R ICH10 Family PCI Express Root Port 5 3448 Intel R ICH10R LPC Interface Controller 3A16 Microsoft ACPI Compliant System Microsoft Generic IPMI Compliant Device Microsoft System Management BIOS Driver Microsoft Virtual Drive Enumerator Numeric data processor PCI Express Root Complex 4 2 7 OpenSM Activation OpenSM is a service required by managed networks in InfiniBand environments and must be activated in one of the machines running on the subnet otherwise the interface link will not come up If the cards are connected to a managed network there is no need to run OpenSM Only one OpenSM should run per subnet In Ethernet interfaces running OpenSM is not required OpenSM does not run as a service during installation as it requires the GUID parameter to decide on which port to work Setting Open
33. he MAC address Step 1 Open a CMD console Step2 Display the MAC address as Physical Address ipconfig all Configuring a static IP is the same for both IPoIB and Ethernet adapters gt To assign a static IP address to a network port after installation Step 1 Open the Network Connections window Locate Local Area Connections with Mellanox devices p Networ gt Search Network Connections 2 Organize He Name Device Name Y Ethernet Broadcom BCMS5709C NetXtreme Il GigE NDIS XJ Ethernet 2 5709 Net Ethernet 3 0 Ethernet 4 U Local Area Connection IBM USB Remote NDIS Network Device Step 2 Right click a Mellanox Local Area Connection and left click Properties Step Select Internet Protocol Version 4 TCP IPv4 from the scroll list and click Properties 34 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Step 4 Select the Use the following IP address radio button and enter the desired IP information General You can get IP settings assigned automatically if your network supports this capability Otherwise you need to ask your network administrator for the appropriate IP settings Obtain an IP address automatically Use the following IP address IP address Subnet mask Default gateway Obtain DNS server address automatically Use the fol
34. ink rate x8 e Auto negotiates to x8 x4 x2 or x1 Support for MSI MSI X mechanisms 2 3 1 C compatible Interface A three pin header on the adapter cards is provided as the I C compatible interface See Figure 7 Mechanical Drawing of the Single port MCX341A Mezzanine Card on page 45 for the loca tion on the board 2 4 LED Interface There are two VO LEDs per port For LED specifications please refer to Section 7 4 Adapter LED Operation on page 48 2 5 Unified Extensible Firmware Interface UEFI The UEFI driver is located on the device option ROM and can be used for network boot for example remote OS installation Mellanox Technologies 15 Rev 1 8 Hardware Installation 3 1 3 1 1 3 2 3 3 3 4 Hardware Installation System Requirements Hardware This card requires a PCI Express connector as specified in the Open Compute Project Intel Moth erboard spec v2 0 Operating Systems Distributions Please refer to Section 1 3 Operating Systems Distributions on page 14 Software Stacks Mellanox OpenFabric software package MLNX_EN for Linux WinOF for Windows and ESX 5 1 for VMware See Chapter 4 Driver Installation Safety Precautions The adapter is being installed in a system that operates with voltages that can be lethal Before opening the case of the system observe the following precautions to avoid injury and prevent damage to system components 1 Remove any
35. is tributions Microsoft Windows Server 2008 CCS 2003 HPC Server 2008 e OpenFabrics Enterprise Distribution OFED e OpenFabrics Windows Distribution WinOF VMware ESX Server 3 5 vSphere 4 0 4 1 1 4 Connectivity Interoperable with 10GigE switches SFP connectors e Passive copper cable and Optical Modules support Powered connectors for optical and active cable support 14 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 2 Interfaces Each adapter card includes the following interfaces e Ethernet SFP Interface PCI Express Interface e 2C compatible Interface LED Interface e Unified Extensible Firmware Interface UEFI Hardware Installation 2 1 Ethernet SFP Interface The network ports of the ConnectX 9 3 adapter cards are compliant with the IEEE 802 3 Ether net standards listed in Table 5 Features on page 13 The SFP port has one Tx Rx pair of SerDes Ethernet traffic is transmitted through the cards SFP connectors 2 2 PCI Express Interface The ConnectX 3 adapter cards support PCI Express 3 0 1 1 and 2 0 compatible through an x8 edge connector The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations The following lists the PCle interface features e PCIe Base 3 0 compliant 1 1 and 2 0 compatible 2 5 5 0 or 8 0GT s l
36. lanox Technologies 37 Rev 1 8 Updating Adapter Card Firmware 5 Updating Adapter Card Firmware Each card is shipped with the latest version of qualified firmware at the time of manufacturing However Mellanox issues firmware updates occasionally and the most recent firmware can be obtained from http www mellanox com gt Support Check that the firmware on your card is the latest found on the Mellanox site if not update to the latest version found on the Mellanox web site Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user manual from the Mellanox Firmware Tools page See http www mellanox com gt Software gt Firmware Tools The following steps describe how to retrieve the PSID firmware identification and programmed firmware version of your adapter card They also describe how to update the card with the latest firmware version available 1 Retrieve the PSID and firmware version a Install the package The package is available at http www mellanox com gt Products gt Software gt Firmware Tools Make sure to download the package corresponding to your computer s operating system b Enter mst start c Get the Mellanox mst device name using the command mst status The mst device name will be of the form dev mst mt4099 pci cr0 d Get the PSID firmware identi
37. lgadas alrededor de las aberturas de ventilaci n 3 Cuando hay rayos peligro de descarga el ctrica No utilizar el equipo ni conectar o desconectar cables durante per odos de actividad de LA rayos 4 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo mente de personal capacitado y competente 5 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 6 C digos el ctricos locales y nacionales LA Este equipo se debe instalar conforme los c digos el ctricos locales y nacionales 66 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 Exposici n a niveles de radiaci n peligrosos Precauci n el uso de controles o ajustes o la realizaci n de procedimientos distintos de los que aqu se especifican podr an causar exposici n a niveles de radiaci n peligrosos PRODUCTO L SER DE CLASE 1 y referencia a las normas de l ser m s recientes IEC 60825 1 2007 03 y EN 60825 1 2007 Mellanox Technologies 67
38. llation procedures Attended Installation An installation procedure that requires frequent user intervention e Unattended Installation Mellanox Technologies 25 Rev 1 8 Driver Installation An automated installation procedure that requires no user intervention Both Attended and Unattended installations require administrator privileges 4 2 4 1 Attended Installation The following is an example of aMLNX_WinOF_win8 x64 installation session Step 1 Double click the exe and follow the GUI instructions to install MLNX_WinOF To configure your setup to contain the logs option please run the following command after opening a CMD console MLNX VPI WinOF 4 40 0 All win8 x64 exe v l vx LogFile If you do not want to upgrade your firmware version run the following command MLNX VPI WinOF 4 40 0 All win8 x64 exe MT SKIPFWUPGRD 1 For further help please run MLNX VPI WinOF 4 40 0 All win8 x64 exe v h Step 2 Click Next in the Welcome screen Welcome to the InstallShield Wizard for MLNX_Y PI The InstallShield R Wizard will install MLNX_YPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties 26 Mellanox Technologies 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Step 3 Read then accept the license agreement and click Next License Agreement Please read the Fol
39. lowing DNS server addresses Preferred DNS server Alternate DNS server Validate settings upon exit Step 5 Click OK Step 6 Close the Local Area Connection dialog Step 7 Verify the IP configuration by running ipconfig from CMD console gt ipconfig Ethernet adapter Local Area Connection 4 Connection specific DNS Suffix TP Address 22222222 2 2 21442 63 Subnet Mask o 02 09 55 10 9 0 Default Gateway 8 4 3 VMware Driver For VMware download and install the latest Mellanox OFED Driver for VMware ES Xi Server software package available via the Mellanox web site at http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers gt Download Follow the installation instructions included in the download package also available from the download page Mellanox Technologies 35 Rev 1 8 Driver Installation 4 3 1 Installing and Running the VBI Driver ESXi 5 x 1 Log into the VMware ESXi server machine as root 2 You can either a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4_en driver VIB package Run gt esxcli software vib install v lt vib url gt c Reboot ESXi server The driver will be loaded automatically OR a Update the driver R
40. lowing license agreement carefully Copyright 2005 2013 Mellanox Technologies All rights reserved Redistribution and use in source and binary forms with or without modification are permitted provided that the following conditions are met Redistributions of source code must retain the above copyright notice this list of conditions and the following disclaimer 2 NTadintribhntrienn in binare Farm mint mm Anun tha share I accept the terms in the license agreement I do not accept the terms in the license agreement InstallShield Step 4 Select the target folder for the installation Destination Folder Click Next to install to this folder or click Change to install to different Install MLNX VPI to C Program Files Mellanox MLNX_ PT InstallShield Step 5 The firmware upgrade screen will be displayed in the following cases Ifthe user has OEM card in this case the firmware will not be updated Mellanox Technologies 27 Rev 1 8 Driver Installation Ifthe user has standard card the firmware version is older than the one specified in WinOF Installation Guide 4 40 the firmware will be updated accordingly However if the user has both OEM card and Mellanox card only Mellanox card will be updated Firmware Upgrade v Upgrade the HCA s Firmware version Recommended Upgrading the Firmware version will reboot your machine Note One or mo
41. metallic objects from your hands and wrists 2 Make sure to use only insulated tools 3 Verify that the system is powered off and is unplugged 4 It is strongly recommended to use an ESD strap or other antistatic devices Pre installation Checklist Verify that your system meets the hardware and software requirements stated above Shut down your system if active After shutting down the system turn off power and unplug the cord u SS Remove the card from its package Please note that the card must be placed on an antistatic surface 5 Check the card for visible signs of damage Do not attempt to install the card if damaged Card Installation Instructions It is recommended to support the cards SFP cages with a front bezel to protect against improper insertions and extractions of cables See Appendix A Bezel ae Assembly Optional on page 49 1 Before installing the card make sure that the system is off and the power cord is not con nected to the server Please follow proper electrical grounding procedures 16 Mellanox Technologies 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 2 Place the card an available connector Figure 1 Connector on Motherboard o 3 Make sure to align the holes of the cards on all four corners with the standoffs on the board Mellanox Technologies 17 Rev 1 8 Hardware Installation Figure 3 Standoffs Mo
42. nOF 25 4 2 4 1 Attended Installation 26 4 2 4 2 Unattended Installation 31 4 2 5 Upgrading MLNX WinOF 1 32 4 2 6 Installation Results 32 4 2 7 OpenSM Activation 33 4 2 8 Uninstalling MLNX_WinOF 34 4 2 8 1 Attended Uninstall 34 4 2 8 2 Unattended Uninstall 34 Mellanox Technologies 3 Rev 1 8 4 2 9 Assigning Port IP After Installation 34 43 Driver nat Ges Coen ee E EL a 35 4 3 1 Installing and Running the VBI Driver ESXi 5 x 36 4 3 2 Installing and Running the offline bundle Driver on 1 5 36 4 3 3 Removing the VIB offline bundle Driver 37 44 FIEXBOOL tU A Hep ew BES GS oad RE e ed 37 Chapter 5 Updating Adapter Card Firmware 38 Chapter 6 Troubleshooting 40 6 1 an en A teas tee eR RE Ed sem Made AS En AS 40 6 2 zin Ie oe 41 6 33 Window ser e
43. nd that returned by fabric driver utilities that read the GUID via device firmware e g using ibstat Mlxburn flint returns as GUID while the utilities return a value derived from the MAC address For all driver firmware software purposes the latter value should be used Please see the below example Mellanox Technologies 55 Rev 1 8 56 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Appendix 0 Interface Connectors Pinout D 1 SFP Pinout Figure 19 SFP Cage Pinout SFP Cage 14 E E E D z o JA Ec E C QE P LM LM RS1 RSO SCL SDA gt RX_LOS MOD ABS TX_Disable TX_Fault VeeT 0 2 PCI Express x8 Connector Pinout The Mezzanine cards use a standard PCI Express x8 Mezzanine connector and the PCI Express x8 standard pinout according to the PCI Express 3 0 specification Mellanox Technologies 57 1 8 Figure 20 PCle Connector Pinout 1 1 gt P12V 2 CND 26 CND 27 P3V3 AUX 2 CND 28 GND 25 5 p3v3 7 D3V3 8 CND 30 LAN 3V3STB ALERT N LAN CLE LAN DAT PCIE RSVD DA D88 CND 31 REVD SATA RSVD SATA TX CND 32 CND 33 CLK 100M MEZZ2 DD CLK 100M MEZZ2 DN GCND 34 GND 35 gu TX DP 0 ga TX DN C 0 GCND 36 CND 37 MEZZ TX DP C 1 MEZZ TX DN C 1 GND 38 CND 35
44. o download the exe according to your Operating System Step 1 Verify the machine architecture 1 Open a CMD console Click start gt Run and enter CMD 2 Enter the following command gt echo PROCESSOR ARCHITECTURE On an x64 64 bit machine the output will be AMD64 Step2 Go to the MLNX WinOF for Windows Web page at http www mellanox com gt Products gt Software gt Ethernet Drivers gt Windows SW Drivers Step 3 Download the exe image according to the architecture of your machine see Step 1 The name of the exe is in the following format MLNX_VPI_WinOF lt ver sion gt _All_ lt OS gt _ lt arch gt exe Installing the incorrect exe file is prohibited If you do so an error message will be dis gt played 3 For example if you try to install a 64 bit exe on a 32 bit machine the wizard will dis play the following or a similar error message Windows Installer x 4 This installation package is not supported by this processor type Contact your product vendor OK 4 2 3 Extracting Files Without Running Installation To extract the files without running installation perform the following steps Step 1 Open a CMD console Click Start gt Run and enter CMD Mellanox Technologies 23 Rev 1 8 Driver Installation Step 2 Enter the following command MLNX VPI WinOF version All OS lt arch gt exe Step3 Click Next to create a server image Welcome to the Ins
45. ogies Ethernet adapter cards based on the Con nectX 3 EN integrated circuit device These adapters connectivity provide the highest perform ing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers High Performance Computing and Embedded environments This chapter covers the following topics e Section 1 1 Product Overview on page 11 Section 1 2 Features and Benefits on page 13 Section 1 3 Operating Systems Distributions on page 14 Section 1 4 Connectivity on page 14 1 1 Product Overview The following tables provide the ordering part number port speed number of ports and PCI Express speed Table 3 Single port 10 Gigabit Ethernet Adapter Cards Ordering Part Number OPN MCX341A XCIC D EJN 10GigE Data Transmission Rate MCX341A XCCN single port SFP IPMI disabled MCX341A XCDN single port SFP IPMI enabled Number of ports MCX341A XCEN single port SFP with IPMI and UEFI IPv6 support PCI Express SERDES Speed PCIe 3 0 x8 8GT s RoHS R6 Adapter IC Part Number MT27518A1 FCCR XE 4099 for Physical Function Device ID decimal 4100 for Virtual Function Mellanox Technologies 11 Rev 1 8 Introduction Table 4 Dual port 10 Gigabit Ethernet Adapter Cards Ordering Part Number OPN Data Transmission Rate Number of ports PCI Express SERDES Speed RoHS Adapter IC Part Number Device ID decimal MCX342A XC C
46. porting 10Gb s Ethernet Mellanox OFED for Linux User Manual Document no 2877 User Manual describing OFED features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Linux SW Drivers gt Mel lanox OpenFabrics Enterprise Distribution for Linux MLNX_OFED WinOF VPI for Windows User Manual Document no 3280 User Manual describing WinOF features performance InfiniBand diagnostic tools content and configuration See http www mella nox com gt Products gt Software gt Windows SW Drivers gt Mellanox OFED for Windows WinOF Mellanox MLX4_EN Driver for VMware README Document no 3527 User Manual describing MLX4_EN driver for VMware features performance diagnostic tools content and configuration See http www mellanox com gt Products gt Software gt Ethernet Drivers gt VMware Drivers INF 8074i Specification for SFP Small Formfactor Pluggable Trans ceiver Rev 1 0 May 12 2001 This specification describes the SFP Small Formfactor Pluggable Transceivers developed by the MSA Multiple Source Agreement group IEEE Std 802 3 Specification This is the IEEE Ethernet specification http standards ieee org getieee802 Mellanox Technologies 9 Rev 1 8 PCI Express 3 0 Specifications Industry Standard PCI Express 3 0 Base and Card Electromechanical Specifications
47. re of your HCA adapters has an old firmware version We recommend upgrading to a newer firmware version to enable improved Functionality and support driver s capabilities InstallShield Step 6 Configure your system for maximum performance by checking the maximum performance box Maximum Performance Check this box to configure your system For maximum performance Y Check this box to configure your system For maximum performance Recommended Note This step requires you to reboot the machine at the end of the installation process InstallShield 28 Mellanox Technologies 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 This step requires rebooting your machine at the end of the installation Step 7 Select a Complete or Custom installation follow Step and on on page 29 Setup Type Choose the setup type that best suits your needs Please select a setup type Complete All program Features will be installed Requires the most disk space Custom Choose which program Features you want installed and where they will be installed Recommended for advanced users InstallShield lt Back Cancel a Select the desired feature to install e OpenSM installs Windows OpenSM that is required to mange the subnet from a host OpenSM is part of the driver and installed automatically e Performances tools install the perfo
48. rmance tools that are used to measure the InfiniBand performance in user environment e Analyze tools install the tools that can be used either to diagnosed or analyzed the Infini Band environment e SDK contains the libraries DLLs for developing InfiniBand application over IBAL e Documentation contains the User Manual and Installation Guide DLLs contains the files for standalone installation of the mlx4nd provider Mellanox Technologies 29 Rev 1 8 Driver Installation Custom Setup Select the program features you want installed Click on an icon in the list below to change how a Feature is installed Performace Tools Analyze Tools SDK Documentation ND FLTR DLLs Install to C Program Files Mellanox MLNX_ PIVIB Tools InstallShield Feature Description Subnet manager SM This Feature requires 1164KB on your hard drive b Click Install to start the installation Ready to Install the Program The wizard is ready to begin installation Click Install to begin the installation IF you want to review or change any of your installation settings click Back Click Cancel to exit the wizard InstallShield 30 Mellanox Technologies 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Step 8 Click Finish to complete the installation InstallShield Wizard Completed The InstallShield Wizard has
49. stalled without kmp support the sources would be located under usr srs minx_en 1 5 10 Loading the Driver Step 1 Make sure no previous driver version is currently loaded Run gt modprobe r mlx4 en Step 2 Load the new driver version Run modprobe mlx4 en The result is a new net device appearing in ifconfig a output Unloading the Driver To unload the Ethernet driver run modprobe mlx4 en Uninstalling the Driver To uninstall the mlnx_en driver run gt sbin mlnx_en_uninstall sh Windows Driver For Windows download and install the latest Mellanox WinOF VPI for Windows software pack age available via the Mellanox web site at http www mellanox com gt Products gt Software 22 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 gt Ethernet Drivers gt Windows SW Driver gt Download Follow the installation instructions included in the download package also available from the download page 4 2 4 Hardware and Software Requirements Table 7 Software and Hardware Requirements Requirements Description Required Disk Space for 100 MB Installation Operating Systems Windows Server 2012 64 bit only Windows Server 2008 R2 64 bit only Installer Privileges The installation requires administrator privileges on the target machine 4 2 2 Downloading MLNX WinOF Follow these steps t
50. successfully installed MLNX VPI Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin glog Cl Show release notes Cancel If the firmware upgrade fails the following message will be displayed InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX VPI Click Finish to exit the wizard You chose to run performance tuning The log file can be found at C windows System32 LogFiles PerformanceTunin g log Firmware upgrade Failed with error 1003 Error Description We could not Find a new version For the chosen device Please refer to the UM for Show release notes lt Back Finish 4 2 4 2 Unattended Installation The following is an example of a MLNX_WinOF_win8 x64 unattended installation session Step 1 Open the CMD console click Start gt Run and enter Step 2 Install the driver Run gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn Mellanox Technologies 31 Rev 1 8 Driver Installation Step 3 Optional To configure your setup to contain the logs option please run the following com mand gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v l vx LogFile Step 4 Optional If you do not want to upgrade your firmware version run the following command gt MLNX VPI WinOF 4 40 0 All win8 x64 exe S v qn v MT SKIPFWUP
51. t Adapter Cards 12 Tabl 5 ESAS cut etii ptem d e em de eser lage OS qe na a tote Me 13 Table 6 Software and Hardware Requirements 21 Table 7 Software and Hardware Requirements 23 Table8 MCX341A Specifications Table 43 Table 9 2 Specifications Table 44 Table 10 Physical and Logical Link Indication 48 Table 11 I2C compatible Connector 2 59 6 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Revision History This document was printed on July 3 2014 Table 1 Revision History Table Date Rev Comments Changes July 03 2014 1 8 Updated Air Flow See Table 8 MCX341A Specifications Table on page 43 and Table 9 MCX342A Specifications Table on page 44 e Added NC SI pinout See Appendix D 4 NC SI Interface Pin out on page 59 May 15 2014 1 7 e Added Section 3 4 Card Installation Instructions on page 16 e Removed changes MCX341A XCEN Rev to Rev A6 from Appendix B Change History on page 51 e Added Note on labels Added Section 7 3 1 Mylar Mechanical Drawing and Dimen sions on page 47
52. tallShield Wizard for MLNX_VPI The InstallShield R Wizard will install MLNX_VPI on your computer To continue click Next WARNING This program is protected by copyright law and international treaties Step 4 Click Change and specify the location in which the files are extracted to Network Location Specify a network location For the server image of the product Enter the network location or click Change to browse to a location Click Install to create server image of MLNX VPI at the specified network location or click Cancel to exit the wizard Network location Change InstallShield 24 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 Step 5 Click Install to extract this folder or click Change to install to a different folder Network Location Specify a network location for the server image of the product Enter the network location or click Change to browse to a location Click Install to create a server image of VPI at the specified network location or click Cancel to exit the wizard Network location InstallShield Step 6 To complete the extraction click Finish InstallShield Wizard Completed The InstallShield Wizard has successfully installed MLNX_ PI Click Finish to exit the wizard 4 2 4 Installing MLNX_WinOF This section provides instructions for two types of insta
53. tions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 55 C 131 F Un flux d air de 200LFM a LA cette temp rature ambiante maximale est n cessaire En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 4 Installation du mat riel qualifi h Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et 5 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur 6 Codes lectriques locaux et nationaux LA Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 62 Mellanox Technologies ConnectX 3 Ethernet Single and Dual SFP Port Adapter Card User Manual for OCP Rev 1 8 7 Exposition au rayonnement grave Mise en garde l utilisation de commandes ou de r glages ou l ex cution de proc dures autres que ce qui est sp cifi dans les pr sentes peut engendrer une exposition au rayonnement grave PRODUIT LASER DE CLA
54. u LM o hh 42 Chapter 7 Specifications 43 JA 0MEX3ATA Specifications ran opened 43 7 2 MEX342A Specifications s eer RE os eke dae e e ae eee 44 7 3 Board Mechanical Drawing and Dimensions 45 7 3 1 Mylar Mechanical Drawing and Dimensions 47 7 4 Adapter LED Operation ces 2 0 dan pU gra eaten a BAL Ra en etd ES alten 48 Lov Regulatory Statements us an aa ace Ba Med aed de T en 48 Appendix Bezel Assembly Optional 49 Appendix Card Change History cece cece cece cee eH nn nn nn 91 B 1 Changes in Rev A2 from Rev Al MCX341A XCCN 51 B 2 Changes in Rev from Rev A2 MCX341A XCCN 51 Changes in MCX341A XCDN from 51 Changes in MCX341A XCEN from MCX341A XCDN 51 Appendix C Finding the MAC GUID and Serial Number on the Mezzanine Card 52 Retrieving Adapter Card 55 Appendix D Interface Connectors
55. un esxcli software vib update lt vib url gt b Reboot ESXi server The driver will be loaded automatically To verify that the driver is loaded run gt vmkload mod 1 grep mlx4_en To query network uplinks installed on your machine run gt esxcli network nic list The number of uplinks claimed by EN driver should be displayed In Non Multifunction Mode port 2 is identified as a pseudo device Therefore devices are not seen by vSphere when added as uplink For further information on how to manipulate the uplink please refer to Section 5 3 Adding the Device as an uplink to an Existing Vswitch using the CLI on page 10 of the VMware User Manual See Table 2 Documents List on page 9 Aud 4 3 2 Installing and Running the offline bundle Driver on ESXi 5 x 1 Copy the offline bundle zip file to ESXi 5 0 machine and extract its contents 2 Youcan install the driver in one of the following ways a Remove any earlier version of the driver from your VMware ESXi server machine prior to installing the new version Run gt esxcli software vib list gt esxcli software vib remove n net mlx4 en b Install the mlx4 en driver offline bundle package Run gt esxcli software vib install d lt path gt mlx4 en mlnx 1 6 1 2 offline bundle 471530 zip c Reboot ESXi server The driver will be loaded automatically OR 36 Mellanox Technologies ConnectX 3 Ethernet Single

ConnectX-3 10GbE User Manual for OCP

Contents

Download Pdf Manuals

Related Search

Related Contents