ConnectX IB Dual Port InfiniBand Adapter Card User`s Manual
Contents
1. Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3 200LFM 55 C Typ Power 11 05W Air Flow InfiniBand Copper current rating 0 5A Maximum Power 13 09W 4X 20Gb s Connector max with active media adapter support Temperature 0 C to 55 C Protocol Support Regulatory Mellanox Technologies Rev 1 2 Table 18 Specifications for MHGH28 XSC XTC Physical Power and Environmental InfiniBand QoS RDMA Support Data Rate PCI Express IBTA v1 2 Auto Negotiation 20Gb s 5Gb s or 10Gb s 2 5Gb s 8 InfiniBand Virtual Lanes for each port Yes All Ports DDR 2 0 SERDES 2 5 GT s EMC Safety Environmental RoHS FCC 47 CFR part 15 2005 subpart B class A ICES 003 2004 Issue 4 class A VCCI V 3 2005 04 class A MIC BCC class A EN 55022 1998 1 2000 A2 2003 class A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A1 2001 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Direc tive 89 336 EEC AS NZS 3548 IEC EN 60950 1 2001 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 RoHS RS 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specifica tion Release 1 2 A 5 MHGH29 XSC and MHGH29 XTC Specifications Table 19 Specifications for MHGH29 XSC XTC Physical Power2. A 6 MHJH29 XSC MHJH29 XTC Specifications Table 20 Specifications for MHJH29 XSC XTC Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3V Typ Power 12 22W Air Flow 200LFM 55 C Maximum Power 14 32W 4X 40Gb s Connector InfiniBand Copper current rating 0 5A Temperature 0 C to 55 C max with active media adapter support Protocol Support Regulatory InfiniBand IBTA v1 2 Auto Negotiation 40Gb s EMC 5Gb s or 20Gb s 5Gb s or 10Gb s XTC only 2 5Gb s ICES 003 2004 Issue 4 class A QoS 8 InfiniBand Virtual Lanes for each port VCCI V 3 2007 04 class A RDMA Support Yes All Ports AS NZSCISPR 22 2006 class A Data Rate QDR EN 55022 1998 A1 2000 A2 2003 class PCI Express 20 SERDES 5 0 GT s A EN 61000 3 2 2006 EN61000 3 3 1995 A 1 2001 A2 2005 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Direc tive 2004 108 EC Article 6 2 AS NZS 3548 XTC only MIC BCC class A XTC only Safety Environmental RoHS RoHS R5 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specifica tion Release 1 2 Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 33 Appendix B Interface Connectors Pinout B 1 P C Compatible Connector Pinout Figure 7 I2C
3. PCB revision Later revisions of the HCA card will have the same format Mellanox Technologies Rev 1 2 Adapter Card Interfaces Table 9 VPD Layout for MHEH28 XTC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHEH28 XTC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Note A1 was used as the HCA card PCB revision Later revisio
4. two compliant 4X InfiniBand ports 1 and 2 each having four Tx Rx pairs of SerDes Each of the HCA cards listed in Table 3 on page 8 based on this device provides access to these ports by means of two InfiniBand connectors These cards are also compliant with the IBTA specification 1 2 Connector 1 connects to port 1 of the device while connector 2 connects to port 2 Figure 2 Port Numbering Port 1 Port 2 Each of the HCA cards is embedded with a media detect circuit that supports active cables and external InfiniBand fiber solutions to be connected to the InfiniBand port connectors Fiber Solutions require the use of active media con verters Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 15 The cards can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus opera tions 4 1 2 LED Assignment The board has four LEDs located on the I O panel 2 LEDs per port The green LED when lit indicates that the InfiniBand driver is running and a valid physical connection between nodes exists The yellow LED when lit indi cates a valid data activity link this is the logical link The yellow LED illuminates when the InfiniBand network is discovered over the physical link A valid data activity link without data transfer is designated by a constant yellow LED indication A valid data activity link with data transfer is
5. Appendix I All ConnectX IB HCA adapter cards share the same PCI VPD layout A1 was used as the HCA card PCB revision Later revisions of the HCA card will have the same format Table 8 VPD Layout for MHEH28 XSC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHEH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00X XXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Note A1 was used as the HCA card
6. ESD protected 2 Hold the label with light pincers See Figure 14 Figure 14 Hold Kapton Label With Pincers Mellanox Technologies Rev 1 2 38 3 Gently place the label as shown in Figure 15 Make sure to align the center hole of the label with the drilled hole in the board Figure 15 Place Label on Print Side With Label s and Card s Holes Aligned 4 After placing the label complete the process by gently sweeping your thumb on the label to assure the label is well attached onto the board See Figure 16 Figure 16 Ensure That Label is Well attached GUID 0002C90200258384 See also Figure 21 Print Side View After Short Bracket Assembly With Kapton Label on page 40 Now your card is ready for a short bracket assembly C 3 Assembling Short Bracket The short bracket can now be assembled onto the HCA card Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 39 1Gently place the bracket onto the card fitting the connectors through the bracket connector holes The tab on the bracket should be pointing in the same direction as the PCI connector Make sure the LEDs are aligned into their intended bracket holes Figure 17 Place Short Bracket onto Card LED Holes 2 Insert a screw along with a washer into each of the two holes on the card intended for holding the bracket 3 Use a torque screwdriver to apply up to 2 Ibs in torq
7. cards covered in this User s Manual have the same mechanical drawing and share the same dimen sions as depicted in Figure 6 Note All dimensions are in millimeters Figure 6 Schematic of the ConnectX IB HCA Card with CX4 Connectors 136 47 i m J5 I2C Connector 15 00 33 a Tnm 65 43 17 57 15 J6 Flash Jumper Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual A 2 EMC Certification Statements A 2 1 Table 16 lists the approved certification status per HCA card in different regions of the world Table 16 HCA Cards Certification Status FCC EN ICES C Tick HCA Card P N Class Class Class noe Australia oss IEC EN cTUVus CB USA Europe Canada P New Zealand MHEH28 XSC A A A A MHEH28 XTC A A A A A A A A MHGH28 XSC a di A A A A MHGH28 XTC ut wa di SY A A A A MHGH29 XSC ae SY w A A A A MHGH29 XTC a va we di MHJH29 XSC A A A A A A A A MHJH29 XTC T d A 2 2 FCC Statements USA Class A Statements 8 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsi ble for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tes
8. compliance Using the legend e field M M to indicate a Mellanox Technologies product e field H H to indicate an InfiniBand Adapter Card field T 2 G to indicate CX4 DDR field S H to indicate the ConnectX family field 2 2 to indicate two ports e field I 8 to indicate PCI Express 2 0 x8 running at 2 5GT s field X X to indicate no on board memory field B 5 to indicate a short bracket and field R C to indicate RoHS R5 w Exemptions compliance Mellanox Technologies Rev 1 2 Overview 10 1 3 Finding the GUID and Serial Number on the Adapter Cards All mellanox HCA adapter cards have a label on the printed side of the adapter card that has the card serial number and the card GUID S N MT0710X01085 LR X1 P N MHGH28 XTC JULI LIL EO TET GUID 0002C90300000424 LI _ Made In IL Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 11 2 HCA Card Installation 2 1 Hardware and Software Requirements Before installing the HCA Adapter card please make sure that the system meets the hardware and software require ments listed in Table 5 Table 5 Hardware and Software Requirements Requirement Description Hardware PCI Express x8 or x16 slots Software e For Windows see WinIB ReadMe at https docs mellanox com dm WinIB ReadMe html Operating e For Linux see Mellanox Op
9. 3 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 13 VPD Layout for MHGH29 XTC Pon Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle DDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHGH29 STR_SPC XSC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe Gen2 STR_SPC x8 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 13 VPD Layout for MHGH29 XTC Continued nee Item Value Format Description 143 Length Ox6
10. 9 XSC 5 0 GT s QDR Tall RoHS R5 MT25408A0 FCC CXSC CXTO with JI exemp MHJH29 XTC 5 0 GT s Tall tion 1 The HCA cards have a similar form and fit The main visible difference is in the bracket height Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 1 2 Mellanox Part Numbering Legend Table 4 describes the Mellanox Technologies adapter cards part numbering legend Table 4 Mellanox HCA Cards Part Numbering Key EA Field Decoder M Mellanox Technologies H Adapter Type H InfiniBand Host Channel Adapter N Ethernet Network Interface Card S Express Module T Media E CX4 SDR G CX4 DDR J CX4 QDR K XFP SR M SFP SR N SFP LRM O SFP LR Q QSFP QDR R QSFP DDR T UTP S Silicon ConnectX InfiniHost Lx T InfiniHost A InfiniHost Ex ports 121222 Host Interface X PCI X 4 PCIe x4 8 PCIe x8 9 PCIe SerDes 5 0 GT s G Generation lt blank gt Initial product generation Separator X Memory Size X MemFree 1 128MB 2 256MB 3 512MB B Bracket S Short T Tall N None R RoHS lt blank gt non RoHS C RoHS w Exemption R RoHS Lead Free For example the part number MHGH28 XSC describes Mellanox Technologies ConnectX IB HCA card with dual CX4 ports a PCIe2 0 x8 2 5GT s interface no on board memory mem free a short PCI bracket and RoHS R5
11. AM Mellanox TECHNOLOGIES ConnectX IB Dual Port InfiniBand Adapter Card User s Manual P N MHEH28 XSC MHEH28 XTC MHGH28 XSC MHGH28 XTC MHGH29 XSC MHGH29 XTC MHJH29 XSC MHJH29 XTC Rev 1 2 Mellanox Technologies Copyright 2008 Mellanox Technologies Inc All Rights Reserved Mellanox Technologies InfiniHost and Connectx 2 are registered trademarks for Mellanox Technologies Inc All other marks and names mentioned herein may be trademarks of their respective companies ConnectX IB Dual Port InfiniBand Document Number 2802 Mellanox Technologies Inc 2900 Stender Way Santa Clara CA 95054 U S A www mellanox com Tel 408 970 3400 Fax 408 970 3403 Mellanox Technologies Ltd PO Box 586 Hermon Building Yokneam 20692 Israel Tel 972 4 909 7200 Fax 972 4 959 3245 Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table of Contents Table of Contents List of Tables List of Figures Revision History About this Manual Chapter 1 Chapter 2 Chapter 3 Chapter 4 AppendixA AppendixB AppendixC AppendixD Overview 1 1 Adapter Cards 1 2 Part Numbering Legend 1 3 Finding the GUID and Serial Number on the Adapter Cards HCA Card Installation 2 1 Hardware and Software Requirements 2 2 Installation Instructions Driver Software and Firmware 3 1 Driver Software 3 2 Updati
12. Compatible Connector Plug and Pinout Table 21 I2C compatible Connector Pinout Connector Pin Number HCA Signal Name 1 SPSDA 2 SPSCL 5 1 234 3 GND al 4 NC 5 NC B 2 InfiniBand Connector Pinout Figure 8 InfiniBand CX4 Connector Pinout Conn 4X S10 Btxop 3 IBtxip a S9 IBtxon 3 S8 S12 IBtxOp 2 2 55 S11 IBxin 2 6 _514__ 1 IBtxlp 1 S3 13 iBixon t IBtxin 1 lt S16 iBtxop 0 0 St S15 iBtxon o IBtxin o S2 QIAO oOo ooo ao 2222 222262 222 Ooooooooooo Mellanox Technologies Rev 1 2 34 Table 22 Connector Pin Name and Number to Signal Name Correspondence Connector Pin Connector Pin IB Port A Signal Number Name Name 51 IBtxIp 0 Rx_Al S2 IBtxIn 0 Rx AO S3 IBtxIp 1 Rx A3 54 IBtxIn 1 Rx A2 S5 IBtxIp 2 Rx A5 S6 IBtxIn 2 Rx A4 S7 IBtxIp 3 Rx A7 S8 IBtxIn 3 Rx A6 S9 IBtxOn 3 Tx A6 S10 IBtxOp 3 Tx A7 S11 IBtxOn 2 Tx A4 S12 IBtxOp 2 Tx_A5 S13 IBtxOn 1 Tx_A2 S14 IBtxOp 1 Tx_A3 S15 IBtxOn 0 Tx AO S16 IBtxOp 0 Tx A1 G1 G6 G9 H1 H2 Signal Ground GND G7 Sense 3 3V SENSE_P1 G8 MC POWER 1 The Sense 3 3V signal is used to enable the Vcc power supply pin G8 used to provide power to the active media adapter B 3 PCI Expres
13. F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 14 VPD Layout for MHJH29 XSC Offset Item Value Format Description Decimal 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle QDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 Mellanox Technologies Rev 1 2 Adapter Card Interfaces 24 Table 14 VPD Layout for MHJH29 XSC 100 Data N A STR_SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR_SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Table 15
14. Mellanox HCA Cards Part Numbering Key Hardware and Software Requirements LEDs Jumper Configuration VPD Layout for MHEH28 XSC VPD Layout for MHEH28 XTC VPD Layout for MHGH28 XSC VPD Layout for MHGH28 XTC VPD Layout for MHGH29 XSC VPD Layout for MHGH29 XTC VPD Layout for MHJH29 XSC VPD Layout for MHJH29 XTC HCA Cards Certification Status Specifications for MHEH28 XSC XTC Specifications for MHGH28 XSC XTC Specifications for MHGH29 XSC XTC Specifications for MHJH29 XSC XTC 12C compatible Connector Pinout Connector Pin Name and Number to Signal Name Correspondence Mellanox Technologies 15 16 17 18 19 20 21 22 23 24 27 29 29 30 32 33 34 ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 Figure 18 Figure 19 Figure 20 Figure 21 HCA Card Port Numbering Physical and Logical Link Indications I2C Connector Flash Jumper Schematic of the ConnectX IB HCA Card with CX4 Connectors I2C Compatible Connector Plug and Pinout InfiniBand CX4 Connector Pinout Tall Bracket of a Dual IB Port HCA Card Connector Retention Clip Extracting Connector Clip Bracket Screws Rotate the Bracket to Detach it From the Card Hold Kapton Label With Pincers
15. Place Label on Print Side With Label s and Card s Holes Aligned Ensure That Label is Well attached Place Short Bracket onto Card Attach Bracket onto Card using Screws Sliding Connector Clip Evenly Assembled Short Bracket Print Side View After Short Bracket Assembly With Kapton Label Mellanox Technologies 14 15 15 16 26 33 33 35 35 36 36 37 37 38 38 39 39 40 40 40 Rev 1 2 4 Revision History This document was printed on August 6 2008 10 52 am Table 1 Revision History Table Date Rev Comments Changes August 2008 1 2 Removed watermark May 2008 0 31 Added CTUVUS and CB report status to Cert Table in Appendix A April 2008 0 30 Preliminary release User s Manual for ConnectX IB Adapter Cards Added C tick a nd Mic for relevant cards Changed OFED link to Mellanox OFED Removed Confidential Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual About this Manual This User s Manual describes Mellanox Technologies ConnectX IB PCI Express x8 HCA Adapter cards It provides details as to the interfaces of the board specifications required software and firmware for operating the board and relevant documentation Intended Audience This manual is intended for the installer and user of these cards The manual assumes basic familiarity with Infiniband networks and architecture specifications Rel
16. VPD Layout for MHJH29 XTC Offset Item Value Format Description Decimal 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 Data Eagle QDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB Ox4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber JSTR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR_SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 15 VPD Layout for MHJH29 XTC 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Mellanox Technologies Rev 1 2 26 Appendix A Specifications A 1 Board Mechanical Drawing and Dimensions All of the HCA
17. and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3V 200LFM 55 C Typ Power 11 62W Air Flow InfiniBand Copper current rating 0 5A Maximum Power 13 68 W 4X 20Gb s Connector max with active media adapter support Temperature 0 C to 55 C Protocol Support Regulatory Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 19 Specifications for MHGH29 XSC XTC Physical Power and Environmental InfiniBand QoS RDMA Support Data Rate PCI Express IBTA v1 2 Auto Negotiation 20Gb s 5Gb s 10Gb s 2 5Gb s 8 InfiniBand Virtual Lanes for each port Yes All Ports DDR 2 0 SERDES Q 5 0 GT s EMC Safety Environmental RoHS FCC 47 CFR part 15 2007 subpart B class A ICES 003 2004 Issue 4 class A VCCI V 3 2007 04 class A AS NZSCISPR 22 2006 class A EN 55022 1998 A1 2000 A2 2003 class A EN 61000 3 2 2006 EN61000 3 3 1995 A 1 2001 A2 2005 EN 55024 1998 A1 2001 A2 2003 standards harmonized under EMC Direc tive 2004 108 EC Article 6 2 AS NZS 3548 XTC only MIC BCC class A XTC only IEC EN 60950 1 2001 IEC 60068 2 64 29 32 RoHS R5 1 The auto negotiation protocol is proprietary of Mellanox Technologies and compliant with the InfiniBand Architecture Specifica tion Release 1 2 Mellanox Technologies Rev 1 2 32
18. ard firmware update page See http www mellanox com under Firmware downloads A firmware binaries table lists a binary file per HCA card The file name of each such binary is composed by combin ing the firmware name the firmware release version and the card part number Please contact your assigned Field Application Engineer if you cannot find the firmware binary for your adapter card This may happen if the product is not yet available for general distribution 3 4 HCA Card Firmware Update as Part of a Cluster Firmware Update If the HCA card is part of an InfiniBand cluster its firmware can be updated as part of the entire cluster firmware update using the ibfwmgr tool of the IB administration IBADM tools package IBADM is available for download from http www mellanox com through the Management Tools download link Check the Firmware downloads link for cluster update instructions 1 Currently only the Linux distributions support updating firmware for an entire InfiniBand cluster Mellanox Technologies Rev 1 2 Adapter Card Interfaces 14 4Adapter Card Interfaces 4 1 I O Interfaces Each HCA card includes the following interfaces e Two 4X InfiniBand Copper Connectors e PCI Express x8 edge connector e I O panel LEDs Cc compatible connector for debug 4 1 1 InfiniBand Interface The ConnectX IB MT25408 device is compliant with the InfiniBand Architecture Specification Release 1 2 It has
19. ated Documentation Table 2 Documents List Mellanox Firmware Tools MFT User s Manual Document no 2204UG User s Manual describing the set of MFT firmware management tools for a single InfiniBand node See http www mellanox com under Firmware downloads Mellanox MST User s Manual Document no 2125SM This manual describes various tools and utilities included in the Mel lanox Software Tools MST package for accessing burning firmware and tracing Mellanox silicon devices InfiniBand Administration IBADM Package User s Manual Document no 2130UM User s Manual describing the utilities included in the IBADM tools package for system administration of an InfiniBand cluster See http www mellanox com under Management Tools IB Specifications Release 1 0 a Infiniband Architecture Specifications PCI Express 2 0 Specifications Industry Standard PCI Express 2 0 Card Electromechanical Specifica tion Rev 1 3 Online Resources e Mellanox Technologies Web pages http www mellanox com e Mellanox Technologies Firmware download Web page http www mellanox com under Firmware downloads Mellanox Technologies Document Distribution System DDS http docs mellanox com requires a customer login account Document Conventions When discussing memory sizes MB and MBytes are used in this document to mean size in mega bytes The use of Mb or Mbits small b
20. cal and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 2 2 2 Installation Instructions The adapter cards listed in Table 3 on page 8 are standard PCI Express 8 cards each with a standard x8 edge connec tor Please consult the host machine documentation for instructions on how to install a PCI Express card Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 3Driver Software and Firmware 3 1 Driver Software For Linux download and install the latest OpenFabrics Enterprise Distribution OFED software package available via the Mellanox OpenFabrics Web site at http www mellanox com products ofed php Follow the installation instructions included in the download package For Windows download the appropriate software from https docs mellanox com dm WinIB ReadMe html 3 2 Updating HCA Card Firmware Each HCA card is shipped with the latest version of qualified firmware at the time of manufacturing Firmware is updated occasionally and the most recent firmware can be obtained from http www mellanox com through the Firmware downloads link 3 3 Single HCA Card Firmware Update Firmware can be updated on the standalone single card using the flint tool of the Mellanox Firmware Tools MFT package This package is available for download along with its user s manual from the single HCA c
21. chnology Equipment VCCI If this equipment is used in a domestic environment radio interference may occur in which case the user may be required to take corrective actions Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual A 3 MHEH28 XSC and MHEH28 XTC Specifications Table 17 Specifications for MHEH28 XSC XTC Physical Power and Environmental Size 2 54in x 5 37in 64 4mm x 136 47mm Voltage 12V 3 3 200LFM 55 C Typ Power 11 01W Air Flow InfiniBand Copper current rating 0 5A Maximum Power 12 62W 4X 10Gb s Connector max with active media adapter support Temperature 0 C to 55 C Protocol Support Regulatory InfiniBand IBTA v1 2 Auto Negotiation 10Gb s EMC FCC 47 CFR part 15 2006 subpart B 2 5Gb s class A QoS 8 InfiniBand Virtual Lanes for each port ICES 003 2004 Issue 4 class A RDMA Support Yes All Ports VCCI V 3 2007 04 class A Data Rate SDR EN 55022 1998 1 2000 A2 2003 class PCIExpress 2 0 SERDES Q 2 5 GT s A EN 61000 3 2 2000 A2 2005 EN61000 3 3 1995 A2 2005 EN 55024 1998 A1 2001 A2 2003 stan dards harmonized under EMC Directive 2004 108 EC Article 6 2 AS NZS 3548 Safety IEC EN 60950 1 2001 ETSI EN 300 019 2 2 IEC 60068 2 64 29 32 Environmental RoHS RoHS R5 A 4 MHGH28 XSC and MHGH28 XTC Specifications Table 18 Specifications for MHGH28 XSC XTC
22. designated by a blinking yellow LED indication If the LEDs are not active either the physical link or the logical link or both connections have not been established Figure 3 Physical and Logical Link Indications Table 6 LEDs Port Number LED Name Port 1 Physical Link Green Port 1 Constant on indicates a good physical link Blinking indicates a problem with the Physical link Data Activity Yellow Blinking indicates Data Transfer Constant on indicates no Data Transfer Port 2 Physical Link Green Constant on indicates a good physical link Blinking indicates a problem with the Physical link Data Activity Yellow Blinking indicates Data Transfer Constant on indicates no Data Transfer Note The short bracket has the same port and LED footprint as the tall bracket 4 2 PC Compatible Interface A three pin header on the HCA card is provided as the re compatible interface See Figure 6 on page 26 for the loca tion on the board Figure 4 PC Connector 4 3 Power All adapter cards receive power from the PCI Express Edge connector All other required power voltages are gener ated by on board switch mode regulators See Specifications on page 26 Mellanox Technologies Rev 1 2 Adapter Card Interfaces 16 4 4 Memory The HCA cards support multiple memory devices through the PCI Express Flash and I2C compatible interfaces 4 4 1 System Memory Each of the HCA cards utilizes
23. enFabrics Enterprise Distribution OFED software package available via the Systems Distributions Mellanox OpenFabrics Web site http www mellanox com products ofed php 2 2 Installation Instructions Read all installation instructions before connecting the equipment to the power source 2 2 1 Installation Safety Warnings 1 Installation Instructions Read all installation instructions before connecting the equipment to the power source gt 2 Over temperature This equipment should not be operated in an area with an ambient temperature exceed ing the maximum recommended5 C F Moreover to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings N 3 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or discon nect cables 4 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warnings and instructions Mellanox Technologies Rev 1 2 HCA Card Installation 12 5 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel 6 Equipment Disposal Disposal of this equipment should be in accordance to all national laws and regulations 7 Lo
24. er ou d brancher les cables 4 Branchement d branchement des c bles InfiniBand en cuivre Les cables InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fabri cant des cables pour connaitre les mises en garde et les instructions sp ciales gt 5 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 6 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gislations et r glementations nationales en vigueur Mellanox Technologies Rev 1 2 42 7 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux Rev 1 2 Mellanox Technologies
25. indicates size in mega bits Mellanox Technologies Rev 1 2 Overview amp 1 Overview This document is a User s Manual for Mellanox Technologies host channel adapter HCA Cards based on the MT25408 ConnectX IB integrated circuit device The cards described in this manual have the following main fea tures e IBTA v1 2 compliant e Two4X InfiniBand copper ports for connecting InfiniBand traffic IB connectors e PCI Express 2 0 1 1 compatible through an x8 edge connector up to 5GT s Media detect circuit with powered connectors supporting the use of active cables and external PHY fiber solu tions e EU Restriction of Hazardous Substances RoHS compliant e SDR DDR QDR IB10Gb s SDR 20Gb s DDR 40Gb s DDR e Two bracket heights short or tall 1 1 Adapter Cards e Table 3 lists the InfiniBand HCA cards described in this manual Table 3 HCA Cards PCI Ordering Part Express IBSDR eu ue HCA IC T Number OPN SERDES DDR 2 eee Part Number HCA Card Photo Bracket ce Speed MHEH28 XSC 2 5 GT s SDR Short ROHS RS MT25408A0 FCC Figure 1 HCA Card with SI exemp MHEH28 XTC 2 5 GT s Tall tion MHGH28 XSC 2 5 GT s DDR Short RoHS R5 MT25408A0 FCC with DI exemp MHGH28 XTC 2 5 GT s Tall tion MHGH29 XSC 5 0 GT s DDR Tall RoHS R5 MT25408A0 FCC with GI exemp MHGH29 XSC 5 0 GT s Tall tion Short Tall Bracket Bracket MHJH2
26. ng HCA Card Firmware 3 3 Single HCA Card Firmware Update 3 4 HCA Card Firmware Update as Part of a Cluster Firmware Update Adapter Card Interfaces 4 1 Interfaces 4 2 1 Compatible Interface 4 3 Power 4 4 Memory 4 5 VPDs Specifications A 1 Board Mechanical Drawing and Dimensions 2 EMC Certification Statements 28 and MHEH28 XTC Specifications A4 MHGH28 XSC and MHGH28 XTC Specifications A 5 29 5 and MHGH29 XTC Specifications MHJH29 XSC and MHJH29 XTC Specifications Interface Connectors Pinout 1 2C Compatible Connector Pinout B 2 InfiniBand Connector Pinout B 3 PCI Express x8 Connector Pinout Replacing a Tall Bracket With a Short Bracket on HCA Cards C 1 Removing Tall Bracket C 2 Placing a Kapton Polyimide Label C 3 Assembling Short Bracket Avertissements de s curit d installation Mellanox Technologies coo 0 dU p pd Fm m RP Re Ww W 02 02 pi EN et pt ADARA N N WWNNNN D Lo I Ge Oe WwW 0 RW Oe Un Ww WwW 09 tA EN pi Rev 1 2 8 List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Table 18 Table 19 Table 20 Table 21 Table 22 Rev 1 2 Revision History Table Documents List HCA Cards
27. niBand Adapter Cards With PCI Express x8 User s Manual Table 12 VPD Layout for MHGH29 XSC Offset Item Value Format Description Decimal 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle DDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN PN STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 SerialNumber STR_SPC 00 00X XXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe Gen2 STR_SPC x8 90 VPD Keyword RV STR 92 Length Ox1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A Mellanox Technologies Rev 1 2 Adapter Card Interfaces 22 Table 12 VPD Layout for MHGH29 XSC 141 VPD Keyword RW STR Remaining read write area 14
28. ns of the HCA card will have the same format Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Table 10 VPD Layout for MHGH28 XSC Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHGH28 XSC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Note A1 was used as the HCA ca
29. rd PCB revision Later revisions of the HCA card will have the same format Mellanox Technologies Rev 1 2 Adapter Card Interfaces 20 Table 11 VPD Layout for MHGH28 XTC Mellanox Technologies Offset Decimal Item Value Format Description 0 Large Resource Type ID String Tag 0x02 0x82 1 Length 7 0 LSB 0x9 2 Length 15 8 MSB 0x0 3 Data Eagle SDR STR 12 Large Resource Type VPD R Tag 0x10 0x90 13 Length 7 0 LSB 0x4F 14 Length 15 8 MSB 0x00 15 VPD Keyword PN STR Add in Card Part Number 17 Length 0x15 18 PN MHGH28 XTC STR_SPC 39 VPD Keyword EC STR Engineering Change Level of the card rev 41 Length 0x2 42 Revision Al STR PCB revision 44 VPD Keyword SN STR Serial Number 46 Length 0x18 47 Serial Number STR_SPC 00 00XXXX XX 71 VPD Keyword vo STR Misc Information 73 Length 0x10 74 Data PCIe x8 STR SPC 90 VPD Keyword RV STR 92 Length 0x1 93 Data 0 92 CSO 94 Large Resource Type VPD W Tag 0x11 0x91 95 Length 7 0 LSB Ox9E 96 Length 15 8 MSB 0x00 97 VPD Keyword VI STR EFI Driver version 99 Length 0x6 100 Data N A STR SPC 106 VPD Keyword YA STR Asset Tag 108 Length 0x20 109 Data N A STR SPC N A 141 VPD Keyword RW STR Remaining read write area 143 Length Ox6F 144 Data STR_ZERO Reserved 0x00 255 Small Resource Type END Tag 0x11 0x78 Rev 1 2 ConnectX IB Dual Port Infi
30. s x8 Connector Pinout These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2 0 specification Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 35 Appendix C Replacing a Tall Bracket With a Short Bracket on HCA Cards This appendix provides instructions on how to remove the tall bracket of a standard Mellanox Technologies HCA card and replace it with a short one It includes the following sections e Removing Tall Bracket e Placing a Kapton Polyimide Label e Assembling Short Bracket Figure 10 shows the bracket side view of an HCA card Figure 9 Tall Bracket of a Dual IB Port HCA Card C 1 Removing Tall Bracket Figure 10 shows a connector retention clip and the designated names of its sections Figure 10 Connector Retention Clip Clip Top Clip Center embraces the connector Clip Hook grips onto the connector 1Using a small flat head screwdriver gently push up one hook of a connector s clip toward the connector s top side as shown in Figure 11 on page 36 a 2 Then push the other hook With both hooks unlatched push the clip towards the connector s top side see Figure 11 b Finally pull the clip away from the bracket see Figure 11 c Mellanox Technologies Rev 1 2 36 Figure 11 Extracting Connector Clip a Gentl
31. ted and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equip ment generates uses and can radiate radio frequency energy and if not installed and used in accor Mellanox Technologies Rev 1 2 27 28 dance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 2 3 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures A 2 4 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian I CES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada A 2 5 VCCI Statements Japan Class A Statement COREL PHUBRES TER C B SENI RE VCCI OS CE 2 722AAITSSSIINZEIEB CT COREE RE CERT 9 CBR D esse TL coUtd ONSE XT EIE BEDPBRKENSCECPHVET Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Te
32. the PCI Express interface to store and access IB fabric connection information and packet data on the system memory 4 4 2 Flash Each of the HCA cards includes one 2MB SPI Flash device M25P80 device by ST Microelectronics accessible via the Flash interface of the MT25408 ConnectX IB device There is a jumper on each adapter card that indicates to the device whether an on board Flash device exists or is to be used Table 7 provides information on this jumper See Figure 6 on page 26 for the jumper location Table 7 Jumper Configuration Card Default Description Option on Comments Flash present not present connection open Flash present connection open Header 1x2 connection shorted Flash not present Flash present Figure 5 Flash Jumper CLOSED FLASH_NOT_PRESENT Eu 2 E Tz J 4 4 3 EEPROM Each board incorporates an EEPROM that is accessible through the I2C compatible interface The EEPROM is used for storing the Vital Product Data VPD The VPD format adheres to the PCI Local Bus specification rev 2 3 VPD definition The EEPROM capacity is 512 bytes 4 5 VPDs Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual The PCI VPD Vital Product Data layout for each of the described Mellanox Technologies ConnectX IB HCA adapter cards complies with the format defined in the PCI 2 3 Specification
33. ue on each screw Figure 18 Attach Bracket onto Card using Screws Mellanox Technologies Rev 1 2 40 4 Gently push the clip onto the connector Make sure to slide both clip hooks sides around the connector evenly as shown in Figure 19 Figure 19 Sliding Connector Clip Evenly 5 Use a small flat head screwdriver to gently slide the clip hooks towards the connector s base side as shown in Figure 19 6 Repeat this step for the second clip See Figure 20 for the assembled short bracket side view and Figure 21 for the Print Side view showing the Kapton label Figure 20 Assembled Short Bracket Figure 21 Print Side View After Short Bracket Assembly With Kapton Label Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual Appendix D Avertissements de s curit d installation 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambiante d passant le maximum recommand de 55 C 131 F En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouver tures de ventilation 3 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas branch
34. y Push One Hook of Clip b Gently Push Other Hook of Clip c Pull Clip Away 3 Repeat the above actions for the second connector s clip 4 Unscrew both screws from the card using a torque screwdriver as shown in Figure 13 Figure 12 Bracket Screws These two screws hold on the bracket Grip the bracket as shown in Figure 13 placing your thumb on the LED component In a rotating move toward the component side of the card slide the bracket out of the connector Figure 13 b Gently hold your thumb on the LED component 9e c3 QM pA At the same time extract the bracket as shown in Figure 13 b Make sure to protect the LED while extracting the bracket Rev 1 2 Mellanox Technologies ConnectX IB Dual Port InfiniBand Adapter Cards With PCI Express x8 User s Manual 37 Figure 13 Rotate the Bracket to Detach it From the Card b Rotate the bracket toward the Component Side a Grip the Card in preparation for Detachment c Bracket Separated From Clips and Screws C 2 Placing a Kapton Polyimide Label Prior to assembling the short bracket you need to apply a Kapton polyimide round label on the board s Print Side Note Check to see if the label is already installed as this label may have been installed at the factory Note The label can be provided by Mellanox Technologies P N MEC000821 The following steps are instructions for placing the polyimide label 1 Make sure your working area is
Download Pdf Manuals
Related Search