User Manual
Contents
1. 2 7 QDR InfiniBand Switch Platform User Manual GND AX n GND TX3n i TX3p GND LPMode Vcc1 VccTx IntL ModPrsL 18 35 Top GND RX4p RX4n View into Rear of Connector GND 2 RX2n GND N Sy M GND TX2n TX2p GND TX4n TX4p GND ModSelL ResetL VccRx SCL SDA T pP 18 35 1 GND RX3p RX3n GND RX1p RX1n GND View into Front of Cage 8 50 53 Mellanox Technologies Rev 2 7 Appendix Replacement Parts Ordering Numbers Table 12 Replacement Parts Ordering Numbers Part Description OPN Power Supply Unit PSU MTP000018 This Replacement part is for both the PSU 1 and PSU 2 Power supply blank MTM003600 Rack installation kit MTR003600 Fan Unit for QSFP MTF000960 RS232 Cable DB9 to DB9 Harness HAR000040 I2C DB9 or RJ45 to USB Adapter MTUSB 1 Power cord Type C13 C14 ACC000251 Power cord Type B for USA Canada Mexico Taiwan ACC000204 Power cord Type H for Israel ACC000205 Power cord Type E F for Sweden France Germany Netherlands Russia ACC000207 Power cord Type G for UK ACC000208 Power cord Type D for India ACC000209 Power cord Type I for China ACC000210 Power cord Type J for Switzerland ACC000211 Power cord Type B for Japan ACC000212 Power cord Type I for Australia ACC000213 54 Mellanox Technologies2. Flash memory The manager also has the ability to burn new FW on the switch 40 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 6 FabriclT Management FabricIT EFM is a software based management system that can be run with either a command line interface or with a GUI interface The GUI interface can be run through the Web A license is required for full use of FabricIT The standard management module without a license allows for managing the switch using the Web interface or the command line interface and software supplied on the management chip Make sure to register your FabricIT license to enable all of the available commands and functions See the FabricIT EFM User Manual for instructions and commands available to manage the switches and fabric 6 1 Downloading FabriclT Software and Documents To download FabricIT software and documentation please visit the following Web page FabricIT http www mellanox com content pages php pg fabric_it_login Note that you will need to enter the License Entitlement number and the Switch S N to obtain the software and documentation and also to generate an EFM license that enables extended FabricIT management and inspection features on your switch system 6 2 Installing Licenses A license for extended FabricIT management and inspection features with FabricIT EFM software is available for an additional licensing fee 6 2 1 Getting the License Key FabricIT E
3. Reference Documents Switch Firmware and Firmware Update Tools See http www mellanox com gt Support gt Download Firmware Tools Note that the Switch System described in this manual is based on Mellanox Technologies InfiniScale IV switch device Mellanox OFED Stack for Linux User s Manual See http www mellanox com gt Support gt InfiniBand Software and Drivers Click Mellanox OpenFabrics Enterprise Distribution for Linux MLNX OFED Select the Linux User s Manual The embedded OS and tools on the CPU in the management module is a subset of the Mellanox OFED stack FabricIT Enterprise Fabric Management Software CLI User s Manual Talk to your Mellanox representative for information regarding licensing and implementation of the FabricIT Enterprise Fabric Management Soft ware System Mellanox Firmware Tools MFT User s Manual Document 2329 The MFT Mellanox Firmware Tools package is a set of firmware tools The manual supplied with this package provides an overview of the firm ware its installation and replacement The MFT can be downloaded with its documentation at http www mellanox com gt Support gt Download Firmware Tools 8 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Conventions Throughout this manual the name MTS3600 and the term switch are used to describe both the 36 port QSFP 40Gb s InfiniBand Switch and the 3
4. mst ib add runs the ibdiagnet tool to discover the InfiniBand fabric and then lists the discovered IB nodes as an mst device under dev mst directory These devices can be used for access by other MFT tools root mymach gt mst ib add I Running ibdiagnet to discover the fabric Loading IBDIAGNET from usr local lib ibdiagnet1 2 W Topology file is not specified Reports regarding cluster links will use direct routes Loading IBDM from usr local lib ibdml 2 I Using port 1 as the local port I Discovering 3 nodes 2 Switches amp 1 CA s discovered I Bad Guids LIDs Info 36 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 I skip option set no report will be issued I Links With Logical State INIT I skip option set no report will be issued I PM Counters Info I skip option set no report will be issued I Fabric Partitions Report see ibdiagnet pkey for a full hosts list I skip option set no report will be issued I IPoIB Subnets Check 32 I skip option set no report will be issued I Bad Links Info I No bad link were found I Stages Status Report STAGE Errors Warnings Bad GUIDs LIDs Check 0 0 Link State Active Check 0 0 Performance Counters Report 0 0 Mel
5. Availability and Serviceability Features Switching Perfor Simultaneous wire speed any port to any mance port Hot Swappable Fan Module and Power Supplies 1 1 Redundant Power Supplies Addressing 48K Unicast Addresses Max per Subnet 16K Multicast Addresses per Subnet Switching Capacity 1 44Tb s A 1 EMC Certification Statements Table 10 lists the approved certification status per switch in different regions of the world Table 10 Switch Certification Status FCC EN ICES VCCI Switch P N Class Class Class cTUVus CB USA Europe Canada MTS3600Q 1UNC A A A A MTS3600R 1BNC A A A A MTS3600Q 1BNC A A A A 48 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 A 1 1 FCC Statements USA Class A Statements 15 19 a 4 This device complies with Part 15 of the FCC Rules Operation is subject to the following two conditions 1 This device may not cause harmful interference and 2 This device must accept any interference received including interference that may cause undesired operation 15 21 Statement Warning Changes or modifications to this equipment not expressly approved by the party responsible for compliance Mellanox Technologies could void the user s authority to operate the equipment 15 105 a Statement NOTE This equipment has been tested and found to comply with the limits f
6. FW version 7 045 138 Burning second FW image without signatures OK Restoring second signature OK I Image burn completed successfully 4 3 3 How to Get Mellanox Firmware Tools MFT Mellanox Firmware Tools MFT and documentation are available for download via http www mellanox com gt Downloads gt Firmware Tools The MFT kit includes mlxburn flint spark Bspark debug utilities See Related Documentation on page 7 4 3 4 Open SM To manage the Mellanox switch system using OFED download Mellanox Open Fabrics from http www mellanox com gt Downloads gt InfiniBandSW Drivers Be sure to read and follow all of the instructions regarding the installation and use of these tools Mellanox Technologies 39 Rev 2 7 Switch Management Tools 5 Switch Management Tools This chapter describes the management tools The management module allows the switch to be managed either locally or remotely The switch can also be managed using Out of Band manage ment of the switch via FabricIT EFM The managed switch comes with a module that will com pletely manage your switch An optional management module in combination with the licensing fee for full access to the FabricIT EFM software can manage a fabric of up to 2000 nodes 5 1 Chassis Management The chassis manager will give the user access to switch temperatures power supply voltages fan unit information power unit information
7. Mellanox Technologies 5 Rev 2 7 List of Tables Table 1 Revision History Table 7 Table 2 Reference Documents Table 3 Part Numbering Legend 9 Table 4 Switch Management 12 Table 5 System Status LED Configurations 15 Table 6 Status LED Configurations 16 Table 7 Fan Status LED Configurations 17 Table 8 MTS3600Q Specification Data 46 Table 9 MTS3600R Specification Data 47 Table 10 Switch Certification Status 48 Table 11 InfiniBand QSFP Connector Pinout 52 Table 12 Replacement Parts Ordering Numbers 54 6 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Revision History Table 1 Revision History Table Date Revision Description February 2011 Rev 2 7 Updated power numbers Changed Grounding section July 2010 Rev 2 6 1 Added QSFP Cable Power Budget Classification Added active power cable power level in Appendix A Specs Added Getting the License Key May 2010 Rev 2 6 Updated Section 3 2 to explain the I2C connector use in managed switches Added safety warnings in Spanish Added Note ETH Ports Must be 10 100M November 2009 2 5 removed reference to dual air flow for MTS3600 August 2009 Rev 2 4 Added that CONSOLE Ethernet and USB interfaces do not work in Unmanaged switches August 2009 Rev 2 3 Updated Figures 4 and 5 Updated interfaces to reflect CONSOLE interface August 2009 Rev 2 2 Updated references to initial start up c
8. PSU System LED is on 3 Remove and reinstall the power cable Check that the power outlet in the wall is working Remove and reinstall the PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles 5 Replace the PSU The power LED for the switch shuts off 1 Check that the there is adequate ventilation Are the fan LEDs showing that the fans are all up and running 2 Make sure that there is nothing blocking the front or rear of the chassis and that the fan mod ules and ventilation holes are not blocked especially dust over the holes 3 If you find dust blocking the holes it is recommended to clean the fan unit and remove the dust from the front and rear panels of the switch using a vacuum cleaner The green power LED for the fans does not come on 1 Check that the Power LEDs are on Mellanox Technologies 43 2 7 Troubleshooting Remove and reinstall the fan unit Make sure the mating connector of the new unit is free of any dirt and or obstacles See Section 3 5 7 Extracting and Inserting the Fan Unit on page 31 Caution Do not run the switch if the System Status LED for the Fans is Yel low The link LED for the InfiniBand connector does not come on 1 2 3 4 Check that both ends of the cable are connected Check that the locks on the ends are secured Make sure that the latest FW version is installed on both the HCA and the switch If media adapte
9. com gt Downloads gt Firmware Be sure to read and follow all of the instructions regarding the updating of the firmware on your switch system Firmware for the HCA cards connected to this switch system can be downloaded from the same site 4 3 Unmanaged Externally Managed Switch 4 3 1 12C Connector The I2C connection provides access to Flash and EEPROMs This connection allows access to the switch for updating FW when in band FW updating is impossible Unmanaged Externally managed switches that is the MTS 3600 UNC do not get configured On unmanaged switches the CONSOLE Ethernet and USB connectors do not work The I2C connec tor should only be used when the FW cannot be updated in band The unmanaged switches are Plug and Play and all firmware updates should be done in band The I2C connection should only be used if the FW image was A corrupted to the point that the regular FW tools cannot successfully reburn the correct image When you install the switch it comes with the latest firmware burned on the board All firmware updates should be done in band This is only done when you receive an email that a newer FW version for your switch is available Download the latest FW from http www mellanox com gt Downloads gt Firmware For instructions on downloading FW see http www mellanox com gt Downloads gt Firmware Tools Be sure to download the user man ual appropriate to your OS Read the instructions in the User m
10. 2 kB I Cache 32 kB D Cache Board Mellanox PPC460EX Board FDEF No 12C ready DRAM 2 GB ECC enabled 400 MHz FLASH 16 MB NAND 1024 MiB PCI Bus Dev Venld Devid Class Int PCIEO link is not up PCIE1 successfully set as root complex 01 00 15b3 bd34 0c06 00 Net ppc 4xx ethO ppc 4xx eth1 Hit Ctrl B to stop autoboot 0 Mellanox Fabricl T Boot Menu 1 EFM PPC M460EX EFM 1 1 1000 2010 06 24 16 32 03 ppc 2 EFM_PPC_M460EX EFM 1 1 1200 2010 06 25 18 00 03 ppc 3 U Boot prompt Choice 5 Select the image to boot Mellanox Technologies 45 Rev 2 7 Appendix A Specification Table 8 MTS3600Q Specification Data Environmental Acoustic Sound power level Physical Power and Environmental Size 1 73 1U H x 17 5 W x23 8 D Input Voltage 100 240 VAC 50 60Hz 44mm X 444mm X 605mm Weight 22 045 Ibs 1 PSU Power Consump 24 60 Ibs 2 PSUs tion Mounting 19 Rack mount No management CPU Typical Max in W Air Flow 70 5 CFM Passive cables 136 05 145 06 Active cables 213 51 248 33 SerDes Speeds 10 20 or 40 Gb s per port 405EXR manage ment Typical Max in W Connector Types QSFP Passive cables 142 01 156 97 Active cables 219 46 260 24 460 EX CPU man agement Typical Max Passive cables 147 96 162 92 Active cables 225 41 266 19 QSFP Max Power per port for active cables 3 5W Temperature 10 to 45 Celsius
11. 6 port QSFP 20Gb s InfiniBand Switch unless explicitly indicated otherwise Mellanox Part Numbering Legend Table 3 Part Numbering Legend Place Field Decoder MT Mellanox Technologies S System Type S Switch MM Model 36 InfiniScale IV FF Form factor 00 Rack mounted switch 10 Chassis C InfiniBand Port Config G CX4 DDR J CX4 QDR QSFP QDR QSFP DDR Separator P Power Supplies 0 0 1 1 2 2 M Management U Unmanaged B Managed Y Module Config N IB modules R RoHS RoHS w Exemption RoHS Lead Free Mellanox Technologies 9 Rev 2 7 Overview 1 Overview Mellanox MTS3600 switch systems provide the highest performing fabric solution by delivering high bandwidth and low latency to Enterprise Data Centers High Performance Computing and Embedded environments Networks built with MTS3600 systems can carry converged traffic with the combination of assured bandwidth and granular quality of service Built with Mellanox s 4th generation InfiniScale IV InfiniBand switch device MTS3600 systems provide up to 40Gb s full bidirectional bandwidth per port With 36 ports these systems are among the densest switching systems available These stand alone switches are an ideal choice for top of rack leaf connectivity or for building small to medium size clusters The switch comes pre installed with all necessary firmware and is configured for standard op
12. Chassis sollte nicht auf andere Ger te gestapelt werden Wenn das Chassis herunterf llt Kann es zu Verletzungen und Besch digungen an Ger ten f hren gt 4 Redundanter Stromversorgungsanschluss Elektrische Gefahr Dieses Produkt verf gt ber eine Abdeckung ber dem Bereich f r die redun dante Stromversorgung Betreiben Sie das Produkt nicht wenn diese Abdeck ung nicht sicher festsitzt oder entfernt wurde gt 5 Bei Gewitter Elektrische Gefahr Arbeiten Sie w hrend eines Gewitters und Blitzschlag nicht am Ger t schlie en Sie keine Kabel an oder ab gt 6 Anschlie en Trennen von InfiniBand Kupferkabel InfiniBand Kupferkabel sind schwer und nicht flexible Deshalb m ssen sie vorsichtig an die Anschl sse angebracht bzw davon getrennt werden Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers gt 7 Rack Montage und Wartung Wenn dieses Produkt in einem Rack montiert oder gewartet wird sind beson dere Vorsichtsma nahmen zu ergreifen um die Stabilit t des Systems zu gew hrleisten Im Allgemeinen sollten Sie das Gestell von unten nach oben mit Ger ten f llen N 8 Ger teinstallation Diese Ger t sollte nur von geschultem und qualifiziertem Personal installiert ausgetauscht oder gewartet werden Mellanox Technologies 57 Rev 2 7 9 Ger teentsorgung Die Entsorgung dieses Ger ts sollte unter Beachtung aller nationalen Gesetze Bestimmungen erfolge
13. E Connector 23 EE eS m Pin Rees Signal Description Rx4n Rx3n Number Pun E ES 1 GND Ground PAR EN ia 2 Tx2n Transmitter Inverted Data Input 28 11 3 2 Transmitter Non Inverted Data Input gt 4 GND Ground 30 A 9 3 Tx4n Transmitter Inverted Data Input 6 Tx4p Transmitter Non Inverted Data Input LPMode ModSelL gt J OND NN 8 ModSelL Module Select 34 Tx3n Txan 5 9 ResetL Module Reset 35 GND GND 10 3 3 V Power supply receiver 1 2 11 SCL 2 wire serial interface clock 38 8 e 1 12 SDA 2 wire serial interface data 13 GND Ground 14 Rx3p Receiver Non Inverted Data Output 15 Rx3n Receiver Inverted Data Output 16 GND Ground 17 Rxlp Receiver Non Inverted Data Output 18 RxIn Receiver Inverted Data Output 19 GND Ground 20 GND Ground 21 Rx2n Receiver Inverted Data Output 3 22 Rx2p Receiver Non Inverted Data Output 3 23 GND Ground 24 Rx4n Receiver Inverted Data Output 3 25 Rx4p Receiver Non Inverted Data Output 3 26 GND Ground 27 ModPrsL Module Present 28 IntL Interrupt 29 Tx 3 3 V Power supply transmitter 30 Vcc 1 3 3 V Power Supply 31 LPMode Low Power Mode 32 GND Ground 33 Tx3p Transmitter Non Inverted Data Input 34 Tx3n Transmitter Inverted Data Input 35 GND Ground 36 Txlp Transmitter Non Inverted Data Input 37 Txin Transmitter Inverted Data Input 38 GND Ground 52 Mellanox Technologies
14. ES PIN XXXXXXXX A IN MAC HHHHHHHHHHHH NN Rev XX MII made in ie Mellanox Technologies 11 2 7 Internally Managed vs Unmanaged Externally Managed 2 Internally Managed vs Unmanaged Externally Man aged The following table shows which switches come with a management CPU and which do not Man aged switches can be upgraded from simple chassis management to full fabric management by purchasing an additional license For more information on buying a license see Section 6 Unmanaged switches are plug and play out of the box All switches come with the latest FW burned on the Flash and EEPROM Update the FW on unmanaged switches in band only When new FW is available you will receive an email with the link to the Mellanox FW download site The download site has the Mellanox FW tool package and full instructions for updating FW All managed switches have internal chassis management Managed switches need an initial con figuration before they will start working See the Installation Guide for initial configuration instructions With the purchase of a license this management module can manage the IB fabric See Table 4 for details Table 4 Switch Management Managed 5 Family end Management Connections MTS3600 UNC Unmanaged Plug and play All firmware updates should be done in band using Mellanox Firm ware Management Tools I2C port access using MTUSB 1 device is required for firmwar
15. Humidity 10 90 non condensing Shock and Vibra tion ETSI EN 300 019 2 2 1999 09 Protocol Support Regulatory Compliance InfiniBand Auto Negotiation of 40Gb s 20Gb s Safety US Canada cTUVus 10Gb s EU IEC60950 QoS International CB 8 InfiniBand Virtual Lanes for all ports EMC Emissions USA FCC Class A Management Baseboard Performance and Device Canada ICES Class A management Agents for full InfiniBand EU EN55022 Class A In Band Management EU EN55024 Class A EU EN61000 3 2 Class A Data Rate EU EN61000 3 3 Class A Japan VCCI Class A EU IEC 60068 2 64 Ran dom Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test ISO 7779 ETS 300 753 70 5 dB A or 7 1 Bel 46 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Physical Power and Environmental Scalability and Performance Reliability Availability and Serviceability Features Switching Perfor Simultaneous wire speed any port to any mance port Hot Swappable Fan Module and Power Sup 1 1 Redundant plies Addressing 48K Unicast Addresses Max per Subnet Power Supplies 16K Multicast Addresses per Subnet Switching Capacity 2 88Tb s Table 9 MTS3600R Specification Data Physical Power and Environmental Size 1 73 1U H x 17 5 W x23 8 D Input Voltage 100 240 VAC 50 60Hz 44mm X 444mm X 605
16. Mellanox TECHNOLOGIES InfiniScale IV 36 Port QSFP 40 Gb s InfiniBand Switch User Manual P N MTS3600Q 1BNC MTS3600Q 2BNC MTS3600Q 1UNC MTS3600Q 2UNC MTS3600R 1BNC MTS3600R 2BNC MTS3600R 1UNC MTS3600R 2UNC Rev 2 7 www mellanox com Rev 2 7 NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING I
17. N ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 PO Box 586 Hermon Building Sunnyvale CA 94085 Yokneam 20692 U S A Israel www mellanox com Tel 972 4 909 7200 Tel 408 970 3400 Fax 972 4 959 3245 Fax 408 970 3403 Copyright 2011 Mellanox Technologies All rights reserved Mellanox BridgeX ConnectX Virtual Protocol Interconnect InfiniBlast InfiniBridge InfiniHost InfiniRISC InfiniScale and InfiniPCI are registered trademarks of Mellanox Technologies Ltd CORE Direct FabricIT and PhyX are trademarks of Mellanox Technologies Ltd other trademarks are property of their respective owners other marks and names mentioned herein may be trademarks of their respective companies QDR InfiniBand Switch Platform User Manual 2 Mellanox Technologies Document Number 2983 QDR InfiniBand Switch Platform User Manual Rev 2 7 Table of Contents Table of Contents 3 List of Figures 5 List of Tables 6 Revision History 7 About this Manual 8 Intended Audience 8 Related Documentation 8 Conventions 9 Mellanox Part Numbering Legend 9 Chapter 1 Overview 10 1 1 Serial Number and Product Version Information 10 Chapter 2 Internally Managed vs Unmanaged Externally Managed 12 Chapter 3 Basic Operation and Installation 13 3 1 Switch Platfor
18. QDR InfiniBand Switch Platform User Manual Rev 2 7 Appendix Avertissements de s curit d installation French 1 Instructions d installation Lisez toutes les instructions d installation avant de brancher le mat riel la source d alimentation lectrique 2 Temp rature excessive Ce mat riel ne doit pas fonctionner dans une zone avec une temp rature ambi ante d passant le maximum recommand de 45 C 113 F En outre pour garantir un bon coulement de l air laissez au moins 8 cm 3 pouces d espace libre autour des ouvertures de ventilation 3 Empilage du ch ssis Le ch ssis ne doit pas tre empil sur un autre mat riel Si le ch ssis tombe il peut provoquer des blessures corporelles et des d gradations de biens 4 Connection d Alimentation electrique excedentaire dangers lectriques Ce produit comporte un couvercle transparent sur l espace pour l alimentation lectrique redondante Ne pas faire fonctionner le produit si le couvercle transparent n est pas solide ment fix ou s il est enlev 5 Orages dangers lectriques Pendant un orage il ne faut pas utiliser le mat riel et il ne faut pas brancher ou d brancher les c bles 6 Branchement d branchement des c bles InfiniBand en cuivre Les c bles InfiniBand en cuivre sont lourds et ne sont pas flexibles il faut donc faire tr s attention en les branchant et en les d branchant des connecteurs Consultez le fabrica
19. active cables with a max power per module of 3 5W This is power level 4 according to the QSFP Public Specification 3 3 Interfaces 3 3 1 Port Connector Interfaces 3 3 1 1 36 Port Switches The Connector side of the switch has 36 QSFP ports These are placed in two rows 18 ports to a row The ports are labelled as shown in Figure 7 The bottom row ports are flipped from the top row See Figure 8 for bottom row top row port orientation Mellanox Technologies 17 Rev 2 7 Basic Operation and Installation Figure 7 Port Numbering MTS3600 InfiniBand Switch 3 3 1 2 Top and Bottom Orientation Figure 8 Top and Bottom Ports 3 3 2 Management and FW updating Interfaces There are five interfaces to connect to the MTS3600 They are I2C DB9 interface is labelled 2 On managed switches this interface is a service interface only Update FW in managed switches by using FabricITTM EFM Web User Inter Do not use the I2C DB9 interface to update the FW in the managed switches face WebUI or FabricITTM EFM Command Line Interface CLI Ethernet connector is labelled Use this connector to connect to the Ethernet for remote management This connector does not work in unmanaged switches 2 USB ports are labelled These connectors do not work in unmanaged switches serial CONSOLE port Use this to connect to a local host PC This connector does
20. anual for the FW update procedure Mellanox Technologies 35 Rev 2 7 Management and Tools Overview 4 3 2 Updating Firmware The switch is delivered with the latest Firmware available at the time of production New firmware versions will be posted on the Mellanox firmware download page http www mellanox com gt Downloads gt Firmware You will need the Mellanox Firmware Tools package available in the Mellanox OpenFabrics soft ware package to update firmware for this switch It can also be downloaded from http www mellanox com gt Downloads gt InfiniBand SW Drivers You will also need to download and unzip the firmware binary image This is provided in the Mel lanox Web site at http www mellanox com gt Downloads gt Firmware and go to the InfiniScale IV Switch systems Click in the Table for the FW image that you need 4 3 2 1 Instructions for Reprogramming Over the InfiniBand Network To update an InfiniScale IV switch device having a specific GUID for example 0x00000006660abcd0 or LID the following are the recommended steps to update the device firmware 1 Make sure all subnet ports are in the active state One way to check this is to run opensm the Subnet Manager root mymach gt etc init d opensmd start opensm start OK 2 Make sure the local ports are active by running ibv_devinfo 3 Obtain the device LID There are two ways to do that I Using the mst ib add command The
21. ch for the First Time get configured On unmanaged switches the CONSOLE Ethernet and USB Unmanaged Externally managed switches that is the MTS 3600 UNC do not connectors do not work See the Installation Guide of the MTS3600 switch Configuring the Switch for the First Time The port labelled CONSOLE must be connected to a local host PC This must be used the first time the switch is connected This must be done before any remote management is available Accessing the Switch Locally For local access out of band management of the switch system use the two procedures below Sec tion 4 2 2 1 for initial configuration and Section 4 2 3 to start a remote connection Accessing the CPU via a host PC Hook up the supplied harness cable HARO00040 from the connector labelled CONSOLE to the DB connector of the local host PC Figure 20 Host Connection Connect the host PC to here 12C c os 34 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 8 Follow the instructions in the Installation guide Rev 1 3 Configuring the Switch for the First Time 4 2 3 Starting a Remote Connection to the Switch 4 2 3 1 Accessing the CPU via the Ethernet Connector Once the initial configuration is completed the management tools can be accessed through SSH Telnet the WEB 4 2 4 Downloading Firmware Firmware for this switch system can be found at and downloaded from http www mellanox
22. ch system includes an internal management module then to activate extended FabricIT management and inspection features you need to install the license that was purchased along with the switch system Without the license you will not be able to run a Subnet Manager on the switch nor run advanced fabric diagnostics For information on installation please refer to the FabricIT Enterprise Fabric Management Soft ware CLI User s Manual 42 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 7 Troubleshooting As soon as a switch is plugged in make sure that the green power LEDs on the PSUs are on Status LED and or Status Health LED If either of these two LEDs is red unplug the switch and call your Mellanox representative Power supply unit If the AC power LED is off check that the power cable is plugged into a working outlet 1 Check that the power cable has a voltage within the range of 100 240 volts AC If the LED on the PSU is not lit or is red Remove and reinstall the power cable Remove and reinstall the PSU u PON If the AC power LED is green but the OK power LED is off or the A LED is on Replace the PSU The Status green power LED for PSU1 or PSU2 does not come on 1 If the OK LED on the PSU is on but the system LED is off remove and reinstall the PSU Make sure the mating connector of the new unit is free of any dirt and or obstacles If the OK LED on the PSU is off check if the
23. con figuration to operate as a basic InfiniBand switch and includes all of the necessary functionality to operate with external standard InfiniBand Subnet Management software The managed switch platform requires initial configuration to operate as an InfiniBand switch An Internal Management option is available for the switch platform which includes InfiniBand Subnet Management software embedded internally in the switch This section describes the installation process and basic operation of the switch platform Please first read the warnings sub section care fully before carrying on with installation procedures 3 5 1 Installation Safety Warnings For Safety Warnings in French see Section E Avertissements de s curit d installation French on page 55 for German see Section F Installation Sicherheitshinweise German on page 57 and for Spanish see Section G Advertencias de seguridad para la instalaci n Spanish on page 59 For special regulations regarding Finland Sweden Denmark and Norway see Section H Special Regulations Regarding Finland Sweden Denmark and Norway on page 62 1 Installation Instructions Read all installation instructions before connecting the equipment to the N power source 20 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 2 Over temperature This equipment should not be operated in an area with an ambient tem perature exceeding the maximu
24. configuration either PSU may be extracted without bringing down the system Make sure that the PSU that you are NOT replacing is showing all green for both the PSU and status indicators 28 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Figure 17 Power Supply Unit Extraction To extract a PSU 1 Remove the power cord from the power supply unit 2 Grasping the handle with your right hand push the latch release with your thumb while pull ing the handle outward As the PSU unseats the PSU status indicators will turn off 3 Remove the PSU To insert a PSU 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles Do not attempt to insert a PSU with a power cord connected to it Insert the PSU by sliding it into the opening until a slight resistance is felt 3 Continue pressing the PSU until it seats completely The latch will snap into place confirm ing the proper installation Insert the power cord into the supply connector 5 Insert the other end of the power cord into an outlet of the correct voltage The green PSU indicator should light and the yellow indicator should remain off If not repeat the whole procedure to extract the PSU and re insert it Mellanox Technologies 29 Rev 2 7 Basic Operation and Installation 3 5 6 InfiniBand Cable Installation All cables can be inserted or removed with the unit powered on To insert a cable p
25. de 1 5 metros pero no m s de 4 5 metros gt En conexiones europeas seleccionar un cable de alimentaci n armonizado internacionalmente y marcado lt gt de tres conductores hilo de 1 0 mm2 como m nimo 300 voltios nominal con cobertura protectora aislante de PVC El cable debe tener un enchufe moldeado con capuch n de 250 voltios nomi nal 10 A ADVERTENCIA Alta corriente de fuga Es esencial efectuar la conexi n a tierra antes de conectar la alimentaci n 60 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 13 A adir conexi n a tierra Antes de conectar el dispositivo a la l nea de alimentaci n los tornillos del terminal de la puesta a tierra de protecci n del dispositivo se deben conectar a la puesta a tierra de protecci n de la instalaci n del edificio Informaci n de conexi n a tierra La instalaci n del edificio deber prover un medio para la conexi n con la puesta a tierra de protecci n y un t cnico de servicio deber conectar permanentemente el equipo a dicho medio de conexi n Un T CNICO DE SERVICIO comprobar si la toma el ctrica de la que se suministrar corriente al equipo provee una conexi n con la puesta a tierra de protecci n del edificio De no ser as el TECNICO DE SERVICIO se encarga r de instalar un CONDUCTOR DE CONEXI N A TIERRA DE PROTEC CI N del terminal de puesta a tierra de protecci n separado al conductor de tierra de protecc
26. e internal status of the unit Figure 6 PSU Status LEDs NX 5 MS Table 6 PSU Status LED Configurations LED Configuration Status AC This LED when lit indicates input voltage between 100 and 240 Volts This LED when lit indicates a fault in the power sup ply OK This LED when lit indicates that the output from the power supply is 12VDC Figure 4 on page 15 shows the explanation of the PSU Status LED colors 3 1 1 3 Fan Status Indicators The indicator labeled Fans is located just to the left of the primary power supply unit next to PSU 1 The following fan status conditions are possible 16 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Table 7 Fan Status LED Configurations LED Configuration Status O OK All fans are operating Green Off Error One or more fans is not operating Repair or replace the fan unit Off Yellow Off The fan unit is not receiving any power C Check that the fan unit is properly and completely inserted Off Off fans must be operating while the power supply is plugged in minutes and replug in the switch For more information See Troubleshooting If the switch shuts down due to over temperature unplug the switch wait 5 on page 43 3 2 QSFP Cable Power Budget Classification MTS3600 QSFP switches are designed for a
27. e updates if in band burning is not possible No Management MTS3600 BNC Managed RS232 cable DB9 to DB9 included in the box to connect to host PC for initial configuration of the switch After initial configuration the switch can be managed through the ethernet using remote connec tion Chassis Management is included with purchase and an additional fab ric management option FabricIT EFM can be used with purchase of an additional license Should your unmanaged switch have CONSOLE Ethernet and USB connectors they will not work Only the I2C connector will work on unmanaged switches 12 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 3 Basic Operation and Installation 3 1 Switch Platform Hardware Overview Figure 3 shows the power side panel and connector side panel views of the QSFP switch The fig ure shows port configurations for the switch systems the dual hot swap power supplies and hot swap fan module I2C connector USB connectors Ethernet port connector RS232 DB9 connec tor and various status LEDs Unmanaged switches have an I2C DB9 connector Figure 3 QSFP Switch System Power and Connector Side Panels Power Side panel 12C OK gt O WEM O a System omg ms mgo co TEES PSU 2 MTS3600 InfiniBand Connector Side Panel All InfiniBand connectivity is via the connector side panel All connectors can support active cables 3 1 1 Status LEDs 3 1 1 1 Syste
28. een 14 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Figure 4 Power Fan System LEDs ME 90 264 V 8A MAX IN 47 63 Hz DC output from the PSU is OK System Status Indicators STATUS amp A Sys There is a fault in the PSU Switch Fan Status Indicators Input AC voltage is good PSU Status Indicators Table 5 System Status LED Configurations LED Configuration Status e OK The system is up and running Green Off O O Error A fault in the system Off Yellow O O Off The system has no power Off Off 3 1 1 2 Power Supply Status Indicators The MTS3600 36 Port Switch is available with one or two factory installed Power Supply Units For switches with only one unit installed a second Power Supply Unit can be added to increase Mellanox Technologies 15 Rev 2 7 Basic Operation and Installation security hot swap ability and to add redundancy See Section D Replacement Parts Ordering Numbers on page 54 for ordering part numbers Figure 5 MTS3600 QSFP Power Side Panel Place I Primary Power Power Supply Unit Fan Unit Supply Unit The primary power supply unit PSU1 is located on the right side of the power side panel with PSU2 on the left side Status indicators are located to the left of the primary power supply unit Each PSU also has three status LEDs on the right side of the PSU that indicate th
29. era tion within an InfiniBand fabric This switch requires an InfiniBand compliant Subnet Manager running from one of the hosts or Fabric Management software running on the switch All that is required for normal operation is to follow the usual precautions for installation and to connect the switch to the HCAs Once connected the Subnet Management software automatically configures and begins utilizing the switch It is recommended that the Mellanox OpenFabrics software package be installed on all nodes con nected to the MTS3600 The software package provides a subnet manager and network manage ment tools as well as connectivity software for servers and storage and is available on the Mellanox web site See Chapter 3 for more information Basic installation hot swapping components and hardware maintenance is covered in Basic Operation and Installation on page 13 1 1 Serial Number and Product Version Information The serial number and product version information are found on the label seen in the figure below Figure 1 Generic Product label FSSSSS S N MTYYWWI Rev XX PIN MTS36000 1UNC QUIL GUID HHHHHHHHHHHHHHHH A N Made In IL Also on this label is the GUID identifier for the switch There is also a label with the MAC for the management module 10 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Figure 2 Management Module Label SSSS SIN MTYYWW
30. g as the ambient temperature is below 45 Celsius Operation without a fan unit should not exceed two minutes During fan hot swap if both indicators are OFF then the fan unit is discon nected To extract a Fan Unit 1 Loosen the two thumb screws locking the fan tray in place One of the fan status indicators either green or yellow will remain on 2 Pull the fan unit out by pulling on the thumbscrews As the fan unseats the fan status indica tor will turn off To insert a FAN Unit 1 Make sure the mating connector of the new unit is free of any dirt and or obstacles 2 Insert the fan unit by sliding it into the opening until slight resistance is felt Continue press ing the fan unit until it seats completely Mellanox Technologies 31 Rev 2 7 Basic Operation and Installation 3 Turn the two thumb screws to lock the fan unit in place remain off If not extract the fan unit and reinsert it After two unsuccessful attempts to install the fan unit power off the switch before attempting any sys tem debug The green fan status indicator should light and the yellow fan status should 3 6 Disassembly of the Switch from the Rack To disassemble the switch from the rack 1 Unplug and remove all connectors 2 Unplug all power cords 3 Remove the ground wire 4 Unscrew the 4 bolts from the side of the switch with the rail slides Support the weight of the switch when you remove the screws so that the switch d
31. head screws 2rail slides e 4 caged nuts e 8 pan head screws M6 e 2 metal washers 22 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Figure 10 Rack Installation Kit Parts rail slide Before you install your new MTS3600 switch unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damages that may have occurred during shipping If anything is damaged or missing contact your customer representative immediately Procedure 1 Place the ESD mat on the floor where you will be working and put on the ESD strap Make sure the ESD strap is touching your skin and that the other end is connected to a verified ground 2 Choose which side of the switch you want even with the vertical rack support Either the side with the power supply units or the side with the IB connectors can be even with the vertical rack support The other side of the switch will be further inside of the rack Things to consider before choosing where to mount the rails and rail slides Mellanox Technologies 23 2 7 Basic Operation and Installation Figure 11 Which Side of the Rack Do You Want the Connectors a LA Hu The distance between the rack and the door can be as little as 4 cm on one side of the rack and as much as 18 cm on the other side of the rack Keep in mind that there can be as many as 36 cables connected to the s
32. hten the pan head screws that hold the rails to 9 2 Nm or 81 5 pound inches Figure 14 Screwing in the Rails 6 Place the four bolts for the caged nuts within reach Slide the switch into the rails 8 Putthe switch into place and screw the bolts into the nuts from step 4 Tighten the bolts to 9 2 Nm or 81 5 pound inches 9 Tighten all of the screws 10 Plug in the power cables 11 Check the Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation 12 You can start connecting all of the cables to the switch Make sure that the Rail kit is compatible with your rack Mellanox Technologies 25 Rev 2 7 Basic Operation and Installation Figure 15 Which Side of the Rack Do You Want the Connectors The figure above shows the power side next to the door and the connector side away from the door This configuration has more room for the cables and a larger bending radius The figure below shows the connector side next to the door and the power side away from the door This configuration may be necessary to conform to your rack configuration loooooooooooooooooooooooooooo 0000000000 Warning Any yellow status LEDs is cause for concern and must be dealt with immediately It can take up to 3 minutes to boot up during which time the status LED may indicate red 26 Mellanox Technologies QDR InfiniBand Switch Platform User Ma
33. i n del edificio El equipo se instalar en un rea donde haya conexi n equipotencial como por ejemplo un centro de telecomunicaciones o una sala de computadoras dedicada 14 C digos de instalaci n Este dispositivo se debe instalar conforme a la versi n m s reciente de los c d igos el ctricos nacionales del pa s en cuesti n En Am rica del Norte el equipo se debe instalar de acuerdo con las disposiciones vigentes del C digo El ctrico Nacional de los EE UU y del C digo El ctrico de Canad 15 Directiva sobre RAEE Conforme a la Directiva 2002 96 CE sobre RAEE todos los residuos de equipos el ctricos y electr nicos EEE se deben recolectar por separado y no se deben eliminar junto con residuos dom sticos A deshacerse de este producto y de todas sus partes h galo de una manera res ponsable y respetuosa con el medio ambiente Mellanox Technologies 61 Rev 2 7 Appendix Special Regulations Regarding Finland Swe den Denmark and Norway Denmark This unit is class I and must be connected with an AC cord compliant with all national electrical codes in Denmark The AC cord shall have an integral ground wire and can only be plugged into a fully grounded outlet Do not connect this unit to any outlet that is not fully grounded gt Finland Laite on liitett v suojamaadoituskoskettimilla varust ettuun pistorasiaan gt Norway Apparatet m tilkoples jordet stik
34. ilizar el equipo ni conectar desconectar cables durante periodos de actividad de rayos gt 7 Montaje y mantenimiento de bastidores Al instalar realizar el mantenimiento de este aparato en un bastidor es preciso adoptar precauciones especiales para garantizar que el sistema se mantenga estable En general en un bastidor los equipos se deben insta lar comenzando desde abajo hacia arriba gt Mellanox Technologies 59 Rev 2 7 8 Instalaci n de equipos La instalaci n el reemplazo y el mantenimiento de este equipo estar n a cargo nicamente de personal capacitado y competente gt 9 Asegurar confinamientos adecuados El fabricante del producto final o el usuario final deber n suministrar un confinamiento adecuado para componentes el ctricos y mec nicos y contra incendio 10 Eliminaci n de equipos La eliminaci n definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales 11 C digos el ctricos locales y nacionales Este equipo se debe instalar conforme a los c digos el ctricos locales y nacio nales gt 12 Cable de alimentaci n homologado por UL y con certificaci n CSA Fuga gt 3 5 mA En conexiones de Am rica del Norte seleccionar un cable de alimentaci n homologado por UL y con certificaci n CSA de tres conductores 16 AWG terminado en un enchufe moldeado con capuch n de 125 voltios nominal 13 A con una longitud m nima
35. irmware updates should be Do not use this connector on managed switches Update FW in managed switches by using FabricITTM EFM Web User Inter Do not use the I2C DB9 interface to update the FW in the managed switches face WebUI or FabricIT EFM Command Line Interface CLI 3 4 Package Contents Before you install your new MTS3600 switch unpack the system and check to make sure that all the parts have been sent check this against the parts list Check the parts for visible damage that may have occurred during shipping Mellanox Technologies 19 Rev 2 7 Basic Operation and Installation The switch comes packed with the following items e 1 switch 1 power cable for each PSU Type 6ft US 125V 10 chord See Replacement Parts Order ing Numbers on page 54 to order power cords for various countries A single power cord for each power supply unit can be ordered at no extra charge rail kit MTR003600 harness HAR000017 or HAR000040 1 CD Quick Start Guide Installation Guide Note If anything is damaged or missing contact your customer representative immediately 3 5 Switch Platform Installation and Operation Installation and initialization of the switch platform are straightforward processes requiring atten tion to the normal mechanical power and thermal precautions for rack mounted equipment The unmanaged externally managed switch platform does not require any programming or
36. kontakt Unit is intended for connection to IT power systems for Norway only gt Sweden Apparaten skall anslutas till jordat uttag gt 62 Mellanox Technologies
37. lanox Technologies 37 Rev 2 7 Management and Tools Overview Partitions Check 0 0 IPoIB Subnets Check 0 0 Please see tmp ibdiagnet log for complete log I Done Run time was 1 seconds I Added 3 in band devices root mymach gt To list the discovered mst inband devices run mst status root mymach gt mst status MST modules MST PCI module loaded MST PCI configuration module loaded Inband devices dev mst CA_MT25418_sw005_HCA 1_1id 0x0001 dev mst SW MT47396 lid 0x0011 dev mst SW MT48438 lid 0x0003 root mymach gt II Using the ibnetdiscover tool Run root mymach ibnetdiscover grep 00000006660abcd0 grep w Switch Switch 24 S 00000006660abcd0 MT47396 Infiniscale III Mellanox Technologies base port 0 lid 17 Imc 0 38 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Note The resulting LID is given as decimal number 4 Run mlxburn with the LID retrieved in step 3 above to perform the In Band burning opera tion Burn the InfiniScale IV switch root mymach gt mlxburn d dev mst SW MT48438 lid 0x0003 fw fw IS4 mlx qq I Querying device I Using auto detected configuration file MTS3600Q 1UNC_Al ini PSID MT 0C20110003 I Generating image WARNING Running quick query Skipping full image integrity checks Current FW version on flash 7 0 135 New
38. m Hardware Overview 13 3 1 1 Status LEDs 13 3 2 QSFP Cable Power Budget Classification 17 3 3 Interfaces 17 3 3 1 Port Connector Interfaces 17 3 3 2 Management and FW updating Interfaces 18 3 4 Package Contents 19 3 5 Switch Platform Installation and Operation 20 3 5 1 Installation Safety Warnings 20 3 5 2 Mechanical Installation 22 3 5 3 Grounding the Switch 27 3 5 4 Power Connections and Initial Power On 27 3 5 5 Extracting and Inserting the Power Supply Unit 28 3 5 6 InfiniBand Cable Installation 30 3 5 7 Extracting and Inserting the Fan Unit 31 3 6 Disassembly of the Switch from the Rack 32 3 7 Disposal 32 Chapter 4 Management and Tools Overview 33 4 1 Network Management and Clustering Software 33 4 2 Internally Managed Switch System 33 4 2 1 Configuring the Switch for the First Time 34 4 2 2 Accessing the Switch Locally 34 4 2 3 Starting a Remote Connection to the Switch 35 4 2 4 Downloading Firmware 35 4 3 Unmanaged Externally Managed Switch 35 4 3 1 I2C Connector 35 4 3 2 Updating Firmware 36 4 3 3 How to Get Mellanox Firmware Tools 39 4 3 4 Open SM 39 Chapter 5 Switch Management Tools 40 5 1 Chassis Management 40 Chapter 6 FabricIT Management 41 6 1 Downloading FabricIT Software and Documents 41 Mellanox Technologies 3 Rev 2 7 6 2 Installing Licenses 41 6 2 1 Getting the License Key 41 6 2 2 FabricIT Management and Inspection License 42 Chapter 7 Troubleshooting 43 Appendix A S
39. m Status Indicators The System Status Indicators are located just to the left of the primary power supply unit and labeled System The system status indicators should display as follows When the switch is plugged in within three minutes the STATUS LED should light up green The PSU LED for the plugged in PSU should light up green The PSU LED for the second PSU should light up green only if a second PSU is installed in the switch for redundancy and Hot Swap ability and it is connected to a power source If two PSUs are installed and only one PSU is connected to a power supply the second PSU LED will be red If only one PSU is installed in the switch the second PSU LED will be off Mellanox Technologies 13 Rev 2 7 Basic Operation and Installation f As long as there is power to the switch one PSU is connected and the switch is booted PET up and running the status LED will be green The FAN LED should light up green If the STATUS LED shows red after three minutes unplug the switch and call your Mellanox representative for assistance If the FAN LED shows red troubleshoot the fan module If the switch shuts down due to over temperature unplug the switch wait 5 minutes and replug in the switch For more information See Troubleshooting on page 43 If the PSU LEDs are not green this indicates a problem with the power supplies Only run the switch if at least one of the PSU LEDs is gr
40. m recommended 45 C 113 F More over to guarantee proper air flow allow at least 8cm 3 inches of clearance around the ventilation openings gt 3 Stacking the Chassis The chassis should not be stacked on any other equipment If the chassis falls it can cause bodily injury and equipment damage gt 4 Redundant Power Supply Connection Electrical Hazard This product includes a blank cover over the space for the redundant power supply Do not operate the product if the blank cover is not securely fastened or if it is removed gt 5 During Lightning Electrical Hazard During periods of lightning activity do not work on the equipment or connect or disconnect cables gt 6 Copper InfiniBand Cable Connecting Disconnecting Copper InfiniBand cables are heavy and not flexible as such they should be carefully attached to or detached from the connectors Refer to the cable manufacturer for special warnings and instructions gt 7 Rack Mounting and Servicing When this product is mounted or serviced in a rack special precautions must be taken to ensure that the system remains stable In general you should fill the rack with equipment starting from the bottom to the top gt 8 Equipment Installation This equipment should be installed replaced or serviced only by trained and qualified personnel gt Mellanox Technologies 21 Rev 2 7 Basic Operation and Installation 9 Equipment Dispo
41. mm Weight 22 045 Ibs 1 PSU Power Consumption 24 60 lbs 2 PSUs No management CPU Typical Max in W Mounting 19 Rack mount Passive cables TBD TBD Active cables TBD TBD Air Flow 70 5 CFM 405EXR management Passive cables Typical Max in W SerDes Speeds 10 or 20 Gb s per port Active cables TBD TBD 460 EX CPU manage TBD TBD Connector Types QSFP ment Passive cables Typical Max Active cables TBD TBD TBD TBD QSFP Max Power per port for active cables 3 5W Temperature 10 to 45 Celsius Humidity 10 90 non condensing Shock and Vibration ETSI EN 300 019 2 2 1999 09 Protocol Support Regulatory Compliance Mellanox Technologies 47 Rev 2 7 Physical Power and Environmental InfiniBand Auto Negotiation of 20Gb s or 10Gb s Safety US Canada cTUVus EU IEC60950 8 InfiniBand Virtual Lanes for all ports International CB Management Baseboard Performance and Device EMC Emissions USA FCC Class A management Agents for full InfiniBand Canada ICES Class A In Band Management EU EN55022 Class A EU EN55024 Class A Data Rate DDR EU EN61000 3 2 Class A EU EN61000 3 3 Class A Japan VCCI Class A Environmental EU IEC 60068 2 64 Random Vibration EU IEC 60068 2 29 Shocks Type I II EU IEC 60068 2 32 Fall Test Acoustic ISO 7779 ETS 300 753 Sound power level 70 5 dB A or 7 1 Bel Scalability and Performance Reliability
42. n 10 Regionale und nationale elektrische Bestimmungen Dieses Ger t sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden 58 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Appendix Advertencias de seguridad insta lacion Spanish 1 Instrucciones de instalaci n Antes de conectar el equipo a la fuente de alimentaci n leer todas las instrucciones de instalaci n 2 Instalaci n en un lugar con acceso restringido Esta unidad ha sido ideada para instalar en lugares de acceso restringido gt 3 Sobrecalentamiento No se debe utilizar el equipo en un rea con una temperatura ambiente superior a la m xima recomendada 45 C Adem s para garantizar una circulaci n de aire adecuada se debe dejar como m nimo un espacio de 8 cm 3 pulgadas alrededor de las aberturas de ventilaci n gt 4 Apilamiento del chasis Los chasis no se deben apilar sobre otros equipos La cafda del chasis podria causar lesiones corporales asf como da os al equipo gt 5 Conexi n de fuente de alimentaci n redundante peligro de descarga el ctrica Este producto incluye una fuente de alimentaci n redundante o en su lugar una vac a Si se dispone de una fuente de alimentaci n vac a no utilizar el producto si su tapa est quitada o no est bien cerrada gt 6 Cuando hay rayos peligro de descarga el ctrica No ut
43. not work in unmanaged switches Figure 9 Management Interfaces 12C lt gt ons o CE El 18 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 3 3 2 1 Serial CONSOLE Interface The port labelled CONSOLE is for a local host connection to the management module This is used the first time the switch is connected Connecting to a local host PC and following the instructions in the Installation Guide Configuring the Switch for the First Time must be done before any remote management is available This connector does not work in unmanaged exter nally managed switches All firmware updates should be done in band using Mellanox Firmware Management Tools 3 3 2 2 RJ 45 Ethernet Connector The Ethernet connection provides access for remote management Each Ethernet connector gets connected to Ethernet switches These switches must be configured to 10 100M auto negotiation This connector does not work in unmanaged externally managed switches 3 3 2 3 12C Connector The I2C connection provides access to Flash and EEPROMs This connector is for use on unmanaged switches only On unmanaged switches this connector is used to update FW done in band The I2C connection should only be used if the FW image was corrupted to the point that the regular FW tools cannot successfully reburn the correct image The unmanaged switches are Plug and Play and all f
44. nt des c bles pour connaitre les mises en garde et les instructions sp ciales gt 7 Montage et entretien sur baie Lorsque ce produit est mont ou entretenu sur baie il faut prendre des pr caut ions sp ciales pour s assurer que le syst me reste stable En g n ral il faut remplir la baie avec du mat riel de bas en haut gt Mellanox Technologies 55 Rev 2 7 8 Installation du mat riel Ce mat riel ne doit tre install remplac ou entretenu que par du personnel form et qualifi 9 Elimination du mat riel L limination de ce mat riel doit s effectuer dans le respect de toutes les l gisl ations et r glementations nationales en vigueur 10 Codes lectriques locaux et nationaux Ce mat riel doit tre install dans le respect des codes lectriques locaux et nationaux 56 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Appendix Installation Sicherheitshinweise German 1 Installationsanleitungen Lesen Sie alle Installationsanleitungen bevor Sie das Ger t an die Stromver N sorgung anschlie en 2 bertemperatur Dieses Ger t sollte nicht in einem Bereich mit einer Umgebungstemperatur ber der maximal empfohlenen Temperatur von 45 C 113 F betrieben werden Au erdem sollten mindestens 8 cm 3 in Freiraum um die Bel ft ungs ffnungen sein um einen einwandfreien Luftstrom zu gew hrleisten gt 3 Stapeln des Chassis Das
45. nterprise Fabric Manager EFM license can be purchased from your Mellanox repre sentative at any time A License Enablement Number will be supplied at the time of purchase To generate the License Key enter the License Enablement Number and the Switch Serial Number into the Mellanox online license tool at http license mellanox com The Switch Serial Number can be found on the pull out tab or the box labeling For the serial number location on the chassis see Section 1 1 Serial Number and Product Version Information on page 10 Mellanox Technologies 41 Rev 2 7 FabriclT Management Figure 21 License Key Entitlement Number AM Mellanox Investors Support Company Products Solutions Partners News Education amp Media Where to Buy License Key Generation with Entitlement Number Entitlement Number D ministrator user click here an gt o c w a m w n A Home Company Products Solutions Partners News Investors Support Legal Privacy Figure 22 License Key Generation Form Mellanox Investor Support TECHNOLOGIES Company Products Solutions Partners News Education amp Media Where to Buy Return License Key Generaton Form Entitlement Number Part Number Number of Ports Enter Your Switch Serial 1 Number Generate Home Company Products Solutions Partners News investors Support Legal Privacy Policy 6 2 2 FabriciT Management and Inspection License If your swit
46. nual Rev 2 7 3 5 3 Grounding the Switch Check to determine if your local or national electrical codes require an external ground to all IT components If so connect a ground wire to one of the casing screws and connect the other end to a valid ground If you choose to not use the ground screw make sure that the rack is properly grounded and that there is a valid ground connection between the chassis of the switch and the rack Test the ground using an Ohm meter Z Some national and or local codes may require IT components to be bonded and exter nally grounded not including the power cord ground You must follow all national and local codes when installing this equipment 3 5 4 Power Connections and Initial Power On The switch platform ships with one or two Power Supply Units For switches with only one unit installed a second PSU may be installed at a later time Each supply has a separate AC receptacle The input voltage is auto adjusting for 100 240 VAC 50 60Hz power connections The power cords should be standard 3 wire AC power cords including a safety ground and rated for 15A or higher Caution The switch platform will automatically power on when AC power is applied There is no power switch Check all boards power supplies and fan tray modules for proper insertion before plugging in a power cable Caution After inserting a power cable and confirming the green system status LED light is on make sure
47. oes not fall 5 Slide the switch from the rack Remove the rails from the rack a 7 Remove the eight caged nuts 3 7 Disposal For proper disassembly instructions to comply with the WEEE Directive 2002 96 EC see the Mel lanox website According to the WEEE Directive 2002 96 EC all waste electrical and electronic equipment EEE should be collected separately and not dis posed of with regular household waste Dispose of this product and all of its parts in a responsible and environ mentally friendly way 32 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 4 Management Tools Overview The Ethernet ports for remote management connect to Ethernet switches These switches must be configured to 10 100M auto negotiation 4 1 Network Management and Clustering Software Download and install on all nodes the Mellanox OpenFabric software package for Linux Win dows or other operating systems from the Mellanox software website http www mellanox com gt Downloads gt InfiniBand SW Drivers This software package provides connectivity for server and storage systems utilizing High Perfor mance Computing HPC or enterprise data center EDC applications across an InfiniBand fabric It also provides a subnet manager for simple network configuration and network administration and diagnostic tools for network management 4 2 Internally Managed Switch System The management module r
48. onnecting to a host PC August 2009 Rev 2 1 Removed procedure to replace the battery June 2009 Rev 2 0 Added Management module label and RS232 DB9 to DB9 harness Added part numbers for the two management modules June 2009 Rev 1 9 Made fixes to management section added Disassembly instructions June 2009 Rev 1 8 Added Fabric IT section April 2009 Rev 1 7 inserted procedure to replace the battery April 2009 Rev 1 6 Changes to installation kit and procedure January 2009 Rev 1 5 added installation instructions added ECM Removed Watermark November 2008 Rev 1 4 fixed weight in specs October 2008 Rev 1 3 Minor Formatting changes September 2008 Rev 1 2 Added Reprogramming the MT3600 Through The I2C Port section Added power cord OPNS to table 11 in Appendix D September 2008 Rev 1 1 Fixed link to Software package page September 2008 Rev 1 0 Initial release Mellanox Technologies 7 Rev 2 7 About this Manual This manual describes the installation and basic use of the Mellanox MTS3600 switch which is based on the InfiniScale InfiniBand switch device Intended Audience This manual is intended for users and system administrators responsible for installing and setting up the switch platforms listed above The manual assumes familiarity with the InfiniB and Architecture Specification Related Documentation Additional Documentation available from Mellanox Table 2
49. or a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi ronment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense A 1 2 EN Statements Europe EN55022 Class A Statement Warning This is a class A product In a domestic environment this product may cause radio interference in which case the user may be A 1 3 ICES Statements Canada Class A Statement This Class A digital apparatus complies with Canadian I CES 003 Cet appareil num rique de la classe A est conforme la norme NMB 003 du Canada Mellanox Technologies 49 Rev 2 7 Appendix A 2 VCCI Statements Japan Class A Statement CORB PHUBRES SERRE AERA MRA VCCI 035 CECI A RME CT COREE RE CERT S CBR Hesse TL coUutd ONSE XT ET BEDPBRKENSCECPHVET A 2 1 Translation This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment VCCI If this equipment is used in a domestic environment radio interfe
50. pecification 46 Appendix B Switch Hardware Components 51 Appendix C QSFP Interface 52 Appendix D Replacement Parts Ordering Numbers 54 Appendix Avertissements de s curit d installation French 55 Appendix F Installation Sicherheitshinweise German 57 Appendix Advertencias de seguridad para la instalaci n Spanish 59 Appendix H Special Regulations Regarding Finland Sweden Denmark and Norway 62 4 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 List of Figures Figure 1 Generic Product label 10 Figure 2 Management Module Label 11 Figure 3 QSFP Switch System Power and Connector Side Panels 13 Figure 4 Power Fan and System LEDs 15 Figure 5 53600 QSFP Power Side Panel 16 Figure 6 PSU Status LEDs 16 Figure 7 Port Numbering 18 Figure 8 Top and Bottom Ports 18 Figure 9 Management Interfaces 18 Figure 10 Rack Installation Kit Parts 23 Figure 11 Which Side of the Rack Do You Want the Connectors 24 Figure 12 Screwing the Rail Slide onto the switch 24 Figure 13 Caged Nut Spacing 25 Figure 14 Screwing in the Rails 25 Figure 15 Which Side of the Rack Do You Want the Connectors 26 Figure 16 Two Power Inlets Electric Caution Notifications 28 Figure 17 Power Supply Unit Extraction 29 Figure 18 Top and Bottom Ports 30 Figure 19 Cable connections to the Switch 31 Figure 20 Host Connection 34 Figure 21 License Key Entitlement Number 42 Figure 22 License Key Generation Form 42
51. rence may occur in which case the user may be required to take corrective actions A 3 MIC Certification Korea Korea s Regulation for Certification of Information and Communication Equipment requires EMC testing and certification for many electronic products Korean EMC certifications are issued by Radio Research Laboratory RRL which is organized under the Ministry of Information and Communications MIC EMC testing includes electromagnetic emissions EMI and susceptibil ity EMS Certified equipment is labeled with the MIC mark and certification number 7 7 YFELLE AAA SES amp 717 9 214 BAA EE AA ARE e BE FIAN AR Va HE TUHAT HVE 718892 MESA 7 ch Translation Class A Device This device is registered for EMC requirements for industrial use The seller or buyer should be aware of this If this type was sold or purchased by mistake it should be replaced with a residential use type 50 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev27 Appendix B Switch Hardware Components Place for Optional Secondary Power Primary Power Supply Unit xd Fan Unit Supply Unit Mellanox Technologies 51 Rev 2 7 Appendix QSFP Interface 20__ 12 __ 21 ron Rxin 18 Table 11 InfiniBand QSFP Connector Pinout ON B
52. ress the con nector into the port receptacle until the connector is firmly seated The GREEN LED indicator corresponding to each QSFP port will light when the physical connection is established that is when the unit is powered on and a cable is plugged into the port with the other end of the connec tor plugged into a functioning port After plugging in a cable lock the connector using the latch ing mechanism particular to the cable vendor When a logical connection is made the yellow light will come on When data is being transferred the yellow light will blink Cables in the bottom row should be inserted up side down in relation to the how the cables are inserted in the top row Figure 18 Top and Bottom Ports To remove disengage the locks and slowly pull the connector away from the port receptacle Both LED indicators will turn off when the cable is unseated Care should be taken not to impede the air exhaust flow through the ventilation holes next to the InfiniBand ports Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack 30 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 Figure 19 Cable connections to the Switch 3 5 7 Extracting and Inserting the Fan Unit This switch can operate indefinitely with one of the three fans in the fan module inoperable so lon
53. rs are used check that the all connections are good tight and secure The activity LED does not come on Check that the Subnet Manager has been started The switch is off 1 Unplug the switch 2 Wait 5 minutes 3 Plugin the switch 4 the switch does not come on check the power supplies 5 Ifthe switch comes on Use the FabricIT management CLI or Web GUI to determine the cause of the Shutdown 6 Check the temperature 7 Check the Fan status The switch is not working and unresponsive 1 Reset the switch If resetting the switch does not work 1 n amp gt Unplug the switch Wait 5 minutes Plug in the switch If the switch does not come on check the power supplies If the switch comes on use the FabricIT management CLI or Web GUI to determine the cause of the shutdown The last software update did not succeed 1 2 3 Connect the RS232 connector CONSOLE to a laptop Push the reset button on the switch or management module You will have 5 seconds to stop the U Boot by pressing Control B 44 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 4 Choose the image to upload Only use image 1 or image 2 U Boot 2009 01 minx1 4 May 12 2010 14 08 15 CPU AMCC PowerPC 460EX Rev A at 1000 MHz PLB 200 OPB 100 EBC 100 MHz Security Kasumi support Bootstrap Option Boot ROM Location 12C Addr 0x52 Internal PCI arbiter disabled 3
54. sal Disposal of this equipment should be in accordance to all national laws and regulations 10 Local and National Electrical Codes This equipment should be installed in compliance with local and national electrical codes 3 5 2 Mechanical Installation The switch platform can be rack mounted and is designed for installation in a standard 19 rack The power side of the switch includes a hot swap power supply module a blank cover for an optional second PSU for redundancy a hot swap fan tray system LEDs and management connec tion ports The connector side of the switch has the QSFP ports The switch platform contains auto sensing 100 240 VAC connections for all possible PSUs The installer should use a rack capable of supporting the mechanical and environmental character istics of a fully populated platform The rack mounting holes conform to the EIA 310 standard for 19 inch racks Take precautions to guarantee proper ventilation in order to maintain good air flow at ambient temperature Cable routing in particular should not impede the air exhaust from the chassis This switch can be installed in any standard 19 racks with depths of 670mm to 900mm 3 5 2 1 Installing the Switch in the Rack Tools and Customer Supplied Parts Phillips Screwdrivers 1 and 2 grounding screw ESD strap grounding wire sufficient to reach a valid ground ESD mat Parts included in the installation kit e 2 rails e 8 recessed flat
55. that the Fan Status indicator shows green If the fan status indicator is not green then unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and or obstacles Caution When turning off the switch make sure ALL LEDS are off to ensure a powered down status Mellanox Technologies 27 Rev 2 7 Basic Operation and Installation Do not hot swap power supply if your switch has only one power supply You must power down the system to replace the power supply unit when there is only one PSU in the switch Figure 16 Two Power Inlets Electric Caution Notifications CAUTION Risk of electric shock and energy hazard The two PSUs are indepen dent Disconnect all power supplies to ensure a powered down state inside of the switch platform ACHTUNG Gafahr des elektrischen Schocks Entferrnen des Netzsteckers elnes Netzteils spannungsfrei Um alle Einhi eten spannungsfrei zu machen sind die Netzstecker aller Netzteile zu entfernen 3 5 5 Extracting and Inserting the Power Supply Unit ATTENTION Risque de choc et de danger e lectriques Le de branchment d une seule alimentation stabi lise e ne de branch uniquement qu un module Alimentation Sta bilise e Pour isoler complete ment le module en cause Il faut de brancher toutes les alimenta tions stabilise es With both power supplies installed in the redundant
56. uns SW that manages the switch chassis An optional licensing fee will allow you to manage the whole fabric up to 108 nodes with the chassis management module See your Mellanox representative for information and pricing regarding this upgrade Management modules will have the capability to allow remote monitoring and remote manage ment of the chassis from any host connected to the fabric The managed switch system includes a CPU which contains embedded OS secure in band out band access chassis manager and system BIST SNMP agent 3rd party tool integration GUI subnet manager performance provisioning manager congestion and notification manager QoS manager adaptive routing manager cluster diagnostics manager The Subnet management features include upgrading drivers Mellanox Technologies 33 Rev 2 7 Management and Tools Overview 4 2 1 4 2 2 4 2 2 1 upgrading software monitoring of AC power to the PSUs DC power out from the PSUs board temperature fan module unit efailure in the switch system esystem failure in the switch system querying for board serial numbers and their revisions In addition the tools enable firmware management capabilities such as querying for existing firmware versions burning new firmware from scratch or for recovery from damaged firmware querying for and changing system GUIDs checking for duplicate or bad GUIDs Configuring the Swit
57. witch Do you want the connector side recessed in the rack to allow for larger cable bending radius Will the connector side be recessed past other equipment in the rack and will this be prob lematic The rails are to be installed on the side of the rack where the switch will be recessed into the rack 3 Install the rail slides onto the switch Place the end of the rail slide with the 90 angle on the side of the switch that you want to be even with the vertical support of the 19 rack Use four of the flat head screws for each rail slide There are two sets of holes in the rail slide Select the set of holes to either mount the switch closer or farther away from the rack vertical sup port Tighten the screws to 3Nm or 26 5 pound inches Figure 12 Screwing the Rail Slide onto the switch This side of the switch will be next to the vertical rack support 24 Mellanox Technologies QDR InfiniBand Switch Platform User Manual Rev 2 7 4 Clip the nuts into the holes in the rack you will be using to connect rail slides Check that both sides of the switch are in the same position number on the rack Figure 13 Caged Nut Spacing The caged nuts are separated by a sin gle space 5 Using two of the pan head screws and one washer for each rail install the rails to the other end of the rack Place the rail behind the holes in the rack and screw the screws through the holes in the washer then through the rails Tig
Download Pdf Manuals
Related Search