Home

Sun Datacenter InfiniBand Switch 72 User`s Guide

1. Ox1b01 SyMbOLEFYOrs reia sais eee es aS 0 ViLES5S DROppe Gl ede reee eae eee a 0 AME DA Caio ob ie sce ao gi aia eane e a E stl 2925432 REV DACA SY ic don hed a ee there cd ayece ee eae 2966688 PME PRC Sita ae sacendia set dcr ie nedelacaiie yey aaa cevtne ee 40631 ROVPKE Sto eei dow ate rece doe Gots oo Sw an 41204 Note The output in the example is just a portion of the full output Related Information m Switch Reference perfquery command m Clear Error Counters on page 98 Vv Display Data Counters for a Node If you want to know the data counters for a node the ibdatacounts command provides that subset of the perfquery command output 82 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 On the management controller type ibdatacounts lid port where m lid is the LID of the node m port is the port of the node Note If no port is specified for a switch node the counters are the total for all switch ports This is also defined by port 255 For example ibdatacounts 24 Port counters Lid 24 port 255 XmtData sss siseasi aar aE iE a 22834224 Revbabtal e sieeve send eee oo i aeea 22835016 XMEPKES sds vos Sei EA eA aw ee 317142 RCVPKUS 0 5 a futelanisa ee we eee Ge 317153 Related Information m Switch Reference indatacounts command m Clear Data Counters on page 99 Vv Display Low Level Detailed Information About a No
2. For this example m The route starts at switch with GUID 0x0021283a8620b0 0 and is using port 0 The switch is LID 25 and in the description the switch host s name is o4nm2 72p 2 m The route enters at port 1 of the switch with GUID 0x0021283a8620b0c0 and exits at port 9 The switch is LID 22 Administering the Switch 79 m The route enters at port 2 of the switch with GUID 0x0021283a8620b0e02 and exits at port 8 The switch is LID 24 Note The output for your InfiniBand fabric will differ from that in the example Related Information m Switch Reference ibtracert command m Perform Comprehensive Diagnostics for a Route on page 95 Vv Display the Link Status of a Node If you want to know the link status of a node in the InfiniBand fabric the ibportstate command can tell you the state width and speed of that node On the management controller type ibportstate lid port where m lid is the LID of the node 80 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m port is the port of the node For example ibportstate 24 8 PortiInfo Port info Lid 24 port 8 HINKSCACE seada ee opa aasa Pe ace ec Active PhysLinkState 2004 LinkUp LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 4X LinkSpee dSupported assen 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpee dEnabled d asa
3. If the previous steps do not rectify the problem disable the port See Disable a Port on page 102 For a permanent solution 1 1 Perform the steps for a temporary solution steps 1 to step 4 2 Swap the cable with a known good one or reconnect the cable to a known good remote port and repeat from 1 Depending upon what does or does not rectify the problem replace that component or the switch See Servicing the InfiniBand Cables on page 132 See remote port s documentation for replacement procedures See Remove the Switch From the Rack on page 139 and Installing the Switch on page 1 Clear the error counters See Clear Error Counters on page 98 Start a fabric stress test Identify the suspect links using the ibdiagnet command See Determine Which Links Are Experiencing Significant Errors on page 97 Look for text like the following W 11id 0x0006 guid 0x0021283a8816c0a0 dev 48438 Port 34 Performance Monitor counter Value link_recovery_error_counter 0x1 symbol_error_counter 0x25 Increase by 3 during ibdiagnet For links that are experiencing recovery errors or substantial symbol errors refer to other parts of this table to help identify the cause and rectify the problem Administering the Switch 53 Situation Corrective Steps Output of InfiniBand commands provides only GUID and port not switch chip or CXP connectors Y
4. An easy way to perform a quick check of the switch s operation is with the showunhealthy command On the management controller type showunhealthy OK No unhealthy sensors Related Information m Switch Reference showunhealthy command a Display Switch Environmental and Operational Data on page 71 V Display Power Supply Status The checkpower command performs a simple pass fail test on the power supplies 68 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 1 On the management controller type checkpower PSU 0 present status OK PSU 1 present status Alert 2 If you see the words Alert or Not Present in the command output perform the following steps for the affected power supply a Remove the power cord from the affected power supply See Power Off a Power Supply on page 117 Remove the affected power supply from the switch chassis See Remove a Power Supply on page 118 Wait one minute Reinstall the power supply See Install a Power Supply on page 120 Re attach the power cord See Power On a Power Supply on page 122 If the power supply s Attention LED lights or the checkpower command still reports Alert or Not Present for the power supply replace the power supply See Servicing the Power Supplies on page 113 Related Information m Switch Reference checkpower command m Check Board Level Voltages on page
5. m CXP Connectors and Link LEDs to Switch Chip Port Routes on page 60 m Switch Chip Port to Switch Chip Port Routes on page 61 m Signal Route Through the Switch on page 64 Administering the Switch 64 Display Link Status on page 75 Display Switch Chip Port Status on page 76 Signal Route Through the Switch By combining the information from the tables in Understanding Routing Through the Switch on page 59 it is possible to determine a route through the switch This topic describes a sample situation that might occur 1 2 A route is initiated at connector 2B The left LED P3 blinks Using CXP Connectors and Link LEDs to Switch Chip Port Routes on page 60 it is determined that the link routes to switch chip A through port 36 The Subnet Manager instructs switch chip A to use port 4 to forward the link Using Switch Chip Port to Switch Chip Port Routes on page 61 it is determined that the link routes to I4 switch chip E port 6 The Subnet Manager instructs switch chip E to use port 12 to forward the link Using Switch Chip Port to Switch Chip Port Routes on page 61 it is determined that the link routes to switch chip B port 15 The Subnet Manager instructs switch chip B to use port 24 to forward the link Using Switch Chip Port to CXP Connectors and Link LED Routes on page 62 it is determined that the link exits the switch at connector
6. Installation Sequence on page 16 2 On the management controller type ibnetdiscover Topology file generated on Thu Aug 13 18 16 02 2009 Max of 2 hops discovered Initiated from node 0021283a8620b0 0 port 0021283a8620b0 0 vendid 0x2c9 devid 0xbd36 sysimgguid 0x21283a8620b0 3 switchguid 0x21283a8620b0 0 21283a8620b0 0 Se HH HE Switch 36 S 0021283a8620b0f0 Sun DCS 72 QDR FC switch o4nm2 72p 2 enhanced port 0 lid 25 Imc 0 36 S 0021283a8620b0d0 4 Sun DCS 72 QDR switch 1 2 LC lid 23 4xQDR 35 S 0021283a8620b0b0 5 Sun DCS 72 QDR switch 1 2 LC lid 21 4xQDR 34 S 0021283a8620b0d0 6 Sun DCS 72 QDR switch 1 2 LC lid 23 4xQDR vendid 0x2c9 devid 0xbd36 sysimgguid 0x21283a8620b0e3 switchguid 0x21283a8620b0e0 21283a8620b0e0 Switch 36 S 0021283a8620b0e0 Sun DCS 72 QDR switch 1 1 FC base port 0 lid 24 lmc 0 22 S 0021283a8620b0c0 18 Sun DCS 72 QDR switch 1 2 LC lid 22 4xQDR Note The output for your InfiniBand fabric will differ from that in the example 42 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 3 Perform InfiniBand fabric diagnostics See Perform Diagnostics on the InfiniBand Fabric on page 43 Related Information m Switch Reference inpnetdiscover command m Validate the InfiniBand Fabric and Report Errors on page 44 Perform Diagnostics on the InfiniBand Fabric The ibdiagnet command
7. To use the commands described in these topics you must be the root user of the management controller a Identify All Switches in the Fabric on page 77 a Identify All HCAs in the Fabric on page 78 m Display the InfiniBand Fabric Topology on page 78 a Display a Route Through the Fabric on page 79 76 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m Display the Link Status of a Node on page 80 m Display Counters for a Node on page 81 a Display Data Counters for a Node on page 82 m Display Low Level Detailed Information About a Node on page 83 a Display Low Level Detailed Information About a Port on page 84 m Map LIDs to GUIDs on page 86 m Display Subnet Manager Status on page 87 Related Information m Monitoring the Hardware on page 67 m Controlling the InfiniBand Fabric on page 93 Identify All Switches in the Fabric If you need to know identity information about the switches in the InfiniBand fabric you can use the ibswitches command This command displays the GUID name LID and LMC for each switch The output of the command is a mapping of GUID to LID for switches in the fabric On the management controller type ibswitches Swi tch 0x0021283a8620b0f0 ports 36 enhanced port 0 lid 25 Imc 0 Swi 0 Swi 0 Swi 0 Swi 0 Swi 01 tch lid 23 tch lid 21 tch lid 20 tch lid 22
8. on page 31 Start the Subnet Manager on page 34 Installing the Switch 23 24 Related Information m Understanding Switch Specifications on page 1 m Routing Service Cables on page 7 m Connecting InfiniBand Cables on page 35 Attach the Management Cables The switch has two connectors for network communication or serial communication with the management controller The network management connector labeled NET is a 100 BASE T Ethernet interface This connector is preferred because it permits remote management of the switch over the Ethernet network The USB management connector labeled USB is the second choice for communication with the management controller in the switch The management console can be a serial terminal a system running a TIP connection or other serial device which communicates with the management controller through a USB to serial adapter The serial parameters for communication with the USB to serial adapter is typically 115600 8 N 1 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 Connect the network management cables from the management console to the connectors labeled NET Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 3 Configure the DHCP server with the MAC address of the management controller The MAC address is p
9. 145 front status LEDs 55 G getbaseguid command 105 getfanspeed command 31 71 getportstatus command 76 GUID mapping to CXP connector 74 to switch chip 74 nomenclature 65 H hardware commands 67 problems 48 HCA identity 78 l i4reset command 89 90 ibcheckerrors command 44 ibcheckstate command 99 ibclearcounters command 99 ibclearerrors command 98 ibdatacounts command 82 ibdiagnet command 43 94 97 ibdiagpath command 95 ibhosts command 78 ibnetdiscover command 42 78 ibportstate command 80 100 101 102 103 ibswitches command 77 ibtracert command 79 identifying fan 125 HCAs 78 InfiniBand cable 133 power supply 114 switches 77 idling noise 3 InfiniBand cable attaching 36 bundling 12 cautions 10 connector 5 delivery floor and underfloor 13 overhead 13 features 133 handling guidelines 9 11 inspecting 132 connectors 134 hardware 133 installing 136 length 11 path lengths 12 removing 134 servicing 132 type 11 understanding 9 InfiniBand commands overview 67 InfiniBand fabric diagnosing 43 discovering 42 problems 51 validating 44 verifying 41 inspecting fan 125 connector 127 hardware 126 InfiniBand cable 132 connectors 134 hardware 133 power supply 114 connectors 116 hardware 116 installation preparation 14 responsibilities 15 sequence 16 understanding 14 installing fans 130 Inf
10. 2 ports have errors beyond threshold Note The output for your InfiniBand fabric will differ from that in the example Related Information m Switch Reference incheckerrors command m Discover the InfiniBand Fabric Topology on page 42 m Perform Diagnostics on the InfiniBand Fabric on page 43 Installing the Switch 45 46 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Administering the Switch The following topics describe the administration of the switch Troubleshooting the Switch on page 47 Administrative Command Overview on page 66 Monitoring the Hardware on page 67 Monitoring the InfiniBand Fabric on page 76 Controlling the Hardware on page 87 Controlling the InfiniBand Fabric on page 93 Related Information Installing the Switch on page 1 Servicing the Switch on page 111 Switch Remote Administration Switch Reference Troubleshooting the Switch The following topics help you resolve some basic problems that might occur with your switch Switch Hardware Problems on page 48 InfiniBand Fabric Problems on page 51 Understanding the LEDs on page 54 Understanding Routing Through the Switch on page 59 Switch GUIDs Overview on page 65 Related Information Administrative Command Overview on page 66 47 a Monitoring the Hardware on page 67 a Monitoring the InfiniBand Fabric o
11. Related Information m Power On a Power Supply on page 122 V Remove a Power Supply 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Power Supplies on page 113 2 Locate the power supply to be removed 118 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 3 Press and hold the release tab to the left and pull on the handle of the power supply 4 Continue to pull the handle of the power supply to remove it from the chassis 5 Set the power supply aside Servicing the Switch 119 6 Install a replacement power supply See Install a Power Supply on page 120 Related Information m Remove a Fan on page 127 m Remove an InfiniBand Cable on page 134 m Remove the Switch From the Rack on page 139 m Replace the Battery on page 140 V Install a Power Supply Note For residual power discharge the power supply slot must remain vacant for at least one minute before installing a power supply 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Power Supplies on page 113 2 Inspect the replacement power supply See Inspecting a Power Supply on page 114 3 Verify that the slot where the power supply installs is clean and free of debris 4 Verify that the slot connector
12. Summary 5 nodes cleared 0 errors Related Information m Switch Reference inclearcounters command a Display Data Counters for a Node on page 82 m Clear Error Counters on page 98 Check All Ports To perform a quick check of all ports of all nodes in your InfiniBand fabric you can use the ibcheckstate command On the management controller type ibcheckstate v Checking Switch nodeguid 0x0021283a8620b0 0 Node check lid 25 OK Administering the Switch 99 100 Port check Port check Port check Port check Port check Port check Port check Checking Node check Port check Port check Port check tt Summary lid 25 port lid 25 port lid 25 port lid 25 port lid 25 port lid 22 port lid 22 port Switch nodeguid 0x0021283a8620b0e0 lid 24 OK lid 24 port 22 lid 24 port 30 lid 24 port 29 6 nodes checked 36 35 34 33 324 6 O s OK OK OK OK OK OK OK OK OK OK 142 ports checked 0 bad nodes found 0 ports with bad state found Note The ibcheckstate command requires time to complete depending upon the size of your InfiniBand fabric Without the v option the output contains only failed ports The output in the example is only a small portion of the actual output Related Information m Switch Reference ipcheckstate command a Display Link Status on page 75 V Reset a Port You might need to rese
13. a There must be a clean dry stable work surface Related Information m Suggested Tools on page 15 a Antistatic Precautions on page 15 a Installation Responsibilities on page 15 a Installation Sequence on page 16 14 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Suggested Tools The following tools are necessary or beneficial for installing the switch m Antistatic mat m Antistatic wrist strap m No 2 Phillips screwdriver m No 1 Phillips screwdriver m Flashlight m Gloves m Magnifying glass Related Information a Antistatic Precautions on page 15 Antistatic Precautions When installing the switch chassis take care to follow antistatic precautions m Use an antistatic mat as a work surface m Wear an antistatic wrist strap that is attached to either the mat or a metal portion of the switch chassis Related Information m Suggested Tools on page 15 Installation Responsibilities The personnel who install the switch must be fully capable of the following tasks m Rackmount a heavy object m Perform line voltage verification m Connect delicate cables in tight spaces m Configure network hosts and serial terminals m Perform software tasks of an administrative nature m Interpret screen output as it pertains to InfiniBand fabrics Installing the Switch 15 Related Information a Installation Sequence on page 16 Installation Sequence The process of
14. le cas ch ant a lui en faire part par crit Si ce logiciel ou la documentation qui l accompagne est conc d sous licence au Gouvernement des Etats Unis ou a toute entit qui d livre la licence de ce logiciel ou l utilise pour le compte du Gouvernement des Etats Unis la notice suivante s applique U S GOVERNMENT RIGHTS Programs software databases and related documentation and technical data delivered to U S Government customers are commercial computer software or commercial technical data pursuant to the applicable Federal Acquisition Regulation and agency specific supplemental regulations As such the use duplication disclosure modification and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract and to the extent applicable by the terms of the Government contract the additional rights set forth in FAR 52 227 19 Commercial Computer Software License December 2007 Oracle America Inc 500 Oracle Parkway Redwood City CA 94065 Ce logiciel ou mat riel a t d velopp pour un usage g n ral dans le cadre d applications de gestion des informations Ce logiciel ou mat riel n est pas con u ni n est destin tre utilis dans des applications a risque notamment dans des applications pouvant causer des dommages corporels Si vous utilisez ce logiciel ou mat riel dans le cadre d applications dangereuses il est de votre responsabilit de prendre tou
15. on page 109 m Create the guid txt File on page 105 m Create the opensm conf File on page 106 V Disable the Subnet Manager If your InfiniBand fabric has too many Subnet Managers you can disable and stop the Subnet Manager within the management controller by using the disablesm command Administering the Switch 107 On the management controller type disablesm Stopping IB Subnet Manager OK Related Information m Switch Reference disablesm command m Stop the Subnet Manager With the opensmd Daemon on page 109 Vv Set the Subnet Manager Priority By default the Subnet Manager within the management controller is set to 0 priority If there is more than one Subnet Manager in your InfiniBand fabric you must set the Subnet Managers priorities appropriately The Subnet Manager with the highest priority is the master Subnet Manager Note Setting Subnet Managers of the same fabric to the same priority can have undesirable results 1 On the management controller type setsmpriority priority where priority is 0 lowest to 13 highest For example setsmpriority 3 OpenSM 3 2 6_20090717 Reading Cached Option File etc opensm opensm conf Loading Cached Option routing_engine ftree Loading Cached Option sm_priority 13 Loading Cached Option sminfo_polling_timeout Loading Cached Option polling_retry_number 3 Command Line Arguments
16. 3B The right LED P1 blinks Related Information CXP Connectors and Link LEDs to Switch Chip Port Routes on page 60 Switch Chip Port to Switch Chip Port Routes on page 61 Switch Chip Port to CXP Connectors and Link LED Routes on page 62 Display a Route Through the Fabric on page 79 Perform Comprehensive Diagnostics for a Route on page 95 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 63 Switch GUIDs Overview Global unit identifiers GUIDs are unique 64 bit strings that identify nodes such as switches and channel adapters For the Sun Datacenter InfiniBand switches GUIDs are modified to identify the node s role and location The following table describes the GUID s structure 16 15 12 11 8 7 4 3 0 Mac 48 bits Iype Pos Device Num The five fields of the GUID are described as follows m MAC Bits 63 through 16 are the Machine Allocation Code MAC address A standard for network components the MAC address is typically provided by manufacturers in a 6 byte colon delimited string For example 00 11 22 33 44 55 m Type Bits 15 through 12 is the type of board on which the node resides m Pos Bits 11 through 8 identify the position of the board within the switch m Device Bits 7 through 4 identify which device on the board has that node Num Bits 3 through 0 are numbers reserved for the programs which modify the GUID In most occurrences th
17. 3V 3 3V 12V 5V VBAT 1 8V 1 2V 1 8V Main 3 28 V Standby 3 40 V 12 06 V 5 10 V 3 17 V 1 78 V Standby 1 21 V Standby 1 80 V 32 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Measured 2 5VA 2 51 V Measured 2 5VB 2 51 V Measured 1 2VA 1 22 V Measured 1 2VB 1 22 V Measured 1 2VC 1 21 V Measured 1 2VD 1 21 V Measured 1 2VB 1 21 V Measured 1 2VE 1 21 V Measured 1 2VF 1 21 V Voltage test returned OK Starting PSU test PSU 0 present PSU 1 present PSU test returned OK Starting Temperature test Back temperature 23 00 Front temperature 32 62 ComEx temperature 26 12 I4 A temperature 55 maxtemperature 56 I4 B temperature 48 maxtemperature 49 I4 C temperature 53 maxtemperature 53 I4 D temperature 48 maxtemperature 49 I4 E temperature 53 maxtemperature 54 I4 F temperature 53 maxtemperature 54 Temperature test returned OK Starting FAN test Fan 0 running at rpm 12433 Fan 1 running at rpm 12311 Fan 2 running at rpm 12311 Fan 3 running at rpm 12433 Fan 4 running at rpm 12433 FAN test returned OK Starting Connector test Connector test returned OK Starting I4 test T4 A OK T4 B OK T4 C OK T4 D OK T4 E OK I4 F OK All I4s OK I4 test returned OK NM2 Environment test PASSED 7 Once the switch has an operational status start the Subnet Manager See Start the Subnet Manager on page 34 Installing the Swi
18. 68 features 114 inspecting 114 connectors 116 hardware 116 150 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 installing 120 LEDs 55 powering off 117 on 122 removing 118 servicing 113 powering off power supply 117 switch 117 powering on power supply 122 switch 23 preparation 14 problem hardware 48 InfiniBand fabric 51 R rear status LEDs 55 reboot command 88 recovering switch chip ports 90 recovery errors 97 removing fan 127 InfiniBand cable 134 power supply 118 switch from rack 139 replaceable components 112 replacing the battery 140 resetting port 100 switch chip 89 restarting entire switch 89 management controller 88 retraction strap 133 root node GUID file 105 route CXP to switch chip port 60 switch chip port to CXP 62 switch chip port to switch chip port 61 through switch 59 64 through the fabric 79 routing InfiniBand cables 9 service cables 7 S sequence of installation tasks 16 servicing battery 139 fan 124 InfiniBand cable 132 power supply 113 switch 111 setlinkspeed command 91 setsmpriority command 34 108 setting link speed 91 port speed 101 Subnet Manager priority 108 shipping carton contents 16 showtemps command 70 showunhealthy command 31 68 sminfo command 87 smpquery command 83 84 specifications acoustic noise 3 electrical 3 environmental 3 physical 2 switch 1 ssh command 29 starting Sub
19. 69 a Display Switch Environmental and Operational Data on page 71 V Check Board Level Voltages When you want to know if the voltages used by the various components within the switch are within nominal values use the checkvoltages command If a voltage deviates more than 10 of its nominal value there is a problem This check also reports if a battery has failed Administering the Switch 69 On the management controller type checkvoltages Voltage ECB OK Measured 3 3V Main 3 28 V Measured 3 3V Standby 3 37 V Measured 12V 12 00 V Measured 5V 5 10 V Measured VBAT 3 08 V Measured 1 8V 1 77 V Measured 1 2V Standby 1 21 V Measured 1 8V Standby 1 80 V Measured 2 5VA 2 51 V Measured 2 5VB 2 51 V Measured 1 2VA 1 22 V Measured 1 2VB 1 21 V Measured 1 2VC 1 21 V Measured 1 2VD 1 22 V Measured 1 2VE 1 21 V Measured 1 2VF 1 21 V All voltages OK Related Information m Switch Reference checkvoltages command a Display Power Supply Status on page 68 m Display Switch Environmental and Operational Data on page 71 V Display Internal Temperatures If you are concerned with the internal temperature of the switch you can use the showtemps command to display nominal and measured temperatures If there is a great deviation there might not be enough cooling On the management controller type showtemps Back temperature 22 88 Front tempe
20. FAN I4 A I4 B I4 C I4 D I4 E I4 F 0 BwWDdNY bP mT ct temperature temperature temperature temperature temperature I4 F temperature Temperature Starting FAN test running running running running running Lest at at at at at Starting PSU test PSU 0 presen PSU 1 presen PSU test returned OK Starting Temperature test Back tempera Front temperature 32 ComEx temperature 26 55 48 53 48 533 535 Measured 3 3V Standby 3 40 V Measured 12V 12 06 V Measured 5V 5 10 V Measured VBAT 3 17 V Measured 1 8V 1 78 V Measured 1 2V Standby 1 21 V Measured 1 8V Standby 1 80 V Measured 2 5VA 2 51 V Measured 2 5VB 2 51 V Measured 1 2VA 1 22 V Measured 1 2VB 1 22 V Measured 1 2VC 1 21 V Measured 1 2VD 1 21 V Measured 1 2VB 1 21 V Measured 1 2VE 1 21 V Measured 1 2VF 1 21 V Voltage test returned OK ture 23 00 62 12 maxtemperature maxtemperature maxtemperature maxtemperature maxtemperature maxtemperature returned OK rpm rpm rpm rpm rpm test returned OK OK OK OK OK OK OK All I4s OK 12433 12311 12311 12433 12433 Starting Connector test Connector test returned OK Starting I4 test 56 49 53 49 54 54 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 I4 test returned OK NM2 Environment test PASSED Related Information m Switch Reference env_test command Vv Display Switc
21. InfiniBand fabric These commands are also installed on and run from the management controller which is also the host of the Subnet Manager Use of these commands requires thorough knowledge of InfiniBand architecture and technology After you log in to the root account the shell prompt appears and you can enter shell commands Enter the InfiniBand commands in the following format command option option Related Information m Switch Reference understanding InfiniBand commands m Hardware Command Overview on page 67 Monitoring the Hardware The following topics enable you to display and check the operation and status of the switch Administering the Switch 67 Note To use the commands described in these topics you must be the root user of the management controller m Display Switch General Health on page 68 a Display Power Supply Status on page 68 m Check Board Level Voltages on page 69 a Display Internal Temperatures on page 70 m Display Fan Status on page 71 a Display Switch Environmental and Operational Data on page 71 a Display Switch Firmware Versions on page 73 m Display Switch Chip Boot Status on page 74 a Display Link Status on page 75 m Display Switch Chip Port Status on page 76 Related Information m Monitoring the InfiniBand Fabric on page 76 m Controlling the Hardware on page 87 Vv Display Switch General Health
22. Locate the cable to be removed Grasp the cable connector to support its weight and apply the removal force Pull on the loop of the retractor strap while simultaneously pulling on the cable connector The cable connector comes free Servicing the Switch 135 6 7 Carefully move the cable out of the cable management hardware Open hook and loop fasteners from bundles and securing hard points to gently lower the cable to the floor Caution Do not allow the cable to drop or strike the floor Jerking bending pulling on or dropping the cable can damage the cable 8 Consider your next steps m If you are removing a single cable for replacement install the new cable See Install an InfiniBand Cable on page 136 a If you are disconnecting all cables for switch replacement repeat from Step 4 for all cables Related Information Remove a Power Supply on page 118 Remove a Fan on page 127 Remove the Switch From the Rack on page 139 Replace the Battery on page 140 V Install an InfiniBand Cable 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the InfiniBand Cables on page 132 Determine your next steps m If you are cabling an entire switch after a replacement procedure locate the cable for the connector 11B and go to Step 5 a If you are installing a replacement cable to the swit
23. TX7n 54 RX5n 75 RX4n 13 GND 34 GND 55 GND 76 GND 14 TX8p 35 TX9p 56 RX11p 77 RX10p 15 TX8n 36 TX9n 57 RX11n 78 RX10n 16 GND 37 GND 58 GND 79 GND 17 TX10p 38 TX11p 59 RX9p 80 RX8p 18 TX10n 39 TX1in 60 RX9n 81 RX8n 19 GND 40 GND 61 GND 82 GND 20 SCL 41 VCC3 3 Tx 62 PRSNT 83 VCC3 3 Rx 21 SDA 42 VCC12 Tx 63 Int_L Reset 84 VCC12 Rx L The following table provides descriptions of the CXP signals Signal Description GND VCC3 3 Tx VCC3 3 Rx VCC12 Tx VCC12 Rx Ground for both signal and power return 3 3V standby power from the power supplies 3 3V standby power return 12V power for cable export and future enhancements 12V power return 6 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Signal Description TXportdiff Differential InfiniBand transmit signal where e port 0 to 11 e diff n or p for negative or positive RXportdiff Differential InfiniBand receive signal where e port 0 to 11 e diff n or p for negative or positive SCL PC clock SDA IC data I O PRSNT Connection present Int_L Reset_L Reset line Related Information a InfiniBand Cable Types on page 11 m Connecting InfiniBand Cables on page 35 Routing Service Cables These topics describe cable routing requirements m Power Cord Requirements on page 7 m Management Cable Requirements on page 8 Related Information m Understanding InfiniBand Cabling on page 9 Power Cord Requirements The p
24. adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract and to the extent applicable by the terms of the Government contract the additional rights set forth in FAR 52 227 19 Commercial Computer Software License December 2007 Oracle America Inc 500 Oracle Parkway Redwood City CA 94065 This software or hardware is developed for general use in a variety of information management applications It is not developed or intended for use in any inherently dangerous applications including applications which may create a risk of personal injury If you use this software or hardware in dangerous applications then you shall be responsible to take all appropriate fail safe backup redundancy and other measures to ensure its safe use Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications Oracle and Java are registered trademarks of Oracle and or its affiliates Other names may be trademarks of their respective owners AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices Intel and Intel Xeon are trademarks or reptetered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc UNIX is a registered trademark licensed through X Open Company Ltd This
25. example ports 20 21 and 22 of switch chip I4 A are down 2 If the ports are still down restart the environment daemon etc init d envd stop etc init d envd start Wait one minute for the links to retrain 90 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 3 Verify that the ports are up listlinkup Connector OA Present lt gt I4 A Ports 22 up 21 up 20 up Connector 1A Not present Note If the ports do not come up repeat this procedure a second time Additionally investigate if the respective cable or the InfiniBand device at the other end of the cable is at fault Related Information m Switch Reference listlinkup command m Switch Reference i4reset command m Reset the Switch Chip on page 89 m Display Link Status on page 75 Set Link Speed For some configurations vendor interoperability or to achieve compatibility it might be necessary to set link speeds to values different than default The setlinkspeed command enables you to do so for specific ports 1 On the management controller type setlinkspeed switch_chip port speed where m switch_chip is the letter of the switch chip A F m port is the number of the port 1 36 m speed is the speed of the port 1 x 2 x or 4 x 2 Reset the port enableswitchport swifch_chip port where Administering the Switch 91 m switch_chip is the letter of the switch chip A F a port is th
26. installing the switch has a specific sequence of tasks that must be performed in order The following table describes the switch installation task sequence and provides links to those procedures Step Links 1 Shipping Carton Contents on page 16 Install the Switch in the Rack on page 18 Attach the Management Cables on page 24 Attach the Power Cords on page 27 Accessing the Management Controller on page 28 Verify the Switch Status on page 31 Start the Subnet Manager on page 34 Attach the InfiniBand Cables on page 36 Y JINIAJIAJAJ JN Check Link Status on page 41 p Discover the InfiniBand Fabric Topology on page 42 m m Perform Diagnostics on the InfiniBand Fabric on page 43 m N Validate the InfiniBand Fabric and Report Errors on page 44 Shipping Carton Contents m Sun Datacenter InfiniBand Switch 72 m Cable bracket and rackmount kit Cable management bracket and cover a Two rack mounting rail assemblies a Assortment of screws and captive nuts a Sun Datacenter InfiniBand Switch 72 Getting Started Guide 16 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Item Descripton Switch Front mounting brackets long Front mounting brackets short C shaped brackets Documentation Hardware Cable management extenders Long rails o oo N DD oO FF WO NY Cable management co
27. more secure On the management controller type passwd user where user is the user name who s password is to be changed For example passwd root Changing password for user root New UNIX password new_password Retype new UNIX password new_password passwd all authentication tokens updated successfully Related Information m Access the Management Controller From the Network Management Port on page 29 m Access the Management Controller From the USB Management Port on page 30 Controlling the InfiniBand Fabric You can perform the following tasks to manage the InfiniBand fabric and its components Note To use the commands described in these topics you must be the root user of the management controller m Perform Comprehensive Diagnostics for the Entire Fabric on page 94 Administering the Switch 93 m Perform Comprehensive Diagnostics for a Route on page 95 m Determine Changes to the InfiniBand Fabric Topology on page 95 m Find 1x or SDR or DDR Links in the Fabric on page 97 m Determine Which Links Are Experiencing Significant Errors on page 97 m Clear Error Counters on page 98 m Clear Data Counters on page 99 m Check All Ports on page 99 m Reset a Port on page 100 m Set Port Speed on page 101 m Disable a Port on page 102 a Enable a Port on page 103 m Controlling the Subnet Manager o
28. page 71 V Check Network Management Port Status LEDs The network management port status LEDs are located on the network management connector of the rear panel See Rear Status LEDs on page 55 1 Visually inspect the NET status LEDs 2 Compare what you see to the following table Name Position Color State Activity Left Green On No function Off No activity Flashing Packet activity Link speed Right Amber or green Amber on 100BASE T Green on 1000BASE T Off No link or link down Flashing No function V Check Link Status LEDs The link status LEDs are located at the InfiniBand connectors of the rear panel See Rear Status LEDs on page 55 1 Visually inspect the link status LEDs Administering the Switch 57 2 Compare what you see for a particular link to the following table Name Color State and Meaning Link Green On Link established Off No link or link down Flashing Symbol errors Related Information m Display Link Status on page 75 m Display the Link Status of a Node on page 80 m Display Counters for a Node on page 81 Check Power Supply Status LEDs The power supply status LEDs are located on the power supply at the front of the chassis See Front Status LEDs on page 55 1 Visually inspect the power supply s status LEDs 2 Compare what you see on the power supply to the following table Glyph gt Bl L
29. software or hardware and documentation may provide access to or information on content products and services from third parties Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third party content products and services Oracle Corporation and its affiliates will not be responsible for any loss costs or damages incurred due to your access to or use of third party content products or services Copyright 2009 2010 Oracle et ou ses affili s Tous droits r serv s Ce logiciel et la documentation qui l accompagne sont prot g s par les lois sur la propri t intellectuelle Ils sont conc d s sous licence et soumis des restrictions d utilisation et de divulgation Sauf disposition de votre contrat de licence ou de la loi vous ne pouvez pas copier reproduire traduire diffuser modifier breveter transmettre distribuer exposer ex cuter publier ou afficher le logiciel m me partiellement sous quelque forme et par quelque proc d que ce soit Par ailleurs il est interdit de proc der a toute ing nierie inverse du logiciel de le d sassembler ou de le d compiler except a des fins d interop rabilit avec des logiciels tiers ou tel que prescrit par la loi Les informations fournies dans ce document sont susceptibles de modification sans pr avis Par ailleurs Oracle Corporation ne garantit pas qu elles soient exemptes d erreurs et vous invite
30. tch id 24 0x0021283a8620b0d0 por imc 0 0x0021283a8620b0b0 por imc 0 0x0021283a8620b0a0 por imc 0 0x0021283a8620b0c0 por imc 0 0x0021283a8620b0e0 por imc 0 ts 36 ts 36 ts 36 ts 36 ts 36 Sun DCS 72 Q Sun Sun Sun Sun Sun DCS DCS DCS DCS DCS 72 72 72 72 72 DR switcl R switc R switc R switc R switcl L 2 LC L 2 LC L 2 LC L 2 LC L 1 FC DR FC switch o4nm2 72p 2 base port base port base port base port base port Note The output in the example is just a portion of the full output Administering the Switch 77 Related Information m Switch Reference ibhosts command m Switch Reference ibswitches command V Identify All HCAs in the Fabric Similar to the ibswitches command the ibhosts command displays identity information about the HCAs in the InfiniBand fabric The output contains the GUID and name for each HCA Note Unlike the ibswitches command the ibhosts command does not display the LIDs On the management controller type ibhosts Ca 0x5080020000911314 ports 1 nsn32 50 HCA 1 Ca 0x5080020000911310 ports 1 nsn32 20 HCA 1 Ca 0x50800200008e532c ports 1 ib 71 HCA 1 Ca 0x50800200008e5328 ports 1 ib 70 HCA 1 Ca 0x50800200008296a4 ports 2 ib 90 HCA 1 Note The output in the example is just a portion of the full output Rel
31. 00 PortConfigurationTraining Related Information m Switch Reference ipportstate command a Enable a Switch Chip Port on page 92 m Disable a Port on page 102 m Reset a Port on page 100 Controlling the Subnet Manager The OpenSM Subnet Manager is enabled with the enablesm command or started with the opensmd daemon When the Subnet Manager starts it reads the etc opensm opensm conf file for configuration information The Subnet Manager is disabled with the disablesm command or stopped with the opensmd daemon The following topics describe how to control the Subnet Manager m Create the guid txt File on page 105 m Create the opensm conf File on page 106 m Enable the Subnet Manager on page 107 m Disable the Subnet Manager on page 107 m Set the Subnet Manager Priority on page 108 m Start the Subnet Manager With the opensmd Daemon on page 109 104 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m Stop the Subnet Manager With the opensmd Daemon on page 109 Related Information m Controlling the Hardware on page 87 m Controlling the InfiniBand Fabric on page 93 V Create the guid txt File If your InfiniBand fabric uses the Fat Tree routing algorithm you must provide a root node GUID file for the Subnet Manager The etc opensm guid txt file is such a file This procedure describes how to create the guid txt file 1 Type the followi
32. 010 5 6 Set the fan aside Consider your next steps m If you are removing the fan for replacement install a new fan See Install a Fan on page 130 m If you are removing the fan as a subtractive action you are finished Note If fewer than two fans are operational the switch shuts down to prevent thermal overload Related Information Remove a Power Supply on page 118 Remove an InfiniBand Cable on page 134 Remove the Switch From the Rack on page 139 Replace the Battery on page 140 Install a Fan 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Fans on page 124 Inspect the replacement fan See Inspecting a Fan on page 125 Verify that the slot where the fan installs is clean and free of debris Verify that the slot connector pins are straight and not missing Orient the fan over the opening in the switch chassis with the thumbscrew on the right Firmly slide the fan into the chassis until the fan stops Servicing the Switch 129 The fan might immediately power on 7 Tighten the captive thumbscrew to secure the fan in the switch chassis 8 Verify that the fan Attention LED goes out Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 130 9 Use the get fanspeed command on the management controller to verify the fan s operation F
33. 07 Set the Subnet Manager Priority 108 Start the Subnet Manager With the opensmd Daemon 109 q lt 4 lt 4 lt Stop the Subnet Manager With the opensmd Daemon 109 Servicing the Switch 111 Understanding Service Procedures 111 Replaceable Components 112 Servicing the Power Supplies 113 Inspecting a Power Supply 114 v Identify the Power Supply 114 v Inspect the Power Supply Hardware 116 v Inspect the Power Supply Connectors 116 Power Off a Power Supply 117 Remove a Power Supply 118 lt lt lt Install a Power Supply 120 v Power On a Power Supply 122 Servicing the Fans 124 Contents ix Inspecting a Fan 125 v Identify the Fan 125 v Inspect the Fan Hardware 126 v Inspect the Fan Connector 127 v RemoveaFan 127 v InstallaFan 130 Servicing the InfiniBand Cables 132 Inspecting an InfiniBand Cable 132 v Identify the InfiniBand Cable 133 v Inspect the InfiniBand Cable Hardware 133 v Inspect the InfiniBand Cable Connectors 134 v Remove an InfiniBand Cable 134 v Install an InfiniBand Cable 136 Servicing the Battery 139 v Remove the Switch From the Rack 139 v Replace the Battery 140 Upgrading the Firmware 145 Index 147 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Using This Documentation This user guide provides detailed procedures that describe preparation installation administration and service for the Sun Datacenter InfiniBand Switch 72 from Oracle This document is written for technicians sys
34. 14 Port 14 Pot gt 4 Port 14 Pot gt 4 Port A 1 E 9 C 1 F 9 E 1 A 9 F 1 C 9 A 2 gt F 8 C 2 gt E 8 E 2 gt C 8 2 gt A 8 A 3 gt F 7 C 3 gt E 7 E 3 gt C 7 F 3 gt A 7 A 4 gt E 6 C 4 gt F 6 E 4 gt A 6 F 4 gt C 6 A 5 gt F 5 C 5 gt E 5 E 5 gt C 5 F 5 gt A 5 A 6 gt E 4 C 6 gt F 4 E 6 gt A 4 F 6 gt C 4 A 7 gt F 3 Cc 7 gt E 3 E 7 gt C 3 F 7 gt A 3 A 8 gt F 2 C 8 gt E 2 E 8 gt C 2 F 8 gt A 2 A 9 gt E 1 C 9 gt F 1 E 9 gt A 1 F 9 gt C 1 A 10 gt F 19 C 10 gt E 19 E 10 gt D 13 F 10 gt B 13 A 11 gt E 20 C 11 gt F 20 E 11 gt B 14 F 11 gt D 14 A 12 gt E 21 C 12 gt F 21 E 12 gt B 15 F 12 gt D 15 A 133 gt F 27 C 13 gt E 27 E 13 gt D 10 F 133 gt B 10 A 14 gt E 26 C 14 gt F 26 E 14 gt B 11 F 14 gt D 11 A 15 gt E 25 C 15 gt F 25 E 1 5 gt B 12 F 15 gt D 12 A 16 gt F 24 C 16 gt E 24 E 16 gt D 18 F 16 gt B 18 A 17 gt E 23 C 17 gt F 23 E 17 gt B 17 F 17 gt D 17 A 18 gt F 22 C 18 gt E 22 E 18 gt D 16 F 18 gt B 16 B 1 gt E 28 D 1 gt F 28 E 19 gt C 10 F 19 gt A 10 Administering the Switch 61 14 Pot gt l4 Port 14 Pot gt 14 Port 14 Pot gt l4 Port 14 Pot gt 14 Port B 2 F 29 D 2 E 29 E 20 A 11 F 20 C 11 B 3 gt F 30 D 3 gt E 30 E 21 gt A 12 F 21 gt C 12 B 4 gt E 36 D 4 gt F 36 E 22 gt C 18 F 22 gt A 18 B 5 gt F 35 D 5 gt E 35 E 23 gt A 17 F 23 gt C 17 B 6 gt E 34 D 6 gt F 34 E 24 gt C 1
35. 2004 LinkUp LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 4X LinkSpeedSupported 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 2008 10 0 Gbps After PortInfo set Port info Lid 24 port 8 ji Related Information m Switch Reference ibportstate command m Disable a Switch Chip Port on page 92 m Enable a Port on page 103 m Reset a Port on page 100 V Enable a Port After disabling a port you can enable the port with the ibportstate command On the management controller type ibportstate lid port enable where m lid is the LID of the node Administering the Switch 103 m port is the port of the node For example ibportstate 24 8 enable Initial PortInfo Port info Lid 24 port 8 PinkStateiwsicadwews audited acdkas Down PhysLinkState 22000 Disabled LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X DINKWLACHACEE Ve ions aeaea eek oraa we 4X LinkSpeedSupported 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 00 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 2 eee eee 2 5 Gbps After PortInfo set Port info Lid 24 port 8 LinkStatencccetssdbeeetaaed eee ds Down PhysLinkState 222
36. 29 connecting InfiniBand cables 35 connector CXP 5 InfiniBand cable 5 network management 4 USB management 5 controlling hardware 87 InfiniBand fabric 93 Subnet Manager 104 creating guid txt file 105 opensm conf file 106 current 3 CXP connector 5 mapping from GUID 74 D data counters clearing 99 displaying 82 diagnosing the InfiniBand fabric 43 diagnostics fabric 94 route 95 disablesm command 107 disableswitchport command 92 disabling port 102 Subnet Manager 107 switch chip port 92 discovering the InfiniBand fabric 42 displaying node counters 81 82 information 83 link status 80 port information 84 route 79 Subnet Manager status 87 switch firmware 73 switch health 68 topology 78 E electrical specifications 3 current 3 power 3 voltage 3 enablesm command 34 107 enableswitchport command 92 enabling port 103 Subnet Manager 34 107 switch chip port 92 env_test command 31 71 environment daemon 90 environmental requirements 3 error counters clearing 98 displaying 81 example route through switch 64 F fan checking LEDs 59 speed 71 status 71 features 125 inspecting 125 connector 127 hardware 126 installing 130 LED 55 removing 127 servicing 124 features fan 125 148 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 InfiniBand cable 133 power supply 114 findport command 74 firmware switch 73 upgrading
37. 3 m Servicing the InfiniBand Cables on page 132 124 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m Servicing the Battery on page 139 Inspecting a Fan Before installing a fan inspect its hardware and connector to verify its suitability for installation Step Description Links 1 Identify the fan Identify the Fan on page 125 2 Inspect the hardware Inspect the Fan Hardware on page 126 3 Inspect the connector Inspect the Fan Connector on page 127 Related Information m Inspecting a Power Supply on page 114 m Inspecting an InfiniBand Cable on page 132 V Identify the Fan 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Fan on page 125 2 Use the following illustration to identify the various features of a fan Servicing the Switch 125 FIGURE Fan Features Figure Legend 1 Thumbscrew 2 Status LED 3 Inspect the fan hardware See Inspect the Fan Hardware on page 126 Related Information m Identify the Power Supply on page 114 m Identify the InfiniBand Cable on page 133 V Inspect the Fan Hardware 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Fan on page 125 2 Unwrap the replacement fan from its antistatic packaging 3 Verify that there i
38. 3 19 45 02 375997 B7EF96C0 0x02 gt osmtest_validate_sa_class_port_info SA Class Port Info base_ver 1 class_ver 2 cap_mask 0x2602 cap_mask2 0x0 resp_time_val 0x10 Aug 13 19 45 02 378991 B7EF96C0 0x01 gt osmtest_validate_node_data Checking node 0x0021283a8620b0a0 LID 0x14 Aug 13 19 45 02 379172 B7EF96C0O 0x01 gt osmtest_validate_node_data Checking node 0x0021283a8620b0b0 LID 0x15 Aug 13 19 45 02 480201 B7EF96CO 0x01 gt osmtest_validate_single path_rec_guid_pair Checking src 0x0021283a8620b0f0 to dest 0x0021283a8620b0 f0 Aug 13 19 45 02 480588 B7EF96C0 0x01 gt osmtest_validate_path_data Checking path SLID 0x19 to DLID 0x19 Aug 13 19 45 02 480989 B7EF96C0O 0x02 gt osmtest_run 96 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 KKEKKKKKKKKKKKKKKK ALL TESTS PASS KKEKKKKKKKKKKKEKKKK OSMT EST T EST Validate Inventory PASS Note Depending on the size of your InfiniBand fabric the output from the osmtest command could be tens of thousands of lines long Related Information m Switch Reference osmtest command m Display the InfiniBand Fabric Topology on page 78 V Find 1x or SDR or DDR Links in the Fabric You can use the ibdiagnet command to determine which links are at 1x bandwidth 2 5 Gbps or 5 Gbps data rate On the management controller type ibdiagnet lw 4x ls 10 pc pm skip all In this instance of t
39. 3a8638b0c0 is truncated to b0c0 The b0 means that the node is inside of a Sun Datacenter InfiniBand Switch 72 The c means that the node is located in I4 switch chip C The port is 21 3 Use the tables in Understanding Routing Through the Switch on page 59 to cross reference the routing to other switch chips and ports and to connections For example using Switch Chip Port to CXP Connectors and Link LED Routes on page 62 I4 switch chip C port 21 routes to connector 6A the center LED and the link is carried by P2 Related Information m Switch GUIDs Overview on page 65 Vv Display Switch Chip Boot Status If you believe there is a problem with a switch chip the checkboot command provides a quick pass fail check of the switch chips Should a switch chip fail the test you can use the i4reset command in attempt to reboot the switch chip On the management controller type checkboot I4 A OK I4 B OK I4 C OK T4 D OK 74 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 I4 E OK I4 F OK Related Information m Switch Reference checkboot command m Reset the Switch Chip on page 89 m Display Switch Environmental and Operational Data on page 71 Vv Display Link Status In some situations you might need to know the status of each route through the switch Additionally the list1linkup command displays where InfiniBand cables are connected to the switch On the managem
40. 6 F 24 gt A 16 B 7 gt F 33 D 7 gt E 33 E 25 gt A 15 F 25 gt C 15 B 8 gt F 32 D 8 gt E 32 E 26 gt A 14 F 26 gt C 14 B 9 gt E 31 D 9 gt F 31 E 27 gt C 13 F 27 gt A 13 B 10 gt F 13 D 10 gt E 13 E 28 gt B 1 F 28 gt D 1 B 11 gt E 14 D 11 gt F 14 E 29 gt D 2 F 29 gt B 2 B 12 gt E 15 D 12 gt F 15 E 30 gt D 3 F 30 gt B 3 B 13 gt F 10 D 13 gt E 10 E 31 gt B 9 F 31 gt D 9 B 14 gt E 11 D 14 gt F 11 E 32 gt D 8 F 32 gt B 8 B 15 gt E 12 D 15 gt F 12 E 33 gt D 7 F 33 gt B 7 B 16 gt F 18 D 16 gt E 18 E 34 gt B 6 F 34 gt D 6 B 17 gt E 17 D 17 gt F 17 E 35 gt D 5 F 35 gt B 5 B 18 gt F 16 D 18 gt E 16 E 36 gt B 4 F 36 gt D 4 Related Information m CXP Connectors and Link LEDs to Switch Chip Port Routes on page 60 m Switch Chip Port to CXP Connectors and Link LED Routes on page 62 m Signal Route Through the Switch on page 64 m Display Link Status on page 75 m Display Switch Chip Port Status on page 76 Switch Chip Port to CXP Connectors and Link LED Routes A reverse lookup mapping of the switch chip and port to CXP connector and its link LEDs is provided in the following table The connectors and LEDs are given as connectorletter location where m connector The identifying number of the connector group 0 11 m letter The letter identifying the upper A or lower B connector 62 Sun Datacenter InfiniBand Switc
41. LED 1 2 Power supply Attention LED 3 Power supply OK LED 4 Fan status LEDs Related Information m Check Power Supply Status LEDs on page 58 m Check Fan Status LEDs on page 59 Rear Status LEDs The chassis status LEDs network management status LEDs and link status LEDs are located on the rear of the switch chassis Administering the Switch 55 FIGURE Rear Status LEDs Figure Legend 1 Chassis status LEDs 2 NET status LEDs 3 Link status LEDs Related Information m Check Chassis Status LEDs on page 56 m Check Network Management Port Status LEDs on page 57 m Check Link Status LEDs on page 57 V Check Chassis Status LEDs The chassis status LEDs are located on the left side of the rear panel See Rear Status LEDs on page 55 1 Visually inspect the chassis status LEDs 56 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 2 Compare what you see to the following table Glyph Location Name Color State and Meaning Top Locator White On No function Off Disabled Flashing The switch is identifying itself Middle Attention Amber On Normal fault detected Off No faults detected Flashing No function Bottom OK Green On Switch is functional without fault Off Switch is off or initializing R gt amp Flashing No function Related Information m Display Switch Environmental and Operational Data on
42. Priority 3 Creating config file template tmp osm conf Log File var log opensm log 1000 108 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 2 Restart the Subnet Manager disablesm Stopping IB Subnet Manager OK enablesm Starting IB Subnet Manager OK Related Information m Switch Reference setsmpriority command m Create the opensm conf File on page 106 V Start the Subnet Manager With the opensmd Daemon The opensmd daemon enables you to start the OpenSM Subnet Manager without providing command line configuration parameters On the management controller type etc init d opensmd start Starting IB Subnet Manager OK The Subnet Manager is started Related Information m Switch Reference opensmd command m Create the guid txt File on page 105 m Create the opensm conf File on page 106 m Enable the Subnet Manager on page 107 m Stop the Subnet Manager With the opensmd Daemon on page 109 V Stop the Subnet Manager With the opensmd Daemon You can stop the OpenSM Subnet Manager with the opensmd daemon Administering the Switch 109 110 On the management controller type etc init d opensmd stop Stopping IB Subnet Manager OK The Subnet Manager is stopped Related Information m Switch Reference opensmd command m Disable the Subnet Manager on page 107 m Start the Sub
43. Sun Datacenter InfiniBand Switch 72 User s Guide sS 2 Sun Part No 820 7751 12 May 2010 Revision A VISIO Copyright 2009 2010 Oracle and or its affiliates All rights reserved This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws Except as expressly permitted in your license agreement or allowed by law you may not use copy reproduce translate broadcast modify license transmit distribute exhibit perform publish or display any part in any form or by any means Reverse engineering disassembly or decompilation of this software unless required by law for interoperability is prohibited The information contained herein is subject to change without notice and is not warranted to be error free If you find any errors please report them to us in writing If this is software or related software documentation that is delivered to the U S Government or anyone licensing it on behalf of the U S Government the following notice is applicable U S GOVERNMENT RIGHTS Programs software databases and related documentation and technical data delivered to U S Government customers are commercial computer software or commercial technical data pursuant to the applicable Federal Acquisition Regulation and agency specific supplemental regulations As such the use duplication disclosure modification and
44. age 27 Management Cable Requirements Management of the switch is done at the management console which is either a 10 100 Ethernet connection at the NET ports or a USB to serial device attached to the USB port Typically the NET connection network management is the default means of communicating with the management controller The controller has a DHCP client in operation and requires the Ethernet network to have a DHCP server The DHCP server must be configured with the MAC address of the management controller so the server can provide an IP address to the management controller upon boot If a DHCP server is not available then the USB connection is used 8 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 The advantage of the NET connection over the USB connection is that administration of the switch can happen from anywhere on the network There is no cable length constraint for the network management route because of the re amplification filtering and processing that happens at each hub or switch within the Ethernet network No network management cable should be any longer than 100 meters The USB connection requires a USB to serial adapter The adapter must be configured to communicate with your serial device management console The serial device can be a serial terminal a terminal server or a serial connection running on a system or laptop Because of the nature of the serial signal a serial management cable cannot be
45. agement Connector and Pins The following table lists the pinout of the network management connector 5 Signal TXD TXD RXD Not used Not used RXD Not used o N A a A Wo N e Not used Related Information m Management Cable Requirements on page 8 m Attach the Management Cables on page 24 m Access the Management Controller From the Network Management Port on page 29 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 USB Management Connector and Pins The following table lists the pinout of the USB management connector Pin Signal 1 5 VDC 2 Data 3 Data 4 GND Related Information m Management Cable Requirements on page 8 m Attach the Management Cables on page 24 m Access the Management Controller From the USB Management Port on page 30 CXP Connector and Pins The CXP connector has three InfiniBand port connections 84 44 hl a Np i is Installing the Switch 5 The following table lists the pinout for each connection Pin Signal Pin Signal Pin Signal Pin Signal 1 GND 22 GND 43 GND 64 GND 2 TXOp 23 TX1p 44 RX3p 65 RX2p 3 TX0n 24 TX1in 45 RX3n 66 RX2n 4 GND 25 GND 46 GND 67 GND 5 TX2p 26 TX3p 47 RX1p 68 RXOp 6 TX2n 27 TX3n 48 RXin 69 RXOn 7 GND 28 GND 49 GND 70 GND 8 TX4p 29 TX5p 50 RX7p 71 RX6p 9 TX4n 30 TX5n 51 RX7n 72 RX6n 10 GND 31 GND 52 GND 73 GND 11 TX6p 32 TX7p 53 RX5p 74 RX4p 12 TX6n 33
46. all a Power Supply on page 120 a Install a Fan on page 130 m Replace the Battery on page 140 138 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Servicing the Battery Oracle s Sun Datacenter InfiniBand Switch 72 has a battery on the main board that supports the management controller You can only replace the battery because the management controller is dependent upon the battery You cannot add or subtract the battery Perform these tasks in order to replace the battery Step Description Links 1 Remove all IB cables Remove an InfiniBand Cable on page 134 2 Power off both power supplies Power Off a Power Supply on page 117 3 Remove the switch from the rack Remove the Switch From the Rack on page 139 4 Replace the battery Replace the Battery on page 140 5 Install the switch in the rack Installing the Switch on page 1 Related Information m Understanding Service Procedures on page 111 m Servicing the Power Supplies on page 113 m Servicing the Fans on page 124 m Servicing the InfiniBand Cables on page 132 Remove the Switch From the Rack 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Battery on page 139 Note You must remove all InfiniBand cables from the switch and power down both power supplies by removing both power cords See Rem
47. anagement 30 acoustic noise emissions 3 idling 3 operating 3 administering the switch 47 administrative command overview 66 antistatic precautions 15 attaching InfiniBand cables 36 management cables 24 power cords 27 battery replacing 140 servicing 139 C changing password 93 checkboot command 31 74 checking board temperature 70 voltage 69 chassis status 71 environmental data 71 fan speed 71 status 71 LEDs chassis status 56 fan 59 link 57 network management 57 power supply 58 link status 75 port 99 port status 76 power supply 68 switch chip boot status 74 checkpower command 31 68 122 checkvoltages command 69 122 clear data counters 99 error counters 98 command checkboot 31 74 checkpower 31 68 122 checkvoltages 69 122 disablesm 107 disableswitchport 92 enablesm 34 107 enableswitchport 92 env_test 31 71 findport 74 getbaseguid 105 getfanspeed 31 71 getportstatus 76 i4reset 89 90 ibcheckerrors 44 ibcheckstate 99 ibclearcounters 99 ibclearerrors 98 ibdatacounts 82 ibdiagnet 43 94 97 ibdiagpath 95 ibhosts 78 ibnetdiscover 42 78 ibportstate 80 100 101 102 103 ibswitches 77 ibtracert 79 147 listlinkup 41 75 90 managementreset 89 nm2version 73 opensm 105 opensmd 109 osmtest 86 95 perfquery 81 reboot 88 setlinkspeed 91 setsmpriority 34 108 showtemps 70 showunhealthy 31 68 sminfo 87 smpquery 83 84 ssh
48. ated Information m Inspect the Power Supply Hardware on page 116 Servicing the Switch 133 Inspect the Fan Hardware on page 126 V Inspect the InfiniBand Cable Connectors 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting an InfiniBand Cable on page 132 Verify that the shell is not bent and is parallel to the inner boards Verify that there are no contaminants inside of the connector Verify that the retractor strap adequately operates the hooks The cable is ready for installation See Install an InfiniBand Cable on page 136 Related Information Inspect the Power Supply Connectors on page 116 Inspect the Fan Connector on page 127 V Remove an InfiniBand Cable This procedure describes how to remove the cables from the switch chassis so that the cable can be replaced If you are removing all cables for switch replacement start removing the cables from the left side of the switch working your way to the right 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the InfiniBand Cables on page 132 Loosen the thumbscrews and remove the cover for the cable management bracket 134 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 gt Baeae J A f or ak A 3 3 g d g As OODODooES
49. ated Information m Switch Reference ibhosts command m Switch Reference ibswitches command Vv Display the InfiniBand Fabric Topology To understand the routing that happens within your InfiniBand fabric the ibnetdiscover command displays the node to node connectivity The output of the command is dependent upon the size of your fabric 78 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 On the management controller type ibnetdiscover Related Information m Switch Reference ibnetdiscover command m Determine Changes to the InfiniBand Fabric Topology on page 95 m Perform Comprehensive Diagnostics for the Entire Fabric on page 94 Vv Display a Route Through the Fabric Often you need to know the route between two nodes The ibtracert command can provide that information by displaying the GUIDs ports and LIDs of the nodes along the route On the management controller type ibtracert slid dlid where m slid is the LID of the source node m dlid is the LID of the destination node For example ibtracert 25 24 From switch 0x0021283a8620b0f0 portnum 0 lid 25 25 Sun DCS 72 QDR FC switch o4nm2 72p 2 1 gt switch port 0x0021283a8620b0c0 9 lid 22 22 Sun DCS 72 QDR switch 1 2 LC 2 gt switch port 0x0021283a8620b0e0 8 lid 24 24 Sun DCS 72 QDR switch 1 1 FC To switch 0x0021283a8620b0e0 portnum 0 lid 24 24 Sun DCS 72 QDR switch 1 1 FC
50. bmit comments about this document by clicking the Feedback link at http docs sun com Include the title and part number of your document with your feedback Sun Datacenter InfiniBand Switch 72 User s Guide part number 820 7751 12 xii Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Installing the Switch The following topics describe the installation of the switch The topics are listed in the order of completion m Understanding Switch Specifications on page 1 m Routing Service Cables on page 7 m Understanding InfiniBand Cabling on page 9 m Understanding the Installation on page 14 m Shipping Carton Contents on page 16 m Install the Switch in the Rack on page 18 m Powering On the Switch on page 23 m Connecting InfiniBand Cables on page 35 m Verifying the InfiniBand Fabric on page 41 Related Information m Administering the Switch on page 47 m Servicing the Switch on page 111 m Switch Remote Administration m Switch Reference Understanding Switch Specifications These topics describe the specifications of the switch and the connectors found on the switch chassis and the pinouts of those connectors m Physical Specifications on page 2 m Environmental Requirements on page 3 m Acoustic Noise Emissions on page 3 m Electrical Specifications on page 3 2 a Network Management Connector and Pins on pag
51. cable into the open slot on the cable management bracket 10 Repeat Step 4 through Step 9 for all cables to be installed 11 Replace the cover for the cable management bracket and tighten the thumbscrews Installing the Switch 39 12 Route the InfiniBand cables so that they do not interfere with other cables or with servicing the Sun Datacenter InfiniBand Switch 72 or other systems Use hook and loop fastener straps to bundle and secure the cables Note Do not use cable zip ties to bundle or secure the cable because the ties damage the wires inside the cable 13 Check that the Link LEDs for cabled links are lit green If the Link LED is unlit the link is down If the Link LED flashes there are symbol errors See Check Link Status LEDs on page 57 14 If installed close the rack doors to maintain EMI compliance 15 Check the link status See Check Link Status on page 41 Related Information a Understanding InfiniBand Cabling on page 9 m Attach the Management Cables on page 24 m Attach the Power Cords on page 27 40 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Y Check Link Status 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 On the management controller determine the state of the links listlinkup m If the link for a conne
52. ch start the procedure at Step 3 Inspect the replacement InfiniBand cable See Inspecting an InfiniBand Cable on page 132 Bring the replacement cable to the switch Feed the cable through the cable management hardware 136 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 6 Orient the cable connector to the CXP receptacle squarely and horizontally Ensure that the upper shell just touches the underside of the top of the receptacle on the rear panel 7 Slowly move the connector in As you slide the connector in the top of the shell should scrape against the underside of the top of the CXP receptacle m If the connector stops or binds after about 1 4 in 5 mm travel back out and repeat from Step 6 m If the connector stops or binds with about 1 8 in 2 mm still to go back out and repeat Step 7 8 Continue to push the connector in until the hooks catch onto the top of the receptacle 9 Secure the cable into the cable management hardware Close hook and loop fasteners at bundles and securing hard points Servicing the Switch 137 10 If you are installing all cables as part of a switch replacement procedure repeat from Step 5 for all cables working your way from right to left 11 Replace the cover for the cable management bracket and tighten the thumbscrews Dunnan f 0 f Hi or Ae a is A g d a g A SWOOOOoOoES Related Information a Inst
53. ctor is reported as Not present there is no cable attached or the link at either end of the cable is down m Ifaportis down use the enableswitchport command to bring the port up Alternatively use the i4reset command to reset the entire switch chip See Enable a Switch Chip Port on page 92 and Reset the Switch Chip on page 89 3 Verify the InfiniBand fabric See Verifying the InfiniBand Fabric on page 41 Related Information m Switch Reference listlinkup command m Switch Reference enableswitchport command m Switch Reference i4reset command m Verify the Switch Status on page 31 Verifying the InfiniBand Fabric Use the ibnetdiscover ibdiagnet and ibcheckerrors commands to initially determine the operational status of your switch in the InfiniBand fabric m Discover the InfiniBand Fabric Topology on page 42 m Perform Diagnostics on the InfiniBand Fabric on page 43 m Validate the InfiniBand Fabric and Report Errors on page 44 Related Information m Powering On the Switch on page 23 m Connecting InfiniBand Cables on page 35 Installing the Switch 41 Vv Discover the InfiniBand Fabric Topology The ibnetdiscover command enables you to see the InfiniBand fabric topology and build a topology file which is used by the OpenSM Subnet Manager 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See
54. d Fabric Topology 42 v Perform Diagnostics on the InfiniBand Fabric 43 v Validate the InfiniBand Fabric and Report Errors 44 Administering the Switch 47 Troubleshooting the Switch 47 Switch Hardware Problems 48 InfiniBand Fabric Problems 51 vi Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Understanding the LEDs 54 Front Status LEDs 55 Rear Status LEDs 55 Check Chassis Status LEDs 56 Check Network Management Port Status LEDs 57 Check Link Status LEDs 57 Check Power Supply Status LEDs 58 v Check Fan Status LEDs 59 lt lt lt Understanding Routing Through the Switch 59 CXP Connectors and Link LEDs to Switch Chip Port Routes 60 Switch Chip Port to Switch Chip Port Routes 61 Switch Chip Port to CXP Connectors and Link LED Routes 62 Signal Route Through the Switch 64 Switch GUIDs Overview 65 Administrative Command Overview 66 Hardware Command Overview 67 InfiniBand Command Overview 67 Monitoring the Hardware 67 v DENIE SSES SEES SE SEE E EE 4 Display Switch General Health 68 Display Power Supply Status 68 Check Board Level Voltages 69 Display Internal Temperatures 70 Display Fan Status 71 Display Switch Environmental and Operational Data 71 Display Switch Firmware Versions 73 Locate a Switch Chip or Connector From the GUID 74 Display Switch Chip Boot Status 74 Display Link Status 75 Contents vii v Display Switch Chip Port Status 76 Monitoring the InfiniBand Fabric 76 v Id
55. de If intensive troubleshooting is necessary to resolve a problem the smpquery command can provide very detailed information about a node Administering the Switch 83 On the management controller type smpquery switchinfo lid where lid is the LID of the node For example to see detailed information about a switch with LID 24 type smpquery switchinfo 24 Switch info Lid 24 LaimearFdbCaps 0 ccc tected ead 49152 RandomPdbCap re aae nan ees eek eee os 0 McastFdbCapii on Seen Sew eee esa ee 4096 LinearFdbTOps sii s ese eae ead 25 DefPOorE ears ee a a eee a a 0 DefMcastPrimPort 4 255 DefMcastNotPrimPort 255 Bi Pet Mee cio oe ee oe ahd aera Sees 18 StateChange venice aei e Hause late des 0 LidSPerPort ooa eine salen oe 0 PartEntonceCap ids wd ik OSS d 32 TnboundPar thn five dies pud dee eae Hake 1 QutboundPartln egare os Ses es 1 FilterRawInbound 6 1 FilterRawOutbound 1 FnhancedPor tone viene ce bi aai s ele t 0 Related Information m Switch Reference smpquery command Vv Display Low Level Detailed Information About a Port The smpquery command can provide very detailed information about a port On the management controller type smpquery portinfo lid port where m lid is the LID of the node 84 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m port is the port of t
56. dinated straps to help identify cables and their routing The InfiniBand splitter and 4X copper conductor cables are fairly thick and heavy for their length Consider the retention strength of the hook and loop straps when supporting cables Bundle as few cables as reasonably possible If the InfiniBand cables break free of their straps and fall free the cables might break internally when they strike the floor or are jerked from tension You can bundle the cables using many hook and loop straps Do not bundle more than 12 cables together A fully configured switch has 24 InfiniBand cables so that is at least two bundles 12 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Place the hook and loop straps as close together as reasonably possible For example every 1 ft 0 3 m If a cable breaks free from a strap the cable can not fall far before it is retained by another strap Related Information m Connecting InfiniBand Cables on page 35 Floor and Underfloor Delivery of InfiniBand Cables The switch accepts InfiniBand cables from floor or underfloor delivery The cable management hardware at the rear of the switch supports the weight of the InfiniBand cables Floor and underfloor delivery limits the tension in the InfiniBand cable to the weight of the cable for the rack height of the switch Related Information m Connecting InfiniBand Cables on page 35 Overhead Delivery of InfiniBand Cables For overhead del
57. e on page 145 Replaceable Components The following illustration identifies the replaceable components of the switch FIGURE Replaceable Components Figure Legend 1 Battery 2 Fan 3 Power supply 112 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Related Information m Servicing the Power Supplies on page 113 m Servicing the Fans on page 124 m Servicing the InfiniBand Cables on page 132 m Servicing the Battery on page 139 Servicing the Power Supplies The following table provides a description of a goal you want to achieve and links to the sequential tasks required to reach that goal Description Links Add a power supply Inspecting a Power Supply on page 114 Install a Power Supply on page 120 Power On a Power Supply on page 122 Replace a power supply Inspecting a Power Supply on page 114 Power Off a Power Supply on page 117 Remove a Power Supply on page 118 Install a Power Supply on page 120 Power On a Power Supply on page 122 Subtract a power supply Power Off a Power Supply on page 117 Remove a Power Supply on page 118 Related Information m Understanding Service Procedures on page 111 m Servicing the Fans on page 124 m Servicing the InfiniBand Cables on page 132 m Servicing the Battery on page 139 Servicing the Switch 113 114 Inspecting a Power Supply Before
58. e 4 m USB Management Connector and Pins on page 5 m CXP Connector and Pins on page 5 Related Information m Install the Switch in the Rack on page 18 a Powering On the Switch on page 23 Physical Specifications Dimension Measurements Width 17 52 in 445 0 mm Depth 24 in 609 6 mm Height 1 75 in 44 5 mm Weight 23 0 Ibs 11 4 kg Related Information m Shipping Carton Contents on page 16 m Install the Switch in the Rack on page 18 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Environmental Requirements Parameter Operating Ambient temperature 41 F to 89 6 F 5 C to 32 C Relative humidity 5 to 85 noncondensing 80 F 27 C maximum wet bulb Elevation Sun requirement Maximum 9840 feet 3000 meters at 104 F 40 C Acoustic Noise Emissions Parameter Operating Idling Acoustic power LWAd 1B 10dB 7 1 B 7 2 B Acoustic pressure LpAm 58 9 dBA 59 0 dBA Electrical Specifications Parameter AC Version Requirement Voltage 100 VAC to 240 VAC single phase 47 to 63 Hz Current per input 5 4 A maximum per input at 100 VAC Current total 5 6 A maximum total for all inputs at 100 VAC Power 550 Watts Total input power is approximately equally divided among the operating power supplies Related Information m Power Cord Requirements on page 7 m Attach the Power Cords on page 27 Installing the Switch 3 Network Man
59. e number of the port 1 36 Related Information m Switch Reference setlinkspeed command m Switch Reference enableswitchport command a Understanding Routing Through the Switch on page 59 Vv Disable a Switch Chip Port On occasion you might need to turn off a port For example a cable has become damaged and is causing symbol errors that are affecting the switch chip s performance Use the disableswitchport command to disable a switch chip port On the management controller type disableswitchport switch_chip port where m switch_chip is the letter of the switch chip A F m port is the number of the port 1 36 Related Information a Switch Reference disableswitchport command m Disable a Port on page 102 a Enable a Switch Chip Port on page 92 Vv Enable a Switch Chip Port You can enable a disabled switch chip port with the enableswitchport command On the management controller type enableswitchport switch_chip port where m switch_chip is the letter of the switch chip A F m port is the number of the port 1 36 92 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Related Information m Switch Reference enableswitchport command m Enable a Port on page 103 m Disable a Switch Chip Port on page 92 V Change the Administrator Password The default password for the root user is changeme You can use the passwd command to make the password unique and
60. e value is 0x2 The following table provides values for Type Pos Device Board Fabric card Line card Gateway 36 Port 72 Port Type Position Device OxF 0x0 Fabric card 0 0x8 Fabric card 8 OxA I4 chip 0 OxB I4 chip 1 0x1 0x0 Line card 0 0x8 Line card 8 OxA I4 chip 0 OxD I4 chip 3 0x3 0x0 0xA 0xB I4 chips 0 1 0x0 0x4 Gbe chips OxA 0x0 OxA I4 chip 0 0xB 0x0 OxA I4 chip 0 0xF I4 chip 5 For example given the following output from the ibswitches command Switch 0x0021283a8620b0e0 ports 36 Sun DCS 72 QDR switch 1 1 FC base port 1 lid 24 Imc 0 The GUID is 0x0021283a8620b0e0 or 0021283a8620 B 0 E 0 Using the information provided in this topic m The MAC address is 0x0021283a8620 or 00 21 28 3A 86 20 m The type is 0xB or a 72 port board Administering the Switch 65 The position is 0x0 The device is OxE or 14 switch chip E The number is 0 Related Information Switch Reference ibnetdiscover command Switch Reference ibnodes command Switch Reference ibswitches command Switch Reference ibhosts command Identify All Switches in the Fabric on page 77 Switch Hardware Problems on page 48 InfiniBand Fabric Problems on page 51 Understanding the LEDs on page 54 Understanding Routing Through the Switch on page 59 Administrative Command Overview The following topics provide an overview of administrative tasks and the command sets to p
61. ed to the chassis the management controller is powered on The main power is supplied for the switch chips and fans 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 Ensure that the circuit breakers for the power supplies are switched off 3 Plug a power cord into each power receptacle at the front of the chassis p IMO DRRORRE J O A O BERRA RERA F 4 Route each power cord to its respective facility power receptacle Use cable ties or hook and loop fastener straps to bundle and secure the cord Installing the Switch 27 5 Plug each power cord into the receptacle 6 Energize the circuit breakers so that the power receptacles are live 7 Verify that the status LEDs for each power supply indicates normal operation The AC LED lights and in a moment the OK LED should light The Attention LED should be unlit See Check Power Supply Status LEDs on page 58 Note At this time power is being supplied to the management controller The controller is effectively on and booting up You might see the boot sequence on the management console 8 Verify that the fans spin up You should feel air going into the fans and the fan Attention LEDs should be unlit See Check Fan Status LEDs on page 59 9 Verify that the chassis sta
62. ent controller type listlinkup Connector Connector Connector Connector Connector Connector Connector Link I4 A Link I4 A Link I4 A Link I4 D OA 1A 2A 3A 4A 10B 11B Present lt gt I4 A Ports 22 up 21 up 20 up Not present Not present Not present Not present Not present Not present 01 lt gt I4 E 09 up 02 lt gt I4 F 08 up 03 lt gt I4 F 07 up 18 lt gt I4 E 16 up Related Information m Switch Reference Listlinkup command m Display the Link Status of a Node on page 80 m Set Link Speed on page 91 Administering the Switch 75 Vv Display Switch Chip Port Status If by using other procedures it seems that a particular port of a switch chip is problematic you can use the getportstatus command to provide the port state width and speed On the management controller type getportstatus switch_chip port where m switch_chip is the letter of the switch chip A F m port is the number of the port on the switch chip 1 36 For example getportstatus A 7 Portstate 4 Portphystate 5 LinkWidthActive 2 LinkSpeedActive 4 Related Information a Switch Reference getportstatus command m Display the Link Status of a Node on page 80 Monitoring the InfiniBand Fabric The following tasks enable you to display and check the operation and status of the InfiniBand fabric and components Note
63. entify All Switches in the Fabric 77 Identify All HCAs in the Fabric 78 lt Display the InfiniBand Fabric Topology 78 Display a Route Through the Fabric 79 Display the Link Status of a Node 80 Display Counters fora Node 81 Display Data Counters fora Node 82 Display Low Level Detailed Information About a Node 83 Display Low Level Detailed Information About a Port 84 Map LIDs to GUIDs 86 q 44444464 v Display Subnet Manager Status 87 Controlling the Hardware 87 v Restart the Management Controller 88 Restart the Entire Switch 89 Reset the Switch Chip 89 Recover Ports After Switch Chip Reset 90 Set Link Speed 91 Disable a Switch Chip Port 92 q lt 4 lt 4 lt Enable a Switch Chip Port 92 v Change the Administrator Password 93 Controlling the InfiniBand Fabric 93 v Perform Comprehensive Diagnostics for the Entire Fabric 94 v Perform Comprehensive Diagnostics fora Route 95 v Determine Changes to the InfiniBand Fabric Topology 95 v Find 1x or SDR or DDR Links in the Fabric 97 v Determine Which Links Are Experiencing Significant Errors 97 viii Sun Datacenter InfiniBand Switch 72 Users Guide May 2010 Clear Error Counters 98 Clear Data Counters 99 Check All Ports 99 Reset a Port 100 Set Port Speed 101 Disable a Port 102 Enable a Port 103 qa 44 4 4 Controlling the Subnet Manager 104 Create the guid txt File 105 Create the opensm conf File 106 Enable the Subnet Manager 107 Disable the Subnet Manager 1
64. erform those tasks Administering the switch requires accessing the management controller Access the Management Controller From the Network Management Port on page 29 Access the Management Controller From the USB Management Port on page 30 Hardware Command Overview on page 67 InfiniBand Command Overview on page 67 Related Information Troubleshooting the Switch on page 47 Monitoring the Hardware on page 67 Monitoring the InfiniBand Fabric on page 76 Controlling the Hardware on page 87 Controlling the InfiniBand Fabric on page 93 66 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Hardware Command Overview The management controller uses a simplified Linux OS and file system From the prompt on the management controller you can type hardware commands to perform some administrative and management tasks Hardware commands are user friendly and can perform some testing upon the switch chips enabling greater control of the switch and its operation After you log in to the root account the shell prompt appears and you can enter shell commands Enter the hardware commands in the following format command arguments arguments Related Information m Switch Reference understanding hardware commands a InfiniBand Command Overview on page 67 InfiniBand Command Overview The InfiniBand commands are a means of monitoring and controlling aspects of the
65. ers 7 feet and specifying hardware to support cable routing See InfiniBand Cable Path Lengths on page 12 Carry the entire cable to the points of connection and unroll the cable from the first connection point to the second Keep the coil closed and pause to enable the cable to relax as it is unrolled and moved through turns Ensure that cable route turns are larger than 85 mm 3 4 inches radius for optical cables and 127 mm 5 inches radius for copper cables Find alternative routes for turns that are tighter Secure the cable to hard points and bundle it with soft hook and loop fasteners See InfiniBand Cable Bundling on page 12 Mediate the slack between securing points to maintain minimal cable tension and proper support See Floor and Underfloor Delivery of InfiniBand Cables on page 13 and Overhead Delivery of InfiniBand Cables on page 13 Label the ends of cables to identify their routes Rest the cable connectors on a stable surface when they are not connected Related Information Power Cord Requirements on page 7 Management Cable Requirements on page 8 InfiniBand Cable Types The following table lists the cables available for the switch their length and data rate Cable Type Lengths Data Rate Splitter optical core CXP QSFP x3 10 m 20 m QDR Splitter copper core CXP QSFP x3 lm2m 3m 5m QDR Pass through optical core CXP CXP 10 m 20 m QDR In
66. es commands on the management controller For example to check the power supplies checkpower PSU 0 present status OK PSU 1 present status OK checkvoltages Voltage ECB OK Measured 3 3V Main 3 28 V Measured 3 3V Standby 3 40 V Measured 12V 12 06 V Servicing the Switch 123 Measured Measured Measured Measured Measured 1 1 2VB Measured 1 Measured 1 Measured 1 1 2VF All voltages OK Measured 5V Measured VBAT Measured 1 Measured 1 Measured 1 5VA 5VB 8V 2V 8V 2VA 2VC 2VD 2V Gl 5 10 V 3 17 V 1 78 V Standby Standby 2 PRPRPRPRRPW Related Information m Switch Reference checkpower command ol sol 22 oad 24d wad 21 oak 1 21 V 1 80 V lt a lt sssesccccil m Switch Reference checkvoltages command m Power Off a Power Supply on page 117 Servicing the Fans The following table provides a description of a goal you want to achieve and links to the sequential tasks required to reach that goal Description Links Add a fan Replace a fan Subtract a fan Inspecting a Fan on page 125 Install a Fan on page 130 Inspecting a Fan on page 125 Remove a Fan on page 127 Install a Fan on page 130 Remove a Fan on page 127 Related Information m Understanding Service Procedures on page 111 m Servicing the Power Supplies on page 11
67. ess the management controller through the USB connector power cycle the switch See Power Off a Power Supply on page 117 and Power On a Power Supply on page 122 If the previous steps do not rectify the situation replace the switch See Remove the Switch From the Rack on page 139 and Installing the Switch on page 1 48 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Situation Corrective Steps The Attention LED on the switch chassis is lit or the management controller seems dysfunctional The Attention LED on a fan is lit or the fan seems dysfunctional After installation no links are operational After installation not all links are operational i 1 i 1 If you are unable to access the management controller power cycle the switch See Power Off a Power Supply on page 117 and Power On a Power Supply on page 122 If you are able to access the management controller restart the management controller See Restart the Management Controller on page 88 Check overall switch health See Display Switch General Health on page 68 Verify that the switch is within operating temperatures and voltages See Display Switch Environmental and Operational Data on page 71 If the previous steps do not rectify the situation replace the switch See Remove the Switch From the Rack on page 139 and Installing the Sw
68. h 72 User s Guide May 2010 m location The location of the LED left center or right Note The shaded cell seen in the PDF file is used for the example described in Signal Route Through the Switch on page 64 The shaded cell is not seen in the HTML file Port Switch Chip A Switch Chip B Switch Chip C Switch Chip D 19 OB left P3 3B left P3 6B left P3 9B left P3 20 OA left P3 3A left P3 6A left P3 9A left P3 21 0A center P2 3A center P2 6A center P2 9A center P2 22 OA right P1 3A right P1 6A right P1 9A right P1 23 OB center P2 3B center P2 6B center P2 9B center P2 24 OB right P1 3B right P1 6B right P1 9B right P1 25 1B left P3 4B left P3 7B left P3 10B left P3 26 1A left P3 4A left P3 7A left P3 10A left P3 27 1A center P2 4A center P2 7A center P2 10A center P2 28 1A right P1 4A right P1 7A right P1 10A right P1 29 1B center P2 4B center P2 7B center P2 10B center P2 30 1B right P1 4B right P1 7B right P1 10B right P1 31 2B right P1 5B right P1 8B right P1 11B right P1 32 2B center P2 5B center P2 8B center P2 11B center P2 33 2A right P1 5A right P1 8A right P1 11A right P1 34 2A center P2 5A center P2 8A center P2 11A center P2 35 2A left P3 5A left P3 8A left P3 11A left P3 36 2B left P3 5B left P3 8B left P3 11B left P3 Note When using CXP to QSFP splitter cables the P1 P2 and P3 QSFP connectors are related to the right center and left link LEDs respectively Related Information
69. h Firmware Versions In time newer or updated switch firmware might become available The nm2version command displays the versions of the firmware within the switch chassis On the management controller type nm2version NM2 72p version 0 1 0 1 Build time Aug 24 2009 16 41 03 FPGA version 0x94 ComExpress info Board Name NOW1L Manufacturer Name JUMP Manufacturing Date 2009 02 19 Last Repair Date 1980 01 01 Serial Number NCD2S0240 Hardware Revision 0x0100 Firmware Revision 0x0102 Jida Revision 0x0103 Feature Number 0x0001 Note The output of the nm2version command contains extraneous information In this example only the output that is bold is relevant to the administration of the switch Related Information m Switch Reference nm2version command Administering the Switch 73 V Locate a Switch Chip or Connector From the GUID The output of some InfiniBand commands identify a node by its GUID You can determine the switch chip or InfiniBand cable connection associated with that GUID using this procedure 1 In the output of the InfiniBand command identify the GUID and port in question For example given the following output from the ibdiagnet command W 11id 0x0055 guid 0x0021283a8638b0c0 dev 48438 Port 21 In this example the GUID is 0x0021283a8638b0c0 and the port is 21 2 Truncate all digits of the GUID except the four digits on the right For example GUID 0x002128
70. haking 4 Press the Return or Enter key on the serial device several times to synchronize the connection You might see text similar to the following CentOS release 5 2 Final Kernel 2 6 27 13 nm2 on an i686 nm2name login where nm2name is the host name of the management controller The name might be the word hostname Even if you do not see the text go to Step 5 5 Type root for the login name followed by the root password of changeme nm2name login root Password password The prompt is displayed 30 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Note You can change the password at a later time See Change the Administrator Password on page 93 for instructions on how to change the administrator password 6 Verify the switch status See Verify the Switch Status on page 31 Related Information m USB Management Connector and Pins on page 5 m Management Cable Requirements on page 8 V Verify the Switch Status You can use the following commands on the management controller to check the status of the switch 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 Check the overall health of the switch showunhealthy OK No unhealthy sensors An unfavorable output from the showunhealthy command means a hardware fault with
71. he ibdiagnet command there is a check for all links to be at 4x QDR lw 4x ls 10 and if not to report those which are not Additionally the ibdiagnet 1log file contains the log of the testing Related Information m Switch Reference ipdiagnet command m Display the Link Status of a Node on page 80 m Determine Which Links Are Experiencing Significant Errors on page 97 V Determine Which Links Are Experiencing Significant Errors You can use the ibdiagnet command to determine which links are experiencing symbol errors and recovery errors by injecting packets Administering the Switch 97 1 On the management controller type ibdiagnet c 100 P all 1 In this instance of the ibdiagnet command 100 test packets are injected into each link and the P a11 1 option returns all counters that increment during the test 2 In the output of the ibdiagnet command search for the symbol_error_counter string That line contains the symbol error count in hexadecimal The preceding lines identify the node and port with the errors Symbol errors are minor errors and if there are relatively few during the diagnostic they can be monitored Note According to the InfiniBand specification 10E 12 BER the maximum allowable symbol error rate is 120 errors per hour 3 Also in the output of the ibdiagnet command search for the link_error_recovery_counter string That line contains the recovery error co
72. he node For example to see detailed information about port 8 on the switch with LID 24 type smpquery portinfo 24 8 Port info Lid 24 port 8 MKCVi oh pA ees Se tee se es 0x0000000000000000 GLAPLEE LX tiers ga Enn bite nee DaS e a 0x0000000000000000 Graig Fit edd Biers GS Ae RS 0x0000 SMUT Ae a Ses beet Sele a tens oak a Dayar ele eee 0x0000 GCapMask lt eet ioi sede alae sie evebee a eed 0x0 Drag odet sieht eles is eis kde See Se 0x0000 MkeyLeasePeriod 2220 00s 0 kocar PONE eoa nes E Re Ge ee eae 8 LinkWidthEnabled 1X or 4X LinkWidthSupported 1X or 4X LinkWidthActive 0 4X LinkSpeedSupported 2 5 Gbps or 5 0 Gbps or 10 0 Gbps DIHKSTACSS speeds eat Rats eS eee Bee Active PhysLinkState 2004 LinkUp LinkDownDefState 4 Polling PFOLECEBLES nite sete ee Sees 0 PM Gee es esa a eid rah Sth bs esa ele eh eters E fe ca eva 0 Subnet Timeouts isoa h bea ede cals 0 RespTimeVal 2 2 00 e ceneo 0 THOCALPAY SEL te erre teas See Saw te ees 8 OVETEUNE ET be vea are oblate WN a eh ah als 8 MaxCredt tHint oss bs eee aes 85 ROUNGTEUD Moters a kets aes ee aaae he 16777215 Note The output in the example is just a portion of the full output Related Information m Switch Reference smpquery command Administering the Switch 85 V Map LIDs to GUIDs In an InfiniBand fabric the Subnet Manager and Subnet Adminis
73. iniBand cable 136 power supply 120 switch 1 18 Index 149 L LEDs chassis status 55 56 fan 55 59 front 55 link 55 57 network management 55 57 power supply 55 58 rear 55 understanding 54 link LEDs 57 setting speed 91 status 41 75 status of node 80 listlinkup command 41 75 90 M management cables attaching 24 requirements 8 management controller accessing 28 network management 29 USB management 30 firmware upgrading 145 version 73 restarting 88 managementreset command 89 mapping CXP to switch chip port 60 GUID to CXP connector 74 to switch chip 74 LED to switch chip port 60 LIDs to GUIDs 86 switch chip port to CXP 62 to LED 62 to switch chip port 61 monitoring hardware 67 InfiniBand fabric 76 N network management cable requirements 8 checking LEDs 57 connector 4 nm2version command 73 node counters 81 82 information 83 link status 80 O opensm command 105 opensm conf file 106 opensmd daemon 109 operating noise 3 osmtest command 86 95 overview administrative commands 66 hardware commands 67 InfiniBand commands 67 P paddle boards 133 password 93 perfquery command 81 physical specifications 2 poor performing links 97 port checking 99 disabling 102 enabling 103 information 84 resetting 100 speed 101 status 76 power cord attaching 27 requirements 7 power specifications 3 power supply checking LEDs 58 status
74. iniBand fabric See Validate the InfiniBand Fabric and Report Errors on page 44 Related Information m Switch Reference ipbdiagnet command m Discover the InfiniBand Fabric Topology on page 42 Validate the InfiniBand Fabric and Report Errors The ibcheckerrors command uses the topology file to scan the InfiniBand fabric and validate the connectivity as described in the topology file and to report errors as indicated by the port counters 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 On the management controller type Error Error Error Error ibcheckerrors counter RcvSwRelayErrors 48342 threshold 100 lid 25 port 255 check on lid 25 Sun DCS 72 QDR FC switch o4nm2 72p 2 port all FAILED counter RcvSwRelayErrors 56839 threshold 100 lid 25 port 28 check on lid 25 Sun DCS 72 QDR FC switch o4nm2 72p 2 port 28 FAILED counter RcvSwRelayErrors 56839 threshold 100 lid 25 port 9 check on lid 25 Sun DCS 72 QDR FC switch o4nm2 72p 2 port 9 FAILED counter SymbolErrors 65535 threshold 10 lid 20 port 255 check on lid 20 Sun DCS 72 QDR switch 1 2 LC port all FAILED warn warn warn warn 44 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Summary 6 nodes checked 0 bad nodes found H 144 ports checked
75. installing a power supply inspect its hardware and connectors to verify its suitability for installation Step Description Links 1 Identify the Power Supply Identify the Power Supply on page 114 2 Inspect the hardware Inspect the Power Supply Hardware on page 116 3 Inspect the connectors Inspect the Power Supply Connectors on page 116 Related Information m Inspecting a Fan on page 125 m Inspecting an InfiniBand Cable on page 132 Identify the Power Supply 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Power Supply on page 114 2 Use the following illustration to identify the various features of a power supply Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 FIGURE Power Supply Features Figure Legend 1 AC Connector 2 Release tab 3 Status LEDs 3 Inspect the power supply hardware See Inspect the Power Supply Hardware on page 116 Related Information m Identify the Fan on page 125 m Identify the InfiniBand Cable on page 133 Servicing the Switch 115 Vv Inspect the Power Supply Hardware 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Power Supply on page 114 2 Unwrap the replacement power supply from its antistatic packaging 3 Verify that there is no visib
76. ion farthest from the flange 18 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 d Using a No 2 Phillips screwdriver tighten the two screws e Repeat from Step a for the other cable management extender 4 Attach the cable management extenders and long rails to the rear of the rack a Place the long rail to the mounting location on the rear rack post b Butt the flange of the extender to the flange of the long rail c Secure the assembly to the post with two captive nuts and two screws d Repeat from Step a for the other cable management extender and long rail Installing the Switch 19 5 Attach the long front brackets with cutouts to the switch with 4 screws on each side The flange of the long front brackets point away from the switch 6 Attach the C shaped brackets to the switch with 4 screws on each side The edge of the C shaped bracket is flush to the rear of the chassis 7 Route the power cords through the rack with the female end at the front of the rack where the switch will install Ensure that there is 24 inches 610 mm of power cord slack at the front of the rack to provide an adequate service loop for when removing the switch from the rack 8 Carefully lift the switch and slide it into the rack from the front rearward Ensure that the ends of the long rails slide into the C shaped brackets at the rear of the switch chassis and that the power cords lay into the cut outs of the l
77. ion tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 Loosen the two captive thumbscrews that secure the cover to the cable management bracket A A Og ac gs 6 A A See a e e e e a eleleie 3 Lift the cover off 36 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 4 Remove the protective cap from the connector and visually inspect the cable connector The shell should not be bent and should be parallel to the inner boards If the connector is bent or damaged use a different cable 5 Ensure that the retraction strap is forward 6 Orient the cable connector to the CXP receptacle squarely and horizontally Ensure that the upper shell just touches the underside of the top of the receptacle on the rear panel Installing the Switch 37 BE ee 7 Slowly move the connector in As you slide the connector in the top of the shell should scrape against the underside of the top of the CXP receptacle m If the connector stops or binds after about 1 4 in 5 mm travel back out and repeat from Step 6 m If the connector stops or binds with about 1 8 in 2 mm still to go back out and repeat Step 7 8 Continue to push the connector in until the hooks catch onto the top of the receptacle 38 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 9 Place the
78. itch on page 1 Check the fan speed See Display Fan Status on page 71 If the previous step does not rectify the situation replace the fan See Servicing the Fans on page 124 If the previous step does not rectify the situation install the fan into another available slot See Servicing the Fans on page 124 If no other slots are available replace the switch See Remove the Switch From the Rack on page 139 and Installing the Switch on page 1 Verify that there is at least one Subnet Manager active on the InfiniBand Fabric See Display Subnet Manager Status on page 87 If no Subnet Manager is active start the Subnet Manager within the switch See Start the Subnet Manager on page 34 If the previous steps do not rectify the situation restart the Subnet Manager See Disable the Subnet Manager on page 107 and Enable the Subnet Manager on page 107 Determine which links are non operational See Display Link Status on page 75 For links that are Down disable and re enable the respective ports See Disable a Switch Chip Port on page 92 and Enable a Switch Chip Port on page 92 If the previous steps do not rectify the situation disable the respective port See Disable a Switch Chip Port on page 92 Administering the Switch 49 Situation Corrective Steps There was a power 1 If you are able to access
79. ivery use cable shelves and lattices to support the InfiniBand cables If the overhead delivery has a large drop height consider using an intermediate support for the InfiniBand cables Use of the support can limit the tension in the InfiniBand cable to the weight of the cable for the distance between the supports and the switch Related Information m Connecting InfiniBand Cables on page 35 Installing the Switch 13 Understanding the Installation This section provides you with a quick introduction to the installation process Topics include m Installation Preparation on page 14 m Suggested Tools on page 15 a Antistatic Precautions on page 15 m Installation Responsibilities on page 15 a Installation Sequence on page 16 Related Information m Shipping Carton Contents on page 16 m Install the Switch in the Rack on page 18 m Powering On the Switch on page 23 m Connecting InfiniBand Cables on page 35 m Verifying the InfiniBand Fabric on page 41 Installation Preparation Before installing or servicing the switch you must prepare the following m The environment where the switch is to be installed must conform to the requirements found in Environmental Requirements on page 3 m The rack to receive the switch must have proper power management and InfiniBand fabric cabling brought to it m The rack must have an available location for the switch
80. le damage to the power supply chassis 4 Verify that the release tab moves freely and smoothly 5 Inspect the power supply connectors See Inspect the Power Supply Connectors on page 116 Related Information m Inspect the Fan Hardware on page 126 a Inspect the InfiniBand Cable Hardware on page 133 V Inspect the Power Supply Connectors 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Power Supply on page 114 2 Verify that the connectors are clean and without damage 116 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 3 The power supply is ready for installation See Install a Power Supply on page 120 Related Information m Inspect the Fan Connector on page 127 m Inspect the InfiniBand Cable Connectors on page 134 YV Power Off a Power Supply Note Powering off both power supplies consequently powers off the switch 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Power Supplies on page 113 2 Determine which power supply is to be removed 3 At the front of the switch chassis remove the power cord from the respective power supply Servicing the Switch 117 The power supply is completely powered off 4 Remove the power supply See Remove a Power Supply on page 118
81. le maps the CXP connector and link LED to a switch chip port The second table provides a switch chip to switch chip map The third table maps the switch chip port to a CXP connector and link LED When a command s output provides a switch chip port or CXP connector you can use these tables to determine the route that link is following m CXP Connectors and Link LEDs to Switch Chip Port Routes on page 60 m Switch Chip Port to Switch Chip Port Routes on page 61 m Switch Chip Port to CXP Connectors and Link LED Routes on page 62 m Signal Route Through the Switch on page 64 Note The tables have shaded cells which are used for the example described in Signal Route Through the Switch on page 64 Related Information m Switch Hardware Problems on page 48 a InfiniBand Fabric Problems on page 51 m Understanding the LEDs on page 54 m Switch GUIDs Overview on page 65 Administering the Switch 59 CXP Connectors and Link LEDs to Switch Chip Port Routes The following table provides a mapping of the CXP connector and its link LEDs to the respective switch chip and port The switch chip and port are given as chip port where m chip The identifying letter of the switch chip A D m port The identifying number of the port 19 36 Note The shaded cell seen in the PDF file is used for the example described in Signal Route Through the Switch on page 64 The shaded ce
82. ll is not seen in the HTML file CXP Connector A CXP Connector B Connector Group Left LED Center LED Right LED Left LED Center LED Right LED P3 P2 P1 P3 P2 P1 0 A 20 A 21 A 22 A 19 A 23 A 24 1 A 26 A 27 A 28 A 25 A 29 A 30 2 A 35 A 34 A 33 A 36 A 32 A 31 3 B 20 B 21 B 22 B 19 B 23 B 24 4 B 26 B 27 B 28 B 25 B 29 B 30 5 B 35 B 34 B 33 B 36 B 32 B 31 6 C 20 C 21 C 22 C 19 C 23 C 24 7 C 26 C 27 C 28 C 25 C 29 C 30 8 C 35 C 34 C 33 C 36 C 32 C 31 9 D 20 D 21 D 22 D 19 D 23 D 24 10 D 26 D 27 D 28 D 25 D 29 D 30 11 D 35 D 34 D 33 D 36 D 32 D 31 Note When using CXP to QSFP splitter cables the P1 P2 and P3 QSFP connectors are related to the right center and left link LEDs respectively Related Information m Switch Chip Port to Switch Chip Port Routes on page 61 60 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m Switch Chip Port to CXP Connectors and Link LED Routes on page 62 m Signal Route Through the Switch on page 64 a Display Link Status on page 75 m Display Switch Chip Port Status on page 76 Switch Chip Port to Switch Chip Port Routes The following table provides a mapping of switch chip I4 and port to another switch chip I4 and port Note The shaded cell seen in the PDF file is used for the example described in Signal Route Through the Switch on page 64 The shaded cell is not seen in the HTML file 14 Port gt 14 Port 14 Pot gt
83. n page 104 Related Information m Monitoring the InfiniBand Fabric on page 76 m Controlling the Hardware on page 87 V Perform Comprehensive Diagnostics for the Entire Fabric If you require a full testing of your InfiniBand fabric the ibdiagnet command can perform many tests with verbose results The command is a useful tool to determine the general overall health of the InfiniBand fabric On the management controller type ibdiagnet v r The ibdiagnet 1log file contains the log of the testing Related Information m Switch Reference ibdiagnet command m Switch Reference ibdiagpath command m Display the InfiniBand Fabric Topology on page 78 94 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Vv Perform Comprehensive Diagnostics for a Route Similar to the ibdiagnet command the ibdiagpath command can perform some of the same tests for a particular route On the management controller type ibdiagpath v 1 slid dlid where m slid is the LID of the source node m dlid is the LID of the destination node The ibdiagpath 1og file contains the log of the testing Related Information m Switch Reference ipdiagnet command m Switch Reference ipdiagpath command a Display a Route Through the Fabric on page 79 V Determine Changes to the InfiniBand Fabric Topology If your fabric has a number of nodes that are suspect the osmtest command enables you
84. n page 76 a Controlling the Hardware on page 87 m Controlling the InfiniBand Fabric on page 93 Switch Hardware Problems The following table lists situations that might occur with switch hardware and corrective steps that can be taken to resolve the problem Situation Corrective Steps The Attention LED ona 1 power supply is lit or the power supply seems 2 dysfunctional 3 There is no network 1 communication with the management controller 5 Check the power supply status See Display Power Supply Status on page 68 Unplug the respective power cord wait 15 minutes then reattach the power cord If the previous steps do not rectify the situation replace the power supply See Servicing the Power Supplies on page 113 Verify that the management controller is powered without faults See Check Chassis Status LEDs on page 56 Verify that there is a LinkUp condition at the network management port See Check Network Management Port Status LEDs on page 57 Verfiy the DHCP server is providing the IP address you are using to access the management controller See DHCP server documentation If you are able to access the management controller through the USB connector restart the management controller See Access the Management Controller From the USB Management Port on page 30 and Restart the Management Controller on page 88 If you are unable to acc
85. na esne 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 22005 10 0 Gbps Peer PortInfo Port info Lid 24 DR path slid 25 dlid 65535 0 8 port 2 LINnKkState ceca ee a ee hs Active PhysLinkState 2008 LinkUp LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 0 4X LinkSpee Supported seess see 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 4 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 2 2 eee eee 10 0 Gbps In the output the Active parameters are the current state of the port Related Information Switch Reference inportstate command Display Switch Chip Port Status on page 76 Display Link Status on page 75 Find 1x or SDR or DDR Links in the Fabric on page 97 Set Port Speed on page 101 Display Counters for a Node To help ascertain the health of a node the perfquery command displays the performance error and data counters for that node Administering the Switch 81 On the management controller type perfquery lid port where m lid is the LID of the node m port is the port of the node Note If a port value of 255 is specified for a switch node the counters are the total for all switch ports For example perfquery 24 8 Port counters Lid 24 port 8 PortSel cti ceea aii ena h ed 8 CounterSelect
86. net Manager 34 109 status chassis 71 fan 71 link 41 power supply 68 Subnet Manager 87 stopping Subnet Manager 109 Subnet Manager disabling 107 enabling 34 107 setting priority 108 starting 34 109 status 87 stopping 109 switch GUIDs 65 identity 77 Index 151 installing 18 powering off 117 on 23 restarting 89 servicing 111 specifications 1 troubleshooting 47 verifying status 31 switch chip checking boot status 74 port disabling 92 enabling 92 resetting 89 symbol errors 97 T tools 15 topology change 95 discovery 78 troubleshooting the switch 47 U understanding InfiniBand cable 9 installation 14 LEDs 54 service procedures 111 switch specifications 1 USB management cable requirements 8 connector 5 V validating the InfiniBand fabric 44 verifying InfiniBand fabric 41 switch status 31 voltage 3 152 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010
87. net Manager With the opensmd Daemon on page 109 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Servicing the Switch Once a failed part is identified it can be replaced The topics listed here help you service switch chassis components Understanding Service Procedures on page 111 Servicing the Power Supplies on page 113 Servicing the Fans on page 124 Servicing the InfiniBand Cables on page 132 Servicing the Battery on page 139 Upgrading the Firmware on page 145 Related Information Installing the Switch on page 1 Administering the Switch on page 47 Switch Remote Administration Switch Reference Understanding Service Procedures Servicing the switch means a component addition replacement or subtraction A component addition means installing a component to increase the functionality of the switch Component replacement means removing a failed component and installing a functional one Component subtraction means removing a component These topics provide information to prepare for service procedures Suggested Tools on page 15 Antistatic Precautions on page 15 Replaceable Components on page 112 111 Related Information m Servicing the Power Supplies on page 113 m Servicing the Fans on page 124 m Servicing the InfiniBand Cables on page 132 m Servicing the Battery on page 139 m Upgrading the Firmwar
88. nfig etc opensm opensm conft OpenSM 3 3 2_20090831_ce6f8dd Command Line Arguments Creating config file template etc opensm opensm conf Log File var log opensm log The etc opensm opensm conf file is created 3 Open the etc opensm opensm conf file in a text editor 4 Find the following parameters and change them to as listed m sweep_interval 120 m routing_engine ftree m use_ucast_cache TRUE m root_guid_file etc opensm guid txt 106 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Note If the etc opensm guid txt file does not exist you must create it See Create the guid txt File on page 105 5 Save the etc opensm opensm conf file 6 Set the permissions for the etc opensm opensm conf file chmod 0644 etc opensm opensm conf Related Information m Switch Reference opensm command m Switch Reference opensmd daemon m Create the guid txt File on page 105 m Enable the Subnet Manager on page 107 m Start the Subnet Manager With the opensmd Daemon on page 109 V Enable the Subnet Manager The Subnet Manager within the management controller is not enabled by default Use the enablesm command to enable and start the Subnet Manager On the management controller type enablesm Starting IB Subnet Manager OK Related Information m Switch Reference enablesm command m Start the Subnet Manager With the opensmd Daemon
89. ng command on the management controller to retrieve the node GUID ibstat grep Node For example ibstat grep Node Node GUID 0x0021283a8638b0 0 2 Start a text editor on the management controller and record the GUID twice For example you would see the following in the text editor 0x0021283a8638b0 0 0x0021283a8638b0 0 3 Change the last two digits on the right from 0 to e0 for one of the GUIDs For example 0x0021283a8638b0 0 0x0021283a8638b0e0 4 Save the text editor file on the management controller as the etc opensm guid txt file Administering the Switch 105 5 Set the permissions for the etc opensm guid txt file chmod 0644 etc opensm guid txt Related Information m Switch Reference opensm command m Switch Reference getbaseguid command m Create the opensm conf File on page 106 a Enable the Subnet Manager on page 107 m Start the Subnet Manager With the opensmd Daemon on page 109 Vv Create the opensm conf File The opensmd daemon uses the opensm conf file to configure the OpenSM Subnet Manager You must create an opensm conf file that is optimized for the Sun Datacenter InfiniBand Switch 72 1 On the management controller stop the OpenSM Subnet Manager See Stop the Subnet Manager With the opensmd Daemon on page 109 2 If a etc opensm opensm conf file does not exist create one opensm create co
90. o you want do reboot now yes no yes Broadcast message from root pts 0 Fri Nov 20 17 10 27 2009 The system is going down for reboot NOW Connection to 123 45 67 89 closed by remote host Connection to 123 45 67 89 closed Note By rebooting the management controller the link to the management console is severed You must re access the management controller to regain administrative control Related Information m Restart the Management Controller on page 88 m Reset the Switch Chip on page 89 Reset the Switch Chip If a switch chip does not boot properly or a large quantity of its ports are problematic the i4reset command might correct the situation Administering the Switch 89 On the management controller type i4reset switch_chip where switch_chip is the letter of the switch chip A F For example i4reset a Reset I4 A Related Information m Switch Reference i4reset command a Display Switch Chip Boot Status on page 74 m Display Link Status on page 75 m Reset a Port on page 100 Vv Recover Ports After Switch Chip Reset If after resetting an I4 switch chip using the i4reset command ports are still identified as down restart the environment daemon 1 On the management console identify the ports that are down listlinkup Connector OA Present lt gt I4 A Ports 22 down 21 down 20 down Connector 1A Not present In this
91. ocation Name Color State and Meaning OK Green On 12 VDC is supplied Off No DC voltage is present Flashing No function Middle Attention Amber On Fault detected 12 VDC shut down Off No faults detected Flashing No function Bottom AC Green On AC power present and good Off AC power not present Flashing No function Caution If a power supply has shut down because of a thermal or overcurrent condition signified by the amber Attention LED lighting remove the respective power cord from the chassis Allow the power supply to completely cool for at least 15 minutes A shorter cooling time might cause damage to the power supply when the power cord is reattached If the Attention LED lights amber upon reattaching the power cord replace the power supply 58 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Related Information a Display Power Supply Status on page 68 m Check Board Level Voltages on page 69 Y Check Fan Status LEDs The fan status LEDs are located in the lower right corner of the fans at the front of the switch chassis See Front Status LEDs on page 55 1 Visually inspect the fan status LEDs 2 If the LED is lit there is a problem with that fan Related Information m Display Fan Status on page 71 Understanding Routing Through the Switch The tables in the following topics describe the routing through the switch The first tab
92. of the switch chassis adjacent to power supply 0 3 Open a SSH session and connect to the management controller by specifying the controller s host name as configured with the DHCP server For example ssh 1 root nm2name root nm2name s password password where nm2name is the host name of the management controller The name might be the word hostname Initially the password is changeme Note You can change the password at a later time See Change the Administrator Password on page 93 for instructions on how to change the administrator password 4 Verify the switch status See Verify the Switch Status on page 31 Related Information m Network Management Connector and Pins on page 4 m Management Cable Requirements on page 8 Installing the Switch V Access the Management Controller From the USB Management Port Note The administrator of the switch has the username of root 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 If you have not already done so connect a USB to serial adapter to the USB port of the switch 3 Connect a serial terminal terminal server or workstation with a TIP connection to the USB to serial adapter Configure the terminal or terminal emulator with these settings m 115200 baud m 8 bits No parity m 1 Stop bit a No hands
93. ong front mounting brackets 20 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 SBSSBBHSSonn AeBBEEREREREREEEEER EERE EERE RE p p OOOODOLAPANDUOOOODONDOOOOOOVDOODADOCN 9 Mount the front of the switch chassis to the front rack posts with two captive nuts and two screws at each side Tighten the screws securely Sd mnneeeee eee eeeeneeeneanane 10 Install the cable management bracket to the tabs of the attachment brackets at the rear of the switch tightening the thumbscrews on each side of the cable management bracket Installing the Switch 21 11 Install the cable management bracket cover A A h h h A 2 s A 0 h h fi h fi 12 Tighten the thumbscrews on each side of the cover 22 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 a A A a a ae S A h a ha a a ajajaiaiaiaiaiaiaial 13 Attach the management cables See Attach the Management Cables on page 24 Related Information Shipping Carton Contents on page 16 Powering On the Switch on page 23 Powering On the Switch After installing the components enable powering up of the switch by performing these tasks Attach the Management Cables on page 24 Attach the Power Cords on page 27 Accessing the Management Controller on page 28 Verify the Switch Status
94. ontrolling the InfiniBand Fabric on page 93 V Restart the Management Controller Should the management controller enter an indeterminate state you can reboot it The Subnet Manager and other services are also restarted however the InfiniBand fabric s integrity is not impacted On the management controller type reboot Broadcast message from root pts 0 Mon Nov 23 12 19 17 2009 The system is going down for reboot NOW Connection to 123 45 67 89 closed by remote host Connection to 123 45 67 89 closed The management controller power cycles and reinitializes Note The reboot command severs any management console link to the management controller You must re access the management controller to regain administrative control 88 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Related Information m Restart the Entire Switch on page 89 m Reset the Switch Chip on page 89 Restart the Entire Switch Should you need to restart the entire switch you can use the managementreset command Resetting the entire switch disrupts the InfiniBand fabric All services are re initialized and all links will retrain Perform this task only when absolutely necessary On the management controller type Managementreset Stopping Environment daemon please wait Resetting CPLD please wait Restarting Environment daemon Reboot needed to reconnect to I4 and enable IB ports D
95. or example to check the fans getfanspeed Fan Fan Fan Fan Fan 0 BwWDdY rpm rpm rpm rpm rpm 14270 14270 14270 14270 14434 Note You should see a fan speed for the fan you just installed Related Information m Switch Reference get fanspeed command m Install a Power Supply on page 120 m Install an InfiniBand Cable on page 136 m Replace the Battery on page 140 Servicing the InfiniBand Cables The following table provides a description of a goal you want to achieve and links to the sequential tasks required to reach that goal Description Links Add an InfiniBand cable Inspecting an InfiniBand Cable on page 132 Install an InfiniBand Cable on page 136 Replace an InfiniBand cable Inspecting an InfiniBand Cable on page 132 Remove an InfiniBand Cable on page 134 Install an InfiniBand Cable on page 136 Subtract an InfiniBand cable Remove an InfiniBand Cable on page 134 Related Information m Understanding Service Procedures on page 111 Servicing the Switch 131 m Servicing the Power Supplies on page 113 m Servicing the Fans on page 124 m Servicing the Battery on page 139 Inspecting an InfiniBand Cable Before installing an InfiniBand cable inspect its hardware and connectors to verify its suitability for installation Step Description Links 1 Identify the cable Identif
96. ortstate lid port speed speed where Administering the Switch 101 m lid is the LID of the node m port is the port of the node m speed is the speed of the port 1 for 2 5 Gbyte sec 2 for 5 0 Gbyte sec and 4 for 10 0 Gbyte sec Note Adding speed values enables either speed For example speed 7 is 2 5 5 0 and 10 0 Gbyte sec For example Port info Port info Port info Port info ibportstate 24 8 speed 1 Initial PortInfo Lid 24 port 8 After PortInfo set Lid 24 port 8 ibportstate 24 8 speed 7 Initial PortInfo Lid 24 port 8 LinkSpeedEnabled After PortInfo set Lid 24 port 8 LinkSpeedEnabled 4 2 5 Gbps 2 5 Gbps 2 5 Gbps 2 5 Gbps or 5 0 Gbps or 10 0 Gbps or 5 0 Gbps or 10 0 Gbps Related Information a Switch Reference ipportstate command m Set Link Speed on page 91 a Display the Link Status of a Node on page 80 Y Disable a Port If a port is found to be problematic you can disable it On the management controller type ibportstate lid port disable where m lid is the LID of the node 102 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m port is the port of the node For example ibportstate 24 8 disable Initial PortInfo Port info Lid 24 port 8 LinkStatets 2b2 0b8e eneeaneveaa eu Active PhysLinkState
97. ou can find the location of a node in the switch by deconstructing the node s GUID and port then you can crossreference the node and port to a connector See Locate a Switch Chip or Connector From the GUID on page 74 and Understanding Routing Through the Switch on page 59 Related Information Switch Hardware Problems on page 48 Understanding the LEDs on page 54 Understanding Routing Through the Switch on page 59 Switch GUIDs Overview on page 65 Understanding the LEDs Status LEDs are used on many components of the switch chassis as a means of indicating the component s state You can check the state of the LEDs using the following procedures Front Status LEDs on page 55 Rear Status LEDs on page 55 Check Chassis Status LEDs on page 56 Check Network Management Port Status LEDs on page 57 Check Link Status LEDs on page 57 Check Power Supply Status LEDs on page 58 Check Fan Status LEDs on page 59 Related Information Switch Hardware Problems on page 48 InfiniBand Fabric Problems on page 51 Understanding Routing Through the Switch on page 59 Switch GUIDs Overview on page 65 54 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Front Status LEDs The power supply status LEDs and fan status LEDs are located on the front of the switch chassis FIGURE Front Status LEDs Figure Legend Power supply AC
98. out Loading Cached Option polling_retry_number 3 Command Line Arguments 1000 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Priority 13 Creating config file template tmp osm conf Log File var log opensm log 6 Enable the Subnet Manager enablesm Starting IB Subnet Manager OK 7 Attach the InfiniBand cables See Attach the InfiniBand Cables on page 36 Related Information m Switch Reference setsmpriority command m Switch Reference enablesm command m Switch Reference opensm command Connecting InfiniBand Cables After verifying the switch operational status you can begin attaching the InfiniBand cables a InfiniBand Cable Cautions on page 10 a InfiniBand Cable Guidelines on page 11 m Attach the InfiniBand Cables on page 36 m Check Link Status on page 41 Related Information m Understanding InfiniBand Cabling on page 9 m Powering On the Switch on page 23 m Verifying the InfiniBand Fabric on page 41 Installing the Switch 35 V Attach the InfiniBand Cables Caution InfiniBand cables must never turn tighter than a 5 inch 127 mm radius A tighter radius damages the wires and fibers inside the cable Note When you install the InfiniBand cables connect cables to the lower connectors first then connect cables to the upper connectors 1 Identify the prerequisite and subsequent installat
99. ove an InfiniBand Cable on page 134 and Power Off a Power Supply on page 117 2 Disconnect the management cables 3 Use a No 2 Phillips screwdriver to remove the four screws that secure the front of the switch into the rack Servicing the Switch 139 4 Slide the switch out of the front of the rack 5 Set the switch chassis onto a stable work surface Related Information m Install the Switch in the Rack on page 18 m Remove a Power Supply on page 118 m Remove a Fan on page 127 m Remove an InfiniBand Cable on page 134 m Replace the Battery on page 140 V Replace the Battery 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Battery on page 139 Note You must remove the switch from the rack See Remove the Switch From the Rack on page 139 2 Use a No 1 Phillips screwdriver to remove the eight screws that secure the C shaped brackets at the rear sides of the switch chassis 3 Remove the eight screws that secure the long front brackets at the front sides of the switch chassis 140 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Remove the 16 screws that secure the top cover to the chassis There are five screws on each side and six screws across the top front of the cover Slide the cover forward and lift it off Lift up slightly on the clip that retains
100. ower supplies are in a N N redundancy Line power is provided from two sources A and B Installing the Switch 7 Your switch country kit should contain two power cords that are specific to your country or application The following table describes the power cords available Cable Part Number Description X311L 180 1097 X312E 180 1982 X312F 180 1999 X312G 180 1662 X312L 180 1993 X314L 180 1994 X317L 180 1997 X332A 180 2121 X383L 180 1995 X384L 180 1996 X386L 180 1998 North America Asia IEC 320 C13 to NEMA 5 15P 15A 125V 2 5M Black RoHS Y China IEC 320 C13 to GB 2099 GB 1002 10A 250V 2 0M RoHS Y Argentina IEC 320 C13 to IRAM 2073 10A 250V 2 0M Black RoHS Y Korea IEC 320 C13 to KSC 8305 15A 250V 2 0M Black RoHS Y Continental Europe IEC 320 C13 to CEE 7 7 10A 250V 2 0M Black RoHS Y Swiss IEC 320 C13 to SEV 1011 10A 250V 2 0M Black RoHS Y U K IEC 320 C13 to BS 1363 10A 250V 2 0M Black RoHS Y Taiwan IEC 320 C13 to NEMA 5 15P 10A 125V 2 5M Black RoHS Y Danish IEC 320 C13 to Asfnit 107 10A 250V 2 0M Black RoHS Y Italian IEC 320 C13 to CEI 23 16 VII 10A 250V 2 0M Black RoHS Y Australian IEC 320 C13 to AS 3112 10A 250V 2 0M Black RoHS Y Caution Install and route power cabling only in a manner that complies to federal state and local electrical codes Related Information m Electrical Specifications on page 3 m Attach the Power Cords on p
101. pect cable with a known good cable or reconnect the cable to a known good remote port and repeat Step 1 See Servicing the InfiniBand Cables on page 132 4 If the problem still remains at the cable connection disable and re enable the respective port and repeat Step 1 See Disable a Port on page 102 and Enable a Port on page 103 Temporary solution e If the problem still remains disable the affected port See Disable a Port on page 102 Permanent solution e If the problem still remains replace the affected component or the switch See Servicing the InfiniBand Cables on page 132 See remote port s documentation for replacement procedures See Remove the Switch From the Rack on page 139 and Installing the Switch on page 1 Administering the Switch 51 Situation Corrective Steps An InfiniBand Link LED is blinking 1 Disconnect and properly reconnect both ends of the respective InfiniBand cable See Switch Service servicing an InfiniBand cable If the LED is still blinking determine the significance of the errors through use of the ibdiagnet command See Determine Which Links Are Experiencing Significant Errors on page 97 Determine which connectors map to the affected link by deconstructing the node s GUID and port See Locate a Switch Chip or Connector From the GUID on page 74 If some of the links are running at 1x or SDR use that situation el
102. performs a collection of tests on the InfiniBand fabric and generates several files that contain parameters and aspects of the InfiniBand fabric 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 On the management controller type ibdiagnet In the following example the ibdiagnet command is minimized to determine which links are underperforming ibdiagnet lw 4x 1s 10 skip all Loading IBDIAGNET from usr lib ibdiagnet1 2 W Topology file is not specified Reports regarding cluster links will use direct routes Loading IBDM from usr lib ibdm1 2 I Using port 0 as the local port I Discovering 6 nodes 6 Switches amp 0 CA s discovered Links With links width 4x as set by lw option with width 4x were found Links With links speed 10 as set by ls option I Stages Status Report Installing the Switch 43 STAGE Errors Warnings Bad GUIDs LIDs Check 0 0 Link State Active Check 0 0 Performance Counters Report 0 0 Specific Link Width Check 0 0 Specific Link Speed Check 0 2 Partitions Check 0 0 IPoIB Subnets Check 0 0 Please see tmp ibdiagnet log for complete log I Done Run time was 16 seconds Note The output for your InfiniBand fabric will differ from that in the example 3 Validate and check errors for the Inf
103. pins are straight and not missing 5 Verify that the slot connector receptacles are free from obstructions 6 Orient the power supply with the status LEDs on the left and the release tab on the right 7 Slide the power supply into the open slot pushing at the handle 120 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 8 When the power supply seats push firmly so that the release tab clicks to secure the power supply into the chassis 9 Power on the power supply See Power On a Power Supply on page 122 Servicing the Switch 121 Related Information a Install a Fan on page 130 m Install an InfiniBand Cable on page 136 m Replace the Battery on page 140 V Power On a Power Supply Note For residual power discharge the power cord must remain unattached to the power supply for at least one minute before powering on a power supply 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Power Supplies on page 113 2 Reconnect the power cord to the power supply 122 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 The AC LED lights green to indicate the power supply is connected to facility power A moment later the OK LED lights green to indicate the power supply is at full power 3 Verify the power supply s operation with the checkpower and checkvoltag
104. rature 31 75 Com Express temperature 26 12 I4 A temperature 55 maxtemperature 55 I4 B temperature 47 maxtemperature 49 I4 C temperature 52 maxtemperature 52 I4 D temperature 48 maxtemperature 49 70 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 I4 E temperature 52 maxtemperature 52 I4 F temperature 52 maxtemperature 54 Related Information m Switch Reference showtemps command m Display Fan Status on page 71 m Display Switch Environmental and Operational Data on page 71 Vv Display Fan Status The output of the get fanspeed command can help you determine fan speed inconsistencies which might indicate a future failure On the management controller type getfanspeed Fan 0 rpm 17940 Fan rpm 17940 Fan rpm 17687 Fan rpm 17940 Fan rpm 17687 BwWDdY PP Related Information m Switch Reference getfanspeed command a Display Switch Environmental and Operational Data on page 71 Vv Display Switch Environmental and Operational Data The env_test command enables you to perform several investigative tasks helping you determine the internal environment and operational status of the switch On the management controller type env_test NM2 Environment test started Starting Voltage test Voltage ECB OK Measured 3 3V Main 3 28 V Administering the Switch 71 72 I4 A I4 B I4 c I4 D I4 E Fan Fan Fan Fan Fan
105. rinted on the customer information yellow sheet on the outside of the switch shipping carton and on the pull out tab on the left side front of the switch chassis adjacent to power supply 0 4 Optional Connect the serial management cables from the management console to the USB to serial adapter and from the adapter to the connector labeled with the USB symbol Installing the Switch 25 5 Route the management cables so that they do not interfere with other cables with servicing the switch or with other systems 6 Prepare the management console for communication with the management controller 7 Power on the switch See Attach the Power Cords on page 27 Related Information a Network Management Connector and Pins on page 4 a USB Management Connector and Pins on page 5 a Management Cable Requirements on page 8 m Attach the InfiniBand Cables on page 36 26 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 V Attach the Power Cords The power cords for the switch ship separately and are specific to the country of installation The facility power receptacles for the power cords should be located such that the power cords are routed out of the way either to the sides of the rack or under the floor When live power is delivered to the receptacles at the front of the chassis standby and main power is made available by the power supplies When standby power is distribut
106. s no visible damage to the fan chassis 4 Verify that the thumbscrew spins freely and smoothly 5 Inspect the fan connector See Inspect the Fan Connector on page 127 Related Information m Inspect the Power Supply Hardware on page 116 126 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m Inspect the InfiniBand Cable Hardware on page 133 V Inspect the Fan Connector 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting a Fan on page 125 2 Verify that the connector is clean and without damage 3 Verify that the connector receptacles are free from obstructions 4 Verify that the connector freely floats in its mounting 5 The fan is ready for installation See Install a Fan on page 130 Related Information m Inspect the Power Supply Connectors on page 116 m Inspect the InfiniBand Cable Connectors on page 134 V Remove a Fan Note Fans are hot swappable and do not require powering off Servicing the Switch 127 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Servicing the Fans on page 124 2 Determine which fan is to be removed If a fan has failed its Attention LED lights 3 Loosen the captive thumbscrew at the right side of the fan 128 Sun Datacenter InfiniBand Switch 72 User s Guide May 2
107. sewhere in this table to rectify the problem Disable and re enable the respective ports See Disable a Port on page 102 and Enable a Port on page 103 If the errors are still significant swap the cable with a known good one or reconnect the cable to a known good remote port and repeat from 2 Depending upon what does or does not rectify the problem replace that component See Servicing the InfiniBand Cables on page 132 See remote port s documentation for replacement procedures 52 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Situation Corrective Steps Some InfiniBand links are running at 1x or SDR There are errors on some InfiniBand links For a temporary solution 1 Identify the suspect links using the ibdiagnet command See Find 1x or SDR or DDR Links in the Fabric on page 97 Look for text like the following W link with SPD 2 5 found at direct path 1 19 From a Switch PortGUID 0x00066a00d80001dd Port 19 Tos a Switch PortGUID 0x00066a00d80001dd Port 24 Determine which connectors map to the affected link by deconstructing the node s GUID and port See Locate a Switch Chip or Connector From the GUID on page 74 Verify the cable connection at both ends See Servicing the InfiniBand Cables on page 132 Disable and re enable the respective ports See Disable a Port on page 102 and Enable a Port on page 103
108. stalling the Switch 11 Cable Type Lengths Data Rate Pass through copper core CXP CXP 0 4m 1m 2m 3m 5m QDR Related Information m CXP Connector and Pins on page 5 InfiniBand Cable Path Lengths Cable paths should be as short as possible When the length of a cable path has been calculated select the shortest cable to satisfy the length requirement When specifying a cable consider the following m Bends in the cable path increases the required length of the cable Rarely does a cable travel in a straight line from connector to connector Bends in the cable path are necessary and each bend increases the total length a Bundling increases the required length of the cables Bundling causes one or more cables to follow a common path However the bend radius is different in different parts of the bundle If the bundle is large and unorganized and there are many bends one cable might experience only the inner radius of bends while another cable might experience the outer radius of bends In this situation the differences of the required lengths of the cables is quite substantial a If you are routing the InfiniBand cable under the floor consider the height of the raised floor when calculating cable path length Related Information a InfiniBand Cable Types on page 11 InfiniBand Cable Bundling When bundling InfiniBand cables in groups use hook and loop straps to keep cables organized If possible use color coor
109. t a port to determine its functionality On the management controller type ibportstate lid port reset where m lid is the LID of the node Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 m port is the port of the node For example ibportstate 24 8 reset Initial PortInfo Port info Lid 24 port 8 DinkStatets 2b2088 25 eee eans seas oe Down PhysLinkState 2004 Disabled LinkWidthSupported 1X or 4X LinkWidthEnabled 1X or 4X LinkWidthActive 4X LinkSpee dSupported nesen 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedEnabled 0 2 5 Gbps or 5 0 Gbps or 10 0 Gbps LinkSpeedActive 22008 2 5 Gbps After PortInfo set Port info Lid 24 port 8 TinkStatel 2205 be snag eee a Down PhysLinkState 234 20 ada ee nee Ses Disabled After PortInfo set Port info Lid 24 port 8 Tank States s cos ahd eksd gow enter eae he Down PhyslinkS tate soa creed ara grec Sessa be PortConfigurationTraining Related Information m Switch Reference ipportstate command m Reset the Switch Chip on page 89 m Disable a Port on page 102 m Enable a Port on page 103 Set Port Speed You can manually set the speed of a single port to help determine symbol error generation The ibportstate command can set the speed to 2 5 5 0 or 10 0 Gbyte sec On the management controller type ibp
110. t a portion of the entire file Related Information m Switch Reference osmtest command Vv Display Subnet Manager Status If you want to quickly determine your Subnet Manager s priority and state the sminfo command can also provide the LID and GUID of the hosting HCA On the management controller type sminfo For example sminfo sminfo sm lid 25 sm guid 0x21283a8620b0f0 activity count 44911 priority 13 state 3 SMINFO_MASTER In the output the Subnet Manager s hosting HCA has LID 25 and GUID 0x21283a8620b0 0 The Subnet Manager has a priority of 13 high and its state is 3 master Related Information m Switch Reference sminfo command Controlling the Subnet Manager on page 104 Controlling the Hardware The following topics describe how you can manage the switch and its components Administering the Switch 87 Note To use the commands described in these topics you must be the root user of the management controller m Restart the Management Controller on page 88 m Restart the Entire Switch on page 89 m Reset the Switch Chip on page 89 m Recover Ports After Switch Chip Reset on page 90 a Set Link Speed on page 91 a Disable a Switch Chip Port on page 92 a Enable a Switch Chip Port on page 92 m Change the Administrator Password on page 93 Related Information a Monitoring the Hardware on page 67 m C
111. tch 33 34 Related Information Switch Reference showunhealthy command Switch Reference checkpower command Switch Reference getfanspeed command Switch Reference checkboot command Switch Reference env_test command Start the Subnet Manager If you have no Subnet Managers other than the one within the management controller you can set the Subnet Manager priority to the lowest value If you have other Subnet Managers in your InfiniBand fabric you must decide which is to be the master Subnet Manager The master Subnet Manager has the highest priority All other Subnet Managers are slave Subnet Managers and must have a lower priority 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 Create the root node GUID file See Create the guid txt File on page 105 Create the Subnet Manager configuration file See Create the opensm conf File on page 106 Determine the priority of the Subnet Manager within the management controller Set the priority of the Subnet Manager setsmpriority priority where priority is 0 lowest to 13 highest For example to set the Subnet Manager to priority 13 setsmpriority 13 OpenSM 3 2 6_20090717 Reading Cached Option File etc opensm opensm conf Loading Cached Option routing_engine ftree Loading Cached Option sminfo_polling_time
112. tem administrators authorized service providers and users who have advanced experience installing administering and servicing InfiniBand fabric hardware m Related Documentation on page xi m Documentation Support and Training on page xii m Documentation Feedback on page xii Related Documentation The documents listed as online are available at http docs sun com app docs prod ib switch 72 Application Title Part Number Format Location Getting started Sun Datacenter InfiniBand Switch 72 Getting Started 820 7755 Printed Shipping Guide PDF kit Online Last minute Sun Datacenter InfiniBand Switch 72 Product Notes 820 7753 PDF Online information Installation Sun Datacenter InfiniBand Switch 72 User s Guide 820 7751 PDF Online administration HTML and service Command Sun Datacenter InfiniBand Switch 72 Command 820 7752 PDF Online reference Reference HTML Compliance Sun Datacenter InfiniBand Switch 72 Safety and 820 7754 PDF Online Compliance Guide xi Some commands reference the PICMG 3 0 specification available at http www picmg org Some commands reference the IPMI specification available at http www intel com design servers ipmi Documentation Support and Training These web sites provide additional resources m Documentation http docs sun com a Support http www sun com support m Training http www sun com training Documentation Feedback Su
113. tes les mesures de secours de sauvegarde de redondance et autres mesures n cessaires a son utilisation dans des conditions optimales de s curit Oracle Corporation et ses affili s d clinent toute responsabilit quant aux dommages caus s par l utilisation de ce logiciel ou mat riel pour ce type d applications Oracle et Java sont des marques d pos es d Oracle Corporation et ou de ses affili s Tout autre nom mentionn peut correspondre a des marques appartenant a d autres propri taires qu Oracle AMD Opteron le logo AMD et le logo AMD Opteron sont des marques ou des marques d pos es d Advanced Micro Devices Intel et Intel Xeon sont des marques ou des marques d pos es d Intel Corporation Toutes les marques SPARC sont utilis es sous licence et sont des marques ou des marques d pos es de SPARC International Inc UNIX est une marque d pos e conc d e sous licence par X Open Company Ltd Ce logiciel ou mat riel et la documentation qui l accompagne peuvent fournir des informations ou des liens donnant acc s des contenus des produits et des services manant de tiers Oracle Corporation et ses affili s d clinent toute responsabilit ou garantie expresse quant aux contenus produits ou services manant de tiers En aucun cas Oracle Corporation et ses affili s ne sauraient tre tenus pour responsables des pertes subies des co ts occasionn s ou des dommages caus s par l acc s a des contenus produits ou ser
114. that particular component 3 Check the status of the power supplies checkpower PSU 0 present status OK PSU 1 present status OK A power supply output that is not OK from the checkpower command means that there is a problem with that power supply See Check Power Supply Status LEDs on page 58 for assistance Installing the Switch 31 4 Check the status of the fans Fan Fan Fan Fan Fan 0 Ae UNE rpm rpm rpm rpm rpm getfanspeed 12311 12311 12311 12433 12433 m A stopped or low speed in the output of the getfanspeed command means there is a problem with that particular fan a Ifnot present is in the output of the getfanspeed command yet a fan is installed at the particular slot there is a problem with that particular fan For either condition check the fan See Servicing the Fans on page 124 5 Check the status of the switch chips checkboot I4 A OK T4 B OK T4 C OK T4 D OK I4 E OK I4 F OK If the output of the checkboot command is not OK there is a problem with the particular switch chip Try resetting the switch chip See Reset the Switch Chip on page 89 6 Alternatively you can use the env_test command to perform the preceding checks and more env_test NM2 Environment test started Starting Voltage test Voltage ECB OK Measured Measured Measured Measured Measured Measured Measured Measured 3
115. the battery and release the battery from the main board Servicing the Switch 141 7 Properly dispose of the old battery 8 Unwrap the replacement battery from its antistatic packaging 9 Install the replacement battery into the main board with the side up 142 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 10 Orient the cover over the chassis and lower it in place 11 Slide the cover rearward so that it engages at the rear panel Ensure that the screw holes in the cover align with the holes in the chassis Servicing the Switch 143 12 Use a No 1 Phillips screwdriver to install the 16 screws that secure the cover to the chassis 13 Use eight screws to attach the two long front brackets to the front sides of the chassis Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 144 14 Use eight screws to attach the two C shaped brackets to the rear sides of the chassis 15 Install the switch into the rack See Installing the Switch on page 1 Upgrading the Firmware Periodically a newer version of the firmware for management controller or the 14 switch chips might become available Refer to the Sun Datacenter InfiniBand Switch 72 Product Notes part number 820 7753 for more information Servicing the Switch 145 146 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Index A accessing management controller 28 network management 29 USB m
116. the management controller restart the management outage during a controller firmware update See Restart the Management Controller on page 88 2 If you are unable to access the management controller power cycle the switch See Power Off a Power Supply on page 117 and Power On a Power Supply on page 122 3 Reperform the firmware upgrade See Upgrading the Firmware on page 145 Related Information a InfiniBand Fabric Problems on page 51 m Understanding the LEDs on page 54 m Understanding Routing Through the Switch on page 59 a Switch GUIDs Overview on page 65 50 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 InfiniBand Fabric Problems The following table lists situations that might occur with the InfiniBand fabric and corrective steps that can be taken to resolve the problem Situation Corrective Steps Performance of the 1 Determine if there are errors or problems with the InfiniBand fabric InfiniBand fabric See seems diminished Perform Comprehensive Diagnostics for the Entire Fabric on page 94 Find 1x or SDR or DDR Links in the Fabric on page 97 Determine Which Links Are Experiencing Significant Errors on page 97 2 Locate the affected nodes by the GUID provided in the output of the ibdiagnet command See Locate a Switch Chip or Connector From the GUID on page 74 3 If the problem is at a cable connection swap the sus
117. to open a kink If it is not severe open the kink by unlooping the cable Do not straighten the cable to correct a bend that is too tight Leave the cable bend as is Do not drop the cable or connectors from any height Gently set the cable down resting the cable connectors on a stable surface Do not drag the cable or its connectors over any surface Carry the entire cable to and from the points of connection Related Information m Connecting InfiniBand Cables on page 35 Do not step on the cable or connectors Plan cable paths away from foot traffic or rolling loads Do not bend the cables to a radius tighter than 85 mm 3 4 inches Ensure that cable turns are as wide as possible Do not pack the cable to fit a tight space Use an alternative cable route Do not hang the cable for a length more than 2 meters 7 feet Minimize the hanging weight with intermediate retention points Do not cinch the cable with hard fasteners or cable ties Use soft hook and loop fastener for bundling and securing cables Do not force the cable connector into the receptacle by pushing on the cable Apply connection or disconnection forces at the connector only InfiniBand Cable Guidelines Proper InfiniBand cable installation requires the following 1 Plan the cable routes and cable length needs identifying problematic cable route bends minimizing the length of continuous vertical runs to no more than 2 met
118. to take a snapshot inventory file of your fabric and at a later time compare that file to the present conditions Note Though this procedure is most useful after initializing the Subnet Manager it can be performed at any time 1 Initiate the Subnet Manager 2 Take a snapshot of the fabric topology osmtest f c Command Line Arguments Done with args Flow Create Inventory Aug 13 19 44 53 601222 B7D466C0 Ox7 gt Setting log level to 0x03 Aug 13 19 44 53 601969 B7D466C0 0x02 gt osm_vendor_init 1000 pending umads specified using default guid 0x21283a8620b0 0 Administering the Switch 95 Aug 13 19 44 53 612312 B7D466C0 0x02 gt osm_vendor_bind Binding to port 0x21283a8620b0 0 Aug 13 19 44 53 636876 B7D466C0 0x02 gt osmtest_validate_sa_class_port_info SA Class Port Info base_ver 1 class_ver 2 cap_mask 0x2602 cap_mask2 0x0 resp_time_val 0x10 OSMTEST TEST Create Inventory PASS 3 After an event compare the present topology to that saved in the inventory file osmtest f v Command Line Arguments Done with args Flow Validate Inventory Aug 13 19 45 02 342143 B7EF96C0 0x7f gt Setting log level to 0x03 Aug 13 19 45 02 342857 B7EF96C0 0x02 gt osm_vendor_init 1000 pending umads specified using default guid 0x21283a8620b0 0 Aug 13 19 45 02 351555 B7EF96C0O 0x02 gt osm_vendor_bind Binding to port 0x21283a8620b0 0 Aug 1
119. trator assign subnet specific LIDs to nodes Often in the use of the InfiniBand commands you must provide a LID to issue a command to a particular InfiniBand device Alternatively the output of a command might identify InfiniBand devices by their LID You can create a file that is a mapping of node LIDs to node GUIDs which can help with administrating your InfiniBand fabric The following procedure creates a file that lists the LID in hexadecimal the GUID in hexadecimal and the node description Note Creation of the mapping file is not a requirement for InfiniBand administration 1 Create an inventory file osmtest f c i inventory txt Note The inventory txt file can be used for other purposes too besides this procedure 2 Create a mapping file cat inventory txt grep e 41id e port_guid e desc sed s lid nlid gt mapping txt 3 Edit the latter half of the mapping txt file to remove the nonessential information The content of the mapping txt file looks similar to the following lid 0x14 port_guid 0x0021283a8620b0a0 node_desc Sun DCS 72 QDR switch 1 2 LC lid 0x15 port_guid 0x0021283a8620b0b0 node_desc Sun DCS 72 QDR switch 1 2 LC lid 0x16 port_guid 0x0021283a8620b0c0 node_desc Sun DCS 72 QDR switch 1 2 LC 86 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Note The output in the example is jus
120. tus OK LED lights See Check Chassis Status LEDs on page 56 10 Access the management controller See Accessing the Management Controller on page 28 Related Information m Power Cord Requirements on page 7 a Electrical Specifications on page 3 m Attach the InfiniBand Cables on page 36 Accessing the Management Controller With power applied you can now access the management controller m Access the Management Controller From the Network Management Port on page 29 m Access the Management Controller From the USB Management Port on page 30 Related Information a Network Management Connector and Pins on page 4 a USB Management Connector and Pins on page 5 m Management Cable Requirements on page 8 28 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 V Access the Management Controller From the Network Management Port Note The administrator of the switch has the username of root 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 If you have not already done so configure the DHCP server with the MAC address and new host name of the management controller inside of your switch The MAC address is printed on the customer information yellow sheet on the outside of the switch shipping carton and on the pull out tab on the left side front
121. unt in hexadecimal The preceding lines identify the node and port with the errors Recovery errors are major errors and the respective links must be investigated for the cause of the rapid symbol error propagation Note Additionally the ibdiagnet 1log file contains the log of the testing Related Information m Switch Reference ibdiagnet command m Find 1x or SDR or DDR Links in the Fabric on page 97 m Display Counters for a Node on page 81 Vv Clear Error Counters If you are troubleshooting a port the perfquery command provides counters of errors occurring at that port To determine if the problem has been resolved you can reset all of the error counters to 0 with the ibclearerrors command 98 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 On the management controller type ibclearerrors Summary 5 nodes cleared 0 errors Related Information m Switch Reference inclearerrors command m Switch Reference perfquery command m Display Counters for a Node on page 81 m Clear Data Counters on page 99 Clear Data Counters When you are optimizing the InfiniBand fabric for performance you might want to know how the throughput increases or decreases according to changes you are making to the fabric and Subnet Manager The ibclearcounters command enables you to reset the data counters for all ports to 0 On the management controller type ibclearcounters
122. used reliably if it is more than 10 meters long The USB to serial adapter is not included with your switch You can purchase such an adapter from computer and electronics stores Related Information m Network Management Connector and Pins on page 4 m USB Management Connector and Pins on page 5 m Attach the Management Cables on page 24 Understanding InfiniBand Cabling These topics describe InfiniBand cabling m InfiniBand Cable Cautions on page 10 a InfiniBand Cable Guidelines on page 11 m InfiniBand Cable Path Lengths on page 12 a InfiniBand Cable Types on page 11 m InfiniBand Cable Bundling on page 12 m Floor and Underfloor Delivery of InfiniBand Cables on page 13 m Overhead Delivery of InfiniBand Cables on page 13 Related Information m Routing Service Cables on page 7 m Connecting InfiniBand Cables on page 35 Installing the Switch 9 InfiniBand Cable Cautions To prevent InfiniBand cable damage you must follow these cautions amp HH 10 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 Do not uncoil the cable as a kink might occur Hold the coil closed as you unroll the cable pausing to allow the cable to relax as it is unrolled Do not pull the cable out of the shipping box through any opening or around any corners Unroll the cable as you lay it down and move it through turns Do not twist the cable
123. ver Cable management assembly p ay Attachment brackets a N Attachment plates Power cords not pictured After verifying the package contents install the switch into the rack See Install the Switch in the Rack on page 18 Related Information m Install the Switch in the Rack on page 18 Installing the Switch 17 Y Install the Switch in the Rack Caution The airflow through the switch is in from the fans through the chassis and out at the connector panel The front of the switch chassis fan end intakes from the cold aisle and the rear of the switch chassis connector end exhausts to the hot aisle This flow direction requires you to install the switch in an orientation that is the opposite of what you might assume 1 Identify the prerequisite and subsequent installation tasks that you must perform in conjunction with this procedure See Installation Sequence on page 16 2 If installed open the rack doors 3 Assemble the cable management extenders a Slide the attachment bracket over the extender so that the tab on the bracket is opposite the flange on the extender The open end of the tab is toward the flange The flat end of the tab is toward the rear of the extender b Place the attachment plate on the flange side of the extender opposite the attachment bracket c Use two screws to sandwich the attachment bracket and plate to the extender in the posit
124. vices tiers ou a leur utilisation Qa m Adobe PostScript Contents Using This Documentation xi Installing the Switch 1 Understanding Switch Specifications 1 Physical Specifications 2 Environmental Requirements 3 Acoustic Noise Emissions 3 Electrical Specifications 3 Network Management Connector and Pins 4 USB Management Connector and Pins 5 CXP Connector and Pins 5 Routing Service Cables 7 Power Cord Requirements 7 Management Cable Requirements 8 Understanding InfiniBand Cabling 9 InfiniBand Cable Cautions 10 InfiniBand Cable Guidelines 11 InfiniBand Cable Types 11 InfiniBand Cable Path Lengths 12 InfiniBand Cable Bundling 12 Floor and Underfloor Delivery of InfiniBand Cables 13 Overhead Delivery of InfiniBand Cables 13 Understanding the Installation 14 Installation Preparation 14 Suggested Tools 15 Antistatic Precautions 15 Installation Responsibilities 15 Installation Sequence 16 Shipping Carton Contents 16 v Install the Switch in the Rack 18 Powering On the Switch 23 v Attach the Management Cables 24 v Attach the Power Cords 27 Accessing the Management Controller 28 v Access the Management Controller From the Network Management Port 29 v Access the Management Controller From the USB Management Port 30 v Verify the Switch Status 31 v Start the Subnet Manager 34 Connecting InfiniBand Cables 35 v Attach the InfiniBand Cables 36 v Check Link Status 41 Verifying the InfiniBand Fabric 41 v Discover the InfiniBan
125. y the InfiniBand Cable on page 133 2 Inspect the hardware Inspect the InfiniBand Cable Hardware on page 133 3 Inspect the connectors Inspect the InfiniBand Cable Connectors on page 134 Related Information m Inspecting a Power Supply on page 114 m Inspecting a Fan on page 125 Y Identify the InfiniBand Cable 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting an InfiniBand Cable on page 132 2 Use the following illustration to identify the various features of the InfiniBand cable 132 Sun Datacenter InfiniBand Switch 72 User s Guide May 2010 FIGURE InfiniBand Cable Features Figure Legend 1 Retraction strap 2 Paddle boards 3 Inspect the InfiniBand cable hardware See Inspect the InfiniBand Cable Hardware on page 133 Related Information m Identify the Power Supply on page 114 m Identify the Fan on page 125 V Inspect the InfiniBand Cable Hardware 1 Identify the prerequisite and subsequent service tasks you must perform in conjunction with this procedure See Inspecting an InfiniBand Cable on page 132 2 Verify that the cable is not cut or damaged 3 Verify that the cable is not kinked or has a fold 4 Verify that the cable is of the correct type from its label 5 Inspect the cable connectors See Inspect the InfiniBand Cable Connectors on page 134 Rel

Sun Datacenter InfiniBand Switch 72 User`s Guide

Contents

Download Pdf Manuals

Related Search

Related Contents