Home

Voltaire Switch User Manual ISR 9024, ISR 9096, and ISR

1. O008f1 04004006f8 ooostt 040040068 00081040041 065e ISR 9024 37 1169 25 23 Figure 3 21 Switches List GUID Displays the Global Unique Identification number GUID of the switch Name Displays the user assigned switch name The switch name is either set by the ATS mechanism or can be set manually via the CLI or the Voltaire Switch Manager LID Displays the Local ID of the switch IP Address Displays the IP address of the switch Ports Displays the total number of ports contained in the switch Node Displays the name of the switch Description Vendor Displays the vendor name Switch ID Displays the switch vendor ID Table 3 17 Switches List Information Parameters Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Information Views 3 7 4 Switch Line Board Information Double click on a switch or a Line Board to drill down to that switch view 7 Down 0 Down o Down o Down o Down o Down o IB Active 11 lax 2 5Gbps 2048 bytes YLO VL Down o _ 3 Down o 10 i Down 0 7 Down 0 12 Down jo 13 Down o 14 Down o 15 7 Down 0 16 IP Active 10 4x 2 5Gbps 2048 bytes VLO VL 17 Down o 18 Down 0 19 Down o 20 Down o
2. e D e gel P acl increase debug level use direct route SMP dest GUID dest GID prefix Direce E T Formate e MNO mS default 200 r retries set retrLies Ele default 0 default lid route Selena elena sl POLE a ve set RPC timeout value per retry in msec default 3 Derav port p Usage formats madstat madstat madstat madstat madstat madstat madstat madstat S N Ir X A C D Z ae a e ERU oeeo RANE Oa ia E S EI lt lt T T ay with counter mask Mache were eb lech Oat madora Re irc pa i with counter mask Get Switch settings Get Node settings e pore SE cial Jenn simi eer lt port gt Get Advanced port settings Get SMA port settings Sjociee E sjo iec hell Gene 4oior1e OU powe mask Acro pers COumEers ois e GSic E EH Sis pore Vitask Zero EAE L CO ers Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode MeaGsito ti lich ie nui wi eor E Ils gt le NOLeen DO ine TrainingSequence I ell ony madsrtar w window E H HE I e orien soe M lt lidipati k port Monitor port performants madora e aaea i lt portk a eee I Get Sen port Cut IIE ser val ro a E e HT E Cleve L Be E E ESPI RUT G E E E R S ee 1 a EE L OE S E E O T aXc mao clue a eea e eee TH a e a Court ers Vi mack weee e UW o ea a I ee e e Sait some state sek erato vali
3. Fabric Manager windows and dialog boxes The menu options displayed depend on where you performed the right click menu actions are identical to those described in Table 3 4 Voltaire Switch User Manual 3 15 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 4 7 Subnet Tree The Subnet Tree displays all the elements monitored by the Fabric Manager in a top down hierarchical format Voltaire Manager Fie view Fabric Mode Port Help i Fabric Hosts 4dapter nodes ibhost 6 J ibhost 7 J ibhost 4 a ibhost 1 a ibhost 16 a ibhost 3 a ibhost 3 a ibhost 13 a ibhiost 4 a ibhost 12 a ibhiost 11 a ibhost 10 a ibhost 9 a ibhost 15 a ibhost 5 ZS Ibnozst 1 4 I Switches o K srg R ieee as Figure 3 6 Subnet Tree The IP address displayed at the bottom of the subnet tree window is the fast Ethernet IP address of the switch 3 16 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Fabric Manager Operation Table 3 5 describes the objects displayed in the Subnet Tree Object Hosts Adapter nodes Table 3 5 Subnet Tree Objects Voltaire Switch User Manual Description The highest level representing the entire fabric managed by the Voltaire Fabric Manager When this object is selected the main window is displayed as shown in Figure 3 4 Logical object which represents the hosts adapter nodes and switches in the fabric This object has no di
4. Figure 4 2 Login window a eas IP Address The IP address of the switch 1s displayed in the read only IP Address field User type The Voltaire Device Manager supports three types of users each with a different authorization level e Administrator user type This user can perform all configuration and provisioning operations Default password 123456 Manager user type Enables you to set the parameters Default password voltaire Guest user type This user can only view device configuration and status information Default password voltaire Password The passwords for each type of user can be defined in the Users window described on page 4 42 Table 4 2 Login Window Fields Description Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Getting Started with the Device Manager To Login into the Device Manager Step 1 Select the User Type Step 2 Type in the Password and click OK The Voltaire Device Manager Main window is displayed as shown in Figure 4 4 and Figure 4 5 PTE Login password is case sensitive 4 2 2 Terminating the Voltaire Device Manager You terminate the Voltaire Device Manager by performing an Exit operation The exit operation shuts down the Voltaire Device Manager application event logging continues to work even after the GUI is shut down To terminate the Voltaire Device Manager Step 1 Select File Exit from the main menu The following warning message 1s
5. Information Views Displays the vendor name Displays the device vendor ID Table 3 15 Hosts List Information Parameters Host Information To drill down to a specific host and view the list of ports and associated information either double click on the host in the Host List or select the host in the main window subnet tree The following window is displayed Fil Active 2048 bytes VLO VL7 l Down E E Figure 3 20 Hosts Ports Port Displays the port number according to the numbering scheme of the switch host State Displays the state of the port which can be one of the following ACT port is active DWN down This field is color coded to indicate the state of the port Displays the Local ID of the port Displays the current link width 1X or 4X Speed Displays the current signaling rate of the link current default is 2 5 Gb s Displays the Maximum Transmission Unit value of the port Displays the VLs supported by the port Current support 1s for VLs 0 to7 Table 3 16 Host Ports Information Parameters Voltaire Switch User Manual 3 37 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 3 Switches List 3 38 To display a list of the switches in the fabric click on the Switch icon in the subnet tree The following window is displayed IP Address Node Description Device ID ISR 9288 Voltaire ISR9024 voltaire Voltaire
6. Found invalid link from s critical enable manual GUID 016Lx port X YZ to s Parameters node_typel string GUID 016Lx port XYZ node_type2 string O NE EVENTE PROE ENIN t Found broken link on s GUID Object port Y warning enable manual 016Lx port XYZ Parameters node_typel string OSM EVENT NO AVAILABLE _LID y a satie hmar Unable to find available LID Object port amp range LMC value is u Parameters Imc l uchar8 OPM FV ENT TMC TOO BIG Unable to configure the subnet Object SM l l Y warning enable manual with LMC XYZ using smaller param Imcl uchar8 Imc2 LMC XYZ uchar8 OSM _ EVENT DUP HCA GUID Found duplicate s port GUID Object port critical enable manual 016Lx port XYZ conflicting Parameters node Yoel string ports or path OSM EVENT DUP SWITCH G UID Found duplicate s port GUID Object port critical enable manual 016Lx port X YZ another s Parameters node_typel string already exists node_type2 string PGIDOUT S S S S S O G GID_ADDR IN SERVICE TRA o t SM N 64 ees Port GID 016Lx 016Lx was Parameters GID prefix uint64 SOUE eOe DNE id uint64 GID_ADDR OUT SERVICE TR AP 65 Object SM 7 S Note for HCA port is 1 or 2 for Y 65 warning enable IB trap 64 G G E M switch port is 0 disconnected from the fabric Parameters GID prefix uint64 id uint64 LINK_INTEGRITY ERROR J a E SRV LOCAL LINK INTEGRI l l TY TRAP 129 mi
7. Rev switch relay error YHS dropped 0 0 MB sec D MBfsec 015 KPfsec 015 KPfsec bo bo C lG lG Lan LH E m MBEzec GByfsec Voltaire Switch User Manual Figure 6 3 Port Counters Information P N 399Z00038 2005 Voltaire Inc Diagnostic Tools Step 2 In the Interval SEC field enter the time interval in seconds After each interval has elapsed the counters are reset and new values are accumulated Step 3 At the bottom of the dialog box select the measurement units to display the counters in MB sec or GB sec Step 4 Click Start To stop counting click Stop Step 5 Check the port s health status Click to refresh the dialog box Fd The counters displayed in the dialog box vary according to the selected port Checking a Port using the Port Counters Graph Port counters information can also be displayed in graphic format as a chart To display the Port Counters Graph Step 1 Click the Port Counters Graph tab The following dialog box is displayed Port Counters S Transmit Bi Receive 9 a ow DL Ka i Transmit Bi Receive Cancel Figure 6 4 Port Counters Graph 6 12 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic Step 2 Step 3 Step 4 6 3 2 In the Interval SEC field enter the time interval in seconds After each interval has elapsed the counters are reset and new values are accumulated At the bott
8. Syntax Command Description Syntax Example Command Description Syntax Example Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Voltaire Switch User Manual Configuration Config Mode sm info smName set Sets the name of the Subnet Manager sm info smName set string sm info sweep interval set Sets the sweep interval parameter sm info sweep interval set int 10 3600 seconds sm info sweep interval set 15 sm info topology set Sets the number of levels in a subnet configured as a Clos fat tree topology This is an odd number from 3 and up or 0 for none sm info topology set Odd number 0 for none Ssm i1nto Copo logy set 3 sm info verbosity set Set the verbosity level to sm log file sm info verbosity set error info verbose debug function frames sm info verbosity Topology set Sets the verbosity parameters to topology log file sm info verbosity Topology set none error info verbose debug function frames sFBs add Adds a sFB to the sFB list Used for Clos topology configuration sFBs add lt row gt lt Guid gt sFBs del Deletes a selected sFB from the sFB list If you wish to edit sFB data you must first perform a delete operation and then an add sFB operation Used for topology Clos configuration SFBs del lt row gt A 53 P N 399Z0
9. The SFU 4 vertical Fan Unit is part of the ISR 9288 air cooling system it provides cooling of the Line boards sLBs It includes four fans with fan speed control fan speed is dynamically controlled as a function of temperature Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc The SFU 8 horizontal Fan Unit is part of the ISR 9096 9288 air cooling system it provides cooling of the Fabric Boards and the sMBs It includes eight fans with fan speed control fan speed is dynamically controlled as a function of temperature Fan speed can be either turbo or normal Refer to the ISR 9096 9288 Fan Speed Algorithm on page 4 36 for further detail To view the fan unit information a M Click the icon from the device toolbar The Fan Units box is displayed Figure 4 26 ISR 9288 Fan Information Figure 4 27 ISR 9096 Fan Information sFU Information Description Green indicates normal operation Should a specific fan unit be faulty the fan unit is displayed in red in the dialog box and the Status bar at the top of the window is also displayed in red When this is the case consult the ISR 9X XX Installation Manual for maintenance instructions Voltaire Switch User Manual 4 35 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM Figure 4 28 Faulty Fan Information After clicking the Advanced Button the board s VPD Vital Product Data is also displayed Additional VPD c
10. sssssssssssscceccccssssceecccoossssseceosoo 4 1 4 1 Device Manager Description x sccsas avecasoaueecnaenaescaznens aee iE ia 4 1 4 2 Getting Started with the Device Manager sese 4 2 4 2 1 Login tothe Device Manat Gr TTT 4 4 4 2 2 Terminating the Voltaire Device Manager esse eee eee eee 4 5 4 3 The Device Manager Main Window cccccccssscccceeeeceeeeeeeeceeeeeeeeeeeeeeeeeees 4 6 4 3 1 ISR 9288 Main W Ind Te 4 6 4 3 2 TSR 909 6 REES Ts Te 4 7 4 3 3 DEVICe Wier ee ee 4 9 4 3 4 Device Rel 4 10 4 3 5 Switch Graphical Representation ccccccceeeseseseessesseseeseeeseeseeeeeeseeees 4 11 4 3 6 Hotspots in Graphical Representation eee 4 14 4 3 7 The TGO ATS ao oie aaa naaa aaa 4 14 4 3 8 Trap able ccs eau eee aaa nae ee 4 14 4 4 Voltaire Device Manager Operating COonvYeniOnS sese esse sees eee eee 4 15 4 4 1 Sortino KTT 4 15 4 4 2 Changing Table Column Sequence ssc cs cxoosz cc cnc cs a cco eee 4 15 4 4 3 Instructional Tool KT 4 15 4 5 Viewing Switch Information aasa 2 aoao a Za 22T s 4 16 4 5 1 De VIC RE Tear Ee TT 4 16 4 5 2 POEL TALOPMAION sin S H 4 18 Voltaire Switch User Manual li P N 399200038 2005 Voltaire Inc Contents 4 5 3 4 5 4 4 5 5 4 5 6 4 6 4 6 1 4 6 2 4 6 3 4 6 4 4 6 5 4 6 6 4 6 7 4 7 4 7 1 4 7 2 4 7 3 4 7 4 4 8 Chapter 5 Sl J2 Chapter 6 6 1 6 1 1 6 1 2 6 1 3 6 1 4 6 2 6 2 1 6 2 2 6 3 6 3 1 6 3 2 6 3 3 6 3 4 6 4 6 4 1 6 4 2 6 4 3 6 4 4
11. 0008110403960134 voltaire HC A 400 voltaire 0008t10403960e50 0008f10403960e50 Voltaire HCA400 voltaire 0008f1 0403960904 0008f1 04039608ds omni11 voltair com omnit 3 voltaire com 192 168 60 11 192 168 60 13 Voltaire HCA400 voltaire HC A 400 Voltaire voltaire 0008f1 04039608a4 jomni 5 voltaire corm 192 168 60 15 voltaire HCA400 Voltaire 0008f1040396087c O008f1 040396087c voltaire HC A 400 Voltaire 0008f1 0403960876 omni16 volttaire com 192 168 60 16 voltaire HC A 400 voltaire K TK TK TEKS TEK TEK TEKS TEA TEKS TEA TEKS TEA TEKS TEA TEKS TEKA TEKS TEKA TEKS TEK TEA TEKS TEA TEKS TEA TEKS N 0008f1 040396086c omni9 vottaire com 192 168 60 9 Voltaire HC4400 Voltaire 0008f1 0403960860 omnit 4 voltaire com 192 168 60 14 Voltaire HC 400 Voltaire 5404 Figure 3 19 Hosts List Parameter Description Name Displays the Global Unique Identification number GUID of GUID the device Displays the user assigned device name The device name is either set by the ATS mechanism or can be set manually via the CLI or the Voltaire Device Manager IP Address Displays the IP address of the device Displays the total number of ports contained in the device Node Description Displays the user defined description given to the device Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc 3 7 2
12. 3 Retype the password identically to the previous step the following message is displayed OK Command persistency state show Description Displays the state of persistency Shows if the sMB acts as a master standalone or standby There can be two sMB modules plugged into the system This command shows the current state of the sMB module Only one sMB can act as a master at given time the other sMB if installed is in standby mode continuously monitoring the active sMB Syntax persistency state show Command ping Description sends echo messages Syntax ping Command qos info show Description Displays the QoS parameters set for the switch For more information regarding QoS arbitration see page 3 46 Syntax gqos info show Example OOS inio eile Gos L GTR arp racon lamie 10 gos configuration enable enable GOs apo ky Combi oura tIon NO Command qos sl vl show Description Displays the QoS SL VL table For more information regarding QoS arbitration see page 3 46 Syntax qos sl vl show Example P N 399200038 2005 Voltaire Inc Voltaire Switch User Manual Privileged Mode jae 6 Sad 7 9 8 8 10 9 9 EG 10 12 11 Ld 13 12 WZ 1a as Le 1s id 14 1G Ils 7 Command qos vl high arbitration show Description Displays the QoS High Arbitration table For more information regarding QoS arbitration see page 3 46 Syntax gos vl high arbitration show Example T 4 TE 0 3 0 4 3 0 5 Al
13. Chapter A CLI Commands Reference Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Example 1 fast iLncertace Command Description Syntax Example config Changes to the Configuration mode which allows the user to configure all switch configuration parameters config debug Changes to Debug mode which is intended for the use of Voltaire personnel only debug disable Changes to Exec mode which allows view only access of the switch configuration parameters disable fast interface show Returns the IP address of the Fast interface its subnet mask the IP subnet to which the Fast interface broadcasts and the active management interface ETH denotes that the Fast interface is the active interface IB denotes that the IB interface is the active interface fast interface show show E ale hI ae E a10Ns SD GT co Sil sh ene oper ono Oa 1G SeOaCeas ies N a oi Sc management interface is eth0 1 front show Displays a list of the modules installed on the front panel of an ISR 9096 9288 switch including the hardware version and serial number of the module front show Deyice s HW Version Le ws Aw Serial Number is eID io Z 2 SE T S Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode SEBS teks HAWN Version I9 TASDS n Coriell Number iS TRASDEZA onon SFB 4
14. PeerName The remote connected Ports node name node name or GUID PeerModuleType This remote connected Ports Module Type see ModuleType PeerModuleIndex The number of the module where the remote connected Port exists see ModuleIndex PeerPort The external port number or internal for internal ports where the remote connection exists OK or ALERT port counter exceed threshold 1X link etc Table B 1 Port Counters Fields Description B 2 CSV Summary Report In addition the CVS summary report fields are detailed below tem peen Table B 2 CVS Summary Report Fields Description B 2 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Available Fabric Port Counters B 3 Available Fabric Port Counters PM Counters Values and Default Thresholds Table B 3 specifies each monitored counter its parameter value and its default threshold Setting the PM threshold to zero 0 or unchecking a counter disables monitoring of the corresponding counter You can also change the default threshold by entering a different value in the respective field Monitored Counter Default Threshold Pm link error recovery counter set U 255 Pm link downed counter set U 255 Pm port rcv remote physical errors 0 65535 Pm port rcv switch relay errors 0 65535 Pm port xmit errors U 65535 Port xmit pkts threshold 0 65535 Table B 3 PM Counters Tresholds 5 5 Voltaire Switch User Man
15. Type this command to access the IB Interface configuration menu which is used to define the settings of the switch InfiniBand interface located on the sMB card See page A 41 for a list of the commands available in this menu interface IB interface INT Type this command to access the INT Interface configuration menu which is used to define the settings of switch s Internal Ethernet interface the internal Ethernet interface is used as the communication channel between redundant sMB cards for data replication purposes The Internal interface has a factory set default IP address which you can see by using the show command however the default IP address can be changed by the administrator in interface INT configuration mode See page Error Bookmark not defined for a list of the commands available in this menu interface INT interface LOCAL Type this command to access the Local Interface configuration menu which is used to define the settings of Local IP address The Local IP address is a secondary IP address of the sMB card Ethernet port and can be used to access the standby sMB card See page A 43 for a list of the commands available in this menu interface LOCAL interface fast Type this command to access the Fast interface configuration menu which is used to define the settings of the switch Ethernet interface located on the sMB card See page A 44 for a list of the commands available in this menu interface fast r
16. VOLTAIRE Voltaire Switch User Manual ISR 9024 ISR 9096 and ISR 9288 Switches Business Headquarters Voltaire Inc 6 Fortune Drive Suite 301 Billerica MA USA 01821 Tel 978 439 5400 Fax 978 439 5401 Release AAA AAA September 2005 P N 399Z00038 Israel Office Voltaire Ltd 9 Hamenofim St Bldg A Herzeliya 46725 Israel Tel 972 9 971 7666 Fax 972 9 971 7660 THIS DOCUMENT IS PROVIDED AS IS WITH NO WARRANTIES WHATSOEVER INCLUDING ANY WARRANTY OF MERCHANTABILITY NONINFRINGEMENT FITNESS FOR ANY PARTICULAR PURPOSE OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL SPECIFICATION OR EXAMPLES Voltaire disclaims all liability including liability for infringement of any proprietary rights relating to use of information in this specification No license expressed or implied by written or otherwise to any intellectual property rights is granted herein This document as well as the software described in it is furnished under license and may only be used or copied in accordance with the terms of the license The information in this manual is furnished for informational use only is subject to change without notice and should not be construed as a commitment by Voltaire Voltaire Ltd assumes no responsibility or liability for any errors or inaccuracies that may appear in this document or any software that may be provided in association with this document Except as permitted by such license no part of this
17. Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuration Config Mode IB Interface Menu The following are the commands of the IB Interface menu IL SIRS Oo SOS ie E abies ss 7 2 command T ast Or a je labile commande Display command s usage string broadcast ib set Set the broadcast address end EXE Sr O Logan Ment kOMm amy e mode via console connection or close telnet session yia telnet connection exit Exit to previous menu ip address ib set Set interface Ip address IO address EI show Show interface Ip address and CONELOUE aE Om management interface set mtu set mtu show ping Set Management Interface to be from fast Set interface mtu PRON LEOLI o Oe Mell Cc Omid REI ae Ton Send echo messages In the IB Interface menu you configure the parameters of the InfiniBand interface of the ISR 9096 9288 sMB card The IB interface can be used as the management interface read the guidelines listed in page 2 9 The following details the commands available in the IB Interface menu Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Voltaire Switch User Manual broadcast ib set Sets the IP address range subnet that the switch broadcasts to through the IB interface broadcast fast set ip address ip address ib set Sets the IP address and subnet mask of the IB interface
18. 10 show Description Displays the router modules installed inside the switch relevant for ISR 9096 9288 switches only Indicates the slot number on the ISR 9096 9288 rear panel in which the router drawer SsRBD is installed indicates the type of router installed in the drawer slots Syntax io show Example akoe Halk OOS Meio a SO higoce kh Cy Pre mia 2 a ZeAg TE DS yA 5 a0 10 In this example an FCR router module is installed in the slot located at the right hand side of the sRBD itself installed in slot 11 of the ISR 9096 9288 rear panel Its IP address is 172 25 22 41 and its subnet mask 1s 255 255 0 0 Command persistency state show Description Returns the persistency state of the sMB card to which the CLI session 1s connected which can be one of the following standalone active or standby standalone indicates that only one sMB card is installed in the chassis active indicates that the sMB card is the primary card standby indicates that the sMB card is in the secondary sMB ina redundant sMB configuration Syntax persistency state show A 6 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Exec Mode Command ping Description Continuously pings a specified network host Press Ctrl C to stop pinging Syntax ping lt IP address gt Example SW GbR ales S Og Mel S Oe EENG EET 2 pO eval Oe eke E let NOW R Otel om ON AB es odbyt oon com Bh I Oe koma eC 0 sil 2 ey Sim D H la iS
19. CPU es UON ee R a R iN lee alata rattan 4 20 Temperature Information ereansiin a a eau ert ialek 4 22 EST RO HEE eT onenn N NAN 4 23 Obtaining Detailed Module Informations sese 4 24 Viewing Module Informatio esnea T 4 26 sB Line Board Informati oN eanan Oak ceenians 4 27 Fabric Board Information seese r E EEE E EEE 4 30 SMB B ardImormal oN oessa AA O 4 31 SCIRG MIONNAN Neoon is eis aaa ea 4 33 SP SU reste seas eed a oe 4 34 Fan UII seca se ear eee cena cnet gue seamen 4 34 Router Module ER eT 4 38 Using the Voltaire Device Manager for Switch Management 0000000000 4 38 TPS ETT 4 39 Remote Management SC ETT 4 40 e ge 801 81 E E A A E EE A A I E A A 4 42 FIP RET Gon ea O 4 43 Reser Operations aaa BR daai RLE abans Ra ade adana eae 4 44 PAE st TTT ING ois 35 oS a ce cea duces baceastecedevekcucdsasueus suuexeuscecerecasencececcsceaes 5 1 Mechanisms Used to Create and Enforce Partitions sss 5 2 Sentina TN EET 5 2 Fabric Din Gnostic issccsseccecesacastessasctssnsaassdcsnbtecsnedaanadsssaaedeacsbassadcsase 6 1 LIGI E H E AE 6 3 The Event Notification Mechanist sss zezas sc Y cadiancdite ens eidnsteeaae i 6 4 When is Fabric Diagnostic Performed sss 6 5 Potential Fabric ee secuen tanta tema 6 5 Procedures and Tools used for Problem Identification sss sese 6 6 Fabre IMSPECE ea TTT 6 8 Configuring the PortCounters csv file via the CLL 6 8 Configuring PM Using the Fabric Manager sss sese 6 9 Diagnose TO Senra a a a tae
20. Down D L I j 22 Fil Active 11 4x 2 5Gbps 2048 bytes YLO VL7 23 Down 0 24 7 Down 0 Displays the port number according to the numbering scheme of the switch or line board Displays the state of the port which can be one of the following ACT port is active DWN down This field is color coded to indicate the state of the port Displays the LID of the port Displays the active width of the link 1X or 4X Displays the signaling rate of the link 2 5 Gb s or 5 Gb s for future use Displays the Maximum Transmission Unit for the link Displays the number of Virtual Lanes enabled on the link Table 3 18 Switch SLB Information Parameters Voltaire Switch User Manual 3 39 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 5 Port Information To drill down and view information for a specific port W Double click a port in the Node Host section in the Topology map You can also arrive at this window by selecting Port Port Info from the main menu this option is only active when a port is selected Port Info Port GUID 0008410403965029 GID Pretix TeaUUUDUOU DUOOUUDU Port LID 230 Port State ATIVE LMC Physical Link Up SM LID Link idth Speed 2 5 Ghps Port MTU 2046 WL VLO VL Has Sh No SM goo0000000000000 Cancel Figure 3 23 Port Information Port GUID Displays the Global Unique ID of the port The GUID is a unique 128 bit num
21. GbE GE GUI GUID GW HCA HPC HQ IB UO IP ISR Application Programming Interface Address Resolution Protocol Constant bisectional bandwidth Compact Disk Command Line Interface Central Processing Unit Completion Queues Direct Memory Access Fibre Channel Fast Ethernet File Transfer Protocol Field Replaceable Unit Gigabit Ethernet Gigabit Ethernet Graphical User Interface Global Unique ID Gateway Host Control Adaptor High performance computing Head of Queue InfiniBand Input Output Internet Protocol InfiniBand Switch Router Voltaire Switch User Manual P N 399Z00004 2005 Voltaire Inc D 1 Tables IT LID LIDSTAT LAN LMC MIB MTU NAS OS PM Qos QP RC RDMA SAN SA SDP SMP sFB12 sLB24 sFU x SL SM SNMP TCP UDP VDM VFM D 2 Information Technology Local ID Properties retrieved from a specific LID Local Area Network LID Mask Control Management Information Base Maximum Transmission Level Network Attached Storage Operating System Performance Management Quality of Service Queue pairs Reliable Connection Remote Direct Memory Access Storage Area Network Subnet Administration Socket Direct Protocol Chapter D Glossary Short for Simple Management Protocol another name for SNMP2 SNMP2 is an enhanced version of the Simple Network Management Protocol SNMP with features required to support larger networks operating at high data transmission rates SNMP2
22. New value is 25149 previous value was 24819 as eyceeder New value ic 24819 nrewin alue swa d g Figure 4 29 Router Module Information The board s VPD Vital Product Data is also displayed For instructions on how to manage the router modules via the Device Manager refer to the JPR User Manual and the FCR User Manual 4 38 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Using the Voltaire Device Manager for Switch Management 4 7 Using the Voltaire Device Manager for Switch Management The Voltaire Device Manager allows you to configure the following device settings R IP Settings on page 4 39 E Remote Management on page 4 40 M User Passwords on page 4 42 E FTP settings on page 4 43 4 7 1 IP Settings The IP Settings dialog box enables you to view the routing mode of your device To View Configured IP Settings Step 1 Select Device IP Settings from the Main Menu The P Settings dialog box opens as displayed in Figure 4 30 1P settings t i Ethernet IP Address Fast f pa raer a Interface fast Defaut Gateway Figure 4 30 IP Settings Voltaire Switch User Manual 4 39 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM Step 2 4 7 2 Step 1 4 40 Table 4 14 describes the IP parameters displayed in the dialog box Parameter Description IP Address Displays the IP address of the switch s Fast Ethernet port Default Display
23. Reset 4 10 Reset operations 4 44 Reset zoom 3 19 Right Click 3 15 route A 36 route default gw show A 7 A 16 Route menu CLI A 46 Router Blade Drawer 1 7 6 5 P N 399Z00038 2005 Voltaire Inc Routing Algorithms 3 53 Routing balanced 3 53 RPM file 3 4 S Search function in Fabric Manager 3 13 security partitioning 1 3 setpkey A 28 Setting Event Filters 3 29 Setting Partitioning 5 2 SFB 1 7 sFBs add A 53 sFU 1 7 SL to VL Mapping 3 49 sLB 1 7 SLB Information viewing in VFM 3 39 sLB sFB chip 6 20 sLBPortCounter reset A 13 sLBPortCounter show A 13 SLs definition 3 49 sm A 36 SM parameters configuration 3 44 sMB 1 7 4 31 sminfo A 28 sm info algorithm set A 51 sm info CLI Commands A 51 sm info show A 7 A 16 SM KEY 3 45 smpdump A 29 smpex A 30 snmp A 36 SNMP 1 11 1 14 4 8 4 40 4 41 A 8 A 17 A 36 A 44 A 65 A 66 A 67 SNMP commands CLI A 65 snmp community show A 8 A 17 software reboot 4 45 software reset 4 44 Sorting tables in Device Manager 3 21 3 26 4 15 Special keys in CLI 2 3 6 6 Chapter E Index sPSU 1 7 sRBD 1 7 Starting a CLI Session 2 7 Statistics Port 6 11 Stop Bits 2 7 Store Configuration 4 43 Subnet Administration 3 2 Subnet Manager 3 2 Subnet Manager menu CLI A 47 Subnet Manager parameters configuration 3 44 Subnet tree VFM 3 16 Sun Java Web Start Application i
24. Switch Lifetime Limit SLL and Head of Queue Lifetime Limit HLL SLL is defined as 4 096us 24LV if 0 lt LV lt 19 5 55 LV is the LifeTimeValue component of the SwitchInfo attribute If LV gt 19 then SLL 1s to be interpreted as infinite HLL is defined as 4 096us 2 HL if 0 lt HL lt 19 5 55 HL is the HOQLife component of the PortInfo attribute Voltaire Switch User Manual A 63 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference If HL gt 19 then HLL 1s to be interpreted as infinite To set these parameters enter the following commands Gon igen ST HEG Hog sert lrs 1 conftig sm rt Smoi1nto sly set vost conrig sm F Sm inio mopvl set vl0 vl0 1 Vv10 3 V710 7 VvL0 14 config sm mc config sm mc mc auto join set disable enable SM Auto Join mode Mini SM like operation When enabled the Subnet Manager will automatically add each active end port HCA SW to the default IPoIB multicast group MGID OxFF12401 BFEFFOOQOQQQQ00000FFFFFFFF A 64 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc SNMP Menu In the SNMP menu you configure the parameters of the switch SNMP interface Configuration Config Mode Using the SNMP interface the switch can be managed by third party managers Refer to Remote Management Setting Section 4 7 2 for further information and for settings information via the GUI The following are the comman
25. To view the Router Module information M Click the desired Router Module FCR or IPR on the Device Tree The Router Module Information Window appears DAR Voltaire Manager View Device Action Help A R E ISR 9096 i fay eS ae i Si i Temp Reset Telnet sFB 1 L j J sFB 2 sFB 3 sFB 4 SLB sRBD sLB 1 S sRBD 2 IPR1 Networks Traffic Flow Man FCR2 Initiators Physical Volumes Volume Groups Logical Volumes Targets FCR3 sLB 3 sLB 4 sMB SMB Left SMB Right S sPSU sPSU 1 sPSU 2 sPSU 3 sPSU 5 Software Version 3 0_build_1 239 built Sun Apr 17 15 35 24 IDT 200 AAA AAA Hardware Version lt 172 25 3 12 Date Time Description Tue May 31 12 51 16 IDT 2005 SR39096 sLB 24 3113 L ounter threshold was exceeded New value is 26872 previous value was 26522 d U Tue May 31 12 48 17 IDT 2005 ISR9O96 sLB 24 3 13 Counter threshold was exceeded New value is 26522 previous value was 26175 Tue May 31 12 45 14 IDT 2005 Counter threshold was exceeded New value is 26175 previous value was 25843 Tue May 31 12 42 14 IDT 2005 Counter threshold was exceeded New value is 25843 previous value was 25522 Tue May 31 12 39 14 IDT 2005 Counter threshold was exceeded New value is 25522 previous value was 25149 Tue May 31 12 36 14 IDT 2005 e his 2 14 IP Litt ISR9096 sLB 24 3 13 RODS n Courier threshold was exceeded
26. default 5 pm excessive buffer overrun errors set ON eS Command Description Syntax pm link downed counter set 0 255 Defines the error threshold for this counter Setting to 0 disables counter monitoring default 1 pm link downed counter set 0 255 Command Description Syntax Voltaire Switch User Manual pm link error recovery counter set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 1 pm link error recovery counter set 0 255 A 57 P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference pm local link integrity errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 pm local link integrity errors set 0 15 pm polling interval set Defines PM polling interval in seconds default 180 pm pollang santecval set T60 65935 pm port rcv constraint errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 PM POLTL LeV CONStraini errors Set 0 239 pm port rcv errors set Defines the error threshold for this counter Setting to 0 d
27. ip address fast set 1ip address netmask ip address ib show Returns the IP address of the IB Interface its subnet mask and the IP subnet to which the IB Interface broadcasts ip address ib show management interface set Sets the IB interface to be the switch management interface management interface set A 41 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Command mtu set Description Sets the Maximum Transmission Unit of the IB interface Syntax mtu 1500 2400 Command mtu show Description Show present MTU setting Syntax mtu show 4 9 an internal interface The following details the commands available in the INT Interface menu Command subnet address int set Description Sets the subnet of the internal interface Used to communicate between the two chassis management boards Syntax subnet address int set subnet address Command subnet address int show Description Show subnet address for an internal interface IP X Y Z 1 or X Y Z 0 X Y Z can be configured by the user 0 or 1 is factory set Syntax subnet address int show A 42 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuration Config Mode LOCAL Interface Menu In the Local Interface menu you configure the Local Interface parameters The Local IP address is a secondary IP address of the sMB card Ethernet port and can be used to access the standby sMB card The following ar
28. positioning the mouse cursor in various locations in the Voltaire Device Manager Context sensitive help data is displayed as relevant Voltaire Switch User Manual 4 15 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 5 Viewing Switch Information This Section describes how to view device information using the Voltaire Device Manager the information is displayed in the Information area of the Main window The information described in this section varies according to the selection as described below Device Information on page 4 16 Port Information on page 4 17 CPU Information on page 4 20 Temperature Information on page 4 22 Local SM Information on page 4 22 4 5 1 Device Information To view device information F oa Step 1 HE On the device toolbar click The Device Info window Figure 4 12 opens in the Information area Device icon Device Mare snap 00 switch Description SR92856 Votre 0 Device Revision founda Software Version 33 0 Hardware Version RARA Vendor ld rotaie Device Id fue Contact ee Location f Figure 4 12 Device Information 4 16 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Table 4 5 describes the parameters displayed for the device Parameter Description rameter eon Vendor ID Displays the vendor identification number Device ID Displays the identification number of the device Contact Displa
29. terminal emulation program is configured as shown in Table 2 4 O o o Terminal Mode VT 100 Baud 38400 Stop Bits 1 Stop Bit Table 2 4 Terminal Emulation Configuration Starting a CLI Session To start a CLI management session via serial connection to the switch Step 1 Connect the PC to the switch via its serial port using the cable supplied by Voltaire Step 2 Start a HyperTerminal client or compatible on the computer Step 3 Configure the terminal emulation parameters as described in Table 2 4 Step 4 Type the appropriate user name password at the logon prompt the default user name is admin and default password is 123456 Step 5 To change to Privileged mode type enable Step 6 Type the password at the prompt the password is voltaire Step 7 Issue the appropriate CLI commands to complete the required actions P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI 2 2 4 Step 1 Step 2 Step 3 Step 4 Optional in ISR 9096 9288 Switches ISR Configuring the Management Interface IP Address The switch must be configured with an IP address to allow remote management Use the following table to enter the appropriate commands To configure the IP address Command Enable password CON BLG interface fast ip address fast set ClO ZOO 9 TT 8 5 lt 255 255 255 0 gt Note The above IP address and mask are used as an example ip address ib set
30. 0 6 65 0 7 6 0 8 7 0 Command qos vl low arbitration show Description Displays the Quality of Service QoS Low Arbitration table For more information regarding QoS arbitration see page 3 46 DV Ube gos vl low arbitration show Example entry vl weight 1 0 0 2 1 4 3 2 al 4 zoo al Z 4 4 6 ae 7 6 4 8 7 a Command rear show Description Displays a list of the modules installed on the rear panel of an ISR 9096 9288 switch including the hardware version and serial number of the module Syntax rear show Voltaire Switch User Manual A 15 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Example Lite m slot ro HW Version 1s AAT erial Numoer rs AAPV30040014 Line in slot 5 HW Version is GGG Serial Number is GGGG12345678 Line in slot 6 HW Version is AAT Serial Number is AAPV43040007 Line in slot 9 BW Version is VAAI Serial Number ig AAPV43040075 IO Drawer in slot 11 HW Version is AAD Serial Number is AARM44040006 Command reload Description Reboots the switch this command is required after performing various operations in order for configuration changes to become effective Syntax reload Command replication status set Description Enable or disable sMB replication When enabled the standby sMB maintains a mirror image of the active sMB and is ready to replace the active sMB in the event of failure In a redundant sMB configuration
31. 12 Figure 1 7 In band Management sss sese eee eee eee 1 12 Figure 3 1 Voltaire Fabric Manager Initiate Window sss eee eee 3 5 Figure 3 2 L gin WN OW rassssdarcnaebentiataclacjeaisustpiseddaraneuabustanimpanctnieedlawaseabervactasleciainmenias 3 6 Feira ea Vy ANE Messa Esenin teren eee niesen EA TEOTEO 3 7 Figure 3 4 Fabric Manager Main Window ccccseeessssnnccceeeeeeeeeeeeeessseeesssnaaeeeeeeeeeeees 3 8 Figure 3 5 Ke is e O TTT 3 10 Paie 3 0 UDINE TOO eea S 3 16 Figure 3 7 Topology Table View TTT 3 20 Fiz re 3 5 Current Alat MS sssrsisisris nnise nA ESEA AEA STERE 3 22 FICO A AT Daea E E A E EERS 3 23 Figore 3 10 Trap PAD T 3 25 Figure 3 11 Events Log Right click eT 3 27 Feu Ae Evens VO C 3 27 PROBS K TO UO NOG TT 3 28 Figure 3 14 Event Filters Segs ssrin aE E rA RR EEan 3 29 Fears io PM SNE Seeerei ergi ean A r a aaea aei E a REE SE 3 30 Figure 3 16 Node Properties isssesessscsasscassscaescansaceanansesaseassatsanstaoneaedasadennanieraebossemaoteneeneneds 3 32 Figure 3 17 Port Properties eerren iein er aE EEEE ENNE 3 33 Figure 3 18 Figure 3 19 Figure 3 20 Figure 3 21 Figure 3 22 Figure 3 23 Figure 3 24 Figure 3 25 Figure 3 26 Figure 3 27 Figure 3 28 Figure 3 29 Figure 3 30 Figure 3 31 Figure 3 32 Figure 4 1 Figure 4 2 Figure 4 3 Figure 4 4 Figure 4 5 Figure 4 6 Figure 4 7 Figure 4 8 Figure 4 9 Figure 4 10 Figure 4 11 Figure 4 12 Figure 4 13 Figure 4 14
32. A 65 for a list of the commands available in this menu snmp Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Voltaire Switch User Manual Configuration Config Mode system name set Sets the system name of this switch This is the string that will appear in the CLI prompt You can enter any combination of letters and digits up to 20 characters in length In order to register the name and prompt change you must log out and log in again to the CLI system name set name system name show Displays the system name of this switch system name show watchdog mode set Enable or disable the watchdog timer on the sMB card The function of the WDT is to reset the switch in the event that no response is detected on the system The watchdog timer is disabled by default watchdog mode set enable disable watchdog mode show Returns the state of the WDT mode enabled or disabled watchdog mode show watchdog timeout show Returns the WDT timeout setting in milliseconds which is the time period that must elapse before the WDT performs the reset The WDT timeout is factory set and can not be changed by the user watchdog timeout show A 37 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Refe
33. All box is checked Check one or more boxes Voltaire Switch User Manual 3 21 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 5 3 5 1 3 22 Managing Alarms and Events The Fabric Manager provides advanced alarm and event reporting capabilities This section describes some of these capabilities The Current Alarms Window The Current Alarms window displays the alarms present in the fabric It is used to perform a fabric diagnostic Refer to Chapter 6 Fabric Diagnostic Section 6 3 3 for further information To display the Current Alarm window M Select Fabric Current Alarms from the Main Menu Current Alarms Filter Severity All Warning Minar Critical Plattorrre Mo Fiter Alart i Port Counter Error ibhost 11 SHOAL 71 ad 9 Critical Pont Counter Error ISROXKK ISR 9096 sFB 4 3 18 19 06 2005 17 31 03 BS Minor Pant Counter Error ibhost 3vHOAY J1 1906005 17 33 45 8 Critical Poi Counter Erat ibhost T HCAS H MNBH2ONS 17 30 08 S i l 9 Critical Port Counter Error SRIXXX ISR 9096 1 sFB 4 3 7 19067005 16 52 50 D Minor Port Counter Error SRIXXX ISR 9096 sFB 4 2 5 19062005 17 30 33 1 Figure 3 8 Current Alarms Filtering the Current Alarms You can filter the Current Alarms to display all alarm states or only specific Alarm States By default the table view shows all nodes regardless of alarm state All box is checked Check one or more boxes Voltaire Switch User
34. Back x E A A Search r Favorites E2 S C ir rel 33 Address F ARA E laizm lnad _one r hour amp s descending amp c Support Clusterah hydras voltaire com amp sh 1 amp hc 4 v Go 7 Links gt Google v R Search Web D EH 1764 boded RJ Options v Ganglia Host Report for Mon 25 Jul 2005 12 28 39 0300 Last hour Node View sourecforge net Grid gt Support Cluster gt hydra3 voltaire com hydra3 voltaire com Overview 1301 71001044 8 hydra3 voltaire com Load last hour This host is up and running E me n 3 SIK LN S S Gn 7 a HORE Eez Time and String Metrics E SENN PUTO SONIC E STO E O PEREON DETER NPL NET 10 boottime Mon 11 Jul 2005 08 33 03 0300 11 40 12 00 D 1 min Load W crus W Running Processes gexec ON Load Procs 3341130 gmond_started Mon 11 Jul 2005 08 33 52 0300 hydra3 voltaire com Memory last hour last_reported 0 days 0 00 11 machine_type x86_ 64 os_name Linux os_release 2 6 9 5 ELsmp 9311130 IJOL 7 1001099 11 40 12 00 12 20 uptime 14 days 3 55 27 B Memory Used R Memory Shared B Memory Cached D Memory Buffered B Memory Swapped E Total In Core Memory Done S Internet 3 48 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 3 12 3 12 1 3 12 2 Configuring Quality of Service QoS Configuring Quality of Service QoS General InfiniBand was specifically designed to support QoS functionality To do thi
35. Chapter A CLI Commands Reference A 28 LAA TOSO E TT E EO switchguids 0x8f1040040300e S I A 5 Ee e OO A O OOE ISR9096 Voltaire sles smedlave 1011 eal POS OOCSm IO AO Os nO 4 es 4 Leo OOOO E OAR Os Oe Actes Pa dl Le MSS MOC Sr OA OWS tO Gar Ay AZ S VO0 sr LO400SEOT Ga 4 beaguLds O Kot LOO Cmie OO Hca ee T ONGNONS se IL OL ONCII OE ORIOKO Voltaire InfiniBand Fiber Channel Router ail NS OOO ele ER EH HET ES Sy clio 0 fee S OCS BLOAVO 40S O00 Za fie elle As Sly ner 610 Command setpkey Description for development purposes only Syntax setpkey d ebug D irect p port t timeout_ms lt path lid gt lt pkey entry gt Command sminfo Description Print info on specific subnet manager state The sminfo utility retrieves subnet manager related information from the IB fabric using LID and direct routed MADs This information includes subnet manager GUID activity count priority and state Syntax sminfo d ebug s state p prio a activity D Lrece t tameout ms lt sm Lae sry path gt Example tomini oO DI SEA E RE a TRER HER HERREG E ove alone A E RD E T OEI E sminfo 1 nE D SiMe Guadiceat OOOO y ara eral vale Vane nE lk Cle TE state SMINFO MASTER 3 sminfo Detailed Description A LID routed MAD is sent to a specific LID in the fabric For example sminfo 1 queries the subnet manager state of the node element correspondi
36. Description Device ID 54 04 0008f10439608904 omnit O votaire com 192 168 60 10 voltaire HCA400 voltaire 0008f10403965224 kip21 voltaire com 192 168 60 21 voltaire HCA400 Voltaire I 00081 0403965028 jomni4 voltaire com 192 168 60 4 voltaire HCA400 voltaire 5404 0008f10403965020 omnit voltaire corn 192 168 60 1 voltaire HCA400 voltaire 5404 0008f1 040396501 4 0008f1 0403965010 momo omniS vottaire com 192 168 60 6 192 168 60 5 Voltaire HCA400 Voltaire HCA400 Voltaire voltaire 0008f1 0403965008 momo 192 168 60 3 Voltaire HCA400 voltaire 0008f10403962e24 kip23 voltaire com 192 168 60 23 Voltaire HCA400 voltaire 0008f1 0403962754 kip22 vottaire com 192 168 60 22 Voltaire HCA400 voltaire 0008f1 0403962740 0008f10403961114 kip24 vottaire com 0008f10403961114 192 168 60 24 Voltaire HCA400 Voltaire HCA400 Voltaire voltaire 0008f104039610e0 0008110403961 0e0 Voltaire HCA400 Voltaire 0008f104039610d8 0008110403961 099 voltaire HC A 400 voltaire 0008t104039610c8 0008t104039610c8 voltaire HC A 400 voltaire 0008f104039610a0 0008f1 0403960fbhO 0008f1 0403961 0a0 0008f1 0403960fbhO Voltaire HCA400 Voltaire HCA400 Voltaire voltaire O008f1 0403960f48 O008f1 0403960f48 voltaire HC A 400 voltaire 0008f1 0403960134
37. Figure 4 15 Voltaire Switch User Manual Figures Er H y PS TNS Src set ee eee 3 35 OS Fes E TT 3 36 Hose TOn eRe creme etme L ne re re 3 37 BIW ILCIICS Lis T 3 38 S WICH SD Mito rma Ms eri ratdats pall Gacuecsacdaeeiicnes EG 3 39 POr ORI O13 hodarcthetccacelito et eae entedobentetcanaelatesnta cleteacaseetabeseutdonesneaaceineses 3 40 Et gt ol Fa 0 51 ET a E E ferret i attra A E T 3 4 Groupine eT Ee 3 42 Node C Om CUT atlOM xc s2s0s5sccnansatesasagsacseansaucsagestacinanrale sazaptaceate meaner 3 43 IVE DG CAIUS TTT 3 45 Cluster Momitor Dialos BOX sssrin eres i sisi doenea naka gccerenynds 3 47 QoS SL to Vib Mapping Table ie viicsece ccc ensetcters oa 3 50 Oos VE Hieh Arbitration Tables ccc tcctscatiasht N 3 52 e TT 3 55 PRG TOA OE B E TTT 3 56 Accessing Voltaire Multi Protocol Switch Window sss 4 2 Ree eT 4 4 Exit W ariine TTT 4 5 ISR 9288 Voltaire Device Manager Main WImdOW 4 6 ISR 9096 Voltaire Device Manager Main WImdOW 4 7 Device Teese ee oes asd eee a eee 4 9 ISR 9096 Switch VDM Graphical Representation Front View 4 11 ISR 9096 Switch VDM Graphical Representation Rear View 000000000n 4 11 ISR 9288 Switch Graphical Representation Front Views 4 12 ISR 9288 Switch Graphical Representation Rear VIeWT 4 13 ISR 9024 Switch Graphical Representation ccccccccccccseceeceeeeeeeeeeeeeeeeees 4 13 Device Ee c Te ressesie a aneteaeesnteanetae 4 16 ISTR Ee en Te sssr a E Doo
38. GULP om 403074 eB IPO oder iar Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Alarm No C 2 Trap Alarm Table Trap Alarm Table The table below provides detailed specification of traps and alarms NOTE S Alarm Trap Name Map Trap IB to Trap Alarm ae ae ae Y minor Severity PORT COUNTERS ERROR PM_SymbolError Object port Parameters current value last value warning PM_LinkErrorRecovery Object port minor Parameters current value last value PM_LinkDownedCounter Object port Parameters current value last value PM_ PortRcvErrors Object port Parameters current value last value PM_PortRcvRemotePhysicalErrors Object port Parameters current value last value PM_PortRcvSwitchRelayErrors Object port Parameters current value last value PM_PortXmitDiscards Object port Parameters current value last value critical warning warning PM_PortXmitConstraintErrors Object port Parameters current value last value PM_PortRcvConstraintErrors Object port Parameters current value last value PM_LocalLinkIntegrityErrors Object port Parameters current value last value PM_ExcessiveBufferOverrunError E Object port Parameters current value last Voltaire Switch User Manual Not all traps and events apply to the current version Default enable enable enable enable P N 399Z00038 2005 Voltaire Inc Correc
39. HW Version is AAA Serial Number is AARM32040006 SMD HW Version is AAA Serial Number is AAAA12345678 SEUS AWN Version 15 E INiblalesics 1s YANO HEES ERGE E SEUA HW Version 1S EE Numobcics is HEE E ERG Command ftp show Description Shows the ftp client configuration Syntax ftp show Example IEE Oe E CACAT remote server 172 25 0 16 users Swget SRG OOo Command sLBPortCounter portReset Description Resets sLB port s counter settings Syntax devicePortCounter portReset slot Fi port number Command sLBPortCounter reset Description Resets counters of specific sLB ports Syntax devicePortCounter reset slot Command sLBPortCounter show Description shows all the external sLB port counters displayed in a table This allows the user to quickly detect counter errors during connection of nodes Syntax sLBPortCounter show slot Command password update Description Change admin or enable password Syntax password update admin enable Example The following is a step by step example of how to update the Privileged mode Enable password l Type password update enable The following message is displayed Insert new 8 characters password 2 Type in the new password for example password The following message is displayed Voltaire Switch User Manual A 13 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Please retype new password for example password
40. Log and or to change PM settings Event Log Displays the Events Log in a browser window Error Log Displays the Errors Log in a browser window Bad Ports Log Displays a log of the bad ports in the switch Fabric Statistics opens a CSV file in a browser window displaying statistics regarding the ports in the fabric this feature must be enabled in the CLI using the pm report enable command Cluster Configuration Future option Quality of Service opens the Quality of Service dialog box which is used to view and configure QoS parameters This window is described on page 3 46 Routing Configuration Opens a dialog box used to configure the switch routing algorithm This window is described on page 3 53 Current Alarms Opens the Current Alarms window which lists the open alarms in the fabric This window is described on page 3 22 This menu is only active when a node HCA or switch is selected in the Fabric Manager main window Node Open opens the Node information view of the node selected in the topology map Launch Device Manager launches the Voltaire Device Manager application with a connection to the switch selected in the topology map See Chapter 4 for descriptions of the relevant Voltaire Device Manager applications Grouping Info shows the internal hardware configuration of a switch system in a single icon Telnet establishes a Telnet session with the switch selected in the topolog Voltaire Switc
41. M Resets the entire switch hardware reset in ISR 9096 9288 switches only R Resets a Line or Fabric Board in ISR 9096 9288 switches only M Reset the CPU software reset available in all switches To reset the entire switch NOTE x This option is only relevant for ISR 9096 9288 switches This operation resets the entire switch It should only be used when you need to reset all switch modules it typically follows a firmware upgrade or when faulty operation of the switch is suspected CAUTION A This is a traffic affecting operation R Right click on a relevant Fabric Board front panel of device view or a relevant Line Board rear panel of device view and select Reset All The following popup window is displayed Warning AN You are about to reset the system Would vou like bo proceed Yes 4 44 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Reset Operations M Click Yes to initiate the reset operation The operation can take several minutes Watch the Trap Table to see when the switch modules are back online To reset a Line Fabric Board in ISR 9096 9288 Switches NOTE Ed This option is only relevant for ISR 9096 9288 switches M Right click on a Fabric Board front panel of device view or a Line Board rear panel of device view and select Reset sFB or Reset sLB CAUTION This is a traffic affecting operation To reset the Management Board software reboot This op
42. MLID Object node 0x X Parameters gid mgid mlid ne OSM EVENT MULTICAST JOI N T normal Port GID 016Lx 016Lx Voltaire Switch User Manual C 5 P N 399Z00038 2005 Voltaire Inc Chapter C Supported Traps and Events N joined MGID 016Lx 016Lx Object node MLID 0x X Parameters gid mgid mlid Pe rc E Assigned new LID XYZ XYZ Ue ae Y normal enable manual for port GUID 016Lx old LID Parameters min lid max lid old A XYZ XYZ min lid old max lid SRV SM LID CHANGE EVE NT Assigned new LID XYZ to SM Object SM Y normal enable manual GUID 016Lx after reconfigure Parameters SM port guid min lid old LID XYZ sm base lid OSM EVENT RESET PORT C OUNTERS normal Reset all Port Counters Object SM OUNTERS COMPLETED normal Reset Port Counters Completed Object SM OSM EVENT GET PORT COU NTERS normal Get Subnet Port Counters Object SM OSM EVENT GET PORT COU Object SM P OSM EVENT MCAST CONFIG _REPORT Object SM normal Parameters mlid number of ports int32 max tree depth int16 wpe LE VENT OSM EVENT SM STATE NOT _ACTIVE N normal enable manual Entering SM Not Active State Object SM OSM EVENT SM STATE STA N NDBY normal enable manual Entering SM Standby State Object SM OSM EVENT SM STATE DIS N COVER normal enable manual Entering SM Discovering State Object SM OSM EVENT SM STATE MAS N TER normal enable manual Entering SM Master State Ob
43. Node Name ibhost ibhost ibhost KDE A 472 25 3 Topology Map Table Description Category Event ID ibhost 2 HCA 1 ourter threshe Value is 22222 previous value was 11111 Port Counters Error Link Down Evert ISROXXX sLB 24 1 Internal Port 10 thresho soldat Sloat value is 22222 previous value was 11111 Port Counters Error Port Rev Remote Physical Errors ISRIXXX sFB 4 279 Counter thresho E value is 22222 previous value was 11111 Port Counters Error Link Error Recovery ISRIXXX sFB 4 4 3 Counter thresho Clear Event value is 22222 previous value was 11111 Port Counters Error Symbol Error ISR9XXX sLB 24 1 Internal Port 21 1x port was fou Clear All Port Counters Error Osm 1x ISROXXX sLB 24 1 Internal Port 7 thresholti Was Exteeded NEW value is 22222 previous value was 11111 Port Courters Error Local Link Integrity Errors LC ibhost 7 HCA 1 ourter threshold was exceeded New value is 22222 previous value was 11111 Port Courters Error Port Rev Switch Relay Errors Sm Comm OK System 0206 lt Figure 3 13 Go to Node 3 28 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Managing Alarms and Events Clearing Events Clearing an event means that you have taken the steps to correct the problem and now wish to remove the event from the Trap Table To clear a specific event M Right click the eve
44. Oyo 2 2 gt Human translation There are two fabric ports whose peers are not responding Conclusion In both cases something connected to the sFB is not responding it could either be SLB the IPR or the GER If it is the sLB try to remove and insert the sLB A device is probably set in continuous reset If the IPR GER are causing the problem verify that the module and sRBD is correctly plugged in After a power cycle it is also possible that modules are still not UP Hint After power cycle wait 4 6 minutes until all modules are UP then run a script Known bugs or limitations none A 22 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode Command findpath Description Prints port path between two nodes Syntax findpath d ebug v erbose s imple f astmode t timeout_ms lt portguid gt Example Example findpath 0x8f10403965lcd Output KOREA REE eZ Human translation 1 Pkt transmitted from CPU 2 Entered first ANAFA chip via port 0 3 Transmitted from ANAFA via port 34 4 Entered second ANAFA chip transmitted fon ANAFA via port 15 5 Entered third ANAFA chip transmitted from ANAFA via port 2 Conclusion on the requested path 3 ANAFA chips one entity voltaire HCA see guid range Command getpathrecord Description Checks the existence of a path between nodes Syntax getpathrecord d ebug p gid_prefix s smlid t timeout ms l
45. Right click the node in the topology map and select Configuration from the context sensitive menu The following window is displayed Node Configuration Mode Mame IPF Address SUID Description shi S26 Yendor Voltaire Device D 5407 Rev OO000040 Figure 3 26 Node Configuration Voltaire Switch User Manual 3 43 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 9 Step 1 3 44 Parameter ATS Manual Node Name IP Address GUID Ports Description Vendor Device ID Rev Description By default the ATS option is selected meaning that node data is discovered and input to the node repository by means of the Automatic Translation Service ATS mechanism However if ATS is not supported you can click the Manual option and enter node data manually If you selected the Manual option enter a user assigned name for the node This is the name that appears in the various Device Manager and Fabric Manager windows in reference to the node If you selected the Manual option you can enter the IP address of a switch you can not enter the IP address for hosts For all hosts and nodes that are set to be discovered using the ATS mechanism this field is read only Displays the Global Unique Identification number GUID of the node Displays the number of ports on the node Displays a description of the node Displays the vendor that manufactures the device Disp
46. San r e L eee Figure 1 4 ISR 9288 InfiniScale Switching Topology Fat Tree Topology The ISR 9288 9096 external ports are configured to work in 12X 4X auto detection mode meaning they are capable of working in 12X mode in case they are cabled to another 12X enabled port The ISR 9288 features six symmetrical Line Boards sLB 24 at both ends and four interconnect Fabric Boards sFB Each sLB includes two 24 4X IB port chips Each sFB has three 4X IB port chips Voltaire Switch User Manual 1 9 P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction Figure 1 5 illustrates the ISR 9096 InfiniBand internal switching matrix Fabric Board sFB 4 Line Board Line Board sLB 24 sLB 24 erp ea A L W 5 a T a 4 c 2 ff V YAN h N d f i N 4 L X L N AN HIT LLU TAN W y WAY VV KY IELI ll li I LI lel Figure 1 5 ISR 9096 InfiniScale Switching Topology Fat Tree Topology The chassis internal links links between sFB 4 and sLB modules are respectively configured for 4X mode when sLB 24 are installed in the system or 12X mode when sLB 8 12 are installed future option LEDs on the sFB 4 modules indicate 4X 12X link operation mode External 4X ports on the ISR 9096 can be grouped as a trio to form a single 12X port even if the physical co
47. Switch User Manual Function The question mark key lists all of the options available at that point in the command line along with a brief description of the command Each word may be truncated at any point after it is unique If you enter a question mark before a partial command use a Space after the mark the system provides a list of commands that begin with that string Completes a partial command name entry When you enter a set of characters that match the beginning of a command name and press the Tab key the system completes the command name If you enter a set of characters that could indicate more than one command the system lists all available options Erases the character to the left of the cursor At the command line pressing the Return key performs the function of processing a command At the prompt on a terminal screen pressing the Return key scrolls down a line Moves the cursor one character to the left When you enter a command that extends beyond a single line you can press the Left Arrow key repeatedly to scroll back toward the system prompt and verify the beginning of the command entry Moves the cursor one character to the right Recalls commands in the history buffer beginning with the most recent command Repeat the key sequence to recall successively older commands Return to more recent commands in the history buffer after recalling commands with the Up Arrow or Ctrl P Repeat the key
48. Temp Z j j BS Chassis Mgt d 2 stp Reset CLE2 Eth 2 Figure 4 24 sCTRL Information SCTRL Information Window Description The sCTRL graphical display shows which sMB is active and whether the Subnet Manager and the Chassis Management are active Feature peen O O O sMB 1 and 2 LEDs show the status of the Subnet Manager and the Chassis Management sFB Shows how many sFBs are installed Ethernet and 2 In the system green link orange active No indication in the GUI Voltaire Switch User Manual 4 33 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 6 5 4 6 6 4 34 Description CLII and CLI2 No indication in the GUI Temperature Lights up when there is an excess of temperature Reset Resets the chassis Table 4 12 sCTRL Information Window Features sPSU Information Power Supply modules are installed at the rear of the ISR 9096 9288 chassis Ideally each Power Supply module line cord should be connected to a different circuit for maximum fault tolerance in the event of power problems To view the Power Supply information M Click the desired sPSU on the Device Tree The sPSU Information Window appears SHE Z HE Figure 4 25 sPSU Information sPSU Information Description The green line around the power connector indicates that the power supply 1s connected to a source of current AC ok DC ok Green LED means that AC and DC are functional Fan Info
49. The lower part of the window displays the status of each port and related events 4 28 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Module Information sLB 24 Line Board Information with MediaConverters The ISR 9096 9288 provides support for optical converters on Infiniband ports When installed on sLB ports optional MediaConverters have a distinctive graphical representation in the VDM to differentiate them from the IB connector as shown below Connected Empty Connected IB Connector IB Connector MediaConverter Figure 4 19 sLB 24 with MediaConverters Further detail is available in the ISR 9XXX Installation Manual Voltaire Switch User Manual 4 29 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 6 2 F Hot Swap Fabric Board Information Up to four Fabric boards can be installed at the front of the ISR 9096 9288 chassis to achieve a fully non blocking system To view the Fabric Board sFB information M Click the desired sFB on the Device Tree Figure 4 20 and Figure 4 21 respectively display the ISR 9288 and ISR 9096 Fabric Information Windows l Hot Swap l Power b info E amp Link i E Link 2 Link 3 Blink 4 Serial Murmber AAMPIZ04001 U Serial Mumber DAH 31 O41 234 Hardware ersior AAA l Hardware wersion AAA Advance Figure 4 20 ISR 9288 Fabric Board Figure 4 21 ISR 9096 Fabric Board Info
50. To update switch software perform the following steps Copy the New Software to the FTP Server Make sure that the FTP server is active If a default gateway has not been configured the FTP server must be located in the same subnet as the switch management port Copy the new version files to the FTP root directory ll Configure FTP Server Settings Identify the location from which to retrieve the updated software If the default download location is not appropriate you can reset it If this location is not the one from which you would normally retrieve updated software use the steps in the following table to set the appropriate download location Voltaire Switch User Manual 2 15 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 1 Step 2 Step 3 x 2 16 Command Description Enable Enter Privileged mode from Exec mode config Enter Config mode from Privileged mode ftp Access the FTP menu from Config mode server Configure the IP address of the FTP server lt PUDsVOLCAtL Se com gt username lt ftp gt Configure the user name for FTP server access password lt 123 gt Set the user password to access the FTP server ftp show After the update is complete execute this command to verify the new settings lll Update Switch Software The update software command in Privileged mode makes a new v
51. Voltaire ISR 9096 9288 high port density switch the Voltaire ISR 9024 enables the formation of clusters scaling to thousand of nodes The ISR 9024 architecture features a passive cooling device with no fans resulting in a noise free high MTBF unit ISR 9024 a SS Se SS SS SS mw R HP NR H 9 eee InfiniBand Figure 1 1 Voltaire ISR 9024 Switch Functional Diagram Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc ISR 9096 9288 System Overview 1 3 ISR 9096 9288 System Overview The Voltaire ISR 9096 9288 InfiniBand Multiprotocol Switch is a modular platform for building high levels of performance and scalability for large high performance computing HPC clusters The Voltaire ISR 9096 9288 enables high performance applications to run on distributed server storage and network resources Elegantly designed with 10 30 Gb s of full bisectional bandwidth for up to 96 or 288 4x 12x ports in a single chassis multiple ISR 9096 9288 can also be interconnected to form very large clusters Possible configurations range from dozens to thousands of nodes with many upgrade paths available Featuring fat tree Clos topology that provides full bisectional bandwidth for each port the Voltaire ISR 9096 leads the InfiniBand industry with the highest switching capacity Providing unmatched levels of IniniBand bandwidth the ISR 9096 and the ISR 9288 respectively support up to 96 and 288 InfiniBand ports which can b
52. Web support voltaire com email Voltaire support center Phone US 1 781 276 1564 Elsewhere 972 54 499 66 33 PS Done Internet Figure 4 1 Accessing Voltaire Multi Protocol Switch Window 4 2 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Getting Started with the Device Manager The window called Accessing Voltaire Multi Protocol Switch contains the following icons eon penan O Launches the Voltaire Graphic Device Manager Launches the Voltaire Fabric Manager which is described in Chapter 4 Click to download the Sun Java Web Start application which must be installed in order to use the Voltaire Vision GUI Table 4 1 Voltaire Initiate Icons NOTE When opening the Voltaire Device Manager Initiate window for the first time click iTA to start the Sun Java Web Start installation The computer that is being used must be connected to the Internet for successful Java Web Start package download from Sun s website To start the Voltaire Device Manager application click from the Voltaire Device Manager Initiate Window Voltaire Switch User Manual 4 3 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 2 1 4 4 Login to the Device Manager After the Voltaire Device Manager application is started the Login window is displayed as shown in Figure 4 2 Voltaire Manager User Login IF Address 223 0021 0 User Type admin Password
53. commands follow a simple structure and are capable of prompting for further information as the user types The CLI provides the following basic commands Command Description exit Leaves current mode and return to previous mode Displays help text and usage for the CLI command command Displays help for the specific command end Terminates the current management session Table 2 2 Basic Commands For complete command descriptions and syntax information refer to 0Appendix A on page A 1 CLI Case Sensitivity The CLI is case sensitive All keywords must be entered in lower case Any user defined strings such as names or descriptions may appear in any case including mixed case Case information for user defined strings is preserved in the configuration CLI Command Prompts The CLI command prompt includes the switch name Configuration changes are automatically saved as they are made The gt or sign respectively show if you are in Exec or Privileged configuration mode as shown in the following table CLI command mode Command Prompt ISRXXxXXxX gt ISRXXXX For use by Voltaire personnel only Voltaire Switch User Manual 2 5 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI 2 2 First Time Configuration First Time configuration is performed via CLI This section describes the procedures for first time configuration It contains the following information Collecting Req
54. displayed SSS ti i lt Q You are about to exit the program Would vou like bo proceed Figure 4 3 Exit Warning Message Step 2 Click Yes to confirm Voltaire Switch User Manual 4 5 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 3 zt 4 3 1 Main Menu Device Tree ISR 9288 Device Graphical Representation Trap Table 4 6 The Device Manager Main Window This section describes the Main window of the ISR 9096 ISR 9288 and the ISR 9024 Voltaire Device Manager application including basic functions and operations The Main window provides access to all Voltaire Device Manager functions It also provides a graphical representation of the connected switch and components with color coded visual indicators that display alarms and important events in the switch PRE The Main window and especially the device graphical representation differs depending on the switch connected to the Device Manager ISR 9288 Main Window Figure 4 4 shows the ISR 9288 Main window Device A O lanager Fim Wie Dente Scion Hea Denice Hame frama Dian Srvtich Deacoin Eevee Votare Deva 6 H ri proosa AE Sothern wWarsiri Bao L Ha ecn e Ve SKL ASAS x Hi Information HEE Wersi ki votare A MELE aa E rance Reset L d a LISLISTISTISELIELI Thina 69 8 6R 86R 86R 8 69 8 17335 2959 Tps Cala Tim Sous Ler Cabegory Everd iD El
55. document may be reproduced stored in a retrieval system or transmitted in any form or by any means without the express written consent of Voltaire Ltd Voltaire is a trademark or registered trademark of Voltaire Ltd or its subsidiaries in the United States and other countries Other names and brands may be claimed as the property of others and have been designated with an asterisk throughout this document Copyright 2005 Voltaire Ltd Voltaire Part Number 3992700038 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc S About this Manual This preface describes the audience organization and command syntax conventions of the Voltaire Switch User Manual It also provides information on how to obtain related documentation and technical assistance Information about hardware installation of Voltaire switches can be found in ZSR 9024 Installation Manual and in the JSR 9XXX Installation Manual which includes the ISR 9096 and the ISR 9288 NOTE Consult Voltaire release notes for last minute updates and restrictions The Voltaire Technical Support Center TSC is at your service You may access Warranty Service through our Web Request Form by using the following link http Awww voltaire com support html Contact Us Please send your documentation related comments and feedback or report mistakes to docs Voltaire com We are committed to constant and never ending improvement Your input wil
56. e EEE E EAA E E AE 6 9 Figure H Port Counters Ten E enirere n R N 6 11 IT 6 4 POr Counters Grapes AA E AE A 6 12 Figure 6 5 Frap Table 1 X Lit eonan e naa eaa aia Ri 6 18 Figure 6 6 VFM Event Data WindoW ssssssssssseeresesessessssssssssserreecsessressssssssssreereesssesse 6 19 Pieun G Nodes Mo W IMGOW seinere aa E N 6 20 VW Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Tables A I CLI Key FUNGON S esiseina 2 3 Taole 22 Base C OAN S ae a A E E E A E E EE 2 5 Table 2 3 Switch Installation Checklist sss essen enen 2 6 Table 2 4 Terminal Emulation Con hgeuration sss ee eee eee 2 7 Table 3 1 Fabric Manager System Requirements sese eee eee eee 3 3 Table 3 2 Voltaire Initiate ICONS scsi ssaneanerastandoendes banner si eirian r aE EE Eni SiS 3 5 Table 3 3 Topology Map Icons sss sss 3 11 Taole oA Main M cni OPHONS siirre ri rn ee E EE EE 3 15 Table 3 5 Subnet Tree 617 CT 3 17 Table 3 6 Fabric Manager Toolbar ICOmS sss sss 3 18 Table 3 7 Topology Table View Felda 3 21 Table 3 8 Current Alarm Window Fields sss 3 23 Table 3 9 Alarm Data Window Fields 3 24 Table 3 10 Trap Table Data sss sss 3 26 Table 3a 1d Events Data TL Ta L 3 28 Table 3 12 Host Node Information Parameters sese sss sese sese 3 33 Table 3 13 Port General Information Parameters esse sese 3 34 Table 3 14 Link Information Parameters sss sese eee eee 3 35 Table 3 15 Hosts List Information Parameters esse 3
57. err Tiu Feb 03 17 39 30 1ST 2005 ITS 3 SHS SAC amd Do Bad Deka 0102 E Toy Trai BU Feb 03 T T 39 29 aT 2005 are 25 32 30 LAP ALA TY MASHK BOOED af OOT GGG LID 173 Z 65 UD 35 Z mead 9 aChdask One Post Lara B Hora Tiu Foes 03 17 39 20 IST 2005 3 35 39 3 TOPOLOGY HFG Applad dalai roulirag Em OCS B nera Arrests 3726 T NEE iras 238 LINK TIMEOUT af STIHSTEIEEET D 258 SubnGel SMinto DA path jii Em E ED rma Tiu Feb 03 17 38 29 IST 2005 jna 2139 aM STATE CHAHGE Eter rna SH Wailer Stele Sm 033 E Her Thu Feb CA ir Sir ZZ ET NE IT a a 30 LIM TOT a LET USRI KET LD Ser Sour Minne De prah Opa Emn RI 5 E path OG Figure 4 4 ISR 9288 Voltaire Device Manager Main Window Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 4 3 2 ISR 9096 Main Window Figure 4 5 shows the ISR 9096 Main window Voltaire Manager File View Device Action Help The Device Manager Main Window ISR 9096 sFB 4 sFB 1 sFB 2 sFB 3 sFB 4 E sLB sRBD sLB 1 E sRBD 2 E FCR 2 E FCR3 sLB3 SLB 4 D gane RT sMB Lett i SMB Right s l sPSU sPSU1 sPSU 2 sPSU 3 SSCS ee eeeeeeee sPSU5 sFU sCTRL ISR 9096 Device Graphical Type Date Time So Description Representation Warning Mon May 30 19 26 34 IDT 2005 ISR9096 sLB 24 3 13 Counter threshold was exceeded New value is 14272 previous value was 14071 Device Name Switch Description Device Revision Software Version Hardware Version Ven
58. ete tad 6 11 Displaying and Checking Port Counters using the GUL 6 11 Fabric Diagnostic using the Event Log sss 6 13 Fabric Diagnostic Using Current Alarm sss sese 6 14 Fabric Diagnostic Using Fabric Statistics portcounters csv file 6 16 Additional Tools and Examples sss sss 6 18 Detecting and Troubleshooting 1X Cable Width Problems 004 6 18 Identifying a Bad Line Board or Fabric Board Port sss 6 20 ba esa ai Wa a Bad POr RP Pon 6 22 Some CEI Digenostic TOOLS erri ets Bees hens ee eee 6 24 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Contents ZELO COUNLETS e R YU 6 24 width heck VSCTIDE aiviaisietesindedstetch Mane lanuca a tibialis 6 24 eror Had SCD nr ee re UT ET OC UT 6 25 Appendix A CLI Commands Reference ssssccssccccccsssssssscssccccscssesees A 1 A 1 CLL Usage KT A 2 A 2 TST VIO GG TTT A 2 A 3 Voltaire lt lt TT Basie CONCEP TTT A 3 A 4 Bee TT A 3 Exec Mode Commands LA exists eins Sviies cdo E ecards A 3 A 5 NRT s Na rea T A 9 Accessing RSH OCG Mode T A 9 Privileged Mode Commands Reference sss sese A 9 UGRESS M 1 T A 19 A 6 Contisuration Contis Modessa ansadenes A 33 JNCCESSING Conto Mode iranan AA ee A 33 Config Mode Commands Referenc A 33 Custer NT O cass R ETRE A 38 IVA er TTT A 39 IS Y aree Me TT A 41 LOCAL Tnierace Menis a A A A 43 Fast Interface Ment ccnrpo nener EEE A 44 Koue Menu sana r RC Ra er
59. fails partition check IP version check or transport header version check Typically will not increase If it is may be an indicator that a parameter should be tweaked Please contact Voltaire Customer Support Total number of packets received on the port that are discarded for the following reasons FilterRawOutbound is true and packet is raw PartitionEnforcementOutbound is true and packet fails partition check IP version check or transport header version check The number of times that the frequency of packets containing local physical errors exceeded local phy errors This counter increasing in number usually indicates a bad link B 5 P N 399Z00038 2005 Voltaire Inc Chapter B Port Counters ExcessiveBufferOverrunEtrrors PortXmitDiscards PortXmitConstraintErrors PortRcvConstraintErrors LocalLinkIntegrity Errors B 6 The number of times that overrun errors consecutive flow control update periods occurred with at least one overrun error in each period see Table 126 PortInfo on page 665 of the IB spec Total number of outbound packets discarded by the port because the port is down or congested Reasons for this include Output port is in the inactive State Packet length exceeded neighbor MTU Switch lifetime limit exceeded Switch HOQ limit exceeded Total number of packets not transmitted from the port for the following reasons FilterRawOutbound is true and packet
60. is executed as follows Retrieve the first block of the multicast forwarding table attribute id 27 attribute modifier is implicitly O of the switch element which corresponds the direct path lt 1 7 gt Follow the direct path start at element 0 which is our 8 port SW 6IB4C Go out of port 1 of element 0 Reach element 1 which is connected to that port Go out of port 7 of element assuming that element 1 has a valid and connected port 7 Reach element 2 which is connected to that port The direct path lt 1 7 gt led us to element 2 The corresponding MAD will retrieve the required data from the element provided this is a switch element Command smpex Description smpex is a powerful tool to create MAD traffic to specified node port smpex is available from the root shell or the ISR 9096 9288 Syntax smpex d ebug s tring_data v erbose a ttr m od D irect t timeout_ms lt dlid path gt lt num gt win Subnet Management Attributes AttributeName ECC e a y AVe cae Wore Biss akene Notice 2 U NodeDescription 16 U NodeInfo Va U S TE Te U GUIDInfo 20 EL Ee ck Portare aA Port Number P_KeyTable Ze Port Number P_Key block SLtoVLMappingTable23 EGL SR GEH pu Orme Niamibetcs VLArbitrationTable 24 Clie Ui Clas Como omens Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode LinearFwdTable US BLOCKA TAGON wee RandomFwdTable 26 PLOCE IO ne ic kere MulticastFwdTable Z
61. lmc sm parameters sm info mode set Fnable or disable subnet manager mechanism ome NEO MODY see Set max number of operational VLs sm info polling retries set Set polling retries sm parameters Si MeO mor LO ie OSE SEE Pruoracy SM paramerers sm info show Show the sm parameters Sli E oe A S E Set switch lifetime value Sm rnar Oem hehehe ellie On ER IhOme us Cle sm info initiate fabric sweep Sm An CO Si hia cite ball rene re TT HE LO ser SM teh Os Osby GOO ON OG a aeo ee cine Sweep the fabric sm info smName set sm info smName set sm info sweep interval set Set sweep interval sm parameters Si UMN bo TOpoLSgy SoL Sman ODO hogy E sm info verbosity set sm info verbosity set sm info verbosityTopology set Sia NEO VSCD Aey a ROPO Loy Ser SFBs add Add sFBs SFBs del Delete sFB SFBs show Show the sFBs Voltaire Switch User Manual A 47 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference SM syntax usage is as follows sm info Sie LIE Si ian Si eine Sin PRELO sm info set sm info sm info sm info sm info Peet Melis cic hog set Ime set mode set mopvl set polling retries DE rOrITY AS OE show sly ser sm initiate Labrie Con rl guiseed On seo sm info sm initiate fia lie eG e OW Ur Aae Om set Si Lie SH set Si EO sm info sm info set smName set Sweep interval topology ser verbosity set verbosityTopology sm info a
62. management Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Voltaire Switch Management 1 4 4 ISR 9096 9288 Management Interface Configuration ISR 9096 9288 switches have several management interfaces resident on the sMB module Serial FAST IB LOCAL Internal The Serial interface RS 232 provides an interface to the CLI The Fast interface is the Ethernet interface of the sMB module In redundant sMB configurations the IP address of the Fast interface 1s applied to the active sMB card In the event of failure on the active sMB the IP address of the Fast interface is moved to the backup SMB The InfiniBand interface is the IPoIB interface of the sMB module When managing the ISR 9096 9288 chassis from the IB fabric in redundant sMB configurations the IP address of the IB interface is used In the event of failure on the active sMB the IP address of the IB interface is moved to the backup sMB The Local interface is an optional IP address which can be attached to the Ethernet interface of the sMB module in addition to the Fast interface IP address and is intended for debug purposes Unlike the Fast interface the Local interface is configured per sMB module regardless of the redundancy configuration and regardless of the sMB mode as active or standby The Local interface 1s attached to a specific sMB module The ISR 9096 9288 sMB module has an additional internal Ethernet connection Th
63. menu Step 2 Click the VL Arbit High tab or the VL Arbit Low tab the procedure 1s identical for both The following dialog box is displayed Voltaire Switch User Manual 3 51 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Quality Of Service Eg Quality Of Service SL To YL YL Arbit High WL Arbit Low SL To YL YL Arbit High VL Arbit Low O Disable Activate Default O Disable Activate Defautt Figure 3 30 QoS VL High Arbitration Table Step 3 In the High Priority Limit field enter the number of high priority data that can be transmitted without an opportunity to send low priority data Specifically the number of bytes that can be sent is High Priority Limit times 4K bytes with the counting done to 4 byte increments A High Priority packet can be sent if current byte count has not exceeded the High Priority Limit A value of 255 indicates that the byte limit is unbounded Note if the 255 value is used forward progress of low priority packets is not guaranteed by this arbitration scheme A value of 0 indicates that only a single packet from the high priority table may be sent before an opportunity is given to the low priority table The High Priority list has a minimum length of one and a maximum of length of 64 The Low Priority list has a minimum length equal to the number of data VLs supported and a maximum of length of 64 The High Priority and Low Priority component lists are allo
64. of the event alarm Category Displays the source of the event Possible values are SA for events that occurred in the Subnet Administration SM for events that occurred in the Subnet Manager Port for events related to port activity Voltaire Switch User Manual 3 25 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Event Description Information Eventld Displays a system generated ID number that identifies the event Refer to Appendix C for a list of the events reported in the system Table 3 10 Trap Table Data Rearranging the Trap Table You can rearrange columns positions within the Trap Table by dragging and dropping a selected column in the new desired position Event Logs The SM log topology log fabric subnet management routing log information and Linux log system log shows info of the OS are available via the CLI as well as via a HyperTerminal Minicom Logs can be retrieved by the means of a browser by entering their IP address or via a terminal Viewing Logs in a Browser You can view error and event logs from a browser by entering the IP address and errorlog txt or eventlog txt as shown in the following example http 172 25 3 16 errorlog txt Sorting Logs You can sort the data in the Trap Table View by clicking the column headers An arrow indicates the sort order ascending or descending Use Ctrl click for a secondary sort order 3 26 Voltaire Swi
65. pipeline 16 sm polling retries 12 sul ECC wiley 200s Sm S SLS lt LSI sm mode enable sm LMC 1 sm hog 13 sm slv 14 Sit mojowl a NOTE Fd Refer to page 3 43 for a description of the Subnet Manager parameters Command snmp community show Description Returns the SNMP community strings Syntax snmp community show Example SW 6IB4C snmp community show Gee e ommuni T PORTII set community private Erap SCM ey PUEI Command version show Description Displays software version information Syntax version show Example ISR 9288 version 3 3 0 date Feb 10 2005 10 51 52 AM build Id 208 A 8 P N 399Z00038 2005 Voltaire Inc Voltaire Switch User Manual Privileged Mode A 5 Privileged Mode The Privileged mode command is used to configure basic parameters such as date time reload reset switch and to perform software and firmware updates It also includes all Exec mode commands If the system administrator has set a password you are prompted to enter it before being allowed access to Privileged mode The password is not displayed on the screen as you enter it and is case sensitive The system administrator uses the password update enable command to set the password that restricts access to privileged mode Accessing Privileged Mode The following example shows how to access Privileged mode ISRXXXX gt enable Password XXXXXXX ISRXXXX Privileged Mode Commands R
66. range of port and node information Voltaire Fabric Manager is available for the following hardware platforms R ISR 9096 switch system R ISR 9288 switch system R ISR 9024 switch system Voltaire Fabric Manager Components The Voltaire Fabric Manager is comprised of the following management components R Subnet Manager SM Provides fabric discovery and configuration E Subnet Administration SA Serves as the subnet database answering queries from other subnet entities R Performance Management PM monitors the port counters of the fabric elements and generates notifications for port errors R Voltaire Fabric Manager User Interface Fabric monitoring and configuration via GUI and CLI M Voltaire Device Manager User Interface Device monitoring and configuration via GUI and CLI Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Fabric Manager Installation 3 1 2 Voltaire Fabric Manager Main Functionality 3 2 3 2 1 Voltaire Fabric Manager provides the following functionality E Alarms Color coded display of alarmed objects in all relevant Fabric Manager display windows Current Alarms reports and ability to manually clear alarms Setting and filtering alarms WE Viewing fabric connectivity E Management and monitoring of Fabric parameters e g LMC value Sweep interval SM info Logging verbosity level etc E Enhanced monitoring of fabric elements e g Node info Port info Port
67. section provides information about the following CLI topics R CLI Usage Tips on page A 2 R CLI Modes on page A 2 M Exec Mode on page A 3 M Privileged Mode on page A 9 a Configuration Config mode on page A 19 Voltaire Switch User Manual A 1 P N 399Z00004 2005 Voltaire Inc Chapter A CLI Commands Reference A 1 CLI Usage Tips R Use the Tab key to complete commands as follows type in the first one or few letters of the command and then key in Tab the CLI completes the current word allowing you to continue entering the command R Use the Question Mark key to list all of the options available at that point in the command line E Commands and keywords can be truncated at any point after they are unique R The CLI is case sensitive All commands and keywords must be entered in lower case user defined strings can appear in any case including mixed case Case for user defined strings is preserved in the configuration A 2 CLI Modes There are four CLI modes in the switch Exec Privileged Configuration and Debug With the exception of the Configuration mode which can be accessed from Privileged mode each mode is password protected Each mode is easily recognized by the accompanying prompt listed in CLI command Command Prompt Access Method mode Privileged ISRXXxx From Exec mode To go back to Exec mode type disable orexit Debug or use by Voltaire personnel only A 2 Voltaire Switch Us
68. set Sets VL to SL mapping as described on page 3 49 gos sl vl set row virtual lane qos sl vl show Displays the QoS SL VL Table gos sl vl show qos vl high arbitration set Sets the High Arbitration table as described on page 3 51 gos vl high arbitration limit set row virtual lane priority weight qos vl high arbitration show Shows the High Arbitration table Gos vieghiqh arbitracion Limic show qos vl high arbitration limit set Sets the Limit of High Priority parameter described on page 3 51 qos vilehigh arbitration limic set number Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Configuration Config Mode Command qos vl low arbitration set Description Sets the Low Arbitration table as described on page 3 51 Syntax qos vl low arbitration set row virtual lane weight Command qos vl low arbitration show Description Shows the Low Arbitration table as described on page 3 51 Syntax qos vl low arbitration show Example of SM Parameter Configuration Max OPerational VL Virtual Lanes operational on this port are indicated as follows 1 VLO 2 VLO VLI 3 VLO VL3 4 VLO VL7 5 VLO VL14 Sets the time a packet can live at the head of a VL queue LifeTimeValue Sets the time a packet can live in the switch HOQ and LifeTimeValue control transmitter queuing as follows The transmitter packet discard is based on among other things two time values
69. the CLI Use the Current Alarms Table to see current problems In the VFM right click and select Alarm Data to get information to help identify where the problem is located Use the Topology Map to identify nodes with a current alarm Proactively look for increasing error counters using the statistics feature and running the Diagnostic scripts using the CLI Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic Fabric Diagnostics Methodology For a successful InfiniBand fabric diagnostic please follow these guidelines l 2 3 4 5 6 7 8 9 10 Voltaire Switch User Manual Enable the port counters reports Run stress traffic across all fabric nodes Get the port counters portcounters csv file and or or Current Alarm Table Review Fabric active port nodes counters Locate problem see list of Potential Problems Section 6 1 3 Fix the problem Reset counters Run Traffic across the fabric again Verify that the problem is fixed Repeat the above until all fabric nodes have No bad link error there might be error counters that are gt 0 and not problem related P N 399Z00038 2005 Voltaire Inc 6 Fabric Inspect Settings 6 2 Fabric Inspect Settings To Configure the System for Fabric Inspection Step 1 Use the CLI to Enable PM collection and PM reports as described in Section 6 2 1 below The system is collecting all port data and errors Step 2 Define PM thres
70. the CLI or the Voltaire Fabric Manager Tool Tips in Topology Map You can display a tool tip for device icons and connecting lines by positioning the cursor above the icon or line The following is an example of a tool tip P 192 168 2 100 For nodes the following information is displayed the device GUID the name of the device and the IP for switches only For lines the number of connections between devices and the link speed are displayed Connections Link width 4z Moving Icons on the Map The location of the device icons can be rearranged in the Topology Map To move a device icon click and drag the icon to the desired location The icon location can be reset by clicking Hierarchical layout button on the toolbar Tee You can select multiple icons by clicking and dragging the mouse cursor over the group you want to select Resizing Icons You can resize icons by positioning the mouse cursor on one of the squares displayed around the icon and dragging it to the desired size Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Search in the Topology Map 2 Type in Search string 1 Select Search 3 Click criteria Search You can perform a search for a specific object in the Topology Map by using the toolbar search box To perform a search Step 1 Select the search criteria you can search according to the node LID according to the node name or according to port GU
71. the Subnet Tree whenever an object has children a icon is displayed to the left of the object icon To expand the object and reveal the children assigned to it click once on the Hl icon The children are immediately displayed underneath the parent and the E icon is displayed to the left of the object icon Click once on the E icon to collapse the parent object hide the children Toolbar The toolbar is displayed at all times at the top of the main window It is divided as follows R General buttons displayed in all Fabric Manager views E Topology Map buttons displayed only when the Topology Map is displayed All toolbars have a tool tip Position the mouse cursor above a toolbar icon to display the name and functionality of the icon The buttons and the functions they invoke are described in Table 3 6 Table 3 6 Fabric Manager Toolbar Icons General Action Buttons Refresh Refreshes the Voltaire Fabric Manager main window with the most updated information Events Log Displays the online Events Log Events Table described on page 3 22 online The Events Log provides information on the events and alarms in the system Back Forward Cycles backward and forward through open Fabric Manager windows Up Moves the view one level up according to the Subnet Tree hierarchy Topology Map Launches the Fabric Manager Topology Map from the topology map option only See page 3 10 for more information Voltaire
72. threshold for this counter Setting to 0 disables counter monitoring default 5 pm symbol error counter set 0 63335 pm trap mask set Defines which event is reported by the event logger Default default pm trap mask set none default all pm vl15 dropped set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 pm vli5 dropped set 0 65535 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc QoS Menu Configuration Config Mode The following are the commands in the Subnet Manager QoS sub menu ROT om Gosia command end exit Goo a PEPI ONT ora EON See gos configuration enable set SIT as how Geos sil vil set goo SE r sinew oe H e elie E a E Gosa ae Nel a pI erae L On Slavens E L T S E E O E E eien A E LTE Goor val Low aa E L QoS syntax usage is as follows EH CoN Igrac ron ser gos configuration enable set GOSS wk Oe Show qos sl vl set qos sl vl1 show Skee e eel O EEN S T e a a a a eO okore an ee R aeo ET ELE TEE Se GOS le lonar one ee On eL GOS VAL ey EE ra dee gh e OR Dresplay hist Or tavairlanlie commands Drep lay commana Ss usage Ern R e T TT EEA orner Moden TT Connection Or T e CEEE Sess benn Vila ee ae eea e E Exit to previous menu Seb the Change flag for Cos Set the Enable Configuration iene hone MONS show the QoS parameters Sane Voln alae Silly Wily sole Sine Ces Sib
73. 0038 2005 Voltaire Inc Chapter A CLI Commands Reference A 54 Command sFBs show Description Displays the sFBs list defined in the switch including entry numbers and GUIDs Syntax SFBs show MC Menu The following are the commands in the Subnet Manager MC sub menu BSORVOIC S009 Leon mtg sma me i 2 Display rot of avaeltlable commands 2 command Display Connandt o msa SEEN end Extn cO login Menus Fron any Orere yia console connection or Glose telner ses ssion via celner connection exit Ex LO prev OUS menu me eee JOIN Set Set the auto join for the MC mc show VOW Me rs Settings The following details the commands available in the Subnet Manager MC sub menu Command mc auto join set Description Enables the Multicast Auto join mode this mode is disabled by default In Multicast Auto join mode a multicast group 1s created by the SM HCAs can join or leave the Multicast group Syntax mc auto join set disable enable Command mc show Description Show Multicast mode setting enabled to disabled Syntax mc show B54 o Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuration Config Mode PM Commands The io E Cons end exi pm pm pm pm pm pm pm pm pm pm pm pm pm pm following are the commands in the Subnet Manager PM sub menu SNOUSHE OOE Sie Oil sine Daksiollene ro t Or naria ie commands ommand Dvsplay command s w
74. 150 Thresholds type Per sample Thresholds Symbol Err Counter 200 Rev Constraint Err Err Recovery Critr Local Link Integrity Err 5 Downed Cntr Excess Buf Ower run Err 5 Port Rew Err VI Z Dropped Rev Rem Phys Err Ym Data d Rev Swich Relay Err o 4 Rev Data xmi Discard m pis 0 Xmit Constraint Err l T Rev Pts D Reset Counters Figure 6 2 PM Settings By default PM counters are monitored for all devices in the fabric Scope field is set to All To change these settings select one of the following values from the Scope field Voltaire Switch User Manual 6 9 P N 399Z00038 2005 Voltaire Inc Step 2 Step 3 Step 4 Step 3 Step 5 Fabric Inspect Settings R Hosts Only PM data is only collected for HCAs in the fabric R Switches only PM data is only collected for switches in the fabric You can also disable selected PM counters by unchecking the box displayed next to the counter In the Sample Interval field set the frequency at which the PM data is collected from the counters range 60 seconds to 65535 seconds The Thresholds Type in the current software release is always set to Per sample i e the Fabric Manager checks for errors in each sampling if the threshold is passed between samples interval The counter is reset when it reaches its maximum value as defined in the InfiniBand specification Disable the Thresholds you don t want to have checked Click Rese
75. 3 4 4 3 4 5 3 4 6 3 4 7 3 4 8 3 4 9 3 5 Jol E 3 6 3 6 1 3 6 2 3 6 3 E Configuring the ISR 9096 9288 with I O Modules sss sese 2 10 Conticurne the Time and Date acid atta hi alidcaceiadetel Ea 2 12 RETER TTS Pass e L 2 13 Ven YI Con oura OI T 2 13 CLI Maintenance Command Sensis a A N 2 14 Pror gU sE Sa Ses eas aie eS 2 14 ACI SS OMG T 2 14 Prerequisites for Updating Software sss 2 15 Backing up the Switch Configuration sese eee eee eee eee 2 17 Software and Firmware Upgrade Process sss sese 2 18 Voltaire Fabric Manager VEM ssscecscessssceceoocoossssccceooossssseee 3 1 F bie Manacer DesciiPUON A A 3 2 Voltaire Fabric Manager Components sss ee eee eee eee eee eee 3 2 Voltaire Fabric Manager Main Functional sss sese 3 3 Fabric Manacer Installation T 3 3 SYS em REGUE eN TTT 3 3 Installing the Sun Java Web Start AppIIcanoOn sees eee eee eee 3 4 Installing the Sun Java Web Start Application on a Linux Workstation 3 4 Getting Started with the Fabric Manager sss 3 5 Login tothe Fabric ET cesera i A E T ai 3 6 Terminating the Fabric Manager sese 3 7 Fabric Manager Operation vieneen A 3 8 The Fabric Manager Main W 1nd OW sss 3 8 Color Coding in the Fabric Managcr sss sese eee 3 8 Finding the Source of Alarms in the SvStenm sese 3 9 Topology MaPinipenuironinien irii A R ou 3 10 Man Mei isi ei a ed ear T E E toad diet aedetiaat 3 13 Right Chiek Context Sensitive TT 3 15 S016 ole taal E gcc pee ne nem rete m
76. 3 Run the pkey update command to update the Pkey Table 5 6 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Setting Partitioning To delete a pkey from a specific row Step 1 Use the pkey set command and set it to 0 zero as shown in the example below ISR9XXX config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkeyo Pkey7 F Q000000000000000 ffff 7554 0 0 0 0 0 0 ISR9XXX config sm pkey set pkeyl 1 0x0 Change will be effective only after pkey update command Step 2 Verify that the pkey 7554 in this example was removed from the row by using the CLI pkey low or high show command as shown below ISR9XXX config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkeyo Pkey7 Ss en ik eae ea eae ea ss Passe es esca 1 0000000000000000 ffff 0 U 0 0 0 0 0 Step 3 Run the pkey update command to update the Pkey Table Voltaire Switch User Manual 5 7 P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic In This Chapter This chapter describes information on logging and monitoring functions that provide customers with advanced fabric debugging and failure detection capabilities This includes information on the following topics WE Overview on page 6 3 The Event Notification Mechanism When is Fabric Diagnostic Performed Potential Fabric Problems Procedures and Tools used for Problem Identification WE Fabric Inspect Setting
77. 37 Table 3 16 Host Ports Information Parameters sese 3 37 Table 3 17 Switches List Information Parameters sss sese 3 38 Table 3 18 Switch SLB Information Parameters sese eee eee eee 3 39 Table 3 19 Port Information Parameter eaz aE 2 e Naa e iaren 3 4 Table 3 20 Node Configuration Parameters ccccccccccccccccecccceceeceeeeeeeeeeeeeceeeeeeeeeeeess 3 44 Voltaire Switch User Manual P N 399200004 2005 Voltaire Inc Chapter 1 Introduction Table S2 IVE Paramete iSu a onha suas ul westlonde 3 46 Table le Voltaire Imitate ICONS arine ncaa anieloanidaunseneunodeevaastandasastaceaucesauns tins denauactontetats 4 3 Table 4 2 Login Window Fields Description sss sese 4 4 Table 4 3 Matn Menu Option S osese E E 4 8 RS ST A 4 Device Kee E and uentenieaanieoeaad 4 10 Table 4 5 Device Information Paraineters ceranae T A a 4 17 able 4 6 Port Intonation TT L nn N A 4 19 Table 4 4 CPU Information Parameters eresse ieena ne a e aaa E esa ENERSEN ai 4 21 Table 4 8 Local SM Information Parameters sss sese 4 24 Table 4 9 Detailed Module Window Fields sss esse 4 25 Table T S EB Ino maosna a deters oon tt east neseasanaiue nnn shiocbenteesgaeace 4 28 Table 4 11 sMB Detailed Module Information Window Fields 4 32 Table 4 12 sCTRL Information Window Features sss 4 34 Table 4 13 Modules Temperatures Treshold Values sss sss 4 37 Table 4214 IP S t tine K ter Seene edness A 4 39 Table 5 12 Pkey Table im REPOSO ss ccnx
78. 4004030Tt ISRO 192 166 0155 ISR S096 Voltaire SADA Switch E Normal joooresnsoos008 jbhost 10 17225421 heal 21 Melanox Technologies 0000 CA B Normal 0001 ca04003t0006 jbhost a 1722536 Heati 4 Melan Technologies 0000 Critical oomcso4003f000e jibhost 2 17225346 Heat_2 4 Melanox Technologies 0000 c B Normal oomreananosto004 jbhost 15 107246148 Heati 3 Melanox Technologies 0000 C E Normal pootesnao03t000e atasi 2 172254554 Heat_2 3 Mellanox Technologies 0000 c E normal oo0csogoostoo02 jbhost 5 1722524 hai 12 Mellanox Technologies 0000 C BD normal _jooorego4nnst000a ooo ca HEL erma ooo ca c V B Normal loootesnson3r0018 ibhost 492488 04 Heat4 1 slana Technologies 000 CA Polosy B normal loooteanso03t0018 jbhost 16_ 1525413416 Heal 3 4 Mellanox Technologies 000 CA 0001 c9040037001 6 ibhost 6 E Normal 17225121 Hcal_4 4 Mellanox Technologies ggg TA Map View P Normal 0001 c3040037001 4 ibhost 3 172 25 1 16 Heal 3 3 Mellanox Technologies 0000 CA E Warning O001cS04003T001c iphiost F 72 25 25 Heal 4 3 Mellanox Technologies DUU cA E Mormal 0001 c3904003T0012 ibhost 8 172 257 1939 Heal 3 2 Mellanox Technologies ggg CA P Normal 0001 cS040037001 4 ibhost 4 172 25210 Hcal_4 2 Melanox Technologies 0000 EA E Morral 0001 cS0400370010 ibhost 13 172 332 13 1 Heal 3 1 Mellanox Technologies DUU cA Ea Topology Map table DateTime DUCE Description Mon Ju
79. 64bytes from 172 25 0 10 icmp_seg 1 ttl 255 time 0 4 ms 64ytes from 172 25 0 10 icmp_segq 2 tt1l 255 time 0 4 ms SW 6IB4C gt sneer Ole Oa PLN Asha s Mies 3packets transmitted 3 packets received 0 packet loss GOUMG E eimai any Wau R sla 2 sins Command replication status show Description Returns the replication status of the sMB card to which the CLI session is connected which can be one of the following enabled or disabled When enabled the standby sMB maintains a mirror image of the active sMB and 1s ready to replace the active sMB in the event of failure In a redundant sMB configuration the replication status must be enabled Syntax replication status show Command route default gw show Description Returns the default gateway configuration Syntax route default gw show Example SW 6IB4C gt route default gw show T E T halen sae uel Command sm info show Description Returns the current parameter settings of the Subnet Manager in the ISRXXXX Syntax sm info show Example Subnet manager info is smName LAVISM port guid 0008f1040040061b topology CLOS stage 3 Voltaire Switch User Manual A 7 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference ace ne EOLO Loa lOS ege algorithm balanced routing active algorithm balanced routing sm KEY OQOO00Q0000000000000 sm priority 2 sm sweep interval seconds 3600 sm verbosity mode info sm topology verbosity none sm mads
80. 7 10 01 GMT 02 00 2004 12 25 3210 PORT COUNTERS ERROR at 000811 0400400467 LID 13 Port GUID O008f 04 0OS0OdbT porte Symbool E Mon Sep 06 17 10 01 GMT 02 00 2004 172 25 3 210 SM INFO Get Subnet Port Counters Mon Sep 06 17 07 01 GMT 02 00 2004 17225 3 210 SM IMF Get Subnet Port Counters Completed Mon Sep 06 17 07 01 GMT 0200 2004 172 25 3 210 PORT COUNTERS ERROR at 000811 0400400467 LID 13 Port GUID O008M 04004004bT porte Symbol E Mon Sep 06 17 07 01 GMT 0200 2004 12 25 3210 SM INFO Get Subnet Port Counters Figure 3 10 Trap Table NOTE The Trap Table only displays the events that occur when the Fabric Manager session is current Events that occur when the Fabric Manager session is not active are not displayed in the Trap Table To display a complete list of events use the Events Log feature described on page 6 13 For each event alarm the Trap Table displays the following information Event Description Information Type Displays the severity of the event alarm which can be one of the following Normal Warning Minor and Critical Date Time Displays the date and time of the event alarm occurrence Note events originating in the fabric from a node or host are time stamped with the date time of the node or host clock which could differ from the PC time Events originating in the PC are time stamped with the PC clock time Displays the IP address of the source of the event alarm Description Displays a short description
81. 9096 9288 Field Replaceable Units FR US 1 7 ISR 9096 9288 Switching Topology sss sese eee eee 1 8 TZ KST 1 10 Voltaire Switch Management cccccccccsccccccccscccccecececccccccccceceseceeees 1 11 Redundancy Mechanism 1 13 Hot Swap Mechanis Miesen onera en iE E R E E 1 13 Management InterfaCES See ee ee ee 1 14 ISR 9096 9288 Management Interface Configuration ccceeeesesneeees 1 15 Voltaire Command Line Interface CLI cccccsssccssccees 2 1 Getting Started with eC LM T 2 2 Interfaces Tor CLI Connection cessaire riea iene NEE AEE Ea 2 2 CLI Command MOGES 1ssssssonsaancronssasacssaseadsarconsssoniaehesansaasavanacadaateomeinzeateaposad 2 2 Fiord TT 2 2 Keyboard Usage in CLI Operation s 2 3 B L Rey ICUS spesso rn EEEE EE EE E ESS 2 5 CLIC Ise SeS VIY oe AA E A E AE 2 5 CLI Command B N vei 2 5 First Time COMM OUT AU OM yo4 s55sc5 csi crssnaactinscticentiohesenchesicaseeaatadactinattmectebestacoesiees 2 6 Collecting Required Information sss sese eee eee eee eee 2 6 COMME CIN S GC OMS OLE xi case scssanszsssacaseaasazenzatapsedeseianmayauoesuavavesacecasesacsesesedeesiess 2 7 AMT es R BR a a E ET EET E E E 2 7 Configuring the Management Interface IP Address sss 2 8 Configuring the Management IP Address in ISR 9096 9288 Switches 2 9 Contents 2 2 6 Dees 22 8 229 23 2 3 1 E S 2 3 4 E Chapter 3 3 1 3 1 1 Sl 3 2 3 2 1 E 3 2 3 3 3 3 3 1 K 3 4 3 4 1 3 4 2 3 4 3
82. B Consistency This routing scheme is applicable for Clos topologies It provides a consistent matching between a LID within an end node LID range and a sFB of the Clos fabric Once activated this routing algorithm will reconfigure the fabric in the following manner Given an end node base lid and LMC value that end node is assigned with lid range baselid base lid 2 1 Define lid 1 base lid 1 SFB sFB in location 1 in the sFB table The algorithm programs the fabric in a way such that the path to lid 1 passes through sFB 1 this is the sFB consistency rule PRE If the algorithm cannot complete due to configuration errors e g user error in SFB GUIDs then the fabric manager programs the fabric using the default routing scheme Indications for such configuration errors and fallback to the default routing scheme are currently available via the SM log file only Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuring Routing Algorithms To configure routing via the Fabric Manager Step 1 Select Fabric Routing Configuration from the main menu The following dialog box is displayed Routing Configuration Routing Into Spines list Index Guid Mame Description Members 1 0001 cso4o03f0004 ibhost 15 2s NOOB H 04004030 ISRO ISR 9096 Add Remove All Activate routing eve Figure 3 31 Routing Voltaire Switch User Manual 3 55 P N 399Z00038 2005 Vo
83. C S ES a ee L L SRV BAD P KEY TRAP 257 Object node P Key 0x X from LID XYZ Parameters p Kev uint16 from GID 016Lx 016Lx QP XYZ lid uint16 from gid prefix id minor enable manual to LID XYZ GID from qp uint32 to lid uint16 to 016Lx 016Lx QP XYZ gt on gid prefix id to qp uint32 sl SL XYZ uint16 SRV BAD Q KEY TRAP 258 Object node Q Key 0x X from lt LID XYZ Parameters q_ key uint32 from GID 016Lx 016Lx QP lid uint16 from gid prefix 1d 258 minor enable manual XYZ gt to LID XYZ GID from qp uint32 to lid uint16 to 016Lx 016Lx QP XYZ gt on gid prefix id to qp uint32 sl SL XYZ uint16 od i rt al a ea w DataValid 0x 04X P Key Obj ae oon 0x X from lt LID XYZ GID Parameters data valid 0 key 259 minor enable manual NOT r oOo eo A o uain Gondil kom doni ee ee a id Somn in o lid to gid prefix i QP XYZ gt on SL XYZ at switch G E gt E LID XYZ port X YZ id to qp sl SRV_SMINFOEVENT o T Do S OSM EVENT OUT OF RESOU RCE STATUS N oot dabe Waa Rejected multicast s for port Object node amp GUID 016Lx Parameters multicast string SRV CAPABILITY MASK MO E At LID XYZ new CapMask Object node normal manual oe Ox 08X Parameters new capability mask value uint32 SRV SYSTEM GUID_MODIFI ED TRAP 145 ee a At LID XYZ new SysGUID 0 Object node 016Lx rr Port GID 016Lx 016Lx left ae normal disable Join below MGID 016Lx 016Lx
84. CAL T fast ping remote logger delete remote logger set remote lLogger show POUT S sm snmp S SROM Aae E system name show watchdog mode set watchdog mode show Display Tist Of aval lable conmands Display command s usage string Change to Cluster configuration mode Exit rto Login menui Cron any G C mode via console connection or close telnet session via telnet connection Exit to previous menu Swikeeh Goaek tO caer y Cera reboot system Change CO LED COnilgurarion mode Change to 1B anterface conitgucarivon mode Change ToO TNE Vikerrace Comm quaat lon mode Change to LOCAL interface configuration mode Change To CaSe E eEr E aee T ONE LJU ae On mode Send echo messages Delete the remote logger of this COMOT cies Set the remote logger of this computer Show the remote logger of this COMPUTE E Change to route configuration mode Change to sm configuration mode Chance tro Samp Conr HRD R em mode Set the system name of this computer Show the system name of this computer Enable or disable Watchdog Show Watchdog Mode and watchdog timeout show Show Watchdog Timeout Voltaire Switch User Manual A 33 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Config syntax usage is as follows CTS E ET Edhie wens Factory deraue fo ccor Ce DEDE ERD EER ine rra eO 1B Iie eS a INS ERNEST aes ETIE ee meee INI interface LOCAL imvertace LOCAL INnterrace Last interface fast
85. Discards XmtConstraint RcevConstraint LocalLinklinteg ExcessBufOvrrun VL15Dropped XmitBytes 1 0 0 0 0 0 0x07964132 RcevBytes Xmit Pkts Rev Pkts PeerLID PeerlBPort PeerPortGUID PeerPlatformType 0x1884ac09 0x001 afaf4 0x005795e8 16 9 0008f1040040061b ISR 9288 PeerModuleType PeerModulelndex PeerPort PeerName Status sFB 12 4 internal 57 ISR9288 0061b ALERT 6 16 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic This example shows the status of the sLB 24 module and its sFB 12 peer Module Using the Port LID and GUID of each board you can use the Node Info Window to identify the specific sLB and sFB board and see whether it is defective or not To save the fabric statistics file Step 1 From the Fabric Manager main menu select Fabric Fabric Inspect Settings Step 2 Specify the name of the statistic file and specify the location where you wish to store the file Step 3 Open the CSV format file with a suitable application e g MS Excel Refer to Appendix B 1 for a complete description of ports counters fields a description of the CSV Summary Report and the port counters currently available in the fabric Voltaire Switch User Manual 6 17 P N 399Z00038 2005 Voltaire Inc Additional Tools and Examples 6 4 Additional Tools and Examples Verification of the fabric is an integral part of running a successful grid Below are some troubleshooting examples for common problems that can
86. E RRR E Voltaire sLB 24 smalid 9 VIS OOH Cherie IOS OOo TOTA iS ERER EE HE HIE EE DR RE nT OTAN ERRER O40 0403002 S O008 1040040300e 3 5 6 Z Z TT I IL 7 as 15 es 22 22 22 Switchguids 0x8f104003f 0764 Switch Ze eS OU OS TO ANO BOT 60 Voltaire sLB 24 smalid 8 UES ae 0 Oop CO Sak OG Aas ike oe VOSA OO sb Og oa aay Po OO moO 40s OZ E mo UO00sStlO4A0040300e 19 LS OOM S E OAT TOE SE Switchguids 0x8f1l04003f074F Switch P oO Oe e OR OAE T Voltaire sLB 24 smalid 7 WS ONO tose MO ANCHO hema I E SO OO ene One OF Anke eZ cn HS OKO neem IOAN O NO 20 SMOG a EL Po OO LOA OO 403 Oe i E mS OOO Sar Od OOS BOs o WS ITE TT ore 2741 Ss 000S T ING A OMCAR ONS OIC CE IL wh Switchguids 0x8f104003f074e Switch 24 PS OUST E R REE eT Voltaire sLB 24 smalid 6 US a OC Ooh ONO Sas R TAT Won VOSEROA00S E0746 Sy Oe ae MONO OA ERE ii wo O008f1040040300e E TORR SHE AOO E oor LE DOR T KOFO ATS 00o 22 baa 7 KS LE 7 switchguids 0x8f10400403002 Switch Zea OOS ELOA0O 403002 SPB AToM Lra O 2 L TO U ick OCOC TROGO KOTONO CE HER LOTOO T Ote T E Oe O O DE T a ESEO ae O OOT e a T Ih 0 0 U 2 1 il 10 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode HEE HERE E Voltaire TSR9288 ISR9096 TSR9I288 ISR9096 TSR9288 ISR9096 TSR9288 ISR9096 ITSR9096 Voltaire A 27
87. EE crr rors Sek UE RE EM SH CE ele E eaS E ESO pm report enable set enable disable ju SSS SCO UIC ies SEC Pi scope seu G ell e Vedula areca akoe pm show pm symbol error counter set IR Doe pm trap mask set Thone derante fall pm vll15 dropped set PER SES 9 The following details the commands available in the Subnet Manager PM sub menu Port counter CSV file alert flag that takes into consideration the enable disable Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Configuration Config Mode pm alert join set Port counter CSV file alert flag that takes into consideration the JOIN issue HCA joined pm alert join set enable disable pm counter operation set Future option In the present software release only the delta option is supported the PM monitors the difference per interval for each counter and generates an error when it passes the predefined threshold pm counter operation set delta avg avg_sum min max Command Description Syntax pm enable set Controls PM operation The default is enabled If disabled PM data viewing will not be available in the Voltaire Fabric Manager pm enable set enable disable Command Description Syntax pm excessive buffer overrun errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring
88. FB board and see whether it is defective or not Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 6 21 6 4 3 Step 1 Step 2 Step 3 Step 4 6 22 Additional Tools and Examples Identifying a Bad Port A switch port may be considered to be a bad port if one of the following conditions occurs M It does not respond to NodeInfo WE In the event of duplicate GUIDs The current status values associated with bad ports are listed below 1 BROKEN LINK the physical link on one side is up but the other side is unreachable INVALID LINK different connectivity on both sides of the link 3 DUPLICATE HCA GUID another HCA or switch with the same GUID as this host 4 DUPLICATE SWITCH GUID another switch with same GUID exists Whenever a bad port is detected the SM sets the port state of the remote port to Down and marks it as Down Polling in the internal fabric database to indicate that this port is not connected This will ensure that bad port shall not be discovered anymore unless the user fixes the problem and clears the bad port table To display the bad port table M Select Device Bad Ports Log from the Voltaire Device Manager main menu A browser window is displayed showing the bad ports To clear the bad port table This procedure is performed via the CLI Command Description enable password Enter Privileged mode from Exec mode Conti Gg Enter Config mode from the Exec mode
89. Host is called hydra3 voltaire com Voltaire Switch User Manual A 65 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference E Ganglia Host Report Microsoft Internet Explorer SEE File Edit View Favorites Tools Help ay x a Le po Search 5 2 Favorites 4 2 lt ee ss Lag 3 rsl 3 172 25 2 175 ganglia m load_one r hour amp s descending amp c Support Cluster amp h hydra3 voltaire com amp sh 1 amp hc 4 i Go Links 7 i B Search Web Q Search Site 51764 blocked fa Options v 2 Ganglia Host Report for Mon 25 Jul 2005 12 28 39 0300 Last hour v Node View SOuUTECforge net Grid gt Support Cluster gt hydra3 voltaire com hydra3 voltaire com Overview a hydra3 voltaire com Load last hour This host is up and running 2 ENE E CAA jun oom Time and String Metrics Lh S E Sere boottime Mon 11 Jul 2005 08 33 03 0300 11 40 12 00 tara D 1 min Load W crus W Running Processes gexec ON gmond_started Mon 11 Jul 2005 08 33 52 0300 last_reported 0 days 0 00 11 machine_type x86_64 os_name Linux ease 2 6 9 5 ELsmp 11 40 12 00 12 20 uptime 14 days 3 55 27 B Memory Used B Memory Shared B Memory Cached D Memory Buffered B Memory Swapped W Total In Core Memory Done Internet Command snmp cluster monitor show Description Shows the SNMP cluster monitor Syntax snmp cluster monitor show Command snmp community get Description Sets the SNMP get community string T
90. ID Step 2 Position the cursor in the text box and type in the node name The search string is case sensitive and there is no support of wild cards the node name must be typed in exactly as displayed in the topology map Step 3 Click the Search button When the search is done the node icon 1s automatically selected in the map and the Node Info and Node Properties areas focus to display the information of selected node 3 4 5 Main Menu The main menu provides access to Voltaire Fabric Manager functions listed in Table 3 4 In general when a menu heading is not available for selection it appears grayed out and in slightly raised type Menu Submenu Action Exit Logs out of Fabric Manager and terminates all Fabric Manager processes View Status Line Shows hides the Status Bar View Tree Shows hides the Subnet Tree View Events Shows hides the online Events Log Voltaire Switch User Manual 3 13 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Fabric Node Submenu Action SM Properties Opens the SM Properties window which is used to view and configure Subnet Manager parameter settings This window is described in Section 3 9 on page 3 44 Fabric Inspect This feature is described both in the VFM and the Diagnostic chapters on pages 3 1 and 6 1 respectively Settings opens the Fabric Inspect dialog box which allows you to select which events are reported in the Event
91. MB board is displayed with a border line around it as shown in the sMBs at the left of Figure 4 10 and Figure 4 9 ISR 9096 Switch Figure 4 7 and Figure 4 8 show the graphical representations of the ISR 9096 switch sFU 8 EEL Giang 4 7 ISR 9096 Switch VDM Graphical fepresentation Front View Click to display the rear view of the ISR 9096 switch shown in Figure 4 10 18 19 20 23 24 Laes Ted P Zn a Tit 1 ked i l i S i H1 J S 4 a E l k i HL 4 _ i Si i TEL imi i p a 19 EE 29 G a fes ITE 1 Fj 4 d B 8 9 ra EST 1 i ga ico i i mi i r 4 44 18 49 FERET lt i n ean a iln n n n a F J 4 E 7 a J w m o a ee ee ee e J BERE 8888 Ee b Di Hin o o ET TES OO HEE FHERR n SRE H S Figure 4 4 8 ISR 9096 Switch VDM Graphical Rea Rear View Voltaire Switch User Manual 4 11 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM ISR 9288 Switch Figure 4 9 and Figure 4 10 show the graphical representations of the ISR 9288 switch ET Figure 4 9 ISR 9288 Switch Graphical Representation Front View ear Click 4 10 to display the rear view of the ISR 9288 switch shown in Figure 4 12 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc The Device Manager Main Window Saas o iik Figure 4 10 ISR 9288 Switch Grap
92. Manual P N 399Z00038 2005 Voltaire Inc Managing Alarms and Events Viewing Alarm Information You can filter and view Switch Host HCA or RLX alarm information by selecting the preferred option from the Platform drop down Menu Table 3 8 describes the parameters displayed in the Current Alarms window Severity Displays the alarm severity which can be one of the following Critical Minor or Warning Alarm Displays a textual description of the event alarm type Source Displays the object where the event originated Date Time The date and time at which the alarm occurred Count How many times an event mapped to this alarm was received since the last clear Displays the information of the remote port of the link or node with which the event is associated Table 3 8 Current Alarm Window Fields To find out more specific details pertaining to an event right click the event listing in the Event Log and select Alarm Data The following of window is displayed Alarm Data X Parar UID ame atform odule ort escription Link Down Event Figure 3 9 Alarm Data Voltaire Switch User Manual 3 23 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM This window displays the following additional data Parameter Description GUID Displays the Global Unique Identification number GUID of the alarmed device port Displays the name of the alarmed object Platform The switc
93. PM pm vli5 dropped set Set the vll5 dropped for the PM Voltaire Switch User Manual A 55 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 56 PM syntax usage is as follows Pi here aabs sce pm alert Join Ser pm counter operation set pm enable set pm excessive buffer overrun SIO ORS S E pm link downed counter set pm link error recovery ERGE E SS D LE eT EE T E error E TEE T ETT H ISTR E E RH T R Ser OME YOU ON ANS IA IONS ISIC PM Ore rOy r MOE p steals Err roOro rOet bil Petr i V onee re a error rone E OOT TEE COS E Ee T JON E dialer E D pm report enable set ul 12 SS Se SCOUMCS IS WSSic IRE TT E E HE pm show pm symbol error counter set pm trap mask set pm vll5 dropped set Command pm alert ats set Description ATS registration Syntax pm alert ats set pm alert ats set enable disable pm alert Join set enable disable pm counter operation set den wave avg avg sunm Pmi Ma pm enable set disable enable pm excessive buffer overrun eeror o See O pm link downed counter set POP ZS pm link error recovery Cormeen Sec Oy 2a eligh elisoxe cull habiel eh igiieiele paak iene Error wseie oe ies ETO T TT T iL sxe loll OO SEK PN H neve COs eh mM be error S Sei 0 255 Dh OCI ON e LOr SSG ONS StS a Pl DpOrhE Lrecy remore physrveal CPES See MON eo asso OROL RE CN D errors oee Coo o PNPOT AME OS
94. R A e P S Q25 052 A Aa management interface 9096 9288 set Switches Step 5 2 8 EXIT Description Enters Privileged mode from Exec mode the default password is voltaire Enters Config mode from the Exec mode Type the password when prompted Enters Fast interface configuration Sets the IP address of the Fast interface Sets the IP address of the IB interface In ISR 9096 9288 switches the active management interface can be set to the sMB board Fast or IB interface This command can be accessed via the Config Interface IB menu See next section for further information Sets the active management interface to the Fast or IB interface See next section for further information Note do not configure the IP address of the IB interface on the same subnet as the IP address of the Fast interface Exits Fast interface configuration mode Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc First Time Configuration 2 2 5 Configuring the Management IP Address in ISR 9096 9288 Switches The active management interface of the ISR 9096 9288 switch sMB module can be set to either the Fast interface or the IB interface not both interfaces simultaneously However CLI sessions can be established through both interfaces simultaneously The IP address of the IB interface should not be configured on the same subnet as the IP address of the Fast interface Configuring the Fast interface as the Manage
95. Sinn AO PS OCHS Se ION Sie Oy abe Domi 24 C gt 1x port was found Port GUID 8f104003f074e IB Port 24 Peer D at OQOO8f104003f074e Pore 22 POr CUID OTL OAO TO Me 1B R A ann Voltaire Switch User Manual 6 13 P N 399Z00038 2005 Voltaire Inc Diagnostic Tools This example shows that the diagnostic tool detected 1X link problems and the ports or the peer node that generated this error Refer to page A 19 for further information on the CLI event log command T The Event Log only saves information since the last device reboot 6 3 3 Fabric Diagnostic Using Current Alarm When the event corresponds to an alarm it is sent to the current alarm mechanism Current Alarms Mechanism When an alarm is detected related traps are issued A database of alarms is created per object The user can view the alarms via CLI amp GUI The current alarm Database includes Unique objects system subsystem port Alarm description Severities Manually corrective actions such as clear alarms The GUI will give a graphic representation of the fabric with color coding and different filtering capabilities Refer to Appendix B 3 for a list of errors Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic RECOMMENDATIONS ON HOW TO WORK WITH CURRENT ALARMS 1 Viewing Alarm Information Alarms can be viewed either through the event log the portcounters csv file or the Alarm Tabl
96. Sun Java Web Start application which must be installed in order to use the Voltaire Vision GUI Table 3 2 Voltaire Initiate Icons Voltaire Switch User Manual 3 5 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Step 3 3 3 1 Step 1 Step 2 3 6 NOT OSS When opening the Voltaire Fabric Manager Initiate Window for the first time click to start the Sun Java Web Start installation The computer that is being used must be connected to the Internet for successful Java Web Start package download from Sun s website To start the Voltaire Fabric Manager application click from the Voltaire Fabric Manager Initiate Window Login to the Fabric Manager After the Voltaire Fabric Manager application 1s started the Login window is displayed Voltaire Manager User Login IP Address 72 25 3 210 User Type admin Password Figure 3 2 Login Window The Fast IP address of the Subnet Manager to which the Fabric Manager is connected is displayed in the read only IP Address field Select the User Type The Voltaire Fabric Manager supports three User Types each with a different authorization level R Administrator user type This user can perform all configuration and provisioning operations Default password 123456 R Manager user type Enables you to set the target parameters Default password voltaire R Guest user type This user can only view device configuration
97. Switch User Manual P N 399Z00038 2005 Voltaire Inc Fabric Manager Operation Topology Map Buttons Pan Interactor Allows you to drag and drop the icons in the topology map Select Allows you to select an object in the Topology Map Zoom in area Enlarges the objects when centering selected area Zoom in Enlarges the size of objects displayed in the Topology Map Zoom out Decreases the size of objects displayed in the Topology Map Fit to Contents Resizes the Topology Map to fit entirely on the screen allowing you to see all objects contained in the map Hierarchical Layout Rearranges the Topology Map to the default arrangement B o Nodes Information Displays the Node Information window Voltaire Switch User Manual 3 19 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 4 9 Topology Table View To display the topology view in textual format M Click the Table tab at the bottom of the Topology map The following window is displayed Voltaire Manager File View Fabric Mode For Help we m m Suret TE Te gt a RTK fon Ce 2 1 al car jew RR s TL Sead By uo Hod C C warning C Minor C critical 1 E Show All m Mortal GUID Mame IP Address Description vendar ID Device ID Type La Minor gogi O
98. TE MC G ROUP Object SM OSM EVENT DELETE MC GR OUP Object SM OSM EVENT COMPLETE SUB NET RECONF Object SM OSM EVENT APPLIED ROUTI NG Object SM Parameters routing scheme char OSM EVENT PORT STATE C HANGE Object node warning enable warning manual N a Ss Rs N minor manual Ms pf a Pf a oo pf aa T I Ol R Pf warning enable Table C 5 Alarm Table Detailed Specification Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Trap Alarm Table Log format Unable to configure all LIDs on Switch GUID 016Lx LinearFDBCap XYZ is too small for current subnet LMC XYZ Up down routing failure No up ports found Possibly invalid sFB configuration switching to default method Found CA port GUID 016Lx at unexpected distance XYZ hops from sFB Number of sFBs found XYZ does not match predefined number of sFBs XYZ Minimal LMC value of XYZ is required for persistent CLOS routing current LMC is XYZ Unable to apply s routing scheme switching to default method Found a Switch GUID 016Lx outside the CLOS Found an HCA Router GUID 016Lx outside the CLOS Create Multicast group MGID 016Lx 016Lx Delete Multicast group MGID 016Lx 016Lx Complete subnet reconfiguration Applied s routing scheme Port State Change Reporting Switch LID XYZ C 7 Appendix D Glossary API ARP CBB CD CLI CPU CQ DMA FC FE FTP FRU
99. Type the password when prompted sm Enter Subnet Manager configuration mode bad ports clear set Clear bad ports table Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic PROT The Bad Ports feature is disabled by default this means that by default the bad x ports will not be reported in the CSV file To enable the Bad Ports feature use the bad ports mode set CLI command in the Subnet Manager submenu under the Config menu Voltaire Switch User Manual 6 23 P N 399Z00038 2005 Voltaire Inc 6 4 4 Some CLI Diagnostic Tools Zero counters script Additional Tools and Examples To clear out all the errors across the fabric use the zero counters script to traverse the fabric and clear out all the port counters on both the switches and HCAs This script is very easy to use and is helpful if you want to start off with a clean baseline of your fabric after many changes have occurred TSRUZSStUuUrTLiLTTes zero counbers Zero All Counters EYED rE 24 KRKEKKKKKKKKKKKKKKKKKKKKKKK TE 72 4 KKEKKKKKKKKKKKKKKKKKKKKKK taku We OTE S a 2A KKEKKKKKKKKKKKKKKKKKKKKKKK IEE Cis MP Oink Sa 24 KKEKKKKKKKKKKKKKKKKKKKKKK Waele 7 lowe sis a KKKKKKKKKKKKKKKKKKKKKKKK Eeo por ms 24 KKEKKKKKKKKKKKKKKKKKKKKKK width check script Another valuable script is the width check script which allows you to easily check the fabric for 1X connect
100. Wsace Ssrering hei cen Ine ERI HE EE other mode via console connection or close telnet session via telnet connection E Exit EO prey ous Menu alert ats set set alert ats for the PM alert join set Set alert join for the PM counter operation set Set the counter operation for the PM enable set Set the Enable for the PM excessive buffer overrun errors set Sep the exceos hve Olimar GNET errors Tor CAS PM LMC TOn e counter cet Seem en ain CLOUMNNe ClasG imines mine ms cla EN link error recovery counter set Set the link error recovery Sorn eee SECIS ERC SREM local link integrity Trors s t Set tenes R R mink E D E E Ae CERNE poltimg imbcerval seu Set Ene polling iuieerveal for the Ee ISE Const rari er rots sep Sen he Port Oy aN oe a mre errors for the PM SOME AG ON E EN Soet tne port Iov err On Si for ENS ER POEL eey e mOr Phyo ea l ecrrOrs ser Set the port rcv remote Paysical errors for the PM port rcv Switch relay crror ser Set tho POTE a V onito N anela errors for the PM Pore xme ON Strae ereo Se Ser Sek Ehe DOr ame e Oon E Ln errors Om tho PMI PM G xme ASCA rds Ser Sets Ele Tp port xmit eca rde Om the PM pm report enable set Set report enable for the PM PETE ES ET o e E Beceri MC OUNTE ES pm scope set Set the scope for the PM pm show Show pm s settings pm symbol error counter set Set the symbol error counter for the PM Si trap misk set set the trap mask for the
101. ach switch port has an SL to VL mapping table that is set by the Subnet Manager to implement the appropriate packet priority with the actual number of VLs Voltaire Switch User Manual 3 49 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Step 1 Step 2 3 50 An InfiniBand link can have one of the following VL configurations Number of Data VL Numbering VLs Supported There are two types of VLs one for normal traffic called a data VL and one reserved for subnet management traffic The subnet management traffic VL is VL15 All other VLs are for normal traffic To set the SL to VL mapping Select the switch to configure in the topology map of the Fabric Manager main window Select Fabric Quality of Service from the main menu The following dialog box is displayed Quality Of Service Virtual Lane U 1 3 Enable Disable Activate Default Figure 3 29 QoS SL to VL Mapping Table 3 80 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Step 3 Step 4 Step 5 Step 6 Configuring Quality of Service QoS This dialog box is used to set the SL to VL mapping Click Disable and the Subnet Manager will not configure the QoS parameters default firmware settings will apply To change the SL to VL mapping position the cursor in the Virtual lane column and click the mouse button Select the new value and click Apply to save the setting Rep
102. aire Subnet Manager supports the following routing algorithms multipath with balanced routing and multi path with sFB consistency sm info algorithm set balanced routing SFB consistency sm info hoq set Sets the time a packet can live at the head of a VL queue hoq is defined as 4 096us 2 hog if 0 lt hog lt 19 5 55 hoq is the HOQLife component of the PortInfo attribute If hog gt 19 then hoq is to be interpreted as infinite Sm anio Neg sec T Command Description Syntax Example Remark Command Description Syntax Command Description Syntax Voltaire Switch User Manual sm info Imc set Sets the number of LIDs per node in the fabric a power of 2 from 0 to 7 Ssm 1nto me see Pant Osa 7 J smM inr o dine Sec 3 8 LIDS set for every node in the fabric sm info mode set Enables or disables the Subnet Manager in the switch When the Subnet Manager is disabled the switch is managed by a Subnet Manager on a remote switch or host on the network sm info mode set enable disable sm info mopvl set Sets the maximum number of operational Virtual Lanes operational on this port The actual number of VL is the minimum between the switch capability VLcap and this parameter set by the user sm info mopvl set PO OSs Oe OT a OA A 51 P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Comma
103. aire Switch User Manual P N 399Z00038 2005 Voltaire Inc Exec Mode NOTE x The IP address of the router module must be on the same subnet as the sMB Zz card with which the CLI session is established Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Example enable Change to Privileged mode enable end Exit to login menu from any other mode via console connection or close Telnet session via Telnet connection end exit Exit to previous menu exit fast interface show Returns the IP address of the Fast interface its subnet mask the IP subnet to which the Fast interface broadcasts and the active management interface ETH denotes that the Fast interface is the active interface IB denotes that the IB interface is the active interface fast interface show fast interface show Voltaire Switch User Manual mesic ae 2525 25 206 MO mists ero A ey E IE IS alien ale I a Zoe A SD management interface is eth0 1 A 5 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Command ftp show Description Shows the name of the FTP server and the user name used by the switch to log in to the server The password is not shown for security reasons Syntax ftp show Example SW 6IB4C gt ftp show PUP CONE L GUA above remote server ftp voltaire com User voltaire Command
104. ally a true indication of the number of times the port has gone down usually for valid reasons This counter should not be increasing and a constantly increasing number probably indicates a bad link This indicates that a problem is occurring ELSEWHERE in the fabric and that this port received a packet that was intentionally corrupted by another switch in the fabric This counter can increase due to valid event occurring in the network Voltaire Switch User Manual Available Fabric Port Counters PortXmitDiscards PortXmitConstraintErrors PortRcvConstraintErrors LocalLinkIntegrity Errors Voltaire Switch User Manual DLID mapping VL mapping Looping output port input port Total number of outbound packets discarded by the port Typically will not increase If it is may be an indicator that HOQ or other parameter should be tweaked Please contact Voltaire Customer Support because the port is down or congested Reasons for this include Output port is in the inactive State Packet length exceeded neighbor MTU Switch lifetime limit exceeded Switch HOQ limit exceeded Typically will not increase If it is may be an indicator that a parameter should be tweaked Please contact Voltaire Customer Support Total number of packets not transmitted from the port for the following reasons FilterRawOutbound is true and packet is raw PartitionEnforcementOutbound is true and packet
105. also supports multiple network management workstations organized in a hierarchical fashion Fabric Board with 12 port support Line Board with 24 ports Fan Unit Service Level Subnet Manager Simple Network Management Protocol Transmission Control Protocol Unreliable Datagram Protocol Voltaire Device Manager Voltaire Fabric Manager P N 399Z00038 2005 Voltaire Inc Voltaire Switch User Manual VL Virtual Lane WQE Work Queue Elements Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter D GlossaryTables D 3 Index Appendix E command A 4 4 4X 12X Auto Sensing 1 10 A Activity Count SM 3 45 Administrator user password 4 42 Administrator user type 3 6 4 4 Alarms Clearing 3 24 Finding the Source of Alarms 3 9 Arbitration setting QoS 3 51 Automatic Path Migration 1 3 B Backing up switch configuration 2 17 Bad Port identifying 6 22 bad ports mode set 6 23 A 48 Baud 2 7 broadcast fast set A 41 A 44 C Case Sensitivity CLI 2 5 Voltaire Switch User Manual Channel Adapter connection 6 20 chip sLB sFB 6 20 Clearing Events 3 29 CLI case sensitivity A 2 Command Reference A 1 Exec mode A 3 FTP configuration A 38 A 39 IB Interface menu A 41 Maintenance Commands 2 14 Modes A 2 Privileged Mode A 9 QoS menu A 61 Route menu A 46 SNMP commands A 65 Subnet Manager menu A 47 Usage Tips A 2 CLI Command Modes 2 2 CLI Comma
106. an be obtained by clicking the Advanced Button refer to Obtaining Detailed Module Information on page 4 24 for further detail ISR 9096 9288 Fan Speed Algorithm After a power cycle the fan speed is initialized to turbo In addition sMB reboot also sets the fan speed to turbo After reboot the sMB scans the temperature sensors of the chassis modules including all sFB modules all sLB modules sMB sFU 8 and sFU 4 modules If all the temperature sensors are below threshold the fan rate is set to Normal If one of the temperature sensors 1s above the thresholds the Fan rate is set to turbo If one of the Fan units is missing the other unit is set to Turbo If one of the Fan trays or one Fan in the unit is failing all the other Fans are set to Turbo Each module has three temperature thresholds E RELAX threshold E WARNING threshold R ALARM threshold 4 36 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Module Information After a successful scan of the sensors of all chassis modules the fan speed is set to normal under the following conditions 1 All Sensors are below the WARNING threshold 2 Both sFU 4 and sFU 8 units are present in the chassis for the ISR 9096 only the sFU 8 should be present 3 All fans within the fan unit are OK no faulty fans The system periodically scans the chassis If any of the above conditions fails fan speed is set to turbo A LED also indicates if the tempe
107. anager Description The Voltaire Device Manager VDM is an embedded application with a web based GUI that provides InfiniBand management functionality for Voltaire ISR 9096 and ISR 9288 and ISR 9024 switches It enables users to view a graphical representation of the modules its LEDs and ports and to monitor and configure device parameters NOTE Common ISR 9096 9288 VDM features are graphically illustrated using either Zz the ISR 9096 OR the ISR 9288 representation Voltaire Switch User Manual gg P N 399Z00004 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 2 Getting Started with the Device Manager To start the application Step 1 Set the Fast Ethernet IP address using the CLI through a serial console connection as described on page 2 6 Step 2 Open the Internet Browser and connect to the fast Ethernet IP address for example http 192 168 1 2 The Voltaire Device Manager Initiate window is displayed Voltaire Paving the way for the InfiniBand era Microsoft Internet Explorer m efx File Edit view Favorites Tools Help ae Q Back Q x a A E Search P Favorites Ea G 9 lt w lt Lu rel Address 4 http 172 25 3 13 v Go Links V voltaire Accessing Yoltaire Multi Protocol Switch Launch Voltaire Graphic Device Manager Z Launch Voltaire Graphic Fabric Manager Download Java tool for Voltaire Graphic Manager Support Contacts USC Voltaire support center on the
108. and status information Default password Voltaire Enter the Password and click OK The Voltaire Fabric Manager main window Figure 3 4 is displayed Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Getting Started with the Fabric Manager NOTE x The login password is case sensitive 3 3 2 Terminating the Fabric Manager The exit feature shuts down the Voltaire Fabric Manager application the switch continues to run with its last configuration even after the GUI is shut down To terminate the Voltaire Fabric Manager Step 1 Select File Exit from the main menu A warning message is displayed Warning You are about bo exit the program Would vou like to proceed Figure 3 3 Exit Warning Message Step 2 Click Yes to confirm Voltaire Switch User Manual 3 7 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 4 3 4 1 Main Menu Subnet Tree Topology Map Topology Map Tab 3 4 2 3 8 T Yaltaire Manager S Fabric Hosts Adapter nodes ce Switches Bi isroxxx sLB1 sLB 2 Sun Jun 19 18 54 00 IDT 2005 ibhost N ibhost ibhost am ze ibhost lt Topology Map Table Sun Jun 19 18 53 59 IDT 2005 Sun Jun 19 18 53 58 IDT 2005 ISR9XXX SFB 4 3 17 SRIXXX sFB 4 3 7 ISR9XXX f SFB 4 3 7 the main areas of interest in the Main Window Toolbar i ibhost ibpost g host Y La M
109. ard at the top of the window shows which ports are active The board s basic VPD Vital Product Data includes the module Serial Number and Hardware Version Additional VPD can be obtained by clicking the Advanced Button refer to Obtaining Detailed Module Information Section 4 5 6 on page 4 24 for further details The center of the window displays the following information on each of the Line Board ports Voltaire Switch User Manual 4 27 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM Port Displays the port number relative to the sLB State Displays the current state of the port Possible values are Down the switch is non operational Initialize the switch has been powered up and is in initializing Armed the port is in armed mode Active the switch is operational The color coding of these fields reflects the status of the port as follows Green indicates that the port is in active state Red indicates that the port is either in Initialized or Armed Status No color indicates that the port is down LID Displays the port Local ID LID Width Displays the current link width 1X 4X or 12X Speed Displays the current signaling rate of the link current default is 2 5 Gb s MTU Displays the Maximum Transmission Unit value of the port VLs Displays the Virtual Lanes supported by the port Current support is for VLs 0 to 7 Table 4 10 sLB Information
110. at an alarm exists view the topology map and look for the node where the alarm originated its color will reflect its alarm status You can also right click the alarm in the Event Log and select Go To Node from the context sensitive menu The map focuses on the alarmed node host small rectangles drawn around the node on the topology map and the focus of the node host information pane of the main window also changes to display the alarmed ports objects For more information on alarm handling refer to Managing Alarms and Events on page 3 22 Also refer to Chapter 5 for information on use of other advanced fabric diagnostic capabilities The Privileged Mode Section A 5 details how to manage Events and Alarms via the CLI Voltaire Switch User Manual 3 9 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 4 4 Topology Map The Topology Map provides a view of the fabric topology using color coded icons and connecting lines to represent the nodes hosts and switches It is fully synchronized with the Subnet Tree and the Node Information J 7 ibhost Pot i post gam E jp a be host jo TE jr E ba Lo ost ibhost ihhost E E Ee E ihhost pay 4 ibhost h E ba inhost ibhost ibhost ibhost Ed Topology Map Table Figure 3 5 Topology Map 3 10 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Fab
111. ation status set replication status show default gw show SM 1nFo show snmp community show update firmware update file Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode Gases update software update software update file lake ENUE DES ETT alxeys version show version show The following details the commands available in Privileged mode Command alarm clear Description Clears the current alarm stored in the alarm buffer Syntax alarm clear entry Command alarm clear all Description Clears all current alarms stored in the alarm buffer Syntax alarm clear all Command alarm show Description Displays a list of the current alarms in the switch alarm buffer Each alarm is listed with its alarm severity alarm type and the object where the alarm has occurred Syntax alarm show Example ISROXXX alarm show JL minor Port Countcer 0008710400400618 YISR 9268 SLS A 11 internal port 19 O 2005 H EE O ET SS E O E EE e ER URTE R E DR GE TRR RE Z e E Calo oul acy mpi E E nar Command clock set Description Sets the time and date registered at the switch in military time format The time and date will appear in relevant reports in GUI based management systems Syntax clock set MMDDhhmmYYYY Command clock show Description Displays the time and date currently set in the switch Syntax clock show Voltaire Switch User Manual A 11 P N 399Z00038 2005 Voltaire Inc
112. be found in the fabric 6 4 1 Detecting and Troubleshooting 1X Cable Width Problems The system tries to detect the maximum link width between any two end points to optimize performance If a 1X link is detected either one of the ports is faulty or the cable is damaged A trap is then issued to notify the user about the 1X link as shown in the example below Type Date Time Description Category Event ID AA Minor Mon Jul 04 17 56 37 IGT 2005 OO08T1 0400400496 sLB 24 4 SIrternal Port 10 1x port was found Port Counters Error Osm 1x QS Minor Mon Jul 04 17 55 20 IDT 2005 O008f104004004a8 sLB 24 4 J internal Port 10 1x port was found Port Counters Error Osm 1x Figure 6 5 Trap Table 1 X Link To troubleshoot 1X cable width problems Step 1 In the VFM right click the trap and select View Event The following Event Data Information is displayed showing which ports are connected on both ends with the 1X link Event Data Port Info Peer Info Mode UID DUUST O4004004a8 Mode GUID poggii O4004004e8 Platform SR 952755 Plattorrn ISR 9255 Module SBE Module sFB 12 4 Logical Port Internal 10 Logical Port Port SUID DUUGTT O400ST0635 Port GUID OUST O4004004b9 IB Port IB Port LID LID Close 6 18 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic Figure 6 6 VFM Event Data Window This trap can be viewed by accessing the Fabric Statistics portcoun
113. ber that identifies the port GID Prefix The GID is a concatenation of a 64 bit subnet prefix an identifier of the subnet and a manufacturer assigned globally unique ID Port LID Displays the Local ID of the port Port State Displays the state of the port which can be one of the following ACT port is active DWN down This field is color coded to indicate the state of the port Defines the LID Mask Control value of the port Possible values are 0 to 7 3 40 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Information Views Parameter Description Port MTU Displays the Maximum Transmission Unit value of the port Displays the VLs enabled on the port Indicates if a Subnet Manager is present in the node Displays the Management Key Value of the port Table 3 19 Port Information Parameters 3 7 6 Link Information To drill down to a specific link and view related information M Select the link connection in the main window topology map The following window is displayed in the main window ink i Connection no 1 amp p008104004006f8 lt L4 FT Port 12 S IT Port 4 Figure 3 24 Link Information This window displays information regarding the ports at both end of the link This includes the alarm status color coded according to the standard convention described on page 3 8 of the ports the switch that the port belongs to and if applicable the board which the port be
114. cord d ebug p gid_prefix TEIL een Omemns S i e HETA L lt desreaguic wi lt nearhs RR IIe EE I HT E R EEE S E ei Vterbose LS I R G tameouroms lt sic I akon N oxen A Madsta IT T ile e eoe Me ET a ee a e HERD inte conve ETIE Ore FoS mA C ounte CrS Eor Cero Monin E cuTthough vlmask exXt_portset Up Esa EES E CDE A e e ene aioa EE josie e A 19 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 20 mcmember MEES nerd scovar setpkey Smm LTTE O smpdump smpex EOLO LOGY OG show vendstat Kon OE RON width Check AJO CONUM eS ets orcii T Pl T EET o a for further detail refer to the madstat section on page A 24 mcmember d ebug t timeout_ms s Seroj PS ep vel Ilo jatonere wabiaiel T E ye ole Ow eee alway eine Meera ce I e Sowg US E E eee sic iie v erbose s imple t timeout_ms lt mlid gt lt src lid gt lt dest lid gt netdiscover d ebug v erbose S how e timeout_ms lt netfile gt seupkey V debug Diareecu o perk Pie TH mel e exeiic ey I lt pKey enEry 442 TRD NE GI G D G DE L ly lass ace jo pre ila SE L H R RE L LE E EL EE E L 1 I TE STE DET I E H HE L DE H EH E e TT TT T T T ii timeout_ms lt dest gt lt atrrid gt mod smpex d ebug s tring data v erbose Seu Ee a Siisnesee am R H HLDR tare lt cuilavel E lt lt a EST BODO lLOG yy log show vead eo
115. counters and performance M Provides advanced Performance Management PM of ports in the fabric R Supports Quality of Service QoS functionality including SL to VL and VL Arbitration mechanisms WE Grouping Information Fabric Manager Installation System Requirements The minimum system requirements specifications for a station running the application are listed in Table 3 1 Pentium IV processor or equivalent Operating System Windows 2000 and above OR Linux At least 17 recommended Table 3 1 Fabric Manager System Requirements Voltaire Switch User Manual 3 3 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 2 2 3 2 3 Step 1 Step 2 Step 3 3 4 Installing the Sun Java Web Start Application Install the Sun Java Web Start application Version 1 0 1 or below on the Workstation Install it via the Internet using the link in the GUI or via the Voltaire Installation CD Installing the Sun Java Web Start Application on a Linux Workstation Installing the Sun Java Web Start Application on a Linux Workstation Execute the file with the command 7280k 1 4 1 03 lanux 1586 rom bain An RPM file is created Execute the RPM file with the command Com ivi WZsedk 1 4 1 USs les Linix 1506 4 7pm Unzip the file javaws 1_0 1 02 linux int zip Run the command install sh Enter the following directory when the program waits usr java j2sdk1 4 1_03 The Java W
116. ction is disabled no recording of bad ports is done Refer to page 6 22 for a description of this feature bad ports mode show Command Description Syntax manual node names show Shows the nodes in the repository In the VFM the user can rename nodes This shows the list of nodes that the user renamed manual node names show Command Description mc Change to Multicast mode which is used to set the Auto join feature See page A 54 for a list of the commands available in this menu mce Syntax Command Description Syntax Command Description Syntax partitioning enable set The Subnet Manager configures the P_Key table in each HCA port based on the user configuration associating a port with a list of P_Key s P Key configuration is arranged as a table in the nodes repository a row per port GUID Every port may be added to up to 16 partitions Current HCA capability is limited to 8 Row 0 in the table corresponds to the default P_Key configuration This entry is protected and can not be removed This command enables or disables partitioning partitioning enable set enable disable partitioning enable show Returns the partitioning status which can either be enabled or disabled partitioning enable show Command Description Syntax pkey delete Delete row in P_Key table pkey delete row gt 1 A 49 Voltaire Switch User Manual P N 399Z00038 2005 Vol
117. ded in the following cases During Fabric installation and during startup Before running an application Performance problems by locating discarded packets and link integrity problems MPI job run problem To locate malfunctioning nodes and get the overall fabric structure Additional problems related to fabric stability blocking or other 6 1 3 Potential Fabric Problems It is critical to verify the that fabric is clean to ensure optimum application performance and prevent premature job completion Cleaning the fabric includes looking for E 1X Links R Bad connections R High Error Rates R Congestion M Other patterns that may indicate an issue Voltaire Switch User Manual 0 8 P N 399Z00038 2005 Voltaire Inc Overview 6 1 4 Procedures and Tools used for Problem Identification Tools available to perform diagnostic Use the Topology Map to see current problems The Error Log The Bad Ports Log The Current Alarms Table The Fabric Statistics portcounters csv file High Level Diagnostic Tools Overview l 2 6 6 Enable the SM Fabric Inspect preferences for debugging Fabric Failure Use the VFM VDM Port Counters Information and Graph window to check a specific port counter s health Use the Event Log to discover that there is a problem in the fabric In the VFM right click and select View Event to get information to help identify where problem is located Alternatively you can show the Event Log from
118. djacent to the sMB The redundancy mechanism maintains consistent replications of the system configuration parameters on both sMB modules The redundancy mechanism consists of three parts e Failover Hardware software or manual decision of when to switch from the master sMB to the standby sMB e Persistency manual or online duplication of information from the Master sMB to the standby in order to continue working with the same information after the failover occurs e Duplicated IP both sMBs will use the same IPs so the management station will always use the same IP address for system management Management boards redundancy is based on keep alive signals that are being sent from the active board to the passive board by the means of software Hot Swap Mechanism ISR 9096 9288 hot swap refers to the ability to insert or remove a board without the need to power down the system That feature is facilitated by a set of micro switches for each sFB 4 12 module and management board The micro switch sends an interrupt to the management CPU each time such insertion or removal of a module occurs A blue LED 1s activated or deactivated to show when it is safe to remove or install the board During Hot swap of Fabric Boards or in case of failure the management software detects and configures the system accordingly Upon removal of an sFB adjacent to an sMB sFB 1 or 4 and if no other sMB is installed in the system the Subnet Management tem
119. dor Id Device Id Contact Location Category Port Counters Error Warning Mon May 30 18 38 33 IDT 2005 ISR9096 sLB 24 3 13 Counter threshold was exceeded New value is 11294 previous value was 11092 Port Counters Error ett Mon May 30 18 14 33 IDT 2005 ISR9096 sLB 24 3 13 Counter threshold was exceeded New value is 9768 previous value was 9563 Port Counters Error AS Warning Jen May 30 18 08 33 IDT 2005 ISR9096 sLB 243 13 Counter threshold was exceeded New value is 9380 previous value was 9178 Port Counters Error Mon May 30 18 05 32 IDT 2005 ISR9096 sLB 24 3 13 Counter threshold was exceeded New anne is 9176 pe value was 8962 Port Counters Error Figure 4 5 ISR 9096 Voltaire Device Manager Main Window Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 4 7 Chapter 4 Voltaire Device Manager VDM Main Menu The main menu provides access to the following Voltaire Device Manager functions Menu Submenu Action Exit Logs out of the Voltaire Device Manager and terminate all Voltaire Device Manager processes View View Tree Shows hides the device tree View Events Shows hides the Events Log Device IP Setting Shows the Fast Ethernet IP and routing mode settings Remote Management Settings Displays and sets the SNMP Manager s IP addresses and general remote management setting details Users C
120. ds in the Subnet Manager SNMP sub menu ISRGSOUSS 3009 eemilg simyo y gt command end exit snmp snmp snmp snmp snmp snmp snmp snmp snmp snmp snmp snmp II TEI SSic elu crer Monitoro don community get community set community show Conmumtuy strap contacts set Contacto Show description set description show tocat ilon sert Tocatcion Show Display list of available commands Display command Ss usage Siring exit eo Login menu trom any other mode via console connection or close telnet session via telnet connection Exit to previous menu Set the cluster monitor string SOW S Ee MOn EO Cr ng Get the snmp get community Set the snmp set community Show the snmp communities Set the snmp trap communtny Set the snmp contacts Show the snmp agent contacts Set the snmp description Show the snmp agent description Set the snmp location Show the snmp agent location The following details the commands available in the Subnet Manager SNMP Sets the path of the Ganglia Monitoring tool In your browser enter the string as follows snmp cluster monitor set string sub menu Command snmp cluster monitor set Description 1 Cluster Head Node IP Address 2 Ganglia tool 3 Cluster Name 4 Name of the Host Syntax Example http 172 25 2 175 anglia m load_one amp r hour amp s descending amp c Support Cluster amp h HOST amp sh 1 amp hc 4 In the example below the
121. e used in any combination of 10 Gb s 4X Ports and or 30 Gb s InfiniBand 12X ports The 12X port incorporates three 4X InfiniBand ports aggregated into a single InfiniBand port Wire speed non blocking switch capacity combined with latency of less than 420 nanoseconds make the Voltaire ISR 9096 9288 the highest performing switch available In addition to scalable InfiniBand switching the Voltaire ISR 9096 9288 offers powerful multi protocol connectivity in a single flexible system In specific configurations the chassis hosts the Voltaire Router Modules providing seamless connectivity between server clusters FC SANs NAS appliances IP SANs and TCP IP networks LANs For further flexibility InfiniBand 4X 10 Gb s and 12X 30Gb s ports InfiniBand to TCP IP Router Modules and InfiniBand to Fibre Channel Router Modules can be mixed and matched upgraded and expanded as necessary allowing users to configure systems to exact requirements Voltaire Switch User Manual 1 5 P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction et T Coens E T E _ InfiniBand GbE Fibre Channe 7 GbE Fibre Channei Figure 1 3 Voltaire ISR 9096 Switch Functional Diagram 1 6 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc ISR 9096 9288 System Overview 1 3 1 ISR 9096 9288 Field Replaceable Units FRUs The following are the ISR 9096 9288 Field Replaceable Units FRUs All FRUs are hot swappable component
122. e cll T TEIL lt Gpio iNfo portCause Perormance gt lt lid gt I vom log show H eea a e H eea e ondal EaD e eaa e ET EDE a a Counters Sh ITE T HIE REE RE I I G The following details the commands available in the Utilities Menu Command error find Description Quick track of Port Counters errors on all fabric All HCA ports are scanned and the errors found are displayed in a table Syntax error find Usage error find OPTIONS error find h for help Can wena dh i ae T R EE RR oe RIE RER Ss Prane Output ao a ET E E e E A a Scan Switches only Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode is show Enis help Example error find sh H Output Scan ALL Counters Hea 2 H 0002f100010c0000 Voltaire IB to TCP IP Router Hoa 2 H OOOUCEIOA02 965224 Joli aire BEAAZIO i TIMID S20 SSO I HEE Sir Ole 2 alore Ero cc silane aera are me er ee ene ne ae ae 13 alerte TUN LSC Oy Iss ees bent er ee ey AL euch a SR e E Cle etek ee ae ate eee ae 8 Human translation All HCA ports are scanned One HCA port displays a problem Conclusion Try to reset this port and retry to run the test Hint There could be a counter increment just after cable plug in so reset the counters and follow up Known bugs or limitations none Command event log show Description Shows the Voltaire Fabric Management event log file Syntax event log show Example ISR9XXX util
123. e of the vendor that manufactured the device Node IP Displays the IP address of the node Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc 3 6 2 Viewing Properties Data Parameter Description Displays the total number of ports in the node Node Description Displays the node type and model Node GUID Displays the Global Unique Identification number GUID of the device Displays the user assigned name of the node Table 3 12 Host Node Information Parameters Port Properties The following information is displayed when selecting a port in the Node Host Information pane Property Value Port GUID O008T1 0403960905 Port Humber 1 Link State ACT oh LID 1 Port LID 42 MTL 2046 Mum VLS VLO VLF H Lite 13 LMC U id Prefix fes00000 00000000 M EY Has SM OOONOOOO00000000 Mo SM Port State Active Port Physical State Link Up Active Link Width 4x Max Supported Link Width 1x Or 4x Active Link Speed 2 Stsbps Max Supported Link Speed 2 olsbps L m Figure 3 17 Port Properties Voltaire Switch User Manual 3 33 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Table 3 13 describes the parameters displayed for each port Parameter Description Port GUID Displays the Global Unique Identification number GUID of the port Port Number Displays the port number according to the numbering scheme of the switch
124. e the commands in the Local Interface menu IL SIRES IONS SONOS evo e abe OIG NIL sas 7 T Dror Of aver lable commands 2 command Dasplay command s Usage SEring end R Co Ogin a gull rOn clan T TE mode via console connection or close telner session yia tcelner connect Lon exit Exit tO previous menu ip address local set Set interface Ip address ip address local show Show interface Ip address and Cont EUr a eE SES send echo messages The following details the commands available in the LOCAL Interface menu Command ip address local set Description Sets the IP address and subnet mask of the Local interface Syntax ip address fast set 1ip address netmask Command ip address local show Description Returns the IP address of the Local Interface its subnet mask and the IP subnet to which the Local interface broadcasts Syntax ip address local show Voltaire Switch User Manual A 43 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Fast Interface Menu In the Fast interface menu you configure the Fast interface parameters The Fast interface is used as the management interface for connection to SNMP managers the FTP server and for Telnet sessions read the guidelines listed in page 2 9 for configuration settings of the Fast interface on ISR 9096 9288 switches The following are the commands in the Fast interface menu E CSO UO Geo mn ng ik trey e eter P command Dis
125. e which are accessible through both the VFM and the CLI The CLI alarm show command can also be used to view Alarms The Trap Alarm Table in Appendix C Section C 2 lists all alarms currently available You can use this table to identify the type of problem and fix it Each alarm is listed with its alarm severity alarm type and the object where the alarm has occurred 2 Look for the Alarm Appearance Traps and alarms are issued according to the degree of severity which can be one of the following Normal Warning Minor Major Critical In the GUI the following color coding is used to indicate the status of objects and events Color coding is defined according to severity 3 Filter Specific Alarms The user may filter the events using the GUI and CLI The filtering policy determines for each event whether it is logged and whether a trap is generated Use the VFM Setting Event Filters window or the CLI to set the desired Current Alarms to display all alarm states or only specific Alarm States By default the table view shows all nodes regardless of alarm state All box is checked Check one or more boxes By applying event filters you can limit the types of events that are reported in the Fabric Manager Event Log 4 Identify the problem and troubleshoot as required Refer to Appendix B 3 for a list of errors 5 Clearing the Alarm Alarms can either be cleared via the GUI by right clicking the Trap Table or via the CLI by u
126. ea og o IE E 0 dphast Ee S ka ibhost ibhost ibhost ibhost Description 1x port was found Fabric Manager Operation The Fabric Manager Main Window Status Bar A Node o BB bhost 14 Fl Port 1 Port 2 Node REV Has SM Node Device ID Node Vendor ID Mellanox Technologies 17254365 2 Node Description Hca1_1 1 Node GUID Node Name libhost 14 oxooo1 904 003f0000 Port Counters Error Counter threshold was exceeded New value is 22222 previous value was 11111 Port Counters Error Counter threshold was exceeded New value is 22222 previous value was 11111 Port Counters Error Sun Jun 19 18 53 57 IDT 2005 ISROXKX f sFB 4 3 7 Counter threshold was exceeded New value is 22222 previous value was 11111 Port Counters Error Sun Jun 19 18 53 56 IDT 2005 ISROXKK f SFB 4 347 Counter threshold was exceeded New value is 22222 previous value was 11111 Port Counters Error Sun Jun 19 18 53 55 IDT 2005 ISROKKX f sFB 4 317 Courter threshold was exceeded New value is 22222 previous value was 111 Port Counters Error Sun Jun 19 18 53 54 IDT 2005 ISR9XXX sFB 4 3 7 Counter threshold was exceeded New value is 22222 previous value was 11 masnim ee ee ee ee aa a aaa aan masimanmi samm AAA Port Counters Error hu AO ADP P AAN IONA a E AT Figure 3 4 Fabr
127. ease contact Voltaire Customer Support The number of times that overrun errors consecutive flow control update periods occurred with at least one overrun error in each period see Table 126 PortInfo on page 665 of IB spec This counter increasing in small increments is not seen Number of incoming VL15 packets dropped due to resource limitations on port selected by PortSelect due to lack of buffers as a problem Total number of data octets divided by 4 transmitted on all VLs from the port selected by PortSelect This includes all octets between and not including the start of packet delimiter and VCRC It excludes all link packets Total number of data octets divided by 4 received on all VLs from the port selected by PortSelect This includes all octets between and not including the start of packet delimiter and VCRC It excludes all link packets Total number of packets excluding link packets transmitted on all VLs from the port Total number of packets excluding link packets received on all VLs from the port Table B 4 Available Port Counters Voltaire Switch User Manual B 7 P N 399Z00038 2005 Voltaire Inc Appendix C Supported Traps and Events An event corresponds to an occurrence within the fabric Traps are generated some of which are translated into alarms per severity according to Voltaire or user defined thresholds C 1 Trap Structure Each trap is charac
128. eat as required Click Activate to send the new settings to the switch 3 12 3 QoS Arbitration VL arbitration refers to the arbitration done for an outgoing link on a switch router or channel adapter Each output port has a separate arbiter VL arbitration is controlled by the following three components High Priority Low Priority and Limit of High Priority The High Priority and Low Priority components are each a list of VL Weight pairs Each list entry shall contain a VL number values from 0 14 and a weighting value values 0 255 indicating the number of 64 byte units that may be transmitted from that VL when its turn in the arbitration occurs The Limit of High Priority component indicates the amount of high priority packets that can be transmitted without an opportunity to send a low priority packet The High Priority table consists of VL Weight pairs that deserve high bandwidth respectively to those in the Low Priority table As long as there are packets to transmit from the High Priority table this transmission will occur regardless of any waiting packets in the Low Priority table unless the Limit of High Priority 1s reached This limit ensures that the Low Priority table items are not starved Within each table the transmission is done with respect to the weight given to each VL Setting QoS Arbitration Step 1 Select the switch to configure in the Fabric manager main window and select Fabric Quality of Service from the main
129. eb Start is ready for use For additional help go to http java sun com 2se 1 4 2 install linux html requirements and http jjava sun com webapps download DisplayLinks Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Getting Started with the Fabric Manager 3 3 Getting Started with the Fabric Manager To start the application Step 1 Configure the IP address of the active management interface using the CLI through a serial console connection Refer to page 2 8 for instructions on these configuration settings Step 2 Open the Internet Browser and connect to the fast Ethernet IP address for example http 192 168 1 2 The Voltaire Fabric Manage Initiate Window is displayed This window varies according to the device type as shown in the example in Figure 3 1 V Voltaire Launch Voltaire Graphic Device Manager Launch Voltaire Graphic Fabric Manager Download Java tool for Voltaire Graphic Manager Support Contacts VSC Voltaire support center on the Web support voltaire com email Voltaire support center Accessing Voltaire ISR 9288 Phone US 1 781 276 1564 Elsewhere 972 54 499 66 33 Intenet Figure 3 1 Voltaire Fabric Manager Initiate Window The Voltaire Fabric Manager Initiate Window contains the following icons Launches Voltaire Graphic Device Manager which is described in Chapter 4 Launches the Voltaire Fabric Manager Click to download the
130. ed Known bugs or limitations none Command mctrace Description Finds multicast port path between nodes MC FDB searches from port to port Useful to test SM abilities to configure MC FDB multi cast forwarding tables Syntax mctrace d ebug D irect src v erbose s imple t timeout_ms lt mlid gt lt src lid gt lt dest lid gt Example Memrage 0 C OOO 24 Output Emon E sme OAO Io TAOTO rE ila Ihakolex O B ai Nollie e alaas HCA400 TS E mone MOA Os Once ks Solel BO S E Om Vodiit annie SiB 24 SGA ine ar Oxo ie OO O67 E ela HE ET UN Orlane See AAO ies E gO NONE eo oy oes ENa o Opicilkcs T oarre AC AAO Human translation A multicast path for mLID 0xc000 exists between portLID 22 and portLID 24 Conclusion portLID 24 joined the MCgroup 0xc000 Hint always check the opposite path Known bugs or limitations none Command netdiscover Description Dumps net configuration for debug or development purposes Syntax netdiscover Example utilities netdiscover it 4 Lopollogy tile generated on Thu Jul 14 19 11 06 2005 T switchguids 0x8f10400403009 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Switch VANS rs ONO sym IOAN ONG ORS ONO Re sFB 4 smalid 1 22 Ege NE fel 2 Hoe ONO E IO RR L dso Si Ll TETE BOATOS Ol Oeil Po OO Si ROA OO SO E in oa OOo MOA Crs eae Ein SE DG Z KECO HRE RRR AER p il Sswitchguids 0x8f104003f0765 Switch AS SOUS
131. eference The commands available in Privileged mode and their descriptions are listed below Note The prompt shows that you are in Privileged Mode TSRI096 3009 7 command alarm clear alarm clear all alarm show Clock Ser clock show CONEA debug disable end exit fast interface show front show ftp show Sipe Obie Comite ts POTTRESET sLBPortCounter reset Voltaire Switch User Manual Ditcolay mor Orn available eonmanes Display command s usage string Clear Current alarms entry Clear all Current alarms entry SNOVE OE E STE aS Sets a new time Display the system clock Change to configuration mode Change to debug mode Change tO EXEC mode Exit To Togrn menu Trom ahy other mode via console connection or close telnet session via telnet connection Ean Co Prev Lous MeN Show interface Ip address and T T IEEE E RE ST PIP e iene I T Reset sLB port s counter settings Reset sLB port s counter settings A 9 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference sLBPortCounter show Show sLB port s counter settings password update admin enable Change admin or enable password persistency state show Display the state of persistency Ping Senel echo messages gos info show SIMUCINn Ele HOS Woe T GOS S LL S how Show QoS SL VL Table ST E a EE SO Show 0S High Arb able GOs Ae Ow e ie ehehO mies Mow Show QoS Low Arb Table rear show Display the rear reload Reboots the syst
132. el E RET E ar E RTE ER GR LE poarten EE RE ERDRE DEHE ERL E HEL EE T 2 Command Description Syntax Example ISR9OXXX tIilitles Counters Zero Zero Zero Zero Zero Zero Zero Zero Zero Zero All e EE ES ES Lael Lidl Lael ie Lil T OOO S P GEE 6G POrt TOSE Ore 2S ama T EEE Wi Meee ma kO Zo 209 259 Zoe NS zero counters mask mask mask mask mask zero counters zero port counters of sw only node s ZeoCcO CoumEctrs Osan Git BE 16 26 ge Obit OnE iin ITE T A miles ITE S E l mask ER RE P N 399Z00038 2005 Voltaire Inc Utility that allows to zero system counters one all hca only Voltaire Switch User Manual A 6 Configuration Config Mode Configuration Config Mode Configuration mode commands apply to system wide features rather than a specific protocol or interface From Config mode you can access other CLI menus for configuration of specific system features protocols and interfaces Available commands for each are described under their respective sub topics Accessing Config Mode The Config mode is entered through Privileged mode by typing the config command Config Mode Commands Reference A list of the commands available and their descriptions in the Configuration mode are listed in the following pages R SIO er ON E ake a 2 command lus wet end exit e L REE I interface IB ie ere eyes INT interface LO
133. em replication status set Change the status of replication replication o tatus show RSK E whe Stiacusy Ob rep DE EHO route default gw show SIO Wee te el ete aE ES SE Show Show the sm parameters snmp community show Show the snmp communities update firmware Update firmware version update software Update software version Add Cemortce server nane via ftp server server name Change to utilities mode Display hardware and software Version E EE 1 ucilicies version show Syntax usage is as follows in Privileged mode Menu allarm elcar alarm clear entry alarm clear all alarm clear all alarm show alarm show E ER clock H MMDDANMANYYY Coe EA COCR sien COn Con g debug debug disable disable fast interface show front show fast interface show front show ftp show ftp show SISO TEC OUN O IPA EEE CoR devicePortCounter portReset Polo ee O Or E AUNDE cavie a Ren t e na ean HI ET SLED emt Coumeor saow a eE a password update admin enable sLBPortCounter reset ShePomeCoumesic Sino password update admin enable persistency state show SE Slovene SEE e lav show gos A hepa rbt roe ven show GOs EA tou e Re show rear show reload roplicat ron E Seu repica t TOn Status show route default gw show SMELE O Savery snmp community show update firmware persistency state show Gos E sow JGOSa olay ohow gos vi malio igh ar prtr a ake las Savon Clos y eres Qui c aeon Slaven rear show reload replic
134. emenadatnanesasnees alex 4 18 Gi ACP ES EST nits ste eee mecce reer ree erent tecrmrrma Tmt anne eset re ct enema T eT 4 20 Temperaire T TTT 4 22 vii P N 399Z00038 2005 Voltaire Inc Figures Figure 4 16 Local SMI Information sesers EN 4 23 Figure 4 17 Detailed Module Information sss sss sss sese see 4 25 Fiege 2 RE SEE eee 4 27 Figure 4 19 sLB 24 with MediaConver ter sese 4 29 Figure 4 20 ISR 9288 Fabric Board Information sese 4 30 Figure 4 21 ISR 9096 Fabric Board Information sss sese 4 30 IST 42 25 VAS E RES T Re 4 3 Figure 4 23 sMB Detailed Module Information Window ccceeceeeeeesssttteeeeeeeeeeees 4 32 Figure 4 24 SC TRE Informatio sesione naas anaa aa a AnaS 4 33 STT 423 SPSU MONNA ON uoa e E E ES A Ea OEE 4 34 Figure 4 26 ISR 9288 Fan Information neisse n a O 4 35 Figure 4 27 ISR 9096 Fan Information sss esse esse sese 4 35 Figure 4 28 Faulty Par KEST ee Te nass a T E 4 36 Figure 429 Router Module Information a EA icra 4 38 e Sg hace oto 0 Poen ranee aT sr nec ne te eee eee 4 39 Figure 4 31 Remote Manager Settings SNMP Tab sese 4 40 Figure 4 32 Remote Manager Settings General tab 4 4 Figure 4 33 Configuring User Authorization sese eee eee 4 42 Ernur 4 34 RT ETT e FI Prices ial al a atone E ated AAEE 4 43 Figure 5 1 Partitioning Block Diagram 0 cccceeceeeeesensecesecsecssssssscssssnsenssseneees 5 1 Figure c H RIT Te RT Block BT T 6 3 PASS E PM SCUDO S
135. emote logger delete Delete the remote logger of this computer remote logger delete A 35 P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference remote logger set Enable the remote logging feature and set the remote logger of this device Remote logging is used to forward system notifications to a remote logger in addition to the internal error log Remote logging is permitted for one remote server at a time Setting the IP address of a remote logger overruns any existing setting of remote logging remote logger set lt IP Address gt remote logger show Display status of remote logging and the IP address of the remote logger if remote logging is enabled remote Logger show route Type this command to access the Route configuration menu which is used to set the default gateway used by the switch Ethernet interface See page A 46 for a list of the commands available in this menu route sm Type this command to access the Subnet Manager configuration menu which is used to set SM parameters See page A 47 for a list of the commands available in this menu sm snmp Type this command to access the SNMP configuration menu which is used for setting parameters for support of third party SNMP managers See page
136. en table is changed or port is reconnected For each fabric node port a Pkey partition may be added to total up to 16 partitions Row 0 in the table corresponds to the default Pkey configuration This entry is protected and cannot be removed Unless configured otherwise all existing node ports are assigned the default Pkey 65535 fffff To view Pkey Commands use the CLI to enter the following information Command Description Enable password Enter Privileged mode from Exec mode the default password is voltaire Contig Enter Config mode from the Exec mode Type the password when prompted sm Enter sm configuration pkey Enter pkey The following commands appear LS DP OOs Om T TOROX CC OME DE E pkey pkey pkey pkey pkey pkey delete high show low show IS n Sele update Voltaire Switch User Manual pkey delete row gt 1 pkey high show pkey low show set pkey port guid set row gt 1 pkey set pkey number pkey update P N 399Z00038 2005 Voltaire Inc row guid Oise rec 5 3 Chapter 5 Partitioning Step 1 Step 2 Step 3 Step 4 5 4 To set partitioning Begin with enabling the partitioning feature by running the command partitioning enable set enable Reviewing the ISR 9096 9288 partition table use the CLI commands pkey high show and pkey low show The low show command will display partition table columns 0 8 The high show command will display partitio
137. er Manual P N 399Z00038 2005 Voltaire Inc Voltaire CLI Basic Concept A 3 Voltaire CLI Basic Concept Exec Mode Read only Menu Login admin Password 123456 Privileged Mode Login admin enable Password default voltaire A 4 _ Configuration Mode Main Menu Exec Mode The Voltaire CLI concept implements a hierarchical organization as described in the diagram below Sub Menu utilities config Sub Menus Cluster Tio interface IB InfiniBand interface INT Internal interface LOCAL interface fast Ethernet Sec sm Sub Menus sm Subnet Manager J mc Multicast snmp pm Performance Manager gos Quality of Service When a CLI session is started on the switch the user enters the Exec mode which allows view only access of switch configuration parameters Exec Mode Commands Reference The commands available in Exec mode and their descriptions are listed below Note The gt prompt shows that you are in Exec Mode INSURES OS oy SUG ie 2 2 command Clock sito connect enable end exit fast interface show Voltaire Switch User Manual Display list of available commands Drop lay conma ndS Weage tse Gun Display the system clock Cone E Change to PRIVILEGED mode EXte Bo vlrogin menu EPrem amy Orne mode via console connection or close telnet session via telnet connection Exit to previous menu Show interface Ip address and E INS W
138. eration will typically be performed when faulty operation of the Voltaire Device Manager software is suspected This operation does not affect the InfiniBand traffic E From the main window toolbar click OR E From the main menu select Action Reset The following popup window is displayed Warning AN You are about to reset the ISR 9288 Would you like to proceed WE Click Yes to complete the reset Voltaire Switch User Manual 4 45 P N 399Z00038 2005 Voltaire Inc Chapter 5 Partitioning Partitioning enforces isolation among systems sharing an InfiniBand fabric Partitioning describes a set of endnodes within the fabric that may communicate Each port of an endnode is a member of at least one partition and may be a member of multiple partitions A partition manager assigns partition keys Pkeys to each channel adapter port Each Pkey represents a partition Reception of an Invalid Pkey causes the packet to be discarded Switches and routers may optionally be used to enforce partitioning Partitioning can be used to associate hosts or devices with common attributes as shown in the diagram below Figure 5 1 Partitioning Block Diagram Voltaire Switch User Manual 5 1 P N 399Z00004 2005 Voltaire Inc Chapter 5 Partitioning 5 1 Mechanisms Used to Create and Enforce Partitions 1 Each HCA port has a partition table defining the partitions that can be accessed by that HCA port 2 The SM ini
139. ere nT Re ee eee ee A 46 Subnet Manager Ment asrina tata eaeaee ecu aceon eae ea A 47 SNMP M SN nas i a a ea E e T E A 65 Appendix B Port BC TT C siisssscissccecsciassccsssaccacansansnsntovecscssaasccscssscaaconsansnsuaesees B 1 B 1 Deseription of Port ounter Piclds T B 1 B 2 CN Sunma REPO aster co ccresices ac T NN B 2 B 3 Available Fabric Port Counters u cecastecncacsusteodandenscsehcaansecnd DEEA EERE B 3 PM Counters Values and Default Thresholds sss B 3 Port Counters List and Description sss sese B 4 Appendix C Supported Traps and Eventts sssccsssssssssssssssssscccccsseseeees C 1 Cul TAI EC UI E E tela salt tucie fine sabe E chanted E E E E C 1 C2 Trap Alar A a0 le a erie cae a acs ccarateadeaaicaaeesaeecees ores C 3 Appendix D GIOSSATY HHT D 1 Appendix E ING L epecsdechadeletecessscccisctbccensecbutiadeletetecesdeciatticceesactethuceke teense 6 1 Voltaire Switch User Manual V P N 399Z00038 2005 Voltaire Inc Figures Figure 1 1 Voltaire ISR 9024 Switch Functional Diagram sese esse 1 4 Figure 1 2 Voltaire ISR 9288 Switch Functional Diagram 2 0 0 0 eee eee eee 1 6 Figure 1 3 Voltaire ISR 9096 Switch Functional Diagram sees eee eee eee 1 6 Figure 1 4 ISR 9288 InfiniScale Switching Topology Fat Tree Topology 1 9 Figure 1 5 ISR 9096 InfiniScale Switching Topology Fat Tree Topology 1 10 Figure 1 6 Out of band Management sss sese eee 1
140. erminating 4 5 viewing device information 4 16 differentiated services 3 49 Discovery 1 2 Disk space requirement VFM 3 3 Display requirement VFM 3 3 E error find A 20 Event Filters setting 3 29 Event Logs 3 26 Event Notification Mechanism 6 4 event log show A 21 A 25 Exec mode CLI A 3 Exit operation Device Manager 4 5 exportFile topology A 39 exportSm A 40 F Fabric Board 1 7 Fabric Diagnostic 1X Cable Width 6 18 1X link 6 18 Bad Line Board or Fabric Board identifying 6 20 Event Log 6 13 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Fabric Statistics 6 16 Port Counters Graph 6 12 Fabric Diagnostic 6 1 Fabric Diagnostic Current Alarm 6 14 Fabric Diagnostics Methodology 6 7 Fabric Inspect 3 46 Fabric Inspect Settings 6 8 Fabric Manager components 3 2 Installation 3 3 main menu 3 13 Online events log 3 25 starting 3 5 Subnet tree 3 16 toolbar 3 18 tooltips 3 12 Topology map 3 10 Fabric Problems 6 5 factory default A 34 Failover 1 13 Fail over 1 2 Fan Units 1 7 Fans information viewing 4 34 fast A 35 Fast interface 1 14 2 2 2 8 fast interface show A 5 A 12 Fat Tree Topology 1 8 FAT Tree topology 1 7 FCR 1 7 Fibre Channel Router 1 5 Filters Event 3 29 find bad ports A 22 findpath A 23 Firmware Updates A 9 Firmware Upgrade 2 18 First Time Configuration of switch 2 6 Fit to contents button 3 19 Flow Contr
141. ersion of switch software available to the switch for boot purposes Use the following steps to verify that the new version of software is available Command Description Enable Enter Privileged mode from Exec mode version show Checks the current software version update Starts the update software process This can take software several minutes Update software version lt Itproot patn gt Pore In ISR 9096 and ISR 9288 switches an additional command update firmware Is required to update the firmware of the switch in ISR 9024 switches the update software command updates both the software and the firmware Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 2 3 4 CLI Maintenance Commands Backing up the Switch Configuration It is recommended that you back up the switch configuration settings and topology file periodically The switch configuration settings can be restored when required by using the import command available in the FTP menu Command Step 1 Enable Step 2 CONTIG Step 3 ftp Step 4 server Step 5 username Step 6 password Step 7 export lt Eeprool path gt lt file name gt Step 8 exportFile topology optional lt ftproot path gt Step 9 Sxportcom lt TC OOt optional path gt Voltaire Switch User Manual Description Enter Privileged mode from Exec mode Enter Config mode from Privileged mode Access the FTP menu from Config mode Configure the IP address of
142. es EIZ omnia o Arme d gt 4 IE meee T CO e eee e ee e l a e ety a joerc hQlife Madsuae Ll T local addr ET is aoaea a value Geb set switkehs O EH D R Nex PETAn lt wie e nc HE HEE eis S CIE an VL mapping table MeaGgseae yy kadar gt il lt Olepowe lock Gere val Bile E EET ET olive Command mcmember Description Creates verifies using SA queries remote port join to different MC groups Syntax memember d ebug t timeout_ms s scope m mordi p portguid Laprotk yi lt a lt a Ks de eave y Example memember mMm OxXFFI2401BFeFFOQOQOQOQQQQQQ0O0FFFFFFEF p Oxo lO OOO Oe Os Output IO OOO Sel Seles Za ol ital eC e lon OOme sill ue Voce HEE Human translation 1 Port 0x8f10400400707 joined MGID 0xFF12401BFeFF000000000000FFFFFFFF 2 The above MGID got the MLID 0xc001 3 pkey 0xFeFF 4 A join is prepared on behalf of the above port proxy join 5 Since the MGID does not exist the SM creates it since a join was sent with lt full membership gt Conclusion very useful utility to test SM abilities to create MCgr and to mass config of MC FDB Voltaire Switch User Manual A 25 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 26 Hint if m option is not specified then default MCegr is used OxFF 12401 BFeFFOOOO0O00000000FFFFFFFF if p option is not specified then self port portguid of connected to CPU anafa is us
143. esco P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference iE e genie LIT Display firmware related information ftp show SH None PEE e ETONE TR EET 1b interface show Show interface IB address and GR iogur Lon 10 show Show existing IOs manual node names show Show the nodes in repository persistency state show Display the state of persistency ping Send echo messages replication status show Display the status of replication route default gw show Show the default Gateway sm info show Show the sm parameters snmp community show Show the snmp communities version show Display hardware and software version INEO rma trOn The following details the commands available in Exec mode Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Example i Displays a list of available commands in the current CLI mode menu Z command Displays brief help text on the specified command lt command gt clock show Displays the time and date currently set in the switch clock show connect Establishes a CLI session with a router module installed in an ISR 9096 9288 switch connect slot number position number connect 12 2 This command establishes a CLI session with a router module mounted in the SRBD installed in slot 12 of the ISR 9096 9288 rear panel in the middle router slot Volt
144. escription The sMB graphical display shows when the Subnet Manager and the Chassis Management are active and the status of the power and info LED The blue Hot Swap LED is active when the sMB board is ready to be removed Voltaire Switch User Manual 4 31 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 32 The board s VPD Vital Product Data is also displayed Additional VPD can be obtained by clicking the Advanced Button Detailed Module Information Module Marne Part Murnber Figure 4 23 sMB Detailed Module Information Window This window displays the following additional data The date on which the module was manufactured GUID The unique identifier of the module CPLD Version The version of the Logic Device PR PMC Version The version of the PowerPC Processor PCI Mezzanine Card Table 4 11 sMB Detailed Module Information Window Fields Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Module Information Refer to Obtaining Detailed Module Information on page 4 24 for further detail 4 6 4 sCTRL Information The sCTRL module is installed on the rear panel of the ISR 9096 9288 chassis It provides the ISR 9096 9288 system management ports Ethernet and Serial It also includes a Reset button for the chassis To view the sCTRL Module information R Click sCTRL on the Device Tree The sCTRL Information Window appears e V dla HS 3
145. etects the system type and executes the suitable device management operations on page 2 19 E Enhanced VFM on page 3 1 able view Users can find out the status of each node through color coded displays based on current alarm information System Grouping ISR 9096 9288 systems are automatically detected grouped as a single entity and represented as a single icon in the fabric topology map hiding its internal structure and simplifying the map Note grouping is not supported in this version for the sLB 8 12 On page 3 42 E Enhanced Device Management on page 44 1 Device Zoom View support Fan rate indication Normal mode or Turbo mode and MediaConverters Graphic display to differentiate between IB media connected to a port iV Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc About this Manual VPD data of each module Users can view the VPD Vital Product Data serial number and hardware versions of each board in a read only field as described in the Obtaining Detailed Module Information Section on page 4 5 6 R Subnet Manager Enhancements Support for partitioning configuration of subnet parameters automatic join mode disabled by default on page 5 1 E Enhanced Fabric Diagnostic on page 6 1 Enhanced VFM GUI for Fabric Diagnostic Sorting to current alarm table in alarm data filtering in current alarm dialog box peer information to current alarm enhanced switch list and host list e
146. evant switch the sLB and sFB board names and there numbers This window allows the user to find the bad sLB or sFB by LID or GUID By combining this node information and portcounters cvs file one can identify any fabric port connectivity failure that occurred during a pre configured interval Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic The example below shows a portcounters csv report LID IBPort PortGUID NodeType Width State MTU NumVLs HOQlife 720 12 0008f104003f0670 Switch 4X ACTIVE 2048 VLO VL7 13 PlatformType ModuleType Modulelindex Port Name NodelP DevicelD ISR 9288 sLB 24 5 internal 12 ISR9288 0061b 192 168 60 200 5a09 MLID JoinedGroups SymErr LinkRecovers LinkDowned ReveErr RcvRemotePhys RcevSwitchRelay 0000 0 0 0 0 0 0 17575 MLID JoinedGroups SymErr LinkRecovers LinkDowned RevErr RcvRemotePhys RcevSwitchRelay 0000 0 0 0 0 0 0 17575 XmtDiscards XmtConstraint RcevConstraint LocalLinkinteg ExcessBufOvrrun VL15Dropped XmitBytes 1 0 0 0 0 0 0x07964132 RcvBytes Xmit Pkts Rev Pkts PeerLID PeerlBPort PeerPortGUID PeerPlatformType 0x1884ac09 0x001 afaf4 0x005795e8 16 9 0008f1040040061b ISR 9288 PeerModuleType PeerModulelndex PeerPort PeerName Status sFB 12 4 internal 57 ISR9288 0061b ALERT This example shows the status of the sLB 24 module and its sFB 12 peer Module Using the Port LID and GUID of each board you can use the Node Info Window to identify the specific sLB and s
147. face default 1 15 IP Setting using the Device Manager 4 39 ip address fast set 2 8 2 11 A 41 A 43 A 44 A 45 ip address fast show A 41 A 42 A 43 A 44 IPR 1 7 ISR 9024 Switch 4 13 ISR 9288 Switch 4 12 4 13 ISR9288 FRUs 1 7 J Java Web Start application Sun 3 4 L LCL SM 4 10 LED 4 11 lidtrace A 23 6 4 Chapter E Index Limit of High Priority 3 51 Line Board 1 7 link aggregation 1 3 Link Information 3 41 Link Speed 3 34 4 19 Link Width 3 34 3 35 3 41 4 19 Linux Workstation 3 4 linux log show A 24 LMC 3 3 3 34 3 40 3 46 3 53 3 54 D 2 Local SM Information viewing 4 23 Login window Device Manager 4 4 M mac address fast show A 45 MADs PIPELINE 3 46 madstat A 24 Main menu VFM 3 13 Maintenance Commands CLI 2 14 Management 1 7 Management Board 1 7 Management interfaces 1 14 Manager user password 4 42 Manager user type 3 6 4 4 mctrace A 26 Memory sharing 1 2 MIB 1 11 MKey 3 34 3 41 Mozilla 3 3 MTU 3 34 3 37 3 39 3 41 4 19 4 28 Multi Path Scheme with SFB Consistency 3 54 Multi PathScheme 3 53 N Navigation tree Device Manager 4 9 netdiscover A 26 Netscape 3 3 Node Information viewing in VFM 3 32 Node Properties 3 32 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Nodes Info Window 6 20 Nodes Information 3 19 O Online Events log in VFM 3 25 events data 3 25 OS bypa
148. for outgoing packets The forwarding databases are configured by the Subnet Manager when the subnet is initially configured and upon network changes In the event that multiple paths exist between a pair of nodes in the subnet the Subnet Manager identifies multiple minimal paths among those end ports selects one path and programs the fabric accordingly However to better utilize network bandwidth the Subnet Manager balances its selections such that the traffic between all end ports is balanced over all possible paths in the fabric InfiniBand Multi PathScheme InfiniBand supports multi path routing in the following manner each end port can have several LID addresses beginning at the base LID and continuing to the base lid 2 _1 The SM can be configured for LMC LID Mask Control values between 0 and 7 If the LMC is not zero the SM assigns a LID range for every end port and configures the switches forwarding tables accordingly The switches themselves are not aware of LID ranges and the multi LID addressing scheme is transparent to them Voltaire Switch User Manual 3 53 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 13 3 3 54 Voltaire balanced routing scheme supports multi path routing in that manner Furthermore in case multiple paths exist between a pair of nodes in the subnet it balances the assignment of the LID range among the possible paths Multi Path Scheme with SF
149. formation load InfiniBand architecture is based upon a very different concept Offering high speed interprocessor communication and memory sharing it enables you to build server clusters with performance comparable to that of large servers at a fraction of the price Instead of one bus through which every device communicates InfiniBand architecture offers a network fabric that incorporates numerous switches and router modules allowing devices to communicate simultaneously Speed is no longer restricted by the shortcomings of the infrastructure InfiniBand operates at 2 5Gb s 1X 10Gb s 4X 20Gb s 4X DDR 30Gb s 12X to 60 Gb s 12X DDR Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc About InfiniBand InfiniBand architecture is modular highly scalable it allows for virtually unlimited network expansion and reduces the workload of the operating system kernel and the CPU freeing their power to run applications Offering advanced fault isolation controls nonexistent in previous protocols InfiniBand provides a high level of fault tolerance And most importantly because the InfiniBand system is modular processing power based on commodity servers can be added as needed 1 1 1 InfiniBand System Elements The following are the InfiniBand system elements R HCA Host Channel Adapter an adapter or silicon chip residing on the host motherboard providing the host side functionality and advanced reliable message pass
150. g Module Information 4 6 1 sLB Line Board Information Up to 12 sLBs Line Boards can be installed at the rear of the ISR 9288 up to four can be installed at the rear of the ISR 9096 chassis To view the Line Board information M Click the desired sLB on the Device Tree The sLB Information Window appears Voltaire Manager File View Device Action Help a RR a0gp 3S sLB sRBD SLB 1 sRBD 2 SMB Right S sPSU sPSU 1 sPSU 2 sPSU 3 Line Board Number 1 Serial Number Port VLS LO VL VLO VL 2048 bytes YLO VL 2046 bytes WLO VL 256 bytes WLO VL Description ounter threshold was exceeded New value is 27587 previous value was 27223 d Counter threshold was exceeded New value is 27223 previous value was 26872 Counter threshold was exceeded New value is 26872 previous value was 26522 Counter threshold was exceeded New value is 26522 previous value was 26175 Counter threshold was exceeded New value is 26175 previous value was 25843 Tue May 31 12 42 14 IDT 2005 ISR9096 sLB 243 13 Counter threshold was exceeded New value is 25843 previous value was 25522 e his l gil Lit Sanibel H AUNTS De Z POC sia C Gele Mesi Sli te pre O Sle sais 4g Figure 4 18 sLB Information SLB Information Window Description The sLB 24 graphical display of the Line Bo
151. ge of time of the total usage time the CPU is waiting to process Table 4 7 CPU Information Parameters Voltaire Switch User Manual 4 21 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 5 4 Temperature Information To display component temperature readings of a switch that has an embedded temperature sensor r Step On the device toolbar click Temp icon The following dialog box is displayed in the main window Information area Farenheit Celsius maar E Farenheit Celsius Farenheit Celsius 100F Farenheit Celsius mmr W 10 1 12 oo ha Figure 4 15 Temperature Readings All fields in this dialog box are read only Module temperatures are displayed both in Fahrenheit and in Celsius The status bar at the top of the dialog box indicates an overheating alarm condition should one exist 4 22 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Switch Information When a module heats up according to preset threshold defaults which can not be adjusted by the user the following occurs WE The switch increases fan speed E The Temp Fault LEDs light up on both the sCTRL and the sFU 8 front panels The fan speed also increases when one of the sFUs is not present or when one of the sFU internal fans is faulty On the toolbar the Temp the Device and the Fan icons upper left corners turn red to display a te
152. h User Manual P N 399Z00038 2005 Voltaire Inc Fabric Manager Operation Menu Submenu Action map once the Telnet session is established you can use the Command Line Interface CLI to view and set device parameters See Chapter 2 for instructions on CLI usage Configuration opens the Node Configuration dialog box which is used to view and configure node parameters Monitor Future option Advanced Routing Mark SFB Add sFBs to the SFB List when configuring routing through the Fabric Manager Unmark SFB Remove sFBs from the SFB List when configuring routing through the Fabric Manager Routing Configuration To configure routing for a node as described on page 3 53 Clear Alarms Clears alarms in the Current Alarm log see page 3 24 for more information This menu is only active when a port is selected in the Fabric Manager main window Port Info Displays the Port Information dialog box with information on the selected node This window is described on page 3 40 Port Counters Displays the port PM counters with information regarding the port performance This window is described on page 3 22 Attached to Displays the nodes which the port is connected to at both ends of the link About Displays the Voltaire Fabric Manager software version Table 3 4 Main Menu Options 3 4 6 Right Click Context Sensitive Menu You can use the right click menu in the Topology map in the event log and in other
153. h platform in which the alarmed object is installed The module board type where the alarm occurred The alarmed port number A textual description of the alarm Table 3 9 Alarm Data Window Fields Alarms types are detailed in the Current Alarm Table in Appendix C on page C 3 Clearing Alarms Clearing an alarm means that you have taken the steps to correct the problem and now wish to remove the event from the Current Alarm list W To clear a specific alarm right click the event listing in the Current Alarms window and select Clear Alarm To clear all the alarms displayed in the Current Alarms window right click the alarm listing in the Current Alarms window and select Clear All PRT x After clearing an alarm in the Fabric Manager if the actual alarm still persists physically the alarm will reappear in the Current Alarms window 3 24 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 3 5 2 Type E Normal E Normal P Normal E Normal EP Normal E Normal Managing Alarms and Events The Trap Table and Event Logs The Trap Table is displayed at the bottom of the main window It provides information on the events and alarms in the system The Trap Table is available in both the VFM and the CLI Refer to page 6 13 for a list of the reported events DateTime SOurce Description Mon Sep 06 17 70 01 GMT 02 00 2004 172 25 5 210 SM IMF Get Subnet Port Counters Completed Mon Sep 06 1
154. he default get community string is public Syntax snmp community get public private Command snmp community set Description Sets the SNMP set community string The default set community string is private Syntax snmp community set public private Command snmp community show Description Shows to which SNMP community string public private the system is connected for both get and set commands Syntax snmp community show A 66 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Voltaire Switch User Manual Configuration Config Mode snmp community trap Sets the SNMP trap community The default trap community string is public snmp community trap public private snmp contacts set Sets the SNMP contacts snmp contacts set contacts snmp contacts show Shows the SNMP agent contacts snmp contacts show snmp description set Sets the SNMP description snmp description set description snmp description show Displays version information on the SNMP agent in the switch snmp description show snmp location set Sets the SNMP location snmp location set location snmp location show Shows the SNMP agent location snmp location s
155. he port Link Speed Displays the supported link speed of the port Supported VL s Displays the Virtual Lane VL range supported by the port HQ Life Displays the Head of Queue lifetime of the port The port discards a packet from transmission if the Head of Queue life has expired Port MTU Displays the Maximum Transmission Unit value of the port Table 4 6 Port Information Parameters Voltaire Switch User Manual 4 19 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 5 3 CPU Information To view CPU information F Step 1 Pid On the Device toolbar click CPU icon The following is displayed in the Information Area User Time TS 74 System Time a Tiy Idle Time waiting Time oon Figure 4 14 CPU Information For ISR 9096 9288 switches the status bar displays the relevant sMB card for which the CPU information is displayed it also indicates whether the sMB card is the active or standby card 4 20 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Switch Information Table 4 7 describes the parameters displayed for the CPU Parameter Description User Time The percentage of time of the total usage time that the CPU is used System Time The percentage of time of the total usage time the system spends using the CPU Idle Time The percentage of time of the total usage time the CPU is idle Waiting Time The percenta
156. he priority When there are two Subnet Managers in the network the one that has the higher priority will be the Master Subnet Manager In the event of a failure on the Master a standby Subnet Manager one with a lower priority will take over Read only field that displays the current SM state Possible values are Standby and Master 3 45 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Parameter Description SM Enable Defines the SM functionality mode Possible values are Enable or Disable Defines the LID Mask Control value for the SM Possible values are 0 to 7 Sweep Interval Defines the sweep interval for the SM seconds Possible values are 70 to 1000 0 indicates that the sweep interval is disabled Verbosity Mode Defines the verbosity mode of the SM log file Possible values are Error Info Verbose Debug Funcs or Frames MADs PIPELINE Read only field that displays the SM Send window that is the maximum number of outstanding MADs on the wire 16 is the default value for switch platforms ISR 9096 ISR 9288 and ISR 9024 Polling Retries Defines the polling retries interval for a standby SM to poll the master SM for its activity If the activity is not incremented within that number of polling retries the standby SM deduces that master SM has failed Possible values are 1 to 1000 Table 3 21 SM Parameters The configuration of Subnet Parameters can also be pe
157. hical Representation Rear View ISR 9024 Switch Figure 4 11 shows the graphical representation of the ISR 9024 switch PORT 13 PORT 14 PORT 15 PORT 16 PORT 17 PORT 186 PORT19 PORT20 PORT21 PORT22 PORT23 PORT 24 am wee ee ee eee 9 T m r a L 7 ACT Voltaire 5 Sys SS u L ISR 9024 SM LNK PORT 1 PORT 2 PORT 3 PORT 4 PORT 5 PORT 6 PORT 7 PORT 8 PORT 9 PORT 10 PORT 11 PORT 12 5 PS t G L 7 ETHERNET RESET Figure 4 11 ISR 9024 Switch Graphical Representation Voltaire Switch User Manual 4 13 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 3 6 4 3 7 4 3 8 Hotspots in Graphical Representation You can click on various points on the graphical representation to view various details in the Information Area This includes the following R IB ports click on one of these ports to obtain further information on the port operating status see page 4 17 for information R ISR 9024 clicking on any point in the graphical representation except the ports displays device general information M ISR 9096 9288 clicking on boards in the graphical representation zooms in on the board and displays information windows on the board Right clicking allows to zoom in or reset boards The Information Area The Information Area is a read only area that displays various information based on user selection in the main and advanced windows When the Device Manager is c
158. his allows you to manage both the IPR itself and the ISR 9096 9288 switch Use the following guidelines when planning this type of configuration M Configure the Ethernet interface of the IPR module on the same subnet as the Fast interface 2 2 7 Configuring the Time and Date Use the following sequence to configure the switch time and date parameters The time and date will appear on event reports that are time stamped Command Description Step 1 Enable password Enter Privileged mode from Exec mode the default password 1s Voltaire Step 2 Clock set Enter time and date in military format For 072805202003 example time 5 20 AM date July 28 2003 2 12 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 2 2 8 Step 1 Step 2 Step 3 First Time Configuration Configuring Passwords Use the following procedure for configuring passwords for Exec and Privileged mode access to the RS 232 console interface and to the Fast Ethernet management interface used for establishing a CLI session via Telnet or SSH NOTE The factory default password for Exec mode is 123456 and for Privileged mode IS voltaire Command Description Enable password Enters Privileged mode from Exec mode password update Sets the Privileged and Exec mode passwords admin enable exit Exits Privileged mode 2 2 9 Verifying Configuration Step 1 Step 2 Step 3 Step 4 From the Exec Privileged mode verify the system
159. holds using the Fabric Manager as described in Section 6 2 2 below Step 3 Set Event Filters as required as described in the Setting Event Filters Section on page 3 29 6 2 1 Configuring the PortCounters csv file via the CLI This allows you to collect all the port data and errors within the system R Telnet the switch IP and Log into the management CLI using Username admin password 123456 Command Description Step 1 enable password Enter Privileged mode from Exec mode Step 2 Contig Enter Config mode from the Exec mode Type the password when prompted Step 3 sm Enter Subnet Manager configuration mode Step 4 pm Enter PM configuration mode Step B pm report enable set Enable PM data collection and failure reports enable Wait for three minutes to allow error counters file to be updated Refer to Appendix A on page A 1 for the complete list of the CLI PM commands 6 8 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic 6 2 2 Configuring PM Using the Fabric Manager In order to manage PM data using the Fabric Manager PM collection must first be z enabled using the CLI as described in Section 6 2 1 This tool is used to configure the current Alarms and Event Log To define the PM settings using the VFM Step 1 Select Fabric Fabric Inspect Settings from the VFM main menu The following dialog box is displayed Fabric Inspect Settings Scope All Sample Interval Sec
160. hout significant maintenance However from time to time you may need to install updated software or modify device configuration The switch stores a software image along with configuration files log files and other information on a local file system This file system is stored on an internal flash memory NOTE Always review the README file before making updated software available to the switch Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc 2 3 3 Step 1 Step 2 CLI Maintenance Commands If you plan to use the CLI update software command to update the switch software from a remote FTP site be sure to set the FTP server IP address through the CLI FTP mode commands and be sure the necessary file is located on the FTP server You can find the FTP mode commands on page A 39 of this manual e The software upgrade updates the entire boot image and not individual components e A configuration merge is performed old values are saved and new fields are added The software upgrade procedure should last approximately three minutes Prerequisites for Updating Software The user receives the update in a predefined media e g CD and copies it to a known location on an FTP server within his LAN The user defines the FTP parameters via the CLI in Config mode and using the FTP menu M ftp server lt ftp server gt M ftp username lt authenticated ftp username gt M ftp password lt ftp password gt
161. how A 67 P N 399Z00038 2005 Voltaire Inc Appendix B Port Counters B 1 Description of PortCounter Fields For each active node port the following info presented in a single line per port tem scription Platform Type ISR9288 ISR9024 HCA400 ModuleType GER 400 Ethernet to InfiniBand router FCR 400 Fiber channel to InfiniBand router sLB24 ISR9288 line board sFB12 ISR9288 Fabric board System ISR9024 HCA400 ModuleIndex The number of the module where the port exists 0 for HCA or ISR9024 External port number or internal for internal ports Node name where possible otherwise the nodes GUID NodelIP The IP address associated where this node where possible otherwise 0 0 0 0 Voltaire product device ID MLID JoinedGroups The multicast group ID that this port belongs to Error and data counters Each port has 16 counters 12 error counters and 4 data counters Error counters include SymErr LinkRecovers XmtDiscards XmtConstraint RevConstraint LocalLinkInteg ExcessBufOvrrun VL15Dropped Data Counters include XmitBytes RcvBytes XmitPkts Rev Pkts Voltaire Switch User Manual B 1 P N 399Z00004 2005 Voltaire Inc Chapter B Port Counters SE E PeerLID39 The remote connected Ports LID PeerIBPort The remote connected Ports internal Port number PeerPortGUID The remote connected Ports GUID PeerPlatformType The remote connected Ports platform ISR9288 ISR9024 HCA400
162. ic Manager Main Window status of objects and events Merd es sch ms Trap Table Color Coding in the Fabric Manager Green indicates a normal operating state or a normal system event pre defined threshold and Minor and Warning types of events event The Main Window provides access to most of the SM functions and provides a view of the topology managed by the system with color coded visual indicators that display alarms and important events in the managed fabric Figure 3 4 shows Node Host Link Information ode Host Properties In the Fabric Manager windows the following color coding is used to indicate the Yellow indicates a Minor problem such as a counter threshold exceeding a Red indicates a Critical problem such as a failed port or board and a Critical Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 3 4 3 Fabric Manager Operation Color coding is defined according to severity For more information refer to Event Notification on page A 1 Finding the Source of Alarms in the System When a Minor or Critical alarm exists in the fabric you can use the Fabric Manager to drill down and detect the source of the alarm When there are no alarms in the fabric the status bar at the top of the Topology map is colored green Should an alarm exist in the fabric the color of the status bar changes to indicate the highest severity alarm in the fabric Once you have detected th
163. ing capabilities R Switch connecting all the nodes in a switched point to point manner R Router Modules providing connectivity to other InfiniBand or IP subnets R Storage native and Fibre Channel access InfiniBand provides a reliable low latency interconnect with typical switch latency of 100nS with fault tolerance mechanisms Automatic Path Migration physical multiplexing Virtual Lanes physical link aggregation 1X 4X 12X and security partitioning The InfiniBand fabric provides central management and configuration controlling discovery failures and resource allocation Voltaire Switch User Manual 1 3 P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction 1 2 1 4 ISR 9024 System Overview The Voltaire ISR 9024 is a high performance low latency fully non blocking switch for high performance computing HPC clusters and grids Offering available bandwidth of 480 Gb s the Voltaire ISR 9024 is a cost effective alternative to proprietary interconnect technologies With twenty four 10 Gb s ports in a 1U chassis the standards based ISR 9024 delivers three times the port density of proprietary offerings Using the Voltaire ISR 9024 you can build high performance clusters and grids that scale from several to hundreds of nodes The ISR 9024 serves as a cost effective building block for Fat Tree topologies of hundred of nodes for scientific commercial HPC and enterprise grid applications When combined with the
164. ion and maintenance functions for the Switch and provides operation instructions Chapter 4 Voltaire Device Manager VDM Describes the Voltaire Device Manager a Graphic User Interface GUI application for device management and switch configuration and provides operation instructions Chapter 5 Partitioning Describes the Partitioning feature used to enforce isolation among systems using an InfiniBand Fabric and provides instructions on how to assign partition keys Pkeys It also describes how Partioning is used to associate hosts or devices that share common attributes Chapter 6 Fabric Diagnostic Provides information on logging and monitoring functions for advanced fabric debugging and failure detection capabilities Appendix A CLI Commands Reference Details the CLI command menus The CLI provides configuration and management functions for Voltaire switches as well as a wide range of provisioning configuration and maintenance functions for the InfiniBand fabric Appendix B Port Counters For Advanced Diagnostics Describes and details the port counter fields of the CSV file details the CSV counters summary reports and lists the ports counters currently available Appendix C Supported Traps and Events Describes the Event Notification Mechanism defines the structure of traps and lists the traps events and alarms supported in this version Appendix D Glossary Appendix D Index Voltaire S
165. ions links While the fabric will work over a 1X connection it will however create a bottleneck and hurt performance within the fabric All links should report no 1X connections when the script is ran Nothing else will be reported other than the LID and GUID if it s a full 4X link TORO Couette nes wkeen cheek WAS TOTES a ERGO ace EES Wid Lic Lael Eed Lael Wiad Tikal 6 24 1 3 4 gt 2 Tl guid OO08ElO4003f0747 ports 24 10 guid O0O008f104003f0746 ports 24 SES Sulake SIS GI gud 0008f104004004qd7 ports O008f104003f0723 ports SERRE Sed OANGKON SEO 7272 PONES OO0OO08f104003f071F ports OO008f1LO4003 f071le ports be printed 24 24 24 24 24 eeu OO CSO AOS POW Sir eel rou 4 P N 399Z00038 2005 Voltaire Inc Voltaire Switch User Manual Chapter 6 Fabric Diagnostic error find script The easiest way to look for errors on every port in the fabric is to run the error find script It will report any non zero port counters found throughout the fabric on both switches and HCAs INS IRS S Cae seve EEE EEE TRER H T EEE E ile xe Joana lid 1 guid GO0SEILO4004004d7 ports 24 e YSr ET nc LTTE SON CLEA Aske Sim Ee bile st Oe male Farr Sirk a asta same an tanner 4 Ore Oo AIM Wie CMe in o ae e ey wire mua s 1 pore o dare ounce H EREE TOON TO L T eS A 1 Oop T Gs Voltaire Switch User Manual 6 25 P N 399Z00038 2005 Voltaire Inc Appendix A CLI Commands Reference This
166. is connection is used for replication and synchronization of the configuration parameters between the two SMB modules in a redundant sMB configuration The user can change the subnet address of the Internal interface which 1s set by default to 10 9 9 0 The IP address of the internal interface of each SMB module is by default 10 9 9 1 for the left hand sMB and 10 9 9 2 for the right hand sMB Voltaire Switch User Manual 1 15 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI In this Chapter This chapter introduces the Command Line Interface CLI which provides commands to perform all necessary management functions including software upgrades and maintenance This chapter contains the following sections M Getting Started with the CLI on page 2 2 E First Time Configuration on page 2 6 R CLI Maintenance Commands on page 2 14 NOTE Fd For the CLI Commands Reference refer to Appendix A on page A 1 Voltaire Switch User Manual 2 1 P N 399Z00004 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI 2A 2 1 1 2 1 2 2 1 3 2 2 Getting Started with the CLI Interfaces for CLI Connection A CLI session can be established via a serial RS 232 connection to the switch or by connecting a Telnet or an SSH session to the switch Fast Ethernet interface on ISR 9096 9288 switches a CLI session can also be established by connecting a Telnet or an SSH sessi
167. is raw PartitionEnforcementOutbound is true and packet fails partition check IP version check or transport header version check Total number of packets received on the port that are discarded for the following reasons FilterRawOutbound is true and packet is raw PartitionEnforcementOutbound is true and packet fails partition check IP version check or transport header version check The number of times that the frequency of packets containing local physical errors P N 399200038 2005 Voltaire Inc Typically will not increase If it is may be an indicator that a parameter should be tweaked Please contact Voltaire Customer Support Typically will not increase If it is may be an indicator that HOQ or other parameter should be tweaked Please contact Voltaire Customer Support Typically will not increase If it is may be an indicator that a parameter should be tweaked Please contact Voltaire Customer Support Typically will not increase If it is may be an indicator that a parameter should be tweaked Please contact Voltaire Customer Support This counter increasing in number usually indicates a bad link Voltaire Switch User Manual Available Fabric Port Counters exceeded local phy errors ExcessiveBufferOverrunEtrrors VL15Dropped PortXmitPackets PortRcvPackets Typically will not increase If it is may be an indicator that a parameter should be tweaked Pl
168. isables counter monitoring default 5 pil POrTE LreCvV Errors Set O 69939 pm port rcv remote physical errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 pm port rcv remote physical errors set 0 CoS 9 pm port rcv switch relay errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 pit DOrE Lev Switch relay errors Ser 0 65535 pm port xmit constraint errors set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 Di POLL xmi tT Cconstrailne errors ser 0 235 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Configuration Config Mode pm port xmit discards set Defines the error threshold for this counter Setting to 0 disables counter monitoring default 5 pm POLE xMLE diecards set 0s 63939 Command Description Syntax pm report enable set Determines whether the PM should generate a port counter report file in CSV format the port counter report file can be viewed through the Fabric Manager The default is enabled pm report enable set enable disable Command Description Syntax pm reset counters set Resets the PM counters on all nodes defined by the pm scope set command Ph Treset CoOuUnLere GOT Command Descriptio
169. itch to the FTP server Periodically backing up the switch configuration settings is recommended The switch configuration settings can be restored when required by using the import command available in the FTP menu export lt ftproot path gt export oub sw contig 19 juney E exportFile topology Uploads a file with the network topology logs of the network nodes to the remote FTP server exportFile topology SCL pPLOOL pat he A 39 P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference exportSm Uploads to the FTP server a log file that stores the Subnet Manager activity You should change the name of the file on the FTP server as the next export action will overwrite the existing file exportsm lt ftproot path gt ftp show Shows FTP client configuration ftp show import Downloads configuration file from the FTP server import remote path file path password Sets user password to access remote FTP server password password server Defines the IP address of the FTP server which the switch will use for the various file transactions server ip address username Defines the user name that the switch uses to access the file server username username
170. itching Topology High performance computing clusters typically utilize Clos networks more commonly known as Fat Tree or Constant Bisectional Bandwidth CBB networks to construct large node count non blocking configurations A Clos network is a switch topology in which integrated non blocking switch elements crossbars with a relatively low number of ports are used to build a non blocking switch topology supporting a much larger number of endpoints The ISR 9096 9288 systems feature symmetrical Line Boards at both ends and interconnect Fabric Boards thus implementing fat tree topology when the system is fully populated sLBs are defined as Line Boards and sFBs as Fabric Boards Each Line Board chip element is connected to each element of the Fabric Board through one or more InfiniBand connections The shortest path from a given source to any other node is performed in three hops hop 1 at the Line Board at one extremity of the system hop 2 at the Fabric Board and hop 3 at the Line Board at the other extremity NOTE The number of available ports is limited when router modules are installed in the system The VFM groups all the Line and Fabric Boards in the Chassis and identifies them as a one switch entity represented as a single icon Figure 1 4 illustrates the ISR 9288 InfiniBand internal switching matrix Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc ISR 9096 9288 System Overview ea i
171. ities event log show Ours 09 492 5550 vem 414 CREATE Me GROUP MELD 16 M2 AYO Woe 12 EG HEE C OOOO G HERE ES ae ce Ie E 1 Ae Jal r3 09 49 56 0 Sem BR T sh STATE CHANGE Entering oM Discovering state Ja TO ao Oan O TMS ATE CHANGE Entering M Master State Ji Ls OF a E SOUS OO eO port 22 Ix port was found Port GUID et lOA00SfU 4e IB Port 22 Peer at 0008f 104003 f074e port 24 Port GUID 8f104003f074e IB E 24 Jit LS OF 849256 6 wem 440 5 COU sr lOACO SiO 4e port 4 1X port was found Port GUID 8f 104003f 074e IB Port 24 Peer at 0008f104003 f074e port 22 Port CUID Sir lO400sr074e TB Porte 2a Voltaire Switch User Manual A 21 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Command find_bad_ports Description Detects bad links ports Allows to pinpoint defective Fabric and peer ports Syntax find_bad_ports Example find_bad_ports Output iInontuaiel EH Oere Ic Ke NSN WEIS leg fan aren eae meee Serta chee ee SNE 0008 1040040061a T e e a A AAA N E E E A R E e a liveth ce sh Ble SLRs Glu re sae Se eh oer carseat Nea tease ene eae earn eee raed seh ae poeta aS H L 1 I Ne darot PALM ron ss elr onae eS ole a3 Pour ooe Ta o E ETT E L O008 10400400706 OROS ame OR SUCH ACE nem A Gece er PRU ket UES PUM ieur ye JIMS iste V okera e R a E U E E E T ee Age ae Ree or eve a 26 HE Gl sete ype ire haven ce HT see a IL Por 2S dire ce paeo ron sel ie Sswapelis
172. ject SM DENES a a a a a ENT Configured MLID 0x X for u ports max tree depth u OSM_EVENT_ILLEGAL c a yd E CONNECTION N minor enable manual E po 6 N Object port port X YZ is connected to port GUID 016Lx port X YZ OSM EVENT ILLEGAL CONN N Found illegal connection between ECTION port GUID 016Lx port X YZ Object port minor enable manual distance XYZ and port GUID Parameters src distance int32 016Lx pori X YZ distance dst distance int32 XYZ OSM EVENT SWITCH UNEXP N ECTED DISTANCE Unexpected distance XYZ hops Object node minor enable manual from Switch GUID 016Lx to Parameters distance expected sFB expected XYZ hops distance C 6 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Alarm Trap Name OSM EVENT TOPOLOGY LIN EAR FDB CAP TO SMALL Object SM Parameters switch guid fdb cap OSM EVENT UPDOWN_ ROUT ING FAILURE Object SM SRV TOPOLOGY WARNING _ EVENT OSM EVENT CA UNEXPECTE D_DISTANCE Object node Parameters distance OSM EVENT INVALID NUM _ SFBS Object SM Parameters number of sFBs int32 predefined number of sFBs int32 OSM EVENT MINIMAL LMC _ VALUE REQUIRED Object SM Parameters minimal Imc required current Imc OSM EVENT UNABLE APPLY _ ROUTING Object SM Parameters routing scheme char SRV TOPOLOGY INFO EVE NT OSM EVENT SWITCH OUT O F CLOS Object node OSM EVENT HCA OUT OF C LOS Object node OSM EVENT CREA
173. l greatly help us in our endeavor Audience This manual is primarily intended for system administrators who are familiar with the fundamentals of router based internetworking and network storage devices but who might not be familiar with the specifics of Voltaire products or the routing protocols supported by Voltaire products It is assumed that readers are familiar with InfiniBand technology and terminology About this Manual Document Conventions NOTE Fd Text set off in this manner presents clarifying information specific instructions commentary sidelights or interesting points of information IMPORTANT Text set off in this manner indicates important information regarding a specific feature Text set off in this manner indicates that failure to follow directions could result in damage to equipment or loss of information ji Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc About this Manual Document Organization This guide contains the following chapters Chapter 1 Introduction Describes the InfiniBand technology and provides an introduction to the ISR 9024 ISR 9096 and ISR 9288 switches Chapter 2 Voltaire Command Line Interface CLI Introduces the Voltaire Command Line Interface CLI Chapter 3 Voltaire Fabric Manager Describes the Voltaire Fabric Manager VFM a Graphic User Interface GUI application that provides a wide range of provisioning configurat
174. lays the device ID Displays the revision number of the node if applicable Table 3 20 Node Configuration Parameters Configuring the Subnet Manager This section describes viewing and configuring SM parameters via the Voltaire Fabric Manager To view Subnet Manager general parameters Select Fabric SM Properties from the main menu The following information is displayed Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuring the Subnet Manager SM Properties d x Port GUID prer 040040060b LMC 0 7 SM KEY pononooo00000000 Sweep Interval seconds 10 1000 15 Activity Court a355 Verbosity Mode Verbose Priority 0 15 fio MADs Pipeline State raster Polling Fetries 3 30 Shi Mlode Enable z Contact EN Location Figure 3 27 SM Details The parameters displayed in the SM Properties window are described in Table 3 21 Port GUID Activity Count Priority Voltaire Switch User Manual Read only field that displays the GUID of the port where the SM resides Read only field that displays the Key of the Subnet Manager SM Read only field that displays the activity count of the Subnet Manager SM The Activity Counter is incremented each time the SM issues an SMP Simple Management Protocol or performs other management activities It is used as a heartbeat indicator by standby SMs Sets the priority of the Subnet Manager the higher the number the higher t
175. lgorithm set balanced LOW LNG S CESC CONSI rOn STET E II H Sil i Om T RH E DD O enable disable ae VAG S aco sm info mode Sm Soo T G ISSE Se LITE sm info polling retries set Seo Oa Sane E Eye Ser ER i omean OSNON sm PACOTES e E a a SN AR O e E TT Coni rgur ra tron Sets pre sm info sm initiate fabric reconfiguration set sm info smName set string sm info sweep interval set LOT SoU ON Seconda sm info topology set eriet ocna UL ioe iain sm inio verbosity sen error ime o verbose debug function frames sm info verbosity Topologyset set none error info verbose debug function frames Dakine The following details the commands available in the SM menu Clears the bad ports table Bad ports will be added if they meet the required conditions as described on page 6 22 if bad ports Enable or disable bad port data collection The Bad Ports feature is disabled by default this means that by default the bad ports will not be reported in the fabric statistics CSV file Command bad ports clear set Description data collection is enabled Syntax bad ports clear set Command bad ports mode set Description Syntax bad ports mode set A 48 enable disable Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Configuration Config Mode bad ports mode show Show the status of bad port data collection When bad port colle
176. line board Link State Displays the link state which can be Active or Not Active SM LID Displays the Local ID of the Subnet Manager Displays the Local ID of the port MTU Displays the Maximum Transmission Unit value of the port Num VL s Displays the VLs supported by the port Current support is for VLs 0 to7 Displays the Head of Queue Life setting for the port LMC Defines the LID Mask Control value of the port Possible values are 0 to 7 GID Prefix The GID is a concatenation of a 64 bit subnet prefix an identifier of the subnet and a manufacturer assigned globally unique ID MKey Displays the Management Key Value of the port Used by an advanced SM mechanism that authorizes subnet management operations based on the Mkey Default value 0 no authentication Has SM Indicates if a Subnet Manager is present in the node Port State Displays the current state of the port which can be one of the following Active Initialize or Down Port Physical Displays the current state of the physical link State Active Link Displays the current link width 1X or 4X Width Max Supported Displays the maximum link width the link can support 1X or 4X Link Width Active Link Displays the current signaling rate of the link current default is 2 5 Gb s Speed Max Supported Displays the maximum signaling rate the link can support current Link Speed default is 2 5 Gb s Table 3 13 Port General Information Parameters 34 Voltaire Swi
177. longs to Voltaire Switch User Manul S A P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 7 7 Grouping Information Boards are grouped into a single icon as shown in the example below i This icon is an example representing a chassis Foe regrouping all its components ee IRINNA The grouping information window allows you to view the information of the various modules in the chassis To display grouping information W Select the switch in the main window topology view and selecting Node Grouping Info from the Main Menu The following is an example of the window displayed ISR9600 Info ISRs283 GUID 0008 104004004a8 0008 104003 f0 b4 0008 104003 EOS 0008 104003 Lo686 0008 104003 0637 0008f 104004004ar 1 nosu 00081040040044 i nosu 0008 104004004ab 1 activen 0008 f104004004bd 1 nosh 0008 f104004004be 1 nou 000g f104004004bf 1 nosu 0008 f104004004b9 A nosm 0008 104004004ba nom 0008 f104004004bb nosm Figure 3 25 Grouping Information PRE Pd Grouping is not supported in this version for the sLB 8 12 3 42 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuring Nodes 3 8 Configuring Nodes To view and set the configuration of a host or switch use one of the following menu options M Select the node in the topology map and select Node Configuration from the main menu OR R
178. ltaire Inc Chapter 3 Voltaire Fabric Manager VFM Step 2 Add sFBs to the SFB List using one of the following options a In the main window topology map right click the node to add and select Mark SFB from the context sensitive menu b Click Add in the Routing dialog box The following dialog box is displayed All syyvitches Index Guid Mame Description 1 0008 oFoosos0ort ISRIXXX ISR 9096 Figure 3 32 Add to SFB List c Click OK to add selected sFB and close the dialog box Step 3 Select the routing topology from the Topology drop down menu Step 4 If you selected the Rearrangeable algorithm the Multi Path with sFB consistency algorithm described previously on page 3 54 then select a value from the Topology field Step 5 Click Activate Step 6 Click Apply 3 56 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM In This Chapter This chapter provides information about working with the Voltaire Device Manager VDM application This includes information on the following topics Device Manager Description on page 4 1 Getting Started with the Device Manager on page 4 2 The Device Manager Main Window on page 4 6 Voltaire Device Manager Operating Conventions on page 4 15 Viewing Switch Information on page 4 16 Viewing Module Information on page 4 26 Using the Voltaire Device Manager for Switch Management on page 4 38 4 1 Device M
179. med by the means of the sMB fast Ethernet or RS 232 console interfaces In band management is derived from the InfiniBand network P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction Figure 1 6 illustrates out of band management which is performed via the sMB Router Module management port Chassis Example IP 1 1 1 2 24 Static Route 5 5 5 0 Example NMS IP 1 1 1 5 24 Example NMS IP 5 5 5 5 24 Static Route 1 1 1 C or Default Gateway Gateway Example IP 1 1 1 3 24 Static Route 5 5 5 0 Router Module IP 1 1 1 1 24 Eth IB Figure 1 6 Out of band Management Figure 1 7 illustrates in band management which is performed via the router port Example NMS IP 1 1 1 5 24 Example NMS IP 5 5 5 5 24 Static Route 1 1 1 0 or Default Gateway Chassis Router Module Gateway Example IP 1 1 1 3 24 Static Route 5 5 5 0 Router Module IP 1 Eth IB Figure 1 7 In band Management 1 12 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 1 4 1 1 4 2 Voltaire Switch Management Redundancy Mechanism The ISR 9096 9288 supports redundant sMBs One sMB serves as the active system manager while the other is a standby management board The latter takes control when it detects a failure of the active sMB Failure can be caused by the Subnet Manager or removal or failure of sMB or the sFB a
180. ment Interface This is the default system configuration To check which interface is the active management interface use the following CLI command ISR 9096 9288 config if fast ip address fast show Local Ip LIZZ oe oo ip mask Is 2554255 405 0 broadest bp 26 1124 25 2 556250 management interface is eth0 1 The last line in the CLI output above specifies that the Fast interface is the active management interface To set the Fast interface as the active management interface ISR 9096 9288 config if fast management interface set Configuring the IB Interface as the Management Interface To check which interface is the active management interface use the following CLI command config if IB ip address ib show ip LS WO hess lhe Ip mask i9 2567252 oon broadcast 2p i9 192 166 1255 management interface is IB Voltaire Switch User Manual 2 9 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI To set the IB interface as the active management interface ISR 9096 9288 config if IB management interface set 2 2 6 Configuring the ISR 9096 9288 with I O Modules When plugged into an ISR 9096 9288 all IPR and FCR hardware modules are automatically identified by the Chassis Management I O module configuration and monitoring 1s embedded within the switch management via the Subnet Management and is performed through the VDM interfaces both GUI amp CLI To manage the ISR 9096 9288 and all i
181. mperature fault and as result a problem with the device 4 5 5 Local SM Information To view local SM information Step CF Click SM icon on the Device toolbar The Local SM window is displayed in the Information area GUID bira 400400431 Sh Key Pooonooonooon000 Activity Count f 09426 SM State active Vendor boost GID Pretix hoooooon 00d4 Oco LID f Haz Sh fic SM Mode Enable VFM Figure 4 16 Local SM Information Voltaire Switch User Manual gen P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 5 6 4 24 Table 4 8 describes the Local SM parameters Parameter Description Read only field that displays the GUID of the port where the SM resides Read only field that displays the Key of the Subnet Manager SM Activity Count Read only field that displays the SM activity count This Counter increments each time the SM issues an SMP or performs other management activities Used as a heartbeat indicator by standby SMs SM State Read only field that displays the current SM state Possible values are Standby Master and Discovering Vendor Read only field that displays the name of the equipment vendor GID Prefix The GID is a concatenation of a 64 bit subnet prefix an identifier of the subnet and a manufacturer assigned globally unique ID LID Displays the Local Identifier LID number of the port where the SM resides Indicates if the SM is p
182. n Options include all all _hca all sw node_list Detaulre is all Syntax pm scope set Defines the scope for PM monitoring all nodes an Che fabric switches and HCAS only HCAs only switches Futuro OO EON nodes r ther ar Ea pm scope set all all_hca all_sw node_list A 59 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Command Description Syntax Example pm show Shows the current PM Performance Management configuration pm show TSE S 2024 Conmmrd sm om i somes now Enable iso eels enable default Polling 2neenrval 180 Seoee Counter operation Symbol error counter threshold bank error recovery Counter unreshold E ED lonned eoun ee we hisecin ola Portare E howd Pore rev remote physical errors threshold Port rev switch relay errors threshold Pore pale I T a voll ROMs seve CONCEAL No idienes TT T Poin a E SOs elec E in Nee o Lo al Iimk anbeqrity errors threshold PMCeo cle Lite lO Vem io eo lak I Mouse VillS arope d threshold Roi ome data CAresShold Portre aaea eNe H lel Rore mnit PRES mages oot lkel Bortar ev el cess tN SNG RE Report enable Command Description Syntax Command Description Syntax Command Description Syntax A 60 eal Q D ct w O O O O n O G O G On o G O S pe acl enable pm symbol error counter set Defines the error
183. n r sabie Manil Local Link Integrity threshold reached Object port FLOW CONTROL ERROR o T SRV EXCESSIVE BUF OVER RUN TRAP 130 minor enable Manual Object port BO OD TROL TME Switch Flow Control Update R EXPIRED TRAP 131 minor enable Manual P watchdo timer expired Object port FABRIC CONFIG ERROR phase IT OSM EVENT 1X LINK ements a 4X link that operates in na eo Y minor enable manual Object port 1X OSM_EVENT 4X_IX_LINK__ Y_ minor enable manual Found a 12X link that operates in C 4 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Excessive Buffer Overrun threshold reached Trap Alarm Table Object port PTs 4K or 1X T AAT D ae Unable to configure all LIDs on E l Switch GUID 016Lx Linear Object SM minor Parameters fdb cap uint16 Imc FDB Cap XYZ is too small for 9 current subnet LMC XYZ uchar8 SRV_BAD_M_KEY_TRAP_256 M Key 0x LX from LID XYZ Object node i l attempted Method s Parameters m_key uint64 256 minor enable manual sip Acorns s aid method string attr string attr AttributeModifier 0x X mod uint32 OSM EVENT UNKNOWN TRA N P Received unknown trap number Object node ere ee XYZ from LID XYZ Parameters trap number uint16 SRV LINK TIMEOUT EVENT N Obj ect port re Link timeout Method s Parameters method string attr amp attribute s DR path s string path SECURITY_ERROR
184. n table columns 9 15 as follows ISR9XXX config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkeyo Pkey7 Fe petra catia ikon horeca SSS emacs eames Veo ata rae Foes heey 1 OKE OONO KOOK BUREL EDO TO OLO EEE 0 0 0 0 0 0 0 Note row 1 is the default relates to each host and cannot be erased ISR9XXX config sm pkey high show entry Port GUID Pkey8 Pkey9 Pkeyl0 Pkeyll Pkeyl2 Pkeyl3 Pkeyl4 Pkeyl5 1 0000000000000000 0 O 0 0 0 0 0 0 Use the pkey set CLI command to add a partition to the ISR 9096 9288 partitioning table When adding a partition you need to specify the P_key number 0 15 table number and the P_ key value Example ISR9XXX config sm pkey set pkey9 1 Oxfl1fl Change will be effective only after pkey update command When setting a Pkey to a specific node port by GUID use the CLI command pkey port guid set to create a new table entry for a specific GUID as shown in the example below EORI Cialis esi Fi7 OSA OCnem Cline sienna a ILL Ia Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Setting Partitioning Step 5 Verify that the entry was added to the table by using the CLI pkey low or high show command as shown below ISR9XXX config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkeyo Pkey7 an an ne aa a ans ln ne nrag ara ls 1 0000000000000000 LEFF 0 0 0 0 0 0 0 2 0x11111111 0 0 0 0 0 0 0 0 Step 6 Once created use the comma
185. n 2011 06 05 IDT 2005 ibhost 2 SHCA 1 Counter threshold was exceeded Hew value is 22222 previous value was 11111 Mon Jun 2011 02 46 IGT 2005 SRIAN i 3LB 24 1 Slrternal Port 10 Counter threshold was exceeded Mew value is 22222 previous value was 11111 Figure 3 7 Topology Table View Topology g critical fea Minor Table Tab Table 3 7 describes the parameters displayed in the Topology table view Description State Indicates the alarm state of the node host The alarm state can be one of the following Normal Minor Warning or Critical GUID The GUID of the node host Name The user assigned name of the node host IP Address The IP address of the node host Description A description of the type of node host 3 20 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Fabric Manager Operation Fett nesne Vendor ID The ID name of the vendor that manufactures the node host Type Another identifier of the node host type Device ID An ID of the node host Table 3 7 Topology Table View Fields Sorting the Topology Table View You can sort the data in the Topology Table View by clicking the column headers An arrow indicates the sort order ascending or descending Use Ctrl click for a secondary sort order Filtering the Table View You can filter the Topology Table to only display nodes with a certain alarm state By default the table view shows all nodes regardless of alarm state
186. nd Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference sm info polling retries set Sets the number of polling retries for a Subnet Manager parameter sm info polling retries set ANE Meee 5 0 sm info priority set Sets the priority of the Subnet Manager the higher the number the higher the priority When there are two Subnet Managers in the network the one that has the higher priority will be the Master Subnet Manager In the event of a failure on the Master a standby Subnet Manager with a lower priority will take over Si O prioruny Set Ine gt sm info show Shows the SM parameters sm info show sm info slv set Sets the time a packet can live in the switch slv is defined as 4 096us 2slv if 0 lt slv lt 19 5 55 slv is the LifeTimeValue component of the SwitchInfo attribute If slv gt 19 then slv 1s to be interpreted as infinite sm info sliv set 1 31 sm info sm initiate fabric configuration set Initiates fabric sweep TR R s SmI ee bete rabric CcOnrLguration set sm info sm initiate fabric reconfiguration set Applies topology changes LIDs could be changed due to new route scheme and initiates fabric sweep sm info sm initiate fabric reconfiguration set Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description
187. nd snmp community show Description Returns the SNMP community strings Syntax snmp community show Example SW 6IB4C snmp community show get community public set community private Crap e onmmunit 1 GEH T Command update firmware Description Updates the firmware version using the firmware located in the specified directory of the designated FTP server Syntax update firmware update file dir NOTE x This command is only applicable for ISR 9096 9288 and ISR 9024 switches Voltaire Switch User Manual A 17 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference S Command Description Syntax update software Updates the software version using the software located in the specified directory of the designated FTP server update software update file dir NOTE In ISR 6000 switches this command also updates the firmware of the switch Command Description Syntax Command Description Syntax Example utilities changes to utilities menu see Utilities Menu Section 0 for additional information utilities version show Displays hardware and software version information SW 6IB4C gt version show eW 61B4 version dv5 09 Aug 2 2003 07 44 04PM Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Privileged Mode Utilities Menu This menu provides advanced scripts and utilities enhancing current diagnostic features and adding diagnos
188. nd Structure 2 5 CLI Special keys 2 3 CLI Debug mode A 12 CLI Fast Interface menu A 44 CLI IB Interface menu A 41 CLI Local Interface menu A 43 CLI sm info commands A 51 clock show A 4 A 11 Clos 3 54 A 53 CLOS topologies 3 54 cluster A 34 Cluster Monitor 3 47 Color Coding 3 8 6 1 P N 399Z00004 2005 Voltaire Inc command utilities A 18 Command alarm clear A 11 alarm clear all A 11 alarm show A 11 clock set A 11 config A 12 Command debug A 12 Command disable A 12 Command rear show A 15 Command reload A 16 Command Modes CLI 2 2 Command Structure CLI 2 5 Configuration mode A 33 Configuration verification CLI 2 13 Configuration Initial 2 6 Configuring Nodes 3 43 Connecting a Console 2 7 constant bisectional bandwidth D 1 Constant Bisectional Bandwidth CBB 1 8 Context Sensitive Menu 3 15 Counters Port 6 11 CPU 4 10 CPU Information 4 20 CPU Information viewing 4 20 Current Alarms Filtering 3 22 Current Alarms window 3 22 D Data center reconstruction 1 2 Data Clusters 1 2 deadlock avoiding 3 49 Debug mode CLI A 12 Default Gateway A 46 6 2 Chapter E Index Default passwords 2 6 default gw delete A 46 default gw fast set A 46 default gw show A 46 Device 4 10 Device Manager device toolbar 4 10 login 4 4 main menu 4 8 main window 4 6 Navigation tree 4 9 operation conventions 4 15 switch graphical representation 4 11 t
189. nd pkey set to define the Pkey value For example ISR9XXX config sm pkey set pkeyO 2 Oxf1f1 Change will be effective only after pkey update command Step 7 Verify that the new pkey was added to the table by using the CLI pkey low or high show command as shown below ISR9XXX config sm pkey low show entry Port GUID Pkey0 Pkeyl Pkey2 Pkey3 Pkey4 PkeyS Pkeyo Pkey7 E ener lanai came m 1 0000000000000000 ffff 0 0 0 0 0 0 0 2 Ox TTT La Eloi O 0 0 0 0 0 0 Step 8 Run the command pkey update to update the Pkey table NOTE Ed The Partition Table update of the all HCA elements in the Fabric takes a few seconds Voltaire Switch User Manual 5 5 P N 399Z00038 2005 Voltaire Inc Chapter 5 Partitioning To erase a row from the Pkey Table Step 1 Use the Pkey delete command as shown in the example below ISR9XX X config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkey6o Pkey7 1 Q000000000000000 ffff 0 0 0 0 0 0 0 2 Ox VAI Ciel 0 g 0 0 0 0 0 TSR9XXX config sm Change will be effective only after pkey update command Row 2 has been deleted from the table in this example Step 2 Verify that the row was removed from the table by using the CLI pkey low or high show command as shown below ISR9XXX config sm pkey low show entry Port GUID PkeyO Pkeyl Pkey2 Pkey3 Pkey4 Pkey5 Pkeyo Pkey7 ae Sees peses aes p C Genes ess 1 0000000000000000 ffff 0 0 0 0 0 0 0 Step
190. ng discovery fail over remote boot I O sharing and other advanced features InfiniBand is the IT industry s solution for expanding data network centers to prepare for the next generation of communications e business Internet use and the demands of the communications age have created the need for increased processing power larger stores of information and a greater more reliable flow of data As the present architecture struggled under the current load of information it was painfully clear that a solution was needed in order to manage the increasing demands of the future The giants of the IT industry Compaq Dell IBM Intel HP SUN and Microsoft followed by more than 230 companies combined the best elements of two competing technological initiatives to create InfiniBand an industry standard that has been adopted with confidence Solving a great number of obstacles that data centers were facing scalability reliability manageability InfiniBand enabled the introduction of data clusters an invaluable next step to data center construction In effect InfiniBand is a fabric based switched network that allows devices such as servers storage and I O to communicate at very high speeds Until the advent of InfiniBand communication devices had to wait in line to send their information through one shared bus The ten year old architecture had reached its limits and did not have enough bandwidth available to meet the rising demands of the in
191. ng to LID 1 in the fabric That should work only if the fabric 1s configured and the node element in the fabric whose LID is assigned to 1 is indeed a subnet manager A direct routed MAD is sent to a node element in the fabric that corresponds to the specified direct path A direct routed mad can work in a fabric that is not configured Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode A path is specified using the following format lt port0 portl port2 gt The path is relative to the SW 6IB4C on which sminfo is running For example sminfo D lt 1 7 gt is executed as follows queries the subnet manager state of the node element which corresponds the direct path lt 1 7 gt Follow the direct path start at element 0 which is our 8 port SW 6IB4C Go out of port 1 of element 0 Reach element 1 which is connected to that port Go out of port 7 of element 1 assuming that element 1 has a valid and connected port 7 Reach element 2 which is connected to that port The direct path lt 1 7 gt led us to element 2 The corresponding MAD will retrieve the required data from the element provided this is a subnet manager element Command smpdump Description The smpdump utility retrieves information from the IB fabric using LID routed MADs and direct routed MADs smpdump is available from the root shell of the ISR 9096 9288 Syntax smpdump d ebug s tring x hex b binary da
192. nhanced Router Blade Drawers SRBD tree pane Enhanced Fabric Summary Tool This tool indicates the specific platform to which a port belongs and identifies the peer on the other end of the port s link This information enables faster location of the troubled link and faster stabilization of the IB fabric Enhanced Event Notification Includes port counter threshold exceed event indication right click options and an event indicating links that are operating with a link width of 1X E Enhanced CLI on page A 1 The user has access to diagnostic utilities by the means of the CLI View Mode for Log Files Chassis Firmware Scan Enhancement of Secure CLI Access with SSH support Secure Shell on pages 1 11 and 2 13 Voltaire Switch User Manual V P N 399Z00038 2005 Voltaire Inc Chapter 1 1 1 11 1 1 2 1 3 1 3 1 1 3 3 1 4 1 4 1 1 4 2 1 4 3 1 4 4 Chapter 2 2 2 1 1 K 2 1 3 2 1 4 2 1 5 2 1 6 2 1 7 22 22A 222 22 0 2 2 4 2 2 3 Contents PU a E E ES oe A E E SN ies T A E E T 1 Document CONV Ee ll D c ment Orsaniza OW T ili Whats New in this OTT iV DOVER O GUC ENON seissen eeren EEEE NEE EE 1 1 About nAn Bandoso a ra iene Eae 1 2 InfiniBand System Elements sess ccss cvescanienssdesievesuaveontnaassivedeansenesdavseveduawseotaewss 1 3 ISR 9024 System Cy CV Wi acavietcecep ne naaomounernsaianuaucsesaancanmonctashiurersiemiucoseneaas 1 4 ISR 9096 9288 System UVYGrYIGW sese 1 5 ISR
193. nition 3 49 Voltaire Device Manage Initiate window 4 2 Voltaire Device Manager 4 1 Voltaire Device Manager terminating 4 5 Voltaire Device Manager Main window 4 5 VoltaireVision 1 11 vsm log show A 31 W watchdog mode set A 37 watchdog mode show A 37 watchdog timeout show A 37 width check A 3 1 Work Queue Elements D 3 Z zero counters A 32 Zoom in 3 19 Zoom out 3 19 6 P N 399Z00038 2005 Voltaire Inc
194. nnectors are separate The firmware configures those three ports as a single 12X group 1 3 3 4X 12X Auto Sensing Three adjacent 4X ports starting from port 1 to 3 can be combined in a 12X port Each 4X port has an orange and a green LED When a group of three 4X ports is operating as a 12X port the LEDs of the lowest numbered 4X are orange and green and function in the same manner as the LEDs on a 4X port On the second and third ports only the orange LEDs will light meaning that the ports belong to the 12X group In order for a group of three 4X links to be detected as a 12X link all links must be connected simultaneously In order for a group of three 4X links to be detected as a 12X link all links must be connected simultaneously or the board must be turned on with all cables already plugged in 1 10 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Voltaire Switch Management The first port LED for each 12X trunk group of three 4X ports blinks for 7 seconds alternating between the orange and the green if the system detects a link that is lower than 12X However this does not affect link behavior Regardless of the blinking LED the link is still operating and fully functional data can flow over the link 1 4 Voltaire Switch Management Management of Voltaire switch models is based on VoltaireVision Voltaire s Grid Interconnect Management Software VoltaireVision management software is a powerful and com
195. nstalling 3 4 Sweep interval 3 3 Sweep Interval 3 46 Switch Configuration backup 2 17 Switch Configuration Parameters View Only A 3 Switch Graphical Representation 4 11 Switch Information viewing in VFM 3 39 Switch Latency 1 3 Switch Maintenance Tasks 2 14 Switch Management 1 11 Switch Management using the Device Manager 4 38 Switches List in VFM 3 38 Switching Topology 1 8 system name set A 37 system name show A 37 System Requirements Fabric Manager 3 3 T Tables sorting in Device Manager 3 21 3 26 4 15 TCP IP Router 1 5 Telnet 4 10 Telnet session 1 11 1 14 2 2 3 14 A 5 Temperature Information 4 22 Terminal Emulation Configuration 2 7 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc terminal emulation parameters 2 7 Terminal Mode 2 7 Time and Date configuration CLI 2 12 Tool Tips 3 12 4 15 Tool tips in VFM Topology map 3 12 Toolbar in VFM 3 18 Topology map in VFM 3 10 moving icons 3 12 resizing icons 3 12 tooltips 3 12 Topology Table View 3 20 topology log show A 31 U update firmware A 17 update software A 18 User name field 4 43 User settings using the Device Manager 4 42 Users window 4 4 V VDM Main Window 4 6 vendstat A 3 1 Voltaire Switch User Manual Chapter E Index verbosity level 3 3 A 53 version show A 8 A 18 VFM 4 31 Virtual Lanes 1 3 VL arbitration 3 49 3 51 3 52 VLs defi
196. nt listing in the Trap Table and select Clear Event To clear all the events displayed in the Trap Table R Right click the event listing in the Trap Table and select Clear All Setting Event Filters By applying event filters you can limit the types of events that are reported in the Fabric Manager Trap Table To display and set the event filters M Select Fabric Fabric Inspect Settings from the main menu The following dialog box is displayed Fabric Inspect Settings Fitters Sm Everts Log Sm Gid Addr In Service Sm Gid Addr Out Service Sm Multicast Group Created Trap Sm Multicast Group Deleted Trap Sm Error Evert Sm Warning Ewent Sm Info Event Sm TimeOut Event Lid Change Event Sm Lid Change Event d L d d d Sm Port Counters Errors Event Reset All Select All NA Figure 3 14 Event Filters Settings Voltaire Switch User Manual 3 29 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Step 1 3 30 By default all events are reported Uncheck an event to filter it Click Default to the return to the default settings Click Reset All to deselect all events Click Select All to select all events Click Apply to save any changes made in the dialog box Click to return to the previously saved settings To display and set the PM Settings Select Fabric Fabric Inspect Settings from the VFM main menu The following dialog box is displayed Fabric Ins
197. nticceesthsaratetede tatsnsie eaaienciatatan dae tease baaeisaeataes 5 2 Table 6 1 Event Severity and Color Coding Table 6 4 Table B 1 Port Counters Fields DGS Crip tii ace aada haz oda E Tadi aZ haz atati B 2 Table B 2 CVS Summary Report Fields Description sss B 2 Table B 3 P MGounters Tresholds T B 3 Table B 4 Avyarlable Port Counters orero rin esac A A B 7 Table C 5 Alarm Table Detailed Spccihcation sss C 7 Po Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Chapter 1 Introduction In this Chapter This chapter provides a brief introduction to InfiniBand technology and describes Voltaire ISR 9024 ISR 9096 and ISR 9288 switches The following information is included in these pages R About InfiniBand on page 1 1 E InfiniBand System Elements on page 1 3 R ISR 9024 System Overview on page 1 4 R ISR 9096 9288 System Overview on page 1 5 E Voltaire Switch Management Interfaces on page 1 11 P N 399Z00004 2005 Voltaire Inc Chapter 1 Introduction 1 1 About InfiniBand InfiniBand technology is a high performance channel based interconnect architecture that provides increased scalability and reliability for servers and other Internet infrastructure equipment InfiniBand architecture defines the entire stack from the physical to the application layers APIs and fabric management InfiniBand has unique capabilities for direct data placement implemented by RDMA and for OS bypassi
198. nts Data The window displays further data on the event for both sides of the link The Port Info left hand pane lists data for the local port with which the event is associated the Peer info right hand pane lists data for the remote port of the link or node with which the event is associated Voltaire Switch User Manual 3 27 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Parameter Description Node GUID Displays the Global Unique Identification number GUID of the device Platform The switch platform in which the port is installed Module The module board type in which the port is installed Logical Port Additional information if applicable regarding the location of the port Port GUID Displays the Global Unique Identification number GUID of the port IB Port Displays the IB port number LID Displays the port LID Table 3 11 Events Data Fields M Select Go to Node The node with which the event is associated is displayed on the right panel Voltaire Manager File View Fabric Node ri Help E l Subnet hA gt m joje l Fabri Sr a Hja B ivhost 2 ibhost Ura ax P P Port 2 L displayed here x E ka host E ka j La noe Node REV ibhost k Has SM i a A F Node Device ID is Node Vendor ID ibhost Node IP ibhost Node Ports 1 o Node Description a E a Node GUID ibhost m E l Eka
199. o Block ITENGE Ner SMEO 32 OES VendorDiag 48 SI Os be e HG E 49 U Example T T E DO re NG Output ees E ehbian mioarchL gisele E D O o cheese eTo TH END we Chetan SiMe O77 R Oy sicaseme iO meres OM T H ote ic Me ROR Command topology log show Description shows the topology log file Syntax topology log show Command vendstat Description vendor specific MAD generator Syntax vendstat d t timeout_ms lt Gpio iNfo portCause Perormance gt lt lid gt Largs iss General options d increase debug level default 0 t timeout_ms set RPC timeout value per retry in msec default 200 r retries set retries number default 3 Usage formats vendstat G lt lid gt Get 1G 2 IR Darta vendstat N lt lid gt Get IS3 General Info vondo tat 8 E D OE o EOE O ue Bire vendstat P lt lid gt lt port gt Get Port Performance Counters Command vsm log show Description shows Voltaire Subnet Manager Log Syntax vsm log show Command width check Description Checks if 1x ports exist in fabric Syntax width check Example ISR9XXX utilities width check Wevalin E R mouiae LT lever soe RR eT PEG ose Puc OO OS RIO A OOAU SOOO pomt ss 74 Voltaire Switch User Manual A 31 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 32 S 2 cpanel ONO as QIAO Sire 0 o S Oeics lid 10 guid 0008f10400403002 ports 24 lid 11 guid OO08f1040040300e ports 24 Ee Sy epblav
200. ol 2 7 front show A 12 Voltaire Switch User Manual Chapter E Index FRUs 1 7 FRUs ISR9288 1 7 ftp A 34 FTP Configuration using the Device Manager 4 43 FTP server 1 14 2 15 2 16 2 17 4 43 4 44 A 6 A 17 A 18 A 34 A 39 A 40 A 44 configuration using CLI A 39 FTP Server field 4 43 ftp show A 6 A 7 A 13 G Gateway Default A 46 getpathrecord A 23 GID Prefix 3 34 4 24 Go to Node 3 28 Grouping Information 3 42 Guest user type 3 6 4 4 H Hardware configuration viewing 3 42 hardware reset 4 44 HCA 1 3 Heartbeat 4 24 Hierarchical Layout 3 19 High Performance Computing HPC 1 5 High Level Diagnostic Tools 6 6 High Priority table 3 51 High speed interprocessor communication 1 2 Host Channel Adapter 1 3 Host Information viewing in VFM 3 37 Hosts List in VFM 3 36 Hot Swap Mechanism 1 13 HPC 1 5 HQ Life 4 19 HQLife 3 34 6 3 P N 399Z00038 2005 Voltaire Inc HTTP session 1 11 1 14 HyperTerminal client 2 7 I O sharing 1 2 IB 1 7 Icons in VFM subnet tree 3 17 Icons in VFM Topology map 3 11 In band Management 1 12 InfiniBand 1 5 Infiniband port information viewing 4 18 physical state 4 19 InfiniBand subnet management 1 11 Information Views 3 36 interface IB A 35 interface INT A 35 interface LOCAL A 35 Internal sMB interface default IP address 1 15 Internet Browser 4 2 IP address of Internal sMB inter
201. om of the dialog box select the measurement units to display the counters in MB sec or GB sec You can also display the counters for TX or RX separately Click Start To stop counting click Stop Fabric Diagnostic using the Event Log The Events Log provides information on the events in the system The Event Log records InfiniBand specific events traps and SM internal events in a file format They are then reported to the user as management via the VFM or the CLI Event Log file To display the contents of the Events Log Step 1 Select Fabric Fabric Inspect Event Log from the VFM main menu A window is displayed with the event log You can copy and paste the information to a text file e g using MS Word OR From the CLI Utilities Menu use the error log show command to display the error log Step 2 Inspect the file for event monitoring or to discover whether there is a problem in the fabric The following is an example of an error log reported through the CLI menu ISR9OXXX utilities event log show Jul 13 09 49 55 0 vsm 414 CREATE MC GROUP MGID EET AO E ae fete te OOOO CGA B G de eee SIG VS Ooo SoG Ol vem TA Ee CHANGE Entering oM Discovering State Oui Ss Oo 49 So0 vem 440 eM slave CHANGE ENTE ring oM Master State RE RI OSS E Sion Ol Sug PAM amp OOS ac IOC Sic 0 L S IX pore was found Pore GUED SIE HE EEE R TB Porte 22 Peer at OO0O08f104003f074e port 24 Port CUID SElO4003rT074e IB Porta E Ue ILS OSS As SS OO
202. on events may be generated in the fabric and sent to the SM by fabric elements The SM reports those events as well The event mechanism can do the following actions with each event a Log the event in the event log b Issue a trap to the GUI session c Ifthe event corresponds to an alarm it is also sent to the current alarm mechanism The GUI Color coding is defined according to traps and events severity as described below GUI Color Event Severit Coding y Description Invalid link Duplicate or conflicting ports or path Broken link Illegal connections between two sLB ports Critical Major Critical means that the system or a system component fails to operate Yellow Warning Minor Warning minor reflects a problem in the fabric but does not prevent its operation A warning 1s asserted when an event is exceeding a pre defined threshold Normal Information Notification provided to the user of normal operating state or a normal system event Table 6 1 Event Severity and Color Coding Table 6 4 P N 399Z00038 2005 Voltaire Inc Complete subnet reconfiguration Create Delete Multicast group Applied routing scheme Port State Change Voltaire Switch User Manual The user may filter the events using the GUI and CLI The filtering policy determines for each event whether it is logged and whether a trap is generated 6 1 2 When is Fabric Diagnostic Performed Diagnostic is recommen
203. on to the IB interface Refer to First Time Configuration on page 2 6 for further information CLI Command Modes The CLI is password protected There are several CLI operation levels each corresponding to a different level of interface operations Each command mode requires a different password The following are the command modes E Exec mode Allows view only access of switch configuration parameters E Privileged mode Allows the user to configure basic parameters such as date time and reload reset switch and to perform software and firmware updates R Config mode Allows the user to configure all configuration parameters E Debug mode Intended for use by Voltaire personnel only Prerequisite Tasks Before configuring system parameters make sure you have completed the hardware installation according to the relevant Installation Guide Information about hardware installation of Voltaire switches can be found in the JSR 9024 Installation Manual and in the ISR 9XXX Installation Manual the latter includes the ISR 9096 and the ISR 9288 Switches Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Getting Started with the CLI 2 1 4 Keyboard Usage in CLI Operation The CLI supports the use of the following special keys for the described functions Table 2 1 CLI Key Functions Key Tab Backspace Return Left Arrow Right Arrow Up Arrow or Ctrl P Down Arrow or Ctrl N Voltaire
204. onfigures user authorization passwords Config FTP Sets the FTP Server parameters Advanced Future Launch Fabric Manager Starts a Fabric Manager session Refer to Chapter 3 of this manual for further information regarding the Fabric Manager Error Log Open a window that displays the error log Event Log Open a window that displays the event log Reset Restarts the CPU Help About Displays the Voltaire Device Manager software version and the switch software and hardware versions Table 4 3 Main Menu Options 4 8 o Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc The Device Manager Main Window 4 3 3 Device Tree The ISR9288 9096 VDM Device tree displays the objects in the managed device in a hierarchical format This includes sFB boards line boards sMB sPSU sFU sCTRL boards sRBDs IO drawers and installed router modules Figure 4 5 shows the Device tree The ISR 9288 was used here as an example 172 25 22 39 Figure 4 6 Device Tree You can expand collapse the navigation tree to display the various components of the switch You can show hide the device tree from view by selecting View View Tree from the Main Menu Voltaire Switch User Manual 4 9 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 3 4 Device Toolbar The Device toolbar provides fast access to device information and to some device actions such as Reset and Telnet The bu
205. onnected to ISR 9096 9288 switches the Information area can be used to zoom in on specific boards Click on a board in the device graphical representation front or rear panel to display a zoomed view Refer to page 4 24 for further information Trap Table You can show or hide the Trap Table from view by selecting View View Events from the Main Menu Resizing the Trap Table You can resize the Online Events Log by positioning the cursor on the top border line of the Events Log The cursor changes into a double headed arrow with two lines in between drag the cursor up or down to increase or decrease the size of the Trap Table area Refer to page 6 13 for further information on the various types of events and alarms Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 4 4 4 4 1 4 4 2 4 4 3 Voltaire Device Manager Operating Conventions Voltaire Device Manager Operating Conventions The following are specific operation conventions that apply to the Voltaire Device Manager Sorting Tables You can sort the data in the Voltaire Device Manager tables by clicking the column headers An arrow indicates the sort order ascending or descending Use Ctrl click for a secondary sort order Changing Table Column Sequence You can adjust the sequence of table columns by dragging the column header to a new location Instructional Tool Tips You can obtain helpful information Voltaire Device Manager operation by
206. parameters using the steps below Command Description version show Displays the version of the current switch software ftp show Optional Displays the FTP client configuration fast interface show Displays IP interface address and configuration clock show Displays the system clock Voltaire Switch User Manual 2 13 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI 2 3 2 3 1 S 2 3 2 CLI Maintenance Commands This section explains how to configure switch parameters and contains information on the following topics M Prerequisites on page 2 14 R Updating Software on page 2 14 E Backing up the Switch Configuration on page 2 17 Prerequisites Before performing any switch maintenance tasks make sure you have configured system parameters as described in First Time Configuration on page 2 6 NOTE Certain configuration tasks such as identifying a location from which to download software are optional and may not have been performed during initial configuration You may perform these tasks at any time via the CLI or the GUI Where necessary this chapter will identify the relevant tasks and commands Updating Software The same software image can be used to manage either an ISR9096 system or ISR 9288 system The software automatically detects the system type and executes the suitable device management operations The switch is designed to run on a continuous basis wit
207. pect Settings Scope Sample Interval sec Thresholds Symbol Err Counter Err Recovery Cntr Downed Cntr Port Rew Err Rev Rem Phys Err d Rev Swich Relay Err Amit Discard amit Constraint Err All 4180 200 ja Thresholds type Rev Constraint Err Local Link Integrity Err Excess But Over run Err VIS Dropped C Xmit Data L Rev Data _ Xtnit Pkts 4 Rev Pkts Reset Counters Figure 3 15 PM Settings By default PM counters are monitored for all devices in the fabric Scope field is set to All To change these settings select one of the following values from the Scope field Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Step 2 Managing Alarms and Events R Hosts Only PM data is only collected for HCAs in the fabric R Switches only PM data is only collected for switches in the fabric You can also disable selected PM counters by unchecking the box displayed next to the counter In the Sample Interval field set the frequency at which the PM data is collected from the counters range 60 seconds to 65535 seconds NOTE The Thresholds Type in the current software release is always set to Per sample i e the Fabric Manager checks for errors in each sampling if the threshold is passed between samples interval The counter is reset when it reaches its maximum value as defined in the InfiniBand specification Step 3 Disable the Thre
208. play krot Of avallable commands SE E K conna n Sr usage E Ng broadcast fast set Set the broadcast address end Exit Ton Eo aena we ON cian toile cua mode ny Ee on o e on e EE On OE close telnet sess Lony eNe l AeL Connect roA exit Exit to previous menu ip address fast set Set interface Ip address ip address fast show Show interface Ip address and CONE LUT aE EON mac address fast show Show interface Mac address management interface set Set Management Interface to be PCOM GE OA Send echo messages The following details the commands available in the Fast Interface menu Command Description Syntax Command Description Syntax Command Description Syntax Example A 44 broadcast fast set Sets the IP address range subnet that the switch broadcasts to through the fast interface broadcast fast set ip address ip address fast set Sets the IP address and subnet mask of the fast interface ip address fast set ip address netmask ip address fast show Returns the IP address of the Fast interface its subnet mask and the IP subnet to which the Fast interface broadcasts ip address fast show ETEO P TTT e K H R SR EE IR I a A Bi a Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Example Command Description Syntax Voltaire Switch User Manual Configuration Config Mode mac address fast show Show inte
209. porarily freezes all applications until the sFB 1s replaced If there is another sMB installed in the system the latter takes over the management of the system Voltaire Switch User Manual 1 13 P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction 1 4 3 Management Interfaces Voltaire switches have the following management interfaces R The Fast interface 10 100 Ethernet provides an interface to the CLI via a Telnet or an SSH session and also provides an interface to third party managers via SNMP The SNMP configuration settings can be made using the CLI as detailed on page A 65 or via the Device Manager as detailed on page 4 40 The Fast interface also provides the interface to a remote FTP server for downloading new switch software versions and for uploading backup files FTP configuration settings are made using the CLI and are detailed on page A 37 The Fast interface also provides an interface to the Voltaire Fabric Manager and Voltaire Device Manager GUI applications via HTTP session Out of band interface R The Serial interface RS 232 provides an interface to the CLI Out of band interface M IPoIB offers the same functionality as the Fast Ethernet interface see above NOTE Ethernet and IPoIB cannot be used simultaneously IPoIB is a native InfiniBand interface of the system No Ethernet is required and cabling is simplified Any machine connected to the IB system can connect to the ISR 9096 9288
210. prehensive application suite that simplifies the management and proactively maximizes the performance and availability of InfiniBand enabled servers networks and storage grid environments VoltaireVision manages all the fabric and networking resources in the grid including the ability to automatically diagnose optimally configure provision monitor traffic partition and fail over Voltaire switches provide the following levels of management E Command Line Interface CLI described in Chapter 2 of this manual The CLI can be connected via a serial connection or a Telnet SSH session W Voltaire Fabric Manager described in Chapter 3 of this manual Provides a fabric wide view of the switch fabric with an advanced Graphic User Interface GUI The Fabric Manager can be connected via HTTP session R Voltaire Device Manager described in Chapter 4 of this manual Provides management of a switch with an advanced Graphic User Interface GUI The Device Manager can be connected via HTTP session The system collects performance statistics and environmental monitoring data and supports an industry standard Management Information Base MIB The Subnet Manager software automatically discovers the fabric topology and configures the hosts and switches for ease of operation Voltaire switches can also be managed with existing management systems using standard protocols like SNMP or InfiniBand subnet management Out of band management is perfor
211. rature is above WARNING threshold on the sFU 8 If the sFU 8 is removed there are no other fault LEDs NOTE state before the fan speed is set back to normal speed and before the temperature x The temperature of all problematic modules must drop below the RELAX threshold LED turns off Whenever a temperature of a specific module goes beyond the WARNING threshold the system generates a trap reporting the high temperature on the problematic module If the temperature of the module further exceeds the alarm threshold for modules where it is defined the system generates a trap reporting the more severe temperature Once the temperature of that module goes back below the relax threshold the system generates a trap reporting the temperature is back to normal The threshold values are Celsius e em sam T enma mem sB osc 70 C fc 149F_ 58 F oF 149 F perc tore fF 212 sru asc soc iF Table 4 13 Modules Temperatures Threshold Values The table of temperatures for each module can be viewed via the VDM See Temperature Information Section 4 5 4 on page 4 22 Voltaire Switch User Manual 4 37 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 6 7 Router Module Information Optional Router Blades Drawers SRBD can be installed with IPR TCP IP router and FCR Fibre Channel Router modules Each Router Blade Drawer can contain up to three router modules
212. regarding the module Port Number The part number if applicable Hardware Version The hardware version of the module if applicable Serial Number The module serial number Production Date The date on which the module was manufactured Table 4 9 Detailed Module Window Fields Voltaire Switch User Manual 4 25 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 6 Viewing Module Information Using the Device Manager you can view graphical information views of the module installed in a switch by selecting the module in the switch graphical representation displayed in the Device Manager main window for certain modules you can also view this information by selecting the module in the tree view of the Voltaire Device Manager main window The LEDs displayed in the graphical information views reflect the actual state of the LEDs on the module For a description of the ISR 9096 9288 module LEDs refer to the ISR 9X XX Installation Manual PRT The windows and views described in this section are mostly relevant to the ISR 9096 9288 switches The information described in this section varies according to the module selected as described below R sLB Line Board Information on page 4 27 Port Information on page 4 17 CPU Information on page 4 20 Temperature Information on page 4 22 Local SM Information on page 4 22 4 26 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewin
213. remote logger delete remote logger set remote logger show remote logger delete remote logger set ip remote logger show route AONE sm sm snmp snmp Spec Mee lem ec Ir system name show watchdog mode set system name set name system name show watchdog mode set enable disable watchdog mode show watchdog mode show The following details the commands available in Config mode Command Description Syntax Command Description Syntax Command Description Syntax cluster Change to Cluster configuration mode which 1s used to assign the IP of router modules installed in the switch See page A 38 for a list of the commands available in this menu Cluster factory default Reverts the switch to its factory default parameter settings and reloads the system factory default ftp Type this command to access the CLI FTP configuration menu which is used to define the settings of the active FTP server Among other uses the active FTP server is used for download of new switch software versions See page A 39 for a list of the commands available in this menu ftp Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Voltaire Switch User Manual Configuration Config Mode interface IB
214. rence Cluster Menu The following are the commands in the Cluster menu oE OOK COE Le Ol US reo fe Dispio hestr siveallelole Connects 2 command Daselay conmnmand sw uS g Str Ing Conn e eE STR GH eo he end Exit to Mogin menu irom any OCNer mode via console connection or close Ke lmset session via tcelner connection exit Exit to previous menu 10 ip address set Set interface Ip address for TO IO SNOW Show existing IOs The following details the commands available in the Cluster Menu Command connect Description Establishes a CLI session with a router module installed in an ISR 9096 9288 switch Syntax connect slot number position number Example connect 12 2 This example shows a CLI connection with a router module installed in the sRBD installed in slot 12 in the rear panel of the ISR 9288 in the middle router slot Pore The IP address of the router module must be on the same subnet as the sMB Z card with which the CLI session is established Command 10 1p address set Description Configures the IP address of router module installed in the switch Syntax io ip address set slot number position number ip address Example io ip address set 12 2 172 25 3 38 This command sets the IP address of the router module installed in the center position of the sRBD which installed in slot 12 of the switch rear panel Command 10 show Description Shows existing I Os Syntax io show A 38 Voltaire Swi
215. resent in the switch SM Mode If an SM is present in the switch displays whether the SM is enabled or disabled Table 4 8 Local SM Information Parameters Obtaining Detailed Module Information VPD vital product data is information about a device that is stored on the device itself This information allows the device to be administered at a system or network level and can be viewed as read only fields either within each individual module information window or by accessing the Detailed Module Information window Typical VPD module information is displayed in the GUI as a read only field and includes a product model number a unique serial number product release level maintenance level and other information specific to the device type Vital product Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Viewing Switch Information data can also include user defined information such as the building and department location of the device The collection and use of vital product data allows the switch system to be understood and service provided more quickly Detailed VPD module information can be obtained by clicking the Advanced button The detailed Module Information Window is displayed Detailed Module Information Module Marne Part Number Serial Mumber Production Date Figure 4 17 Detailed Module Information This window displays the following additional data Module Name More textual information
216. rface MAC address mac address fast show SW SECON ih hast mac address rast show OCR Oe he Ores seal management interface set Sets the current interface to be the switch management interface depending upon which level you are in When under config interface fast menu the current management interface will be set to Fast When under config interface IB menu the current management interface will be set to InfiniBand management interface set A 45 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference Route Menu In the Route menu you configure the default gateway which the switch uses to access remote networks The following are the commands in the Route menu ES 0 G6 OS emer e OMS sip 8 2 Display T Ot aver laole conmands 2 eonmanne Display command s usage string default gw delete Delete the default gw ip address default gw fast set Set the default gw ip address for fast interface default gw show Show the default Gateway end Exit to login menu from any other mode yia console connecrcion or close Ee lL session via tcelner connection exit Exit to previous menu The following details the commands available in the Route menu Command default gw delete Description Deletes the IP address of the default gateway currently set in the switch Syntax default gw delete Command default gw fast set Description Sets the IP address of the switch default gateway Syntax default gw fast se
217. rformed via the CLI Refer to Subnet Manager Menu on page A 47 for more information 3 10 Fabric Inspect Fabric Inspect is used for diagnostics purposes and is fully detailed and illustrated in the Fabric Diagnostic Chapter on page 6 1 3 46 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Cluster Monitor 3 11 Cluster Monitor Use the Cluster Monitor dialog box to set the path of the Ganglia Monitoring tool Step 1 Select Fabric Cluster Config from the main menu The Cluster Monitor dialog box is displayed Cluster monitor Figure 3 28 Cluster Monitor Dialog Box Step 2 Enter the string as follows 1 Cluster Head Node IP Address 2 Ganglia tool 3 Cluster Name 4 Name of the Host Step 3 Click Update and close the window Step 4 On the topology map right click the host and select Monitor The browser window opens with the Ganglia link directed to the specific host displaying a detailed host report as shown in the example below Voltaire Switch User Manual 3 47 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM Example http 172 25 2 175 ganglia m load_one amp r hour amp s descending amp c Support Cluster amp h HOST amp sh 1 amp he 4 In the example below the Host is called hydra3 voltaire com E Ganglia Host Report Microsoft Internet Explorer SEE File Edit View Favorites Tools Help ae S F 4 X l J lt c we z Bs E
218. ric Manager Operation The Topology Map displays the following types of icons depending on the configuration of the fabric Icons are color coded to indicate their alarm status Table 3 3 describes the Topology Map icons tem pean This icon is an example representing a managed Voltaire InfiniBand Switch grouping Voltaire modules in a single icon Represents a Host Represents a switch A number inside the icon indicates the type of switch e g 288 indicates an ISR 9288 switch 96 indicates an ISR 9096 switch Represents a Voltaire Fibre Channel Router FCR module This icon can be displayed on a device icon when displayed it indicates that the device is a Voltaire device this can either be a device manufactured by Voltaire such as an ISR 9288 switch for example or a device with the Vendor ID Voltaire set in the device configuration This icon shows that the device is managed 1 e a Subnet Manager SM is installed in the device Connecting lines between icons represent the physical connections between devices A thin line indicates a single connection while lines of double thickness indicate a multiple connection between devices Table 3 3 Topology Map Icons P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM z The name of the device is displayed beneath each icon If there is no name configured for the device its GUID number is displayed The device name is set via
219. rmation Information 4 30 sFB Information Window Description Each LED corresponds to a group of internal links LEDs represent the state of the link between the fabric board and the appropriate line The blue Hot Swap LED 1s active when the sFB board is ready to be removed The board s VPD Vital Product Data is also displayed Additional VPD can be obtained by clicking the Advanced Button refer to Obtaining Detailed Module Information on page 4 24 for further detail Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 4 6 3 Viewing Module Information SMB Board Information The ISR 9096 9288 chassis supports one or two redundant hot swappable Management Boards SMB for fabric management VFM and chassis management when two sMB boards are installed they support failover capabilities in the event of failure to one of the boards None one or two sMB boards can be installed on the front panel of the ISR 9096 9288 chassis The sMB board provides two RS 232 ports one 12C port for debug purposes by authorized personnel only the second port CLI can be used to establish a CLI connection locally to the chassis To view the Management Board information M Click the desired sMB on the Device Tree The Management Board Information Window appears b Hot Swap Power b Info SM Active Chassis Mgt Serial Murer Hardware Version Figure 4 22 SMB Information SMB Information Window D
220. s Fabric Board sFB The ISR 9096 9288 supports up to four redundant hot swappable Fabric boards which support the switching Fat Tree topology e The ISR 9096 Fabric Board is identified as sFB 4 supporting connectivity of up to 4 sLB sRBD modules e The ISR 9288 Fabric Board is identified as sFB 12 supporting connectivity of up to 12 sLB sRBD modules Line Board sLB Hot swappable sLB 24 board with 24 4X IB ports and sLB 8 12 with 8 12X IB ports e The ISR 9096 can host up to 4 sLB 24 or sLB 8 12 or any combination thereof e The ISR 9288 can host up to 12 sLB 24 or sLB 8 12 or any combination thereof Management Board sMB One or two redundant hot swappable Management Boards Router Blade Drawer SRBD A mechanical and electrical interface with up to three IPR and or FCR Router Modules e The ISR 9096 enclosure can contain up to 4 hot swappable sRBDs e The ISR 9288 enclosure can contain up to 11 hot swappable sRBDs Power Supplies SsPSU e The ISR 9096 supports up to four redundant hot swappable Power Supplies with N 1 or N N redundancy e The ISR 9288 supports up to five redundant hot swappable Power Supplies with N 1 or N N redundancy Fan Units sFU e The ISR 9288 only accommodates a vertical fan unit with 4 fans sFU 4 e The ISR 9288 9096 accommodates a horizontal fan unit with 8 fans sFU 8 Voltaire Switch User Manual 1 7 P N 399Z00038 2005 Voltaire Inc Chapter 1 Introduction 1 3 2 1 8 ISR 9096 9288 Sw
221. s InfiniBand employs Virtual Lanes VL which are separate logical communication links sharing a single physical link Each link can support up to 15 standard VLs and one management lane VL 15 Two mechanisms using Service Levels SLs and VLs are provided to implement Quality of Service QoS related services One is SL to VL mapping the other is data VL arbitration Both are described in detail below Virtual lanes VLs provide a means to implement multiple logical flows over a single physical link Link level flow control can be applied to one lane without affecting the others There are two types of VLs one for normal traffic called a data VL and one reserved for subnet management traffic The subnet management traffic VL is VL15 All other VLs are for normal traffic Service Level SL is an indication of the service class of the packet SLs provide differentiated services improving fabric utilization and avoiding deadlock To avoid having the user configure each port in the subnet with all the QoS parameters templates have been defined and their configuration is sufficient for the configuration of the whole subnet When configuring a subnet port the QoS parameters are derived from the templates according to the number of VLs supported by the port SL to VL Mapping As a packet traverses the subnet its service level SL defines the desired QoS level Because each link along a path can implement a different number of VLs e
222. s on page 6 8 Configuring the PortCounters csv file via the CLI Configuring PM Using the Fabric Manager R Diagnostic Tools on page 6 11 Displaying and Checking Port Counters using the GUI Fabric Diagnostic using the Event Log Fabric Diagnostic Using Current Alarm Fabric Diagnostic Using Fabric Statistics portcounters csv file M Additional Tools and Examples on page 6 18 Detecting and Troubleshooting 1X Cable Width Problems Identifying a Bad Line Board or Fabric Board Port Identifying a Bad Port Some CLI Diagnostic Tools Managing Alarms and Events on page 3 22 provides information on using the Fabric Manager for initial alarm detection and handling Voltaire Switch User Manual 6 1 P N 399Z00004 2005 Voltaire Inc 6 2 Setting Partitioning Appendix B lists the available Port Counters describes the PortCounter Fields and Summary report Appendix C Defines the trap structure and lists the Supported Traps and Events in the Alarm Table Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic 6 1 Overview Voltaire s Fabric Manager enables fabric connectivity debugging using the built in Performance Manager PM PM has two major capabilities E Port Counters Monitoring and Report The PM generates a periodic port counters report file in CSV format that can be loaded to Excel and further analyzed by the user It also monitors port counters errors and report
223. s every port that passed its error threshold limit as configured by the user E Event Logging Creates an event log file for both IB traps and SM internal events The user may filter the events using the GUI and CLI The filtering policy determines for each event whether it is logged and whether a trap is generated The fabric diagnostic tool is essential to identify any problem ports and node connectivity problems prior to running application as well as during standard operation The following diagram illustrates the Diagnostic Features FABRIC TOOLS Traps Events Current Alarms Mechanism DIAGNOSTIC Log File Alarm Trap Portcounters csv OUTPUTS Event Table Table File Log Error VFM CLI VDM Browser CLIVFM VFM Figure 6 1 Diagnostic Block Diagram Voltaire Switch User Manual 6 3 P N 399Z00038 2005 Voltaire Inc Overview 6 1 1 The Event Notification Mechanism Fabric related events can be generated by both the PM Performance Monitor and by the SM Subnet Manager The PM periodically scans the error counters and of all IB elements in the fabric and reports if a counter exceeds its threshold The SM monitors the fabric detects configuration changes and dynamically configures the new elements and new routes in the fabric The SM can detect fabric errors warnings informative events and report them Both the PM and the SM generate events and report them to the event notification mechanism In additi
224. s the Fast Ethernet gateway address of the switch Gateway Table 4 14 IP Setting Parameters Click OK or Cancel to close the screen Remote Management Setting The Remote Management Settings dialog box enables you to set SNMP parameters to interface with remote management systems and to provide general details of the remote management system To Configure Remote Management Settings Select Device Remote Management Settings from the Main Menu The Remote Management Settings dialog box opens as displayed in Figure 4 31 Remote Management Settings Remote Management Settings MMP General ieee 0 0 0 0 E 0 0 0 0 0 0 0 0 0 0 0 0 Figure 4 31 Remote Manager Settings SNMP Tab Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Using the Voltaire Device Manager for Switch Management Step 2 Click the SNMP tab Enter the IP address The relevant check box will be marked automatically Change or reactivate the existing IP address Click the relevant SNMP Manager check box and then type in the relevant IP address Step 3 Click OK Step 4 Click the General tab Remote Management Settings x Remote Management Settings Device Mame snag 0 SNMP VWersior Vottaire SM agent 5 00 Contact Location E Figure 4 32 Remote Manager Settings General tab Step 6 Fill in the following fields Device Name Describes the device name in a read only field SNMP Version Displays the
225. save the new entry Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Using the Voltaire Device Manager for Switch Management 4 7 4 FTP Configuration The FTP configured for use with the switch is used for the following M Upgrading software versions W Uploading log files to the remote server R Uploading downloading the repository file To Configure FTP Settings Step 1 Select Device Config FTP from the Main Menu The Config FTP window opens as displayed in Figure 4 34 Config FIP Contig FTP FTP server I User name Password ee Remote path Get Configuration store Configuration Figure 4 34 Configuring FTP Step 2 To configure the FTP server fill in the configurable fields as follows FTP Server The FTP Server IP Address User name The User Name of the FTP holder Password FTP Server Password Remote Path Remote path processing Step 3 Click Get Configuration to download the switch configuration file also called repository file from the ftp server Step 4 To upload a file that stores the switch configuration for backup click Store Voltaire Switch User Manual 4 43 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM Configuration the switch configuration file is stored on the configured FTP Server Step 5 Click OK and Apply to save the new entry and close the window 4 8 Reset Operations The Voltaire Device Manager has several reset options
226. sequence to recall successively more recent commands 2 3 P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI Key Ctrl A Ctrl B Ctrl D Ctrl E Ctrl F Ctrl K Ctrl T Ctrl W Ctrl Y Esc B Esc C Esc D Esc F Esc L Function Moves the cursor to the beginning of the line Note this button combo does not work when the session 19 established with Minicom Moves the cursor back one character Deletes the character at the cursor Moves the cursor to the end of the command line Moves the cursor forward one character Deletes all characters from the cursor to the end of the command line Transposes the character to the left of the cursor with the character located at the cursor Deletes the word to the left of the cursor Recalls the most recent entry in the delete buffer The delete buffer contains the last ten 1tems you have deleted or cut Ctrl Y can be used in conjunction with Esc Y Moves the cursor from the middle of the word to the beginning Capitalizes the character on which the cursor is pointing and moves it to the end of the word Deletes from the cursor to the end of the word Moves the cursor forward one word Changes the word to lowercase at the cursor to the end of the word 2 4 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Getting Started with the CLI 2 1 5 CLI Command Structure 2 1 6 2 1 7 All CLI
227. serial console or a Telnet terminal Update the sMB software by activating the software upgrade process from the CLI using the Update Software command The upgrade process installs the new software image onto the sMB Flash memory and reboots Upon booting the software automatically detects the firmware version on the sFB 12 unit connected to the sMB If it detects an older version it is automatically upgraded to the new revision Note When performing an automatic firmware upgrade as described above the system issues a request to power cycle the ISR 9X XX chassis This request appears on all terminals On the next boot invoke a firmware upgrade for the entire chassis by using the CLI Update Firmware command Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc CLI Maintenance Commands ll Upgrade Process for Redundant sMB Configuration When a switch configuration includes a second sMB upgrade its software by performing the following steps To upgrade the ISR 9096 9288 with two sMBs installed Step 1 Make sure only one sMB is installed in the chassis If two sMBs are installed remove the second sMB Step 2 Follow the steps described in the previous section Update Procedure for a SINGLE sMB installed in the Switch Step 3 Power down the system and replace the first SMB module with the second sMB Step 4 Follow the steps described in the previous section Update Procedure for a SINGLE sMB installed in the Swi
228. sholds you don t want to have checked Step 4 Click Reset to reset the PM counters in all fabric devices Step 3 Click Apply to save PM configuration Note Data entered appear in red until clicking Apply Step 5 l l l Click J to return to the previously saved settings Refer to Configuring PM Using the Fabric Manager Diagnostic Section 6 2 2 for further information on PM Settings Voltaire Switch User Manual 3 31 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 6 3 6 1 3 32 Viewing Properties Data The Node Host Link Properties pane of the main window displays various configuration data relevant to the object selected in the topology map or in the Node Host Link Information pane This includes the following information M Node Properties R Port Properties R Link Properties Node Properties The following information is displayed when selecting a node host or switch in the main window Mode Device ID S407 Node Vendor ID Mode IF Mode Ports Mode Description ISR 9266 Node GUID Ox 00081104 0040068 Node Name 0008t104004006f8 Figure 3 16 Node Properties Table 3 12 describes the parameters displayed for the node Parameter Description Displays the hardware version of the device Has SM Displays whether or not a Subnet Manager 1s installed in the node Node Device ID Displays the identification number of the device Node Vendor ID Displays the nam
229. sing the following commands alarm clear Clear Current alarms entry alarm clear all Clear all Current alarms entry Voltaire Switch User Manual 6 15 P N 399Z00038 2005 Voltaire Inc Diagnostic Tools 6 3 4 Fabric Diagnostic Using Fabric Statistics portcounters csv file Using the Fabric Manager you can save a file in CSV format that lists a wide range of statistics for all active nodes and their active ports in the fabric This includes information such as Node name InfiniBand IPOIB IP address Chassis component PM data failure reports and a detailed port counters summary as shown in the example below Port Counters Summary Alert Ports 8 Total Switches 22 Switch Ports 8 Switch Alert Ports 7 Total HCAs 1 HCA Ports 1 HCA Alert Ports The csv file also displays the port s peer node showing to which node and port it 1s connected Each active node port information is presented in a single line per port The example below shows a CSV report LID IBPort PortGUID NodeType Width State MTU NumVLs HOQlife 720 12 0008f104003f0670 Switch 4X ACTIVE 2048 VLO VL7 13 PlatformType ModuleType Modulelndex Port Name NodelP DevicelD ISR 9288 sLB 24 5 internal 12 ISR9288 0061b 192 168 60 200 5a09 MLID JoinedGroups SymErr LinkRecovers LinkDowned RevErr RcvRemotePhys RcevSwitchRelay 0000 0 0 0 0 0 0 17575 MLID JoinedGroups SymErr LinkRecovers LinkDowned ReveErr RcvRemotePhys RcevSwitchRelay 0000 0 0 0 0 0 0 17575 Xmt
230. software version of the SNMP in a read only field Contact Details of the contact of the device Location The location of the device Step 7 Click Apply to save parameter changes and OK to save parameter changes and exit the dialog box Voltaire Switch User Manul S 4 41 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 7 3 Step 1 Step 2 Step 3 Step 4 Step 5 4 42 User Settings The Voltaire Device Manager allows you to configure three types of user authorization E The Administrator level enables you to change user password configuration to export or import configurations and more The default password is 123456 E The Manager level enables you to set the parameters The default password is voltaire R The Guest level enables you to view configurations only The default password is voltaire To Configure Authorization Passwords Select Device Users from the Main Menu The Users window opens as displayed in Figure 4 33 Users User authorization Administrator TTT ae Old password TKK KL Hew password FEAR XX Mew paszaword retype C Ce Figure 4 33 Configuring User Authorization Select the user authorization to assign to the user Guest Manager Administrator from the Authorization field Type in old user password Type in the new password and confirm it by typing it again in the New Password retype field Click OK and Apply to
231. ssing 1 2 Out of band Management 1 12 P Pan interactor button 3 19 Parity 2 7 Partitioning 5 1 Password field 4 43 password update A 13 Passwords Default 2 6 Passwords configuration CLI 2 13 Persistency 1 13 persistency state show A 14 ping A 7 A 14 port connection matrix displaying 6 20 Port Counters in VFM 6 11 Port GUID 3 34 Port information Infiniband See Infiniband port information Port Information viewing in VFM 3 40 Port LID 3 34 Port MTU 4 19 Port Properties 3 33 Ports bad 6 22 Power Supplies 1 7 Priority SM 3 45 Privileged mode CLI A 9 Procedures and Tools used for Problem Identification 6 6 Prompts in CLI 2 5 Voltaire Switch User Manual Chapter E Index QoS Arbitration setting 3 51 QoS Arbitration 3 51 QoS Menu CLI A 61 QoS configuration 3 49 qos apply configuration set A 62 qos configuration enable set A 62 qos info show A 14 qos sl vl set A 62 qos sl vl show A 14 qos vl high arbitration set A 62 qos vl high arbitration show A 15 qos vl high arbitration limit set A 62 qos vl low arbitration set A 63 R RAM requirement VFM 3 3 RDMA 1 2 redundancy 1 7 redundancy mechanism 1 13 redundant sMB 1 13 Remote boot 1 2 remote logger delete A 35 Remote Management Setting using the Device Manager 4 40 Remote Path field 4 43 remote logger set A 36 remote logger show A 36 replication status set A 16 replication status show A 16
232. stinct selection options Logical object which represents all the servers HCAs in the fabric Click on this object to display the Hosts List described on page 3 36 Expand this object to display the hosts in the fabric in the subnet tree Underneath the Hosts logical object the subnet tree displays the hosts in the fabric Click on a specific host to display the Host Information window described on page 3 37 Logical object which represents all the switches in the fabric Click on this object to display the Switches Lists described on page 3 37 Expand this object to display in the subnet tree the switches in the fabric Underneath the Switches logical object the subnet tree displays the switches in the fabric Click on a specific switch to display the Switch SLB info window described on page 3 39 This object is only displayed underneath ISR 9096 9288 switches It represents a Line board in the ISR 9096 9288 Click this object to display the Switch SLB info window described on page 3 39 This object is only displayed underneath ISR 9096 9288 switches It represents a Router Blade Drawer which has router modules also referred to as I O modules inserted in the sRBD slots up to three modules per sRBD The user also has the ability to see in which slot they are installed P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 4 8 Expanding and Collapsing Objects in the Subnet Tree In
233. t ip address Command default gw show Description Shows the default Gateway of the fast Ethernet Syntax default gw show B46 05 2 2 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Configuration Config Mode Subnet Manager Menu In the Subnet Manager menu you configure the parameters of the Subnet Manager in the switch The following are the commands in the Subnet Manager menu TORIO SUE CCOn kro Sim te 32 i Display lise sor vavat lab iS commands 2 command Display Commands Usage Srering bad ports clear set Clear Bac POCUS bad ports mode set Enable or disable Bad Ports Mode bad pores mode show Show Bad Ports Mode end EIE TO Login Menn Tron any L C moce via console connection or Close celnertr session via telner connection exit Exit to previous menu manual node names show Show the nodes in repository mc Change to mc mode partitioning enable set Fo a E E EE OEO partitioning enable show Pisplay che stac Lor Ce ne Calica TE pkey celere Wie lere Sieen Wel odio OE 17 pkey high show Show the high part of pkey table pkey low show Show the low part of pkey table pkey port gord set SO POCEN Guid wih Pattie rom ke ya labo pkey set Set key in Partition Key Table pkey update Update Partition Key Table pm Change to pm mode gos CHange eo Oos COn Ura e nom mode show active nodes Show Active Nodes sm Into algoritm Sek SmI NEO da lor NMA Set sm info hog set Set head of queue omaa Oa hme sere set
234. t evn eaten ene E EE E etre eee meri ne eran 3 16 t Keen 3 18 Topology Table Viemme ee oe ea ee eee 3 20 Managing Alarms and Events xcccsccacacsivsdeaxsdoeecsdaceorstundseckaausdedanis se eecssectensartie 3 22 The Current Alarms Wind Wiereonsssrenriio eae R N 3 22 Th Trap Tableand Event E 3 25 Event Ea ren e E E E NS 3 26 Viewine Properes TS 2 A A os 3 32 Node Properties sisccnarnasscasnnbacsnanoainsasegtas seas nauscasaseacinasnainasaogesdassniacieannaleaiao 3 32 OE Properes zespri n raed 3 33 EY 2 ST n RTT 3 35 POTATO V IENS enir setetaatebtindencasodmahoaaiuet getednededieedennuastuushas 3 36 Voltaire Switch User Manual P N 399200038 2005 Voltaire Inc Contents E Leco HO S a enn ee OnE ee 3 36 IT 2 Hos E Ee nT TT 3 37 E SUO Li T 3 38 3 7 4 Switch Line Board Information s ccaciecsecseidssccasdcacavssidatedasnestactetdavceteldacaseees 3 39 SAAS Port RET LT 3 40 3 7 6 ET BER Te 3 4 T EEE T Tee Te 3 42 3 8 COn UDE NOJ Seriean a AOA E OR 3 43 3 9 Configuring the Subnet Manager 3 44 3 10 Fabric spect asmanni AT es 3 46 3 11 CIE MOn TT 3 47 3 12 Configuring Quality of Service OOS 3 49 l OT 3 49 K tO VE Map TTT 3 49 DADS QOS ATION eeraa valves iaethsdea O O S 3 51 3 13 Configuring Routing Algorithms 3c sse 3 53 Bala Balanced ROUNE Scheme serari sett aT A aeacveadantedes 3 53 3 132 AnfimiBand M lti PathSche TTT 3 53 3 13 3 Multi Path Scheme with SFB COonSIStencY sese 3 54 Chapter 4 Voltaire Device Manager VDM
235. t on an ISR 9096 9288 switch can be identified as follows M In the Fabric Manager select the suspected ISR 9096 9288 switch in the topology map of the main window and click the Nodes Information icon on the toolbar A separate browser window as shown in Figure 6 7 opens and displays ISR 9096 9288 node information the sLB sFB chip and the port connection matrix shows each ISR 9096 9288 external port internal chip port and its Channel Adapter connection It also details the system grouping information Nodes Info TBR Node Type Mode GUID Number of ports Node name 000811 0403960904 192 168 60 11 2 prm 7 DUUSTT 0403965008 192 168 60 3 omnis voltaire com 000811 0403965010 192 168 60 5 omnis vottaire com 000811 040396501 4 192 168 60 6 omni6 voltaire com 0008f1 0403965020 IL 0008f10403965020 0008f1 0439608904 168 60 10 2 omnit 0 005442ba00001 180 192 168 60 122 ISR9024 01180 0008f104004004a0 192 168 2 100 ISR9268 40061b ISR 9268 GUID 0006 104004004a0 Switch 1 0008f104003 0784 switch 2 O008f 104003f0765 switch 1 OO008 104003f 0766 switch 2 O008f 104003f 0767 switch 1 O008f 104003f06aa switch 2 O008f104003f06ab switch 1 0008 104003f0670 Switch 2 O008 104003f0671 switch 1 0008 104003f06be switch 2 O0008 104003f06bda Figure 6 7 Nodes Info Window This window shows the GUID LID of the rel
236. t sregquid gt lt destquid gt lt npaths gt Command lidtrace Description Print LID route path between two nodes unicast path only Syntax lidtrace d ebug D irect src v erbose s imple I t tameout ms lt src lid gt lt dest Lid gt Example lidtrace 24 22 Output From e Tooo r IMO E H EE RED H Ee 10D E R VOT are ISH Oe RE E G T Melis BUE IO EE E H T TE G S T H G ETT T IOT e HG T L 00 Sar E L H DE R E Ee HE E eR I TETE aorar e DORE EE E E EET a vd Osx E ER e E aE HCA400 Voltaire Switch User Manual A 23 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 24 Human translation A unicast path exists between the portLID 24 which is local port 1 and portLID22 which is local port 1 Path descr HCA pl pl6 ANAFA pl4 pl1 HCA Conclusion Unicast tables configured at least for this LIDs Hint Always verify that the opposite path exist as well Known bugs or limitations none Command linux log show Description Shows linux log file var log messages Syntax linux log show Command madstat Description retrieves various information from the IB Fabric with LID DIRECT routed MADs Syntax madstat d D irect t timeout_ms i sampling_intrval_ms w stat_window lt Switch Port smA Counters Forcets Monitor cuTt hough vlmasK eXt_portset Up sl_vl_mappinG Vl_a rhitratiLon gt lt lLid pach gt porti res type anport block General options
237. t to reset the PM counters in all fabric devices Click Apply to save PM configuration Note Data entered appear in red until clicking Apply Click to return to the previously saved settings Fabric Port Counters are listed and detailed in Appendix B 3 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Chapter 6 Fabric Diagnostic 6 3 6 3 1 Step 1 time interval as errors if any Diagnostic Tools Displaying and Checking Port Counters using the GUI Both Voltaire s Fabric and Device Managers allow you to view statistics for active To display the Port Counters dialog box and start the counters ports and detect faulty ports the statistics are collected according to a user defined The port Counters Info and Graph show the port traffic transfer in real time as well From the VFM Main Menu select the port in the main window and then select Port Port Counters or right click the port and select Port Counters In the VDM graphical representation right click the active port The following dialog box is displayed Port Counters x Port Counters Info Fort Counters Graph Counter Fort smit data Port rew data Port xmit packets Port rev packets Port rev errors Port xmit discard Symbol errors Link error recovery Link error downed Local integrity error Rev remote physical error Amit constraint error Rey constraint error Excess buf over run error
238. ta D irect t timeout_ms lt dest gt lt atrrid gt mod Subnet Management Attributes AttributeName R E E I IE E Ee Notice 2 U NodeDescription NG U T EE HE U SIE MUINE O 18 U GUIDInfo 20 CUM Bedrock Porc mune Za Poi TETE P_KeyTable 2 Porte Nuno r EEO DIOC k SLtoVLMappingTable 2 Topucy output Port Nunber VLArbitrationTable 24 Cues ome Comoomeni LinearFwdTable 2D Plocka denei ven RandomFwdTable 26 ielkovele Ikeketoue akin EEE MulticastFwdTable 2 Prock E TETEE Sly Aa a Chee VendorDiag 48 OE Ors iy Lec miko 49 U Detailed Description A LID routed MAD is sent to a specific LID in the fabric For example Voltaire Switch User Manual A 29 P N 399Z00038 2005 Voltaire Inc Chapter A CLI Commands Reference A 30 smpdump 1 25 retrieves the first block of the linear forwarding table attribute id 25 the attribute modifier is implicitly 0 meaning the first block of the switch corresponding to LID 1 in the fabric That should work only if the fabric is configured and the node element in the fabric whose LID is assigned to 1 is indeed an IB switch A direct routed MAD is sent to a node element in the fabric that corresponds to the specified direct path A direct routed mad can work in an non configured fabric A path is specified using the following format lt portO portl port2 gt The path is relative to the SW 6IB4C on which smpdump 1s running For example smpdump D lt 1 7 gt 27
239. taire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference pkey high show Show the high part of P_Key table pkey high show pkey low show Show the low part of P_Key table pkey low show pkey port guid set Add a port to a Kev table row okey port guid set row gt 1 guid pkey set Adds a Pkey table pkey set pkey number row 0x hex pkey update Update Pkey table pkey update pm Type this command to access the PM configuration menu which is used for setting Performance Management parameters See page A 55 for a list of the commands available in this menu pm qos Type this command to access the QoS configuration menu which is used to define the Quality of Service QoS settings See page A 61 for a list of the commands available in this menu qos show active nodes Displays a list of the active nodes in fabric show active nodes Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Configuration Config Mode sm info algorithm set Sets the routing algorithm to be implemented by the Voltaire Subnet Manager The Volt
240. tch Step 5 Power down the system and install both sMB modules into the chassis Step 6 Power up the system with both sMBs Voltaire Switch User Manual 2 19 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM In This Chapter This chapter provides system information and requirements for working with the Voltaire Fabric Manager VFM application This includes information on the following topics Fabric Manager Description on page 3 2 Fabric Manager Installation on page 3 3 Getting Started with the Fabric Manager on page 3 5 Fabric Manager Operation on page 3 6 Managing Alarms and Events on page 3 22 Viewing Properties Data on page 3 32 Information Views on page 3 35 Configuring the SM on page 3 41 Configuring Nodes on page 3 43 Configuring the Subnet Manager on page 3 44 Configuring Quality of Service QoS on page 3 46 Configuring Routing Algorithms on page 3 53 Voltaire Switch User Manual 3 1 P N 399Z00004 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 1 3 1 1 3 2 Fabric Manager Description The Voltaire Fabric Manager VFM is an embedded application with a web based GUI that provides InfiniBand fabric management functionality The Voltaire Fabric Manager provides a color coded topology map of the InfiniBand fabric which allows users to view the status of the managed InfiniBand fabric Using the VFM GUI users can drill down to obtain a wide
241. tch User Manual P N 399200038 2005 Voltaire Inc LO Viewing Properties Data 3 6 3 Link Properties The following information is displayed when selecting a link connecting line between hosts switches in the main window topology map Property Number of Connections 1 Link width 2 Link width Figure 3 18 Link Properties Table 3 14 describes the parameters displayed for each line Displays the number of connections between devices a link with more than one connection is depicted as a thick line on Number of Connections the topology map Link Width Displays the connection width in the present software release this field always displays 4x to indicate InfiniBand 4X Table 3 14 Link Information Parameters Voltaire Switch User Manual 3 35 P N 399Z00038 2005 Voltaire Inc Chapter 3 Voltaire Fabric Manager VFM 3 7 3 7 1 3 36 Information Views The Voltaire Fabric Manager allows you to get an overall view of the InfiniBand fabric and to drill down to specific information views all the way down to the individual port At each view different information is displayed List of Hosts To display a list of the hosts in the fabric click on the Host icon in the subnet tree The following window is displayed GUID 0008f1 04039608b0 Name 0008f104039608b0 IP Address Voltaire HCA400 Vendor Voltaire Last Update Wed Feb 16 16 18 45 2005 Node
242. tch User Manual P N 399Z00038 2005 Voltaire Inc Configuration Config Mode FTP Menu In the FTP menu you configure the parameters of the FTP server that the switch uses for various transactions The following lists the commands in the FTP menu E 3 00 Seong tine rt D command end exit eX OGIO Display list oi avar lable reo onnands Display command s usage Strang Exit to login menu from any other mode via console connection or close telnet session via telnet connection Exit to previous menu Upload config to repository new in remote FTP server using previously defined server name user name and password exportFile topology ES OOIOG Silt E Siglo eI O se password server username Upload Topology File to remote FIP server using previously defined server name user name and password Upload sm Log to remote FTP server using previously defined server name user name and password SHOW EP Clie rie COn ae om Download config repository new from the remote FTP server using previously defined server name user name and password Set user password to access remote FTP server Set remote FTP server name ip address Selo Ue iMeiileS CO SOcees enor WIP seiewe is The following details the commands available in the FTP menu Command Description Syntax Example Command Description Syntax Voltaire Switch User Manual export Uploads the configuration settings of the sw
243. tch User Manual P N 399Z00038 2005 Voltaire Inc Managing Alarms and Events Getting More Data for an Event To find out more specific details pertaining to an event see note M Right Click the event listing in the Trap Table The following menu is displayed NOTE x This feature is only supported for certain events when a specific type of event does not support this feature the View Event option is grayed out in the menu ROAM SFo 4 3 Counter threshold was exceeded Mev SRN J SFB 4 See old was exceeded Mew SR XX J sF B 4 Go To Node ehold was exceeded New ERISA J sFB 4 Fleer Evert found SUA Jun 19 18 53 50 IDT 2005 SUA Jun 19 18 53 49 IDT 2005 SUA Jun 19 18 53 48 IDT 2005 SUA Jun 19 18 53 47 IDT 2005 Sun dun 19 18 53 46 IOT 2005 SRIAK J SFB 4 Clear Al E holgd was exceeded Mev SUM Jun 19 16 53 45 OT 2005 ISRO J SFB 4 n Bhold was exceeded Heyy Tore hom JOA 2s ASAT an Peo yy 0 ED Ao 07 C eb ey Tar eea rll Lee Gevelled Kleon Figure 3 11 Events Log Right click Menu M Select View Event The following window is displayed Event Data E x Port Info Peer Info Mode GUID rnar O4004 00698 Node GUID roar O4004006T8 Flattorrn sr 2209 Platforern fer 92565 hodule FeR at sRBD 12 Module S zal Logical Port modue 1 internal 1 Logical Port g Port GUID nns 0001 260004 Port GUID eos O4004006Th IB Port l IB Port kes LID ba LID bs Figure 3 12 Eve
244. tem is configured to manage a chassis with I O modules and IB is the selected management interface In this mode configure the interfaces and I O modules as follows Configure the IB interface IP address subnet mask Configure the I O modules as follows e Make sure the IP address of each module is on the same subnet as the IB interface e Configure each module using the io IP address set CLI The configured modules can be viewed using the io show CLI Use the following table to enter the appropriate commands To set the I O address Use the following table to enter the appropriate commands Command Enable password io show Cone LO Cluster ip address ib set lt slOUu gt lt poOSLuLon gt C192 lt lt a D 0 60 gt T a BT Voltaire Switch User Manual Description Enter Privileged mode from Exec mode the default password is voltaire Use the io show CLI command to view modules Enter Config mode from the Exec mode Type the password when prompted Enter cluster configuration Set the IP address of the I O module in a specific slot and specific position in the ISR 9096 9288 switch Exit Fast interface configuration mode P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI IPR Management Interface Configuration When the system is configured with an IPR module you can use the IPR Ethernet interface as the active management channel t
245. terized by the parameters listed below A Header invisible to the user which includes the following fields E Trap number R Object type E Object WE Object s peer Object fields are as follows Guid Lid Port number Node System Guid Platform type Module type Module Index Subtype logical port number System name Voltaire Switch User Manual C 1 P N 399Z00004 2005 Voltaire Inc Chapter C Supported Traps and Events C 2 The object type could be port node switch router or HCA or SM Object peer is relevant for port object type only meaning connected port In case any of the above fields is unknown or irrelevant it will have null value The object type is required for log formatting Node object format is the same as a port format without port number and connected peer SM object format is as follows SM on lt logical name or Sys GUID gt Parameters The types of parameters below are defined as in trap originator side SM They could be changed to more generic types in API or SNMP trap An example of event log window is shown below ISR9XXX utilities event log show Wi ks OSE ASS aie Crs LAOS OOS ie ICO Sic G ee NOURI Aa te LX Ore wes hound Pore GUlD wo T G OON hy Aes iB Porte 2 Za Peer at OOO08f104003f074e Pome ZA Pore GUD OR O00 7074S Le Pore 2A SKI O E TTT E O E OE 8 Oly Ake 5 port 24 HT wes LOounc I Pore CUIDE T OOO POY E EB Pore 24 Peer at OOO08f104003f074e Ort 22 Pere
246. ters csv file either from the VFM or the CLI Step 2 Reseat both connectors on the 1X link or if this does not fix the problem proceed to step 3 Step 3 Replace the cable that connects between these two ports If none of this works one of the ports may be faulty In that case proceed as described below If the cable is connected between Host and Switch Step 1 Swap cables and connect the 1X cable to a different port in the switch to verify if the problem remains on the switch port Step 2 If it remains use the second port on the HCA if available or use another HCA Step 3 If it remains replace the ISR 9288 9096 Line Board sLB or the ISR 9024 Switch as required If the cable is connected between two switches Switch A and Switch B Step 1 Swap cables on Switch A and connect the 1X cable to a different port in switch A to verify if the problem remains on the switch port If the problem 1s solved replace the faulty ISR 9288 9096 Line Board sLB or the ISR 9024 Switch A Step 2 If it remains swap cables on Switch B and connect the 1X cable to a different port in switch B to verify if the problem remains on the switch port If the problem is solved replace the faulty ISR 9288 9096 Line Board sLB or the ISR 9024 Switch B Voltaire Switch User Manual 6 19 P N 399Z00038 2005 Voltaire Inc 6 4 2 6 20 Additional Tools and Examples Identifying a Bad Line Board or Fabric Board Port A bad Line Board or Fabric Board por
247. the FTP server Configure the user name for FTP server access Set the user password to access the FTP server This command uploads the configuration settings of the switch to the FTP server This command uploads to the FTP server a file with a log of network topology changes of the nodes in the network This is intended mainly for debug testing needs This command uploads a file to the FTP server with the SM log This is intended mainly for debug testing needs P N 399Z00038 2005 Voltaire Inc Chapter 2 Voltaire Command Line Interface CLI 2 3 5 Step 1 Step 2 Step 3 Step 4 Software and Firmware Upgrade Process Requirement for Firmware Upgrade In addition to the software that runs on the Management Boards sMBs there is firmware running on each InfiniBand switch ASIC The software upgrade process automatically detects the firmware level in each ASIC and upgrades it if necessary Switch ASICs are located on each sFB and sLB module Of the four possible sFB 12 Fabric Board modules two are unique in that they are connected to the CPU units on the sMB modules These two are the leftmost and on the rightmost sFB 12 modules slots sFB1 and sFB4 Upgrade Process I Update Procedure for a SINGLE sMB installed in the Switch During the ISR 9X XX upgrade process a single sMB module is installed in the chassis To upgrade the ISR 9096 9288 with a SINGLE sMB installed Connect to the sMB via a
248. the replication status must be enabled Syntax replication status set enable disable Command replication status show Description Shows the status of the sMB replication enabled or disabled When enabled the standby sMB maintains a mirror image of the active sMB and is ready to replace the active sMB in the event of failure In a redundant sMB configuration the replication status must be enabled Syntax replication status show Command route default gw show Description Returns the default gateway configuration Syntax route default gw show Example SW 6IB4C gt route default gw show I aes son es Ol ail Command sm info show Description Returns the current parameter settings of the Subnet Manager in the ISRXXXX Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Privileged Mode Syntax sm info show Example subnet manager info is smName LAVISM port guid 0008f1040040061b topology CLOS stage gt S COPO y LOS stage 0 algorithm balanced routing active algorithm balanced routing sm sm sm sm sm sm sm sm sm sm sm sm sm sm KEY 0000000000000000 priority 2 Sweep interval seconds 3600 verbosity mode info topology verbosity none mads pipeline 16 polling retries 12 ACau 2010 NSxe state master mode enable LMC 1 hog 13 slv 14 mopy l E NOTE Ed Refer to page 3 43 for a description of the Subnet Manager parameters Comma
249. tializes each HCA port partition table 3 The SA answers Pkey path queries by looking for a common Pkey between the two endnodes 4 IPoIB is used to create multiple IP domains one for each partition providing name address resolution Each IPoIB instance uses a separate QP Queue Pair associated with a separate Pkey partition 5 Each Multicast group is associated with a partition In the case of IPoIB there is a well known MGID Pkey mapping 5 2 Setting Partitioning Using ISR 9096 9288 CLI it is possible to configure fabric partitioning by setting the Partition Keys P Key to a partition table The ISR 9096 9288 partitioning Pkey configuration is organized in table format with a row per node port GUID based on user configuration associating a port with a list of Pkeys Pkey configuration is arranged as a table in the repository a row per port guid Every port may be added up to 16 partitions Row 0 in the table corresponds to the default Pkey configuration This entry is protected and cannot be removed Table 5 1 Pkey Table in Repository 5 2 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Step 1 Step 2 Step 3 Step 4 Setting Partitioning The SM either configures the HCA with the Pkey table as configured in the repository for that port GUID or uses the default Pkey table for that port This feature is disabled by default and may be enabled by CLI The SM reconfigures the port wh
250. tic functionality USING OOS E I avers Command end error fiLne event log show exit FINSE screens imalterelerenelal getpathrecord ide race EREE ere ON madstat mcmember MON Bes Kets netdiscover setpkey sminfo smpdump smpex topology log show vendstat vsm log show width check ASO CONESA Display Wrst Or avar able Teona Display command s usage string Exit to login menu from any other mode via console connection or Close telnet session yia celnet connection Perform error find command Show event log file Exit to previous menu Perr rori rind bad porto lt COmme me Berr orm f iiideae tn command Perrorm getpathrecore commanel Perform lidtrace command Show log file Perform madstat command Perform mcmember command Perform mctrace command Perform netdiscover command Perform setpkey command Perform sminfo command Perform smpdump command Perform smpex command Show log file Perform vendstat command Show log file Perform width check command Perron zero cCounters Command Utilities syntax usage is as follows error find event log show R h R Ea E e T getpathrecord IS ee Iii lloe Msi quent madstat Voltaire Switch User Manual error Ird A o e 2 sie o e aa a E OUA rO Era HEE TONS event log show EGILS E T I o r e T help DISABLED P PA T Er a e ae e T TIE Usage findpath d ebug v erbose s imple f astmode t timeout ms lt 2 Om OU er getpathre
251. tive Manual manual manual manual Log format Symbol Error counter threshold was exceeded New value is XYZ previous value was XYZ Link Error Recovery counter threshold was exceeded New value is XXX previous value was XYZ Link Downed counter threshold was exceeded New value is XXX previous value was XYZ PortRcvErrors counter threshold was exceeded New value is XXX previous value was XYZ PortRcvRemotePhysicalErrors counter threshold was exceeded New value is XXX previous value was XYZ PortRcevSwitchRelayErrors counter threshold was exceeded New value is XXX previous value was XYZ PortXmitDiscards counter threshold was exceeded New value is XX Xs previous value was XYZ PortXmitConstraintErrors counter threshold was exceeded New value is XXX previous value was XYZ PortRcvConstraintErrors counter threshold was exceeded New value is XXX previous value was XYZ LocalLinkIntegrityErrors counter threshold was exceeded New value is XXX previous value was XYZ ExcessiveBufferOverrunErrors counter threshold was exceeded New value is XXX previous value was XYZ C 3 Chapter C Supported Traps and Events value a R T SN aS PM Sy Dropped eae counter threshold Object port warning enable manual was exceeded New value is Parameters current value last RTS XXX previous value was XYZ FABRIC CONFIG ERROR ooo o S S O OSM EVENT INVALID LINK Object port
252. ts I O router modules using the single active management interface I O modules ISR 9096 9288 IP Addresses must first be configured via the CLI Once configured the ISR 9096 9288 sets the internal management IP routes Tee x Do not physically connect the I O module local fast Ethernet Management port to the same physical network when using Ethernet as active management Management over the FAST ETH Interface When the system is configured to manage a chassis with I O modules and the FAST Ethernet is the selected management interface the sMB automatically enters into Proxy ARP mode This allows the connection between the manager which resides on the ETH network and the I O modules residing on the IB network In this mode configure the interfaces and I O modules as follows Step 1 Configure the FAST interface IP address subnet mask Step 2 Configure the IB interface to the same values of the FAST interface same IP address and same subnet mask Step 3 Configure the I O modules as follows e Make sure the IP address of each module is on the same subnet as the FAST interface e Configure each module using the io P address set CLI The configured modules can be viewed using the io show CLI 2 10 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Step 1 Step 2 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 First Time Configuration Management over the IB Interface In this case the sys
253. ttons and the functions they invoke are described in Table 4 4 General Buttons Front Rear Toggles the device view in the main window to the front or rear view of the chassis This does not apply to the ISR 9024 Device Displays device data in the Information Area CPU Displays CPU data in the Information Area SM Displays local SM data in the Information Area Temp Displays the temperature readings of the switch components that have an embedded temperature sensor Fan Displays information regarding the fan units Reset Restarts the CPU Telnet Opens a Telnet window allowing you to connect to the device through Telnet VFM Launches a VFM Voltaire Fabric Manager session see Chapter 3 for a description of the VFM Table 4 4 Device Toolbar Notice the little green triangle at the upper left corner of the Icons Green means that the status is normal If a problem exists it will turn red 4 10 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc The Device Manager Main Window 4 3 5 Switch Graphical Representation The graphical representation of the switch provides a graphical view of the switch panels its components its LEDs and ports The ISR 9096 9288 Voltaire Device Manager provides two views of this switch a front view and a rear view This does not apply to the ISR 9024 NOTE When two Management boards sMB are installed in failover configuration x the active s
254. ual B 3 P N 399Z00038 2005 Voltaire Inc Chapter B Port Counters Port Counters List and Description The Port Counters currently supported are as follows SymbolErrorCounter LinkErrorRecoveryCounter LinkDownedCounter PortRcvErrors PortRcvRemotePhysicalErrors PortRcvSwitchRelayErrors B 4 Total number of symbol errors received on one or more lanes Total number of times the Port Training state machine has successfully completed the link error recovery process Total number of times the Port Training state machine has failed the link error recovery process and downed the link Total number of packets containing an error that were received on a port These errors include Local physical errors CRC VCRC FCCRC and all physical errors that cause entry into the BAD PACKET or BAD PACKET DISCARD states of the packet receiver state machine Malformed data packet errors Malformed link packet errors Packets discarded due to buffer overrun Total number of packets marked with the EBP delimiter received on the port Total number of packets received on the port that were discarded because they could not be forwarded by the switch relay Reasons for this include P N 399Z00038 2005 Voltaire Inc This counter can increase without it indicated a significant problem If SymbolErrors are increasing quickly AND this counter 1s increasing it may be indicating a bad link This counter is typic
255. uired Information on page 2 6 Connecting a Console on page 2 6 Starting a CLI Management Session on page 2 7 Configuring the Management Interface IP Address on page 2 8 Configuring the Time and Date on page 2 9 Configuring Passwords on page 2 13 Verifying Configuration on page 2 13 2 2 1 Collecting Required Information Use the Voltaire switch installation checklist Table 2 3 to record the system and network information required for first time configuration The following information is required R Passwords A password for users who will administer the switch An additional password is needed to access the Privileged mode for advance configuration The default password for Exec mode is 123456 and the password for Privileged mode is voltaire R Management interface IP address and subnet mask The IP address and subnet mask of the Fast Ethernet interface 10 100 Ethernet Once you have completed the checklist you are ready start the first time configuration of the switch Exec mode password password 123456 Management interface IP address and network mask 192 168 1 2 Table 2 3 Switch Installation Checklist 2 6 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc 2 2 2 2 2 3 First Time Configuration Connecting a Console Connect a PC with a terminal emulation program to the RS 232 console interface of the switch as described in the Installation Guide Make sure that the Switch
256. valk aoe Set Table Snow OOS Hugh Arb ao Ees set Sew ene Vike avoir ar ak kon HI E E Oe Set Table Show QoS Low Arb Table GeOSS apmiky COM tCur al Lone ser yes no gos configuration enable set enable disable GOS INTO SANON Py qos sl vl set row Ge Saal vas ion Gos Vil hoi ek base ete On Seo Leow wil weasel GO Wile ouieal eh lene ie cle TH Ges a vil Peo er E onan e set number Os iS low arbi T U E REEE vl weight gos Vie low arbore roeL ons Siow In the QoS menu you configure the Quality of Service QoS parameters The following details the commands available in the Subnet Manager QOS sub menu Voltaire Switch User Manual A 61 P N 399Z00038 2005 Voltaire Inc Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Command Description Syntax Chapter A CLI Commands Reference qos apply configuration set Changes made to the QoS configuration will be saved after typing gqos apply configuration set yes qos apply configuration set yes no qos configuration enable set Sets the Enable Configuration flag for QoS gos configuration enable set enable disable qos info show shows the QoS parameter information gos info show qos sl vl
257. wed to be of different length Each list entry contains 1 a VL number values from 0 14 and 2 a weighting value values 0 255 indicating the number of 64 byte units which may be transmitted from that VL when its turn in the arbitration occurs The PktLen field in the LRH is used to determine the number of units in the packet Note the VCRC and also the symbols between packets introduced by the physical layer should not be included in VL arbitration weight calculations The calculation is 3 52 Voltaire Switch User Manual P N 399Z00038 2005 Voltaire Inc Step 4 Step 5 Step 6 3 13 3 13 1 3 13 2 Configuring Routing Algorithms maintained to 4 byte increments Click Activate to set the changes in the fabric Click Apply to save the configuration in the Fabric manager Click Default to display the default settings in the dialog box Quality of Service can also be configured via the CLI Refer to the CLI QoS Menu on page A 61 for further detail Configuring Routing Algorithms The Voltaire Fabric Manager supports the following routing algorithms R Balanced routing R Rearrangeable routing multi path with sFB consistency Balanced Routing Scheme This is the default routing scheme and is applicable to all Fabric configurations InfiniBand routing is done in the following manner each InfiniBand switch maintains a simple forwarding database that defines per destination LID which physical port should be used
258. witch User Manual iii P N 399Z00038 2005 Voltaire Inc About this Manual What s New in this Manual This switch software version includes the following new features WE The Voltaire 96 port chassis ISR 9096 was added to the manual E IPoIB and In band Management Management Boards sMBs Hosts use InfiniBand instead of Ethernet to run the Device Manager on page 1 11 R ISR 9096 9288 External Ports The ISR 99096 9288 external ports are configured to work in 12X 4X auto detection mode meaning they are capable of working in 12X mode in case they are cabled to another 12X enabled port on page 1 10 E Full Hot Swap Support Hot swap support was added for sMB modules and for Fabric Board sFB modules on page 1 13 R Chassis Management with I O Modules Users can fully configure and monitor I O modules embedded within the switch management via the GUI and CLI on page 2 10 E Redundant sMB Configuration The ISR 9096 9288 supports redundant sMB configurations One sMB serves as the active chassis manager while the other is a standby management board The latter takes control when it detects a failure of the active sMB The redundancy mechanism maintains consistent replications of the system configuration parameters on both sMB modules on page 2 19 E Single Software Image for Both Platforms ISR 9288 9096 The same software image can be used to manage either an ISR 9096 system or ISR 9288 system The software automatically d
259. ys the user defined contact person assigned to this device Location Displays the user defined location assigned to this device Table 4 5 Device Information Parameters Step 2 Click advanced to obtain detailed module information see p 4 24 for further information Voltaire Switch User Manual 4 17 P N 399Z00038 2005 Voltaire Inc Chapter 4 Voltaire Device Manager VDM 4 5 2 Port Information To view port information Step 1 In the Main Window Switch graphical representation view click on a port HE The following information is displayed in the information area of the main window Link State Link Up Link Wich Link Width Supported 1 4x or Tex Link Speed 2 Gbps Link Speed Supported ther MIS VLO TE Her Lite Port MTU 2046 bytes Figure 4 13 Port Information 4 18 P N 399200038 2005 Voltaire Inc Voltaire Switch User Manual Viewing Switch Information Table 4 6 describes the parameters displayed for each port Parameter Description Physical Indicates the physical state of the port which could be one of the following Polling port 1s polling for a link Link Up Link is active Link Error Recovery port is attempting to get a link following a problem such as a short cabled pin Link Width Displays the width data of the link Link Width Displays the range of supported width data of the link Supported Link Speed Displays the link speed of t

Voltaire Switch User Manual ISR 9024, ISR 9096, and ISR

Contents

Download Pdf Manuals

Related Search

Related Contents