Home

Cisco Systems SFS 7008 Server User Manual

1. System Power Power Supply Status LED Supply Status LEDs Identify LED aa ooo Ooo oo0000 2 2 900000 Fan status g Ut LEDs Dogg ae ggg o g nnn nnmna ahi m Fabric Controller OoOo0o0gg eo 7 n Status LEDs OOOOT OoOoooodo e aona i J00gog DOUOUUOU Fabric 1Ooo0g0 S ee Controller 0o00000 i Master Status ooooo LEDs OoOo0o000 alaala eo Q gogoro _ fi 4 OOUOUUOU JoUUOU oO joo e o OUOOL i O E ma ml alan OOUOOC oo OOOO0OC If OOUUUOUU UU aes go fi Figure 5 1 Front Panel LEDs System Wide Status LED Locating the System Wide Status LED The system wide status LED is located on the front of the chassis and is visible whether the bezel cover is on or off This LED indicates the overall health of all modules in the system If you are facing the front of the chassis the system wide status LED is found on the top left corner of the switch In addition a mirror of the system wide LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 71 Interpreting the System Status LED Table 5 1 Interpreting the System Wide LED see Figure 5 1 Color State Indication off No system power or LED failure Green solid on System running with no detected errors Yellow solid on Failure withi
2. Table 7 2 Electrical Specifications Category Specification AC Input Auto ranging 90 264VAC 47 63Hz Maximum Power lt 600 Watts consumption EMC Immunity Table 7 3 EMC Immunity Description FCC CFR 47 Part 15 Subpart B Class A ICES 003 Issue 2 EN 61000 3 2 Harmonics EN 61000 3 3 Flicker EN 55022 1998 EN 55024 1998 EN 61000 4 1 2 3 4 5 6 8 11 Japan VCCI V3 02 04 Safety Table 7 4 Safety Description UL60950 3rd ed CSA 22 2 No 60950 2000 IEC60950 EN60950 EN60825 1 and EN60825 2 Acoustics Table 7 5 Acoustics Country Description Deployment Sound Pressure 37dB at 25C ambient ISO 7779 and section 8 5 of ISO 3744 1994 E Sound Power 50dB at 25C ambient ISO 7779 section 8 6 of ISO 3744 1994 E Product Markings UL FCC Statement VCCI CE ICES 105 106 Index A active controller card 11 admin status 81 C card operational state 79 reset 80 card reset 80 chassis ID about 19 LEDs 74 chassis manager launching 32 LEDs 78 controller card identify active 11 upgrade 59 controllerFabric12x 79 core slot about 18 D diag card 90 diag chassis test standard 93 diag power supply 97 F fabric controller LEDs general 70 master status 70 module status 70 fabric controller modules about 18 fabric12x 79 fail over fabric controller 42 management interface module 18 management setup 27 fan LEDs 69 H hot standby 11 I ib12p
3. 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 46 3 Remove the InfiniBand cables from the ports a To remove a cable with a pinch connector pinch both sides of the back of the connector and pull the connector away from the port Figure 3 12 Removing a Pinch Type Connector b To remove a cable with a pull connector grasp the connector with one hand and push it toward the port then pull the latch away from the port with your other hand and gently wiggle the connector away from the port Figure 3 13 Removing a Pull Type Connector 47 4 Use a l1 phillips screwdriver to unscrew the captive screws on both sides of the LIM card o gt Figure 3 14 Loosen the Captive Screws 5 Release the LIM module by pulling both levers toward you then toward the outside of the chassis Figure 3 15 Releasing the LIM Module Levers 6 Use the levers to pull the module from the bay Figure 3 16 Pulling a LIM Module from the Slot 48 7 Install a blanking panel in place of the LIM module or refer to Installing a LIM Module on page 48 for instructions on replacing the LIM module Installing a LIM Module ee Ne Remove the new LIM module from the package Refer to Removing a LIM Card on page 45 if a LIM module is still in place Ground yo
4. 4 h jg 5 Use the LEDs to verify the status of the module Refer to Chassis ID LEDs on page 74 The new Chassis ID has been previously serialized with the information that is unique to your chassis 6 Reboot the chassis by using the GUI or by using the reload command Example Topspin 270 reload System configuration has been modified Save yes default no cfg yes Proceed with reload confirm Topspin 270 Comic tioni ron ne Sit mene shee 53 Administrative Tasks This chapter describes administrative tasks that may be performed on the Topspin 270 system Upgrade all Cards in a Chassis on page 53 Upgrading an Individual Controller Card on page 56 Upgrading Individual Node Cards on page 59 Upgrade all Cards in a Chassis These procedures assumes you are upgrading all fabric controllers and node cards in the chassis Upgrading a Topspin 270 with the CLI For the commands needed to upgrade a Topspin 270 chassis using the CLI refer to the CLI Reference Guide 54 Up grading a Topspin 270 with Element Manager The following procedure describes the upgrade procedure using the Element Manager GUI This procedure also assumes you are upgrading all fabric controllers in the chassis 1 Launch Element Manager for the server switch 2 Delete any old img files to make room for the new image a b c Select Maintenance gt File Management Click ona img file if one
5. If one or more of the following situations occurs remove power to the Topspin system chassis and contact your Topspin support representative The power cable or power plug is damaged e The system has been exposed to water e The system has been dropped or damaged in any way e The system does not operate correctly after following the installation instructions A CAUTION You must ensure the operating environment has adequate air circulation Keep the Topspin system in a cool well ventilated room Do not block cooling vents Install filler panels into all unused slots ix A CAUTION Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin chassis Unused card and module bays should always have a Topspin cover over the bay to ensure proper safety ventilation and cooling General Cautions CAUTION No user is authorized to remove the Topspin system enclosure cover The internal chassis contains no user serviceable components Removing the enclosure cover voids your warranty See the warranty card for further details regarding the servicing of your chassis In general e Do not spill food or liquids on your system components e Protect your Topspin system from sudden power surges and interruptions by using a surge suppressor line conditioner or uninterruptable power supply UPS e Place cables appropriately so that they do not obstruct any egress w
6. required Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Table 5 6 Interpreting the Fabric Controller s Master Status LED see Figure 5 4 Color State Indication Active Master Status Indicator Green on This module is currently the system master Note that only fabric modules in one of the two core slots can become master Green off This module is not the active system master Note that only fabric modules in one of the two core slots can become master 71 Interpreting the Back Panel LEDs The back of the Topspin 270 chassis has the following types of LED indicators e Management Interface Module LEDs on page 71 e Chassis ID LEDs on page 74 LIM Status LEDs on page 75 InfiniBand Port LEDs on page 75 Ethernet Management Master pei pla Management i System Status LEDs chassis Chassis ID Status LEDs Connection LED LEDs DO OUOQODODOOUOGODODOUDOOOUDOOOODOOO OOD OOO OOO OOOOOO0OD oo o ooobpog 8000000 oo o og m og go oo o og ooooo0oog LIM Card OOo00o00 Status LEDs oOO0000 ooooog OO0O000 IB Port ogoogo Status LEDs OO0000 OO0O000 oo000g00g0 oooooogo OO0000 OO0000 OO0000 OQOOO0O000 oooooogo OOo000 oO00000
7. 1 card 1 card 1 carii fan 1 fan 2 Jo SWNPOot gt S IF USE Oe none none none none none none none none none none none none none none 5 Reboot the chassis when the self test is complete Example Topspin 270 reload System configuration is modified Save yes default no filename y Proceed with reload yes default no y Run Power Supply Tests The following tests are available to locate and diagnose fan modules e LED page 96 e Self Test page 97 Run a LED Test on the Power Supply Use the LED test to blink the LED and identify a specific power supply Example Topspin 270 gt enable Topspin 270 config Topspin 270 config diag power supply 1 Topspin 270 config diag power supply Topspin 270 config diag power supply 1 Topspin 270 con ig diag power supply Topspin 270 config diag power supply 1 test led start stop 1 97 Run a Self Test on a Power Supply 1 Show the ID of the power supply that you want to diagnose if you do not already know the number a Enter the show power supply command Example Topspin 270 show power supply Power supply Information ps type admin status oper status WE sLIL aL Src LOA voltage il AC up up n a 48 2 AC up up tS 47 2 Enter the diag power supply lt id gt command and specify the type of test self test is chosen by default Example Topspin 270 gt enable Topspin
8. 270 config Topspin 270 config diag power supply 2 Topspin 270 config diag power supply 2 test self test 3 Start the test Example Topspin 270 config diag power supply 2 start 4 Use the show diag fan command to show the progress of the test Example Topspin 270 config diag power supply 2 exit Topspin 270 config exit Topspin 270 show diag power supply Diagnostic Tests For Power Supply module type power supply module number 2 test self test iterations 1 action 3 Start result success percentage completed 100 result string Self Test Final report PASSED Please reboot syst em Topspin 270 98 5 Use the show diag fru error command to display any errors that appear in the result string Example Topspin 270 show diag fru error ETUS HTEO TEUS LOE PEUT C ELOR card 9 none card 10 none aueel iL i none ca rokai none carrai CRI none card 14 none angel Ges none euecl AL none fan 1 none fan 2 none item 3 none fan 4 none power supply 1 none power supply 2 none Topspin 270 6 Reboot the chassis when the self test is complete Example Topspin 270 reload System configuration is modified Save yes default no filename y Proceed with reload yes default no y Displaying Hardware Errors To display POST results non fatal errors that can are recovered informational logging such as firmware upda
9. ID Module cecceeceecceeseceseceseceeeeeeseecnaeceseeeeeeenseecseeeneenseees 52 4 Administrative TasSkS cccccecceceececeees 53 Upgradeall Cards in a Ch ssis ienirt ra e saad R A r cee ales REES 53 Upgrading a Topspin 270 with Element Managetr ccccccecccecsceceseceeceeeecsseceeeseeeeeseecsaeenseensaes 54 Upgrading an Individual Controller Card si c64 es4isashshauss testsnsstasone dons Seteeessanciot anasto decsialdanlienagvaeda 56 Upgrading a Controller Card with the CUI 0 ccccessasscesssacceodeecsnssediveneeeasdeeatedoeteundediecdaudegeunuosedaacees 57 Upgr ding Individual Node Cards anoct i a R EA ein het E R E a 59 Upgrading a Node Card with Element Managet cccccccecssesssecsseceneceeeeeeseecsaecneeeeseecsaeeneenaees 60 5 Monitoring the Topspin 270 System 65 Interpreting the Front Panel LEDS oy cssdise auc etctu asta Sia ug i Anadis a Ais aes tags Anse a Aas a ees 65 Systema Wide Satis LED neina EE E E ERE acyeens E va RANER 66 P wer supply LEDS pnesca a a E eave tai cal a a ea 67 Beary TBS 255 EEA E E T 69 Fabrice Controller LEDS sirenerne tie ir ioia r E AA TE acs A TAEA 70 Interpreting the Back Panel LEDS 3 s20 053 sicqateaiecs seey cog aces cx tv sake caey toate gaan Segue RE R A en E 71 Management Interface Module IGE Ds yas fosntsacdadniss sccatacus avi spa tla we tbeac vant da npandeatietarerapaghinae 71 Chassis ID EEDS a ea eet ys ocak aden E aa es a Baa Ss Sea RS 74 EIM Status
10. Installing a Fabric Crit WEES esac 2s cemaccetsec telnet acon tacts ws ucssalsouree cea sauecasest ama tuee as eue soba ee aeas 44 Upgrading the Controller Card c3 isascstectvasescetgavneacey lt shacaestaatecaa viteataa satu deegadtiechosdematloatedavieiv ieee 44 Line Interface Management Modules LIM sic7 s5 saserlasacdst oslalete geet eeadache dike tu daieelsgadd naebau eleanor aivetves 45 About LIM Car va sassy cna siccip Se scea ccaas eet be dke Stas a A ina a eee aac ein tag da E uted one ate 45 Removing a LIM C fdisenenri reen nna i eh yarags canton ate Mintas aalny eaten an R anos 45 Installing a WIV WMOdUles case ociccaacugecesaomecucoterteasenahensathaacauce a a a A aiaa 48 Management Interface Modules 295 ca feion vet sas outa cack as dee dat nautical aoa aioe iol lume cae aah icy 49 About Management Interface Modules a iasisisiscahecudbaciieseentiancsiaeasnclssgiacnplardracediaxtgaaeeadeaslanadivades 49 Removing a Management Interface Module cee eecceecceesceceseceseceececaeceeeeeeeeeaeecnaeeneeseeeenseees 49 Installing a Management Interface Module 3 353055 tcd icc sahaevndsandsaobdandonasavde squndaatousghanevseltuusdos 50 Chassis BED IN US 6 ie cae alas Canipcece tulad npc Ses esate yc ta acd gaat gi ied cia atalino vs Bod gee va Os OUR DOE 50 Abo t Chassis ID Modules ri oe ag ies sre eel a aye easy hacen Sia Sayeed cease uea ea ebay 50 Removing a Chassis ID Module mirerien ie i E e i a i i 50 Installing a Replacement Chassis
11. LED Sco i rehesshes cates ies i i oe oi Ria staal aes are ala ee Sod 75 HEP arid Pot Se acct cuales ches oa n a e oe Caves a adie cc tang dated ences 75 Monitoring the System with Chassis Manager ccccccsccsssceesceessecsseceeeeeeeesseceecseeeeeaeecaeceeeeeeeenseeeaeees 76 Monitor the DEVi a cp can esnetan a a A a aaraa E A aa aA ea aar ETa ARTE EAn EESE ERTS 76 Monitor the C tdsessecsniianionn nu naa a a aa aE 79 Monitor the InfiniBand Ports sucks3 cscl conch col casas bcdepl catenin aoaten teen cieaavit aa ne Resa cenay ie ceteaead 80 Monitor the Power Supplies 225s catercs teed ceasecanns Wlases teducee sacs cast atesuayeMeoas ai maswareiar se sause eats 81 Monitorthe Ransicccry cats nt wetnelsvesce id vadiisaet hares seas a Sarasa a oa asta iecedd 82 M nitor the Temper ane fo enhe e a i a E E a A iE Ea sire 83 Monitoring the System with Element Manager cccscccsccssscsssctesscssetssesccsssssssessscssssenaccsseseasesectenaces 84 Interpreting the Summary Tab sisccj ccstccsbaadanlasnceacessadidaaseaadcas SE ARAT ESR AET ENE aai 84 Interpreting the Power Supplies Tab soc ccatsiseiastsauneuaysne heats crrananaes athgosoiicasdecahotanetannnateneniotes 85 Interpreting the Fans VAb x ss0hs mesues Sec tunes san a A E E A EE aA 85 Interpreting the Sensors Fabi iruren a n i ac ed as estas eae i ae es 85 Monitor the System Wath the Cll uersecsi isca aa a a a A 86 6 Hardware Diagnostic Tests 6 89 About Diagnostic
12. Ports s Maintenance InfiniBand Help 83 Monitor the Temperature 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser Press the Enter key The Device Rear View and the Device Front View appear Click open the Chassis file in the left navigation tree The individual components appear beneath the Chassis menu Click Sensors in the left navigation tree The Sensors window appears Sensors 10 3 108 4 gt Health gt Sensors Refresh _ EDERN Current Status Oper Code Current Temp c Alarm Temp c Shutdown Temp c 10 1 up 58 91 98 lit 12 13 15 16 normal 1 up normal 56 91 98 1 up normal 57 91 98 1 up normal 59 91 98 1 up normal 34 65 HS 1 up normal 34 65 75 Data Refreshed At Friday April 23 2004 1 05 28 PM Figure 5 17 Chassis Manager Sensors Window 2 6 Verify that the sensor is Up by checking the Current Status column Compare the Current Temp c column to the Alarm Temp and Shutdown Temp columns The Current Temp should be lower than the Alarm Temp The Shutdown Temp indicates the degree in celsius at sea level at which the switch will reboot Each slot is monitored with individual sensors The Alarm and Shutdown temperatures vary depending on the slot 84 Monitoring the System with Element Manager For information regarding installing the optionally licensed Element Manager GUL refer to the Inf
13. Tests mn erae ien a e E EE E E E E E R E E A R 89 A S BHA KE EEA EE EEA EE EA E E EES E EE OAE A EESE E E A eae 89 Self Tests i n hada a n a a e E E A A enchants whic 89 BR tea A R E AE EAE T E E A E E SE E E T EN ET 90 R na Card Self Test oiannnio inania a a e E E R a a a ERN 90 R Chassis I CB IS catte tbe barstepet A a 92 Ru a Chassis Standard Test ascsins ocaca Cee cctesvcadae cy daseah ve eiii ieies esas iaaea ieia irii acess 93 RUM SPAM TS StS i ssisey tosses age tsb eR 0 nee PASE SE OEE Ee SERENE DEER is 94 R n Set oh hs Fan Ses ad cs Gu aiee eats apa aana ass a ude ana ach a A O R eS 94 R n Power Supply Wests ss ceci aeniea a a aa tak Blow aes dads te aero esha enn 96 Run a LED Test on the Power Supply vesiciacksbacatncedis0G3 choeresacstasdeaatstubcis axa atadeneeeeaiauane 96 Rima Self Teston a Power Supply isch sostavisssiccusk ensgass atiees eaves tanii i 97 Display Hardw re TT io eh sa ta a a la a Sd a dle eel cae 99 vi 7 Specifications and Compliance Certifications 103 Chassis and Management Interface s4 c15 acids san sstehgiaadlanen ious aansieynas ues 103 Electrical Specifications ti25329 crouse egal eco wars eerpa a a E E se E Aisa eit scat 104 EMAC AU tn int yo ees lattes each pe lies ecg Nodes an laces eae asia ela tae das e agenesis 104 ATELY dik cstaceiassenudiicasgtutacta dam buckane diahasassacdsdaceanasaausccun E nae Maid saneadins 104 PR OLS TLC Sas ated tease scene aces r ca at aaa be a Ca at aac at Otte
14. e Data Bits 8 e Parity None e Stop Bits 1 e Flow control None 21 Connect an Ethernet Cable to the Ethernet Management Port 3 Connect an Ethernet cable to the Ethernet port of the left Management Interface module slot number 15 See Figure 2 7 a Ifyou have two Management Interface modules you may also want to connect an Ethernet cable to the standby module slot 16 to maintain Ethernet connectivity in the event of a fail over Recommended Management Configuration The diagram below shows the recommended management port connections To have constant access to the chassis remotely redundantly connect the serial ports on two Management Interface cards to a terminal server Remote Connetion 1 Terminal Server gt y ri Connect left module first F o o o o o o o 00 o copoocoocoooooonoonooonooonooooooooooooonooooon P OD D idia e Ethernet Switch a ooooooooo0oo0oo0oo0o0ooo0ooocooo0ooooooooooooooooooooooooooooo0ooonoo0oo0o0 000 o SSeS I ae EET ee Tee eee eee eT TI Teepe O a a o i st I EIEE o 225223 ooooo0o0o0o00000000000000000000000000000000000000000000000 O ooooo0000000 ooo0o0o00000000 oo00000000 oo oo oo oa oo oa oa oo oo oo oooooooooooocoooocooocooooonoooooooooooooooocooooooooooo0o0oo0o0 o EEEE E ooooo0ooooo0ooooocooooooooooooooooooooooooooonooooooooo0o0oo0o0 EEEE 0 e U oo0o00000 hhh DoooODDOUUDUUDOUDOUDOOO Doo
15. ee elt asec eal 104 PrOGUCt Markmgsiss caneri e sits wen seca ee tae tata ali eae tate Blah neues E aca 105 6 MAOK lem ee ee ote ene 107 vil Regulatory Notices FCC Statement This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense Safety Information e Electrical Cautions on page vii e General Cautions on page ix Electrical Cautions A CAUTION Use only the power cable provided with your Topspin system Inspect the power cord and determine if it provides the proper plug and is appropriately certified for use with your electrical system Discard the cord if it is inappropriate for your country s electrical system and obtain the proper cord as required by your national electrical codes or ordinances Vill A CAUTION Grounding is supplied by the ground prong on the 3 prong power cable Do not attach a separate ground c
16. module becomes active The Master led becomes green 8 Install a blanking panel in the empty bay or refer to Installing a Management Interface Module on page 50 to replace the Management Interface module Installing a Management Interface Module 1 Remove the new Management Interface module from the package 2 Ground yourself appropriately 3 Refer to Removing a Management Interface Module on page 49 if a Management Interface module is still in place 4 Remove the blanking panel if a Management Interface module was not previously in place Align the Management Interface module and push the module completely into the bay 6 Use the LEDs to verify the status of the management interface module and the corresponding core card Refer to Management Interface Module LEDs on page 71 7 Connect the Ethernet or serial cables to the ports 8 Tighten the outside screws with a 1 phillips screwdriver Chassis ID Modules About Chassis ID Modules Refer to Chassis ID Module on page 19 for additional information Failed Chassis ID Module If you believe a chassis ID module has failed contact technical support Refer to Chassis ID LEDs on page 74 and Monitor the Cards on page 79 for monitoring information e A failure of the Chassis ID module will cause the yellow error LEDs to be lit on the Chassis ID module The System Status LED will turn yellow as well A CAUTION Never remove a Ch
17. one set of rack ears e If you have determined that the switch will be justified toward the front of the rack then use a 2 phillips screwdriver to unscrew the rack ears from the back of the switch e If you have determined that the switch will be justified toward the back of the rack then use a 2 phillips screwdriver to unscrew the rack ears from the front of the switch Figure 2 2 Unscrew Extra Rack Ears from the Chassis 6 Separate the rack ears from the chassis when all screws have been loosened 24 7 Use 2 3 people to lift the switch into the rack Figure 2 3 Chassis Set onto Shelf 8 Push the chassis into the rack until the rack ears are flush with rails Figure 2 4 Rack Ears Flush with Rails 25 9 Use screws that fit your rack to attach both rack ears to the rack rails SN i YS RAN Figure 2 5 Use Appropriate Screws to Attach Rack Ears to Rails Install Optional HA Components If you have purchased a High Availability HA package install the following components e Redundant Power Supply Refer to Power Supply Modules on page 35 e Redundant Fan Tray Refer to Fan Tray Modules on page 39 e Redundant Management Interface Module Refer to Management Interface Modules on page 49 26 Connect Management Devices To connect the management ports use either a serial cable and or an Ethernet cable Ethernet Serial Ethernet Serial Port Port Port Port BOO
18. system mastership and run system wide control functions in addition to its local management functions e One fabric controller in the Core slot acts as the Master e One fabric controller in the Core slot acts as the Standby to the Master Fabric controllers in Node slots act as slaves e Note Fabric controllers in Node slots do not act in active standby configuration Node and Core Slots The fabric controller modules can occupy either a Node slot or a Core slot in the chassis The Node Slot There are four Node slots in the Topspin 270 Fabric modules installed in Node slots provide fabric leaf switching as opposed to controller functionality The Core Slot There are two Core slots in the Topspin 270 19 e At least one Fabric Controller module must be installed in the Core slot for the chassis to boot e When a module is installed in a Core slot it acts as either the master or standby controller for the chassis with all modules in the system being slaves to the master controller The subnet manager runs on the fabric module installed in the Core slot that has arbitrated to be the master e The Core slot module also acts as the switch fabric spine interconnecting the end nodes Each Core slot is connected to 1 of 2 ports on the Management Interface module The pairing of core slot to Management I O module is as follows e The Fabric Interface Module in the Core slot 11 is connected to the Management I O module in slo
19. test on Power Supply 1 config mode gt diag card 15 gt test led gt gt start start flashing gt stop stop flashing Table 5 9 Interpreting the Fan Tray Status LEDs see Figure 5 6 Power Supply Module Status LEDs Color State Indication Green over Yellow off No module power or LED failure Green solid on Fan tray is running with no errors detected Green off No power to the fan tray LED failure or yellow LED is On Yellow solid on Failure of the fan tray Operator attention required The label indicates a failure Yellow blinking Identify a fan tray module Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example For example to run the identity test on Power Supply 1 config mode gt diag fan 1 gt test led gt gt start start flashing gt stop stop flashing 74 Table 5 10 Interpreting the Power Supply Status LEDs see Figure 5 6 Color State Indication Green over Yellow loff No fan module installed or LED failure Green solid on Power supply is running with no errors detected Green off No power to the Power supply LED failure or yellow LED is On Yellow solid on Failure of the Power Supply Operator attention required The
20. them without removing power from the chassis However before removing a power supply make sure that the redundant power supply has the corresponding power cord installed Each power supply is supported by a the power cord that is directly behind it on the chassis 6 Determine the power cord that is linked to the power supply module you want to remove The power cord and its corresponding power supply are situated in tandem on the same half of the chassis A CAUTION Make sure you do not remove the power cord of the functioning power supply 37 7 Pull the power cord from the chassis Power sake ai Power Supply front OO0O0000 OO0O000 ogpogogogg o Fog ogg ogg gpoggogogogogggoggg oogoogogogoogogogg oogooogoogoogoog VOOR DOOR H OOOO SSSS gt S gt gt JO e ogg gg gg eo e Figure 3 1 Diagram of Power Supply to the Associated Power Cord 8 Use a l1 phillips screwdriver to unscrew the fasteners that hold the power supply module in place Figure 3 2 Loosening Captive Screws 38 9 Use the handle to pull the power supply module from the bay i Figure 3 3 Pulling the Power Supply Module from the Bay 10 Install the blanking panel in place of the power supply module The device should never be run without either a blanking panel or module in any bay as overheating may occur or Install a new Power Supply modu
21. 1 hwif_log log 34113 Thu Jan 27 01 04 4 Export 11 istartup config config 271 Wed Jan 26 01 04 4 11 tslog llog 1239917 Thu Jan 27 01 08 2 _Install X lt gt Figure 4 2 Watching the File Copy Progress e Click Ok when the Image Install Completed Successfully message appears Configure the image for the next boot a Select Maintenance gt Boot Config b Select the new image from the Image Source drop down menu and click the Apply button Boot Configuration Software Images Current Image Yersion TopspinOS 2 2 Ofbuild530 Image Source for Next Reboot ERE Saon a0 le cy v a Startup Configuration Overwrite startup configuration with Delete startup configuration use factory defaults Figure 4 3 Set the Boot Config c Repeat for all Server Switches you wish to upgrade 56 6 Select Maintenance gt Reboot to reboot the controller cards and activate the image Click Yes at the prompt 7 View the new image on the cards a Select View gt Device Front View b Right click one of the cards and select Properties c Click the Inventory tab and view the version name in the Current Image Source field IB Fabric Core Card 11 10 3 108 6 Card Inventory Used Memory 17672 KB Free Memory 110088 KB Used Disk Space 30879 KB Free Disk Space 71500 KB Current Image Source TopspinOS 2 2 O build531 Image Source for Next Reboot TopspinOS 2 2 Ofou
22. 11 e The Management Interface module in slot 16 is connected to the Fabric Controller Module in the Core slot 12 For example removing the Fabric Controller from the Core slot 11 would cause both the Fabric Controller card and its paired Management Interface module in slot 15 to fail over The new primary Controller would be slot 12 and the primary Management Interface would be slot 16 Fabric Controller Modules See Figure 1 2 on page 14 for a diagram of the chassis Refer to Installing a Fabric Controller on page 44 for Fabric Controller module installation instructions The Topspin 270 contains six fabric control modules The fabric controller modules perform the following roles within the system e System Controller e Switch fabric spine e Switch fabric end nodes e Stores redundant Vital Product Data VPD from the Chassis ID module This allows the chassis to continue functioning and the VPD information to be recovered even if a Chassis ID module fails Each Fabric Controller module runs the management software and contains non volatile storage The behavior and responsibility of each fabric controller module is determined by the type of slot into which it is inserted Each fabric controller module can be installed in either a Node slot or a Core slot refer to Node and Core Slots on page 18 Master Slave and Standby Roles When the CPU of a fabric module detects that is in a Core slot it will arbitrate for
23. 2 on page 14 for a diagram of the chassis Refer to Installing a Management Interface Module on page 50 for installation instructions The Topspin 270 supports redundant hot swappable Management Interface modules each of which is paired to one of the Fabric Controller Core modules The Management Interface modules provide the following functions e RS 232 serial console port e 10 100 Ethernet management port e System real time clock RTC e Intra system Ethernet switch which provides a system configuration management data path to all fabric modules e Provide mirrors of the LED indicators from the front side of the chassis for the power supply fans and over all system status refer to Management Interface Module LEDs on page 71 e Rack Alarm Connection refer to Rack Alarm on page 19 Rack Alarm Redundant Ethernet Serial not supported LEDs Port Port Figure 1 5 Management Interface Module 18 Understanding the Core Slot to Mgmt Interface Module Connection Each Core slot in the chassis refer to The Core Slot on page 18 is connected to 1 port on the Management Interface module The controller fabric interface module inserted in the core slot selects the Management Interface module to which it will connect The pairing of Management Interface module to core slot is as follows e The Management Interface module in slot 15 is connected to the Fabric Controller Module in the Core slot
24. B 9 connector Ethernet Management port RJ 45 jack for unshielded twisted pair connections e Switch card 4X 10 Gbps InfiniBand connectors Manage the 31 Connecting InfiniBand Devices Attaching the IB Cable to the IB Port To connect using InfiniBand to other workstations or switches standard 4X InfiniBand cables are required InfiniBand cables can be used to connect any two InfiniBand devices whether switch or host 1 Plug InfiniBand cables into the InfiniBand switch ports To plug in an InfiniBand cable push the connector into the interface until you hear feel a click Figure 2 9 Removing a Pinch Connector a Ifthe IB cable is already connected to an IB host the physical LED light will turn green on the switch IB port when connected properly b To remove a cable with a pinch connector pinch both sides of the back of the connector and pull the connector away from the port Connecting to Ethernet or Fibre Channel Devices Connecting to the Host The opposite end of the InfiniBand cable will most likely be connected to an InfiniBand enable host refer to the HCA Installation Guide If your host does not provide an ample amount of free space around a given IB port double check that your IB cable connector engages fully Wiggle your connector back and forth to be sure that both sides of the connector have locked firmly into place Connecting to a Router or Switch To connect directly to an Ethernet switch router o
25. Completed Successfully message appears Configure the image for the next boot a Select Maintenance gt Boot Config b Select the new image from the Image Source drop down menu and click the Apply button Boot Configuration Software Images Current Image Version TopspinOS 2 2 O build530 Image Source for Next Reboot MA aE nena nE ey k Apply N rStartup Configuration Overwrite startup configuration with O Delete startup configuration use factory defaults Figure 4 7 Set the Boot Config 62 6 he Activate the image on the node card s by enabling and disabling each node card Do not reboot the chassis as this will not result in the image being applied to the node card s a Right click a node card and select Properties b Click the down radio button and select the Apply button IB Fabric Core Card 11 1 Card Inventory Card Type controllerFabric1 2x Enable Disable Card Current Card Status Operational State normal Card Boot Stage done Card Boot Status success Serial Number 3 PCA Serial Number PY 0340 000003 PCA Assembly Number 95 00036 01 FRU Number 0 Figure 4 8 Disable and Enable the Node Card s c Click the up radio button and select the Apply button View the new image on the node card s when the card comes back online a Select View gt Device Front View b Right click one of the cards and select Properties c Click the Inven
26. January 2005 Printed in the United States of America Regulatory Notices cccccscesseeseeeeeeseeees vil FCO St tement areitan a ead Saas eee ase Sos dasa es ead PG cS eee anes vii Salety nformato ieee aa ce ot a eit since a E a a ae as vii Electrical Cautions nena t a a a a a a A a T vii General Cautions enean e aa a a aaia aa aai a a araa ix C ntact Infotm t Ms ics ss vied cy aid naaa n aa tad at a sue ade aa ix 1 Features Overvi W cccccccececcececeeeecees 11 System Architectures iresi a E R A E E A E cet E E ER RS 11 Hot Standby 235 os orien ccb dan tas cteotcadceeg versie dand casas cductesahead tamede tbascenad otnabeudecasecitucciaanirncon 11 InfiniBand Connectivity sineera ia A cosas aE EE Aa Bag Dae at Aas es Ala ats E Aes atk 12 Flexibility ieina ant Ee ea EN E E REA AO aE aca SEa aia EEES 12 HAVAA Dil iy a eennenr a e a E N T E E rae ee N E 12 No Blocking Architecture tackles chet cere a Seah E Sad Seah E E A 13 Abo tthe Topspin 270 Chassis cuan a a cubes a we ons eaves aa Maw a e wena aa Oana 13 Diagram of the CHASSIS 523g diasceta ahaa zeeg aa a a a Ai E EREE EE 13 Chassis Slot N mbering Slain seco saline Ad ian a a a ti a ola sch A arate ala arataigs 15 IEE TIS TOTS sa hse Satna inn tpt ee tarps A ee ig at Baa ee WN Ga it Ne Atha eens aa bis aa 16 CONNECTIONS esse Sagar elt ate stale shee ta cates iS ial Seti Aaah Na ated ate hd 16 Mia ths E cat ict ccna tacuia Ad cess dlia cash A A E A Aime idea
27. OO0O00 OO0o0o00 oOoOoo0o0o00 OOO0OO000 oOo0000 OO0000 OO0000 OO0OO0O00 OoOO0O000 OO0O000 i R if ht HH oOooooo OOOO00 Figure 5 5 Back Panel LEDs Management Interface Module LEDs The Management Interface modules have six LED yellow over green indicator pairs which show critical system module statuses The six pairs of LED indicators are a mirror of the LEDs that appear on the front of the chassis e System wide status Table 5 7 e Management Module status Table 5 8 e Power Supply 1 and 2 status Table 5 10 e Fan Tray 1 and 2 status Table 5 9 In addition the Ethernet Management port RJ 45 connector on the Management Interface module has built in status LEDs for link and traffic status 72 Location The Management Interface modules are located at the top of back the chassis Refer to Figure 5 5 to view the chassis diagram IN O C E a a e 5 a O5 o Figure 5 6 Management Interface Module LEDs Interpreting the Management Interface LEDs System Status LEDs Table 5 7 Interpreting the Management Interface Module System Status LEDs see Figure 5 6 Color State Indication Green and Yellow off No module power or LED failure Green solid on System running with no errors detected Green off No power to the Management Interface module LED failure or yellow LED is ON Yellow solid on At least one module failure in the system Operator attention required The label
28. OO0o0o00 ooo ooojooo ooo ooo0 ooo ooo omo oom ooooooooD oof ooooog S oof ooo go oss oo g o g o g o oponopnnpnonononnnncnnnccnnnonnonncnnnncnnnnnncananoncnnnnnnnononnnonononn n0opo0o00 o EIEEE O ooo l opoo0o000 DUUUOU oOo0000 DOO000 DOOO00O0 ooo0000 OO000D 0O0000 oo000g0 ogcoogoog OO0O000 ogoogo CECE ELECE ogoogo oo0000 000000 OOO0O00 oooogoog OO0000 oooo0oogno OQO0O000 OOOO OOOOO Oooo oogogggogogogoogoggoggooOogog ono ono oOogoOogoOoogooonggnonong Figure 2 6 Serial and Ethernet Management Ports Attach a Serial Console Cable to a PC or Terminal 1 Attach the RJ 45 console cables from the cable kit that is provided See Figure 2 7 a Connect the cables to the InfiniBand chassis serial console port on the Management Interface card If you have two Management Interface cards make sure that you connect to the left Management Interface module slot 15 which will be the primary Management card upon initial boot up The serial console port is labeled 10101 b Connect the other end of the serial cable to your terminal server or management workstation For detailed information on how to connect the serial console cable please see the documentation included with the serial cable kit 2 Open a terminal emulation window using a program such as HyperTerminal for Windows Set your terminal parameters to the following Baud 9600 bps
29. S EE TSAAR 31 Optional Launch the Chassis Manager csccceccesseeesseeeeceeceseecsaecnseeeeeeeeseecsaeceseeeeeeenseeesaeens 32 Optional Install the Element Manager GUI icles aie wcsgantiass deaoriansedudeessasd ahaa wanentiaranaenas 34 3 Managing Individual Components 35 About the Field Replaceable ii sityscc ccm ctaasugat mnataGae cate areca echaee hut Mcaaenas Sentient aencntavaheetupledinee se ccatams 35 Power Supply MOdUles cciaseauds cer aei suet vane Hatin acta a i aa ati tadedeas ei eek 35 About the Powersupply Modes i c 5scisssesscadiseseied bas Sicshaeesdtcadaedscatoasta cap laxddactdaasceplaxwiasdaaelaaueaees 35 Removing a Power Supply Module ts ccic i sesagsissdaceusk seiniganssacts asceriaanicnisase ga ieaaisucts senaads 36 Installing a New Power Supply Module vives ccsclessecshcee asad asnaetle casalvleccaeea cid ashes matin iautneants 38 By Ura DS aa a a os sda e ve ees ec as states Wo aa n n Slane ase 39 Aboot the Fan Tray Modulesicc cn cco ycasveses nace ete As atin E toed otautenr aust 39 Removmea Fan Modulen cererii rone a a i a E elotend aE iE O E 39 Installing a Parr Mod ler cic re a aE a EE AE Sada Ara aes ea a Ave 41 Fabric Controllers reiris a mesaer aTa Ae EEA cea R Ea a A Wes E AEE AE A N ona a ETE ea 42 About the Fabric Commrouers dacs cred ait ohisec at sauecay vhedauncent areca pioneateav shen mae nesiea aes 42 Removing a Fabric Controller iss casstehsiosncs october tnanotatel celsun E suet 42
30. SS TOPSPIN Cisco SFS 7008 InfiniBand Server Switch Hardware Guide Topspin 270 Hardware Guide Release 2 2 0 Update 1 Part Number OL 7725 01 Product ID 140265 Copyright 2004 2005 Topspin Communications Inc All rights reserved The Topspin Switched Computing System Topspin Host Channel Adapter Topspin Element Manager and collateral software programs and documentation are subject to and made available only pursuant to the license agreement signed by you and Topspin Communications Inc or if no signed license agreement exists the license agreement included with your original media and may only be used in accordance with the terms of that agreement Making copies modifications or compilation works of the software except as specifically allowed in that agreement is prohibited by law and constitutes a punishable violation of the law This documentation may not be copied modified or reduced to any electronic or machine readable form without Topspin Communication Inc s prior written consent As defined in FAR section 2 101 DFAR section 252 227 7014 a 1 and DFAR section 252 227 7014 a 5 or otherwise the collateral software programs provided to you are commercial items commercial computer software and or commercial computer software documentation Consistent with DFAR section 227 7202 FAR section 12 212 and other sections any use modification reproduction release performance display disclosure or dist
31. Server Switch a Select Maintenance gt File Management b Click the Import button c File Type mage gt Select Remote FTP Server or Local File to navigate to the file Import File Copy a file to the system File Type image v Copy From Remote FTP Server Local File salfile Topspin270 TopspinOS 2 2 0 build531 img Copy To File Name on System Topspin2 70 TopspinOS 2 2 0 build531 img Figure 4 5 Copy the img File to Server Switch e Click the Copy button and wait for the message that indicates the copy was successful f Click Ok Install the new image 61 Note all node cards in the must have an oper status of up for the upgrade to be effective a Select Maintenance gt File Management b Click the new img file and click the Install button c Click Yes when prompted File Management x You can import a file from an external FTP server export a file to an FTP server install an image in the system or delete a file from the system Current Files on System Slot ID File Name File Type Size Date Topspin270 TopspinOS 2 2 0 build531 img image TopspinOS 2 2 0 ouild530 image 6684672 Wed Jan 26 01 07 hyif_log log 34113 Thu Jan 27 01 04 4 startup contig lconfig 271 Wed Jan 26 01 04 1 slog log 239917 Thu Jan 27 01 08 2 Figure 4 6 Watching the File Copy Progress d Click Ok when the Image Install
32. a taaecal 16 Power SU pS ica otal soar aan ade bela uate tone ae csc R adele ah coe 16 IPAM TAY Sia A E E A EE EA 17 Management Interface Modules ss nnnsessseeseeseessesseseessessesstssressesstssresseestestosseesesresstesseseessesseesss 17 Fabric Controller Mad Ges yascaisshca cosicsdaaeyantvascssagdi e A E EA E E a Ea 18 INGGE Art Cte O AE EE E E AEE E E A 18 Line Interface Modules LIM sis ar cigs Seis ina A RA E a RRRA S 19 Chassis VEY ITU caus Sects scat Vdc ate Se aneteenitgh a e aac Mea andes eavaltet bated ets a a 19 Rack Alarmi ena rno aces Caspase wasn Ac see is ve ce Ate ees DARD A Sahin Ata et as As ak 19 Administrative E CANIL SS sak ads atedn aisia aas a EAEE REE taslande asst EATE Ea E E aN E AEREAS 20 Real Time Cloc kienean anne n a Aei ted ae eee 20 110184 LON ROR a R a E AN 20 Non Volatile Memory ena nan a a a a a a e e aer 20 Vital Product Data Storage sccasyicusuisauhGa EEE E E E E a 20 Diagnostiese haee ea ich ice Pea E E A A eal clad T E A a 20 2 Installing the Topspin 270 System 21 IAR OINEN TEST II AE PEE ES A AEN TAE S E E E T E E A 21 Read the Cautionary Statements si oa 5 65 sn0s secsyul sears ruassttaead pats gus mas sandiens ueaegente tien aa tohenanenmews 21 Prepare the Physical Environment for the Topspin 270 System cccccesccesceeseeesseeeteeeteeeseees 21 Mount the Topspin 270 Chassis in a Rack 325 s3scayssteseegtasdech idesa eesascssasteleatusedactiiede aged cadscauh tude gasisa
33. able Do not use adapter plugs Do not remove the ground prong from the cable Ensure the ground connection on the power supply is correct and functioning before applying power to the Topspin system CAUTION Always ground yourself before touching any internal Topspin system component to avoid damage from electrostatic discharge ESD A CAUTION The Topspin system has two power cables The operator must disconnect all power cables before attempting to remove the system from the rack A CAUTION Remove power cables by grasping the cable connector and pulling straight out If the chassis comes equipped with a power plug retainer move the bail wire used to retain the plug to the side before attempting to remove the power plug Replace power supply cables be grasping the cable connector align the connector with the power jack and insert the connector straight into the jack If the chassis comes equipped with a power plug retainer move the bail wire back into place to keep the power plug firmly in place CAUTION The Topspin system contains lithium batteries Do not attempt to replace or discard these batteries The batteries may only be serviced by Topspin service personnel CAUTION Observe and follow service markings Do not service the Topspin system except as explained in the Topspin documentation supplied with your system Opening the chassis or removing the enclosure cover exposes you to electrical shock and may damage system components
34. ace last image source primary image source image cpu descr fpga firmware rev ib firmware rev SILOGH 1Cl used memory free memory used disk space free disk space last image source primary image source image cpu descr fpga firmware rev ib firmware rev bak 18092 kbytes 109676 kbytes 30891 kbytes 71488 kbytes MEOSOLAOS 2 2 0 owii dels 31 MES OLiMOS 2 2 10 ioiwin els 3 MEOSO1LiAOS 2 2 0 oyoin els 31 PPC 440GP Rev C Rev 4 129 pvr 4012 0481 ae 0004031fe8 Re T yee 12 18092 kbytes 109676 kbytes 30891 kbytes 71488 kbytes TopspinOS 2 2 0 build487 TopspinOS 2 2 0 buil1d487 TopspinOS 2 2 0 build487 PPC 440GP Rev C Rev 4 129 pvr 4012 0481 ae 0004031fe8 a E Yet 58 2 Show the standby Controller card Example Topspin 270 show card inventory Card Resource Inventory Information slor rel iL ik used memory 18092 kbytes free memory 109676 kbytes used disk space 30891 kbytes free disk space 71488 kbytes IASE LMACa SOuIceS 8 WojosjoimOS 2 2 0 lowricls3i PieaAmeiey iIMACGeS SouUreSs 8 WepsoLiMOS 2 2 0 louiiles3i image s TopspinOS 2 2 0 build531 cpu descr PPC 440GP Rev C Rev 4 129 pvr 4012 0481 fpga fLirmware rev ae ib firmware rev 0004031fe8 slow tel LZ used memory 16017 kbytes free memory 107432 kbytes used disk space 30891 kbyte
35. afa2 POST firmware check PASSED AS Oks OFZ OlO4s ROS SEE PROME PAIS S ED 419207707 2004 POST FPGA PASSED A IDsO7207 ZOO4s POSWs SUMMOMPE YAS SIRID A AOD OFSOS ZOO4S IROSWs EREC PLAS SIMD AES AON see OO 45 ae PO Sra AN PALS SED 19 0751272 2004 Caro srenteg x Card 15 Sicaicciing V 4 19 07 12 2004 Anafa2Init a2update set to IGNORE 4 19 07 34 2004 Anafa2 POST firmware check PASSED Se lesiisats ZOOS ROSS RS EE EROM WAS ISD l6gi9sas 2ZOO4s BOSWs WEA RAS SED 5 loesgiSeae ZOO4As POSS SUMMA Ss IPAS Sad l6gi9sats ZOO4s POSS IUICS RASSiaID l6sil9es3 ZOOAs POSMs WANS IAS SED S 1619 33 2004 Caro Sieiccwyo Cee 1S Seeing Vo 5 16 19 53 2004 Anafa2Init a2update set to IGNORE 5 16 20 15 2004 Anafa2 POST firmware check PASSED amp les2iles 200 POST wo Pte ROMER EAS gt PID OESS 200 ROSIE RHEGI PASOED runcated gt 4 Enter reload to reboot the switch after the test is complete If you do not reboot the switch after running the self test the switch will remain in an unknown state Run Chassis Tests The following tests are available to locate and diagnose the chassis e LED e Standard page 93 e Extended e Extended Memory Run a Chassis Standard Test The chassis standard diag test runs a diagnostic test on all cards in the chassis 1 Enter the diag chassis test standard and start commands to initiate the test Example Topspin 270 gt enable Topspin 270 config Topspin 270 config diag chassi
36. aja a ala a ggogog Slot 11 oo00000 aja a aa a oOo00000 ajaj alala a Slot 12 oOoo0000 ala a aaa alal alala a Slot 13 0 1 OOO000 oOoo0000 ooo0o000 oOooooo0 Slot 14 oS ogogooogqago OOO000 OoOoo0oo00 Ooooo0o0 oOoooo00 Figure 1 3 Chassis Slot Numbering in Front Non port Side 16 Slot 1 p Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Slot 15 Slot 16 Slot 17 ooocoocooocoogoocoooooooooooooooooooogpocoogoooooooo og 5 7 o go o ogpggog ogpggoggo ooogoogoogg ogoogogq Figure 1 4 Chassis Slot Numbering in Back Service Side Dimensions Height 6U unit Width 17 04 Depth 8 23 7 Weight maximum 106 pounds full chassis configuration with 4x copper Line Interface Modules Connections 96 ports of 10 Gbps 4X Copper InfiniBand Two 10 100 Ethernet RJ 45 Management Ethernet ports for out of band management Two RJ 45 Console Port used to configure and monitor the Topspin 270 Rack Alarm All cabling is on the rear of the chassis LEDs LEDs are located on both the front and back of the chassis Refer to Interpreting the Front Panel LEDs on page 65 and Interpreting the Back Panel LEDs on page 71 for more information Power Supplies Refer to Installing a New Power Supply Mod
37. all image image file name img Example Topspin 270 install image Topspin 270 TopspinOS 2 2 0 build531 img ISTE PSEA IS I TSES TE TS TS CVO SICSICILOIN CONAL STS Sucesssituulily 6 Specify the new boot image to be used a Find the name of your new system image by entering the dir image command Copy the name of the new image b Enter boot config primary image source then paste the copied image file name Example Topspin 270 configure Topspin 270 config boot config primary image source TopspinOS 2 2 0 build531 Topspin 270 config exit 7 Reset the individual Controller card without rebooting the whole chassis Example Topspin 270 config card 12 Topspin 270 config card 12 action reset Upgrading Individual Node Cards Upgrading a Node Card with the CLI For the commands needed to upgrade a node card using the CLI refer to the CLI Reference Guide 60 Upgrading a Node Card with Element Manager The following procedure describes the upgrade procedure for individual node cards using the Element Manager GUI This procedure assumes you are upgrading only one or more node cards and not the Controller cards l 2 4 Launch Element Manager for the server switch Delete any old img files to make room for the new image a Select Maintenance gt File Management b Click on a img file if one exists and click the Delete button c Repeat for all Server Switches you wish to upgrade Copy the new image to the
38. anager Power Supplies Window Monitor the Fans 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Log into the chassis The Device Rear View and the Device Front View appear 4 View the Fan status at the top of the window to quickly check the status color and the number of modules installed e Ifonly one number appears then only one fan is installed The actual integer refers to the slot in which the fan module is installed A field that read Fans 1 2 would indicate that a fan trays are installed in both slots e The color of the integer indicates the status of the fan A green number indicates that the fan is Up A red number indicates that a fan has failed 5 Click on the word Fans at the top of the window to view details The Fans view appears IP Address 10 3 108 4 SS TOPSPIN System Name Topspin 270 X 10 3 108 4 Fans a4 Chassis 10 3 108 4 gt Health gt Fans amp Cards 9 Ports amp Power Supplies 8 Bs Current Status Speed 4 ensors 1 up 74 B Backplane 14 Management Ports 2 a 70 aA Maintenance a4 InfiniBand Data Refreshed At Friday April 23 2004 1 18 08 PM Help Figure 5 16 Chassis Manager Fans Window X 10 3 108 4 Chassis amp Cards amp Ports Power Supplies amp Fans Sensors a Backplane Management
39. assis ID unless Technical Support has specifically instructed you to do so Locating the Chassis ID Module The Chassis ID module is located on the back of the chassis Refer to Figure 1 2 for a diagram of the chassis Removing a Chassis ID Module 1 Ifyou must remove a Chassis ID module verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 2 Ground yourself appropriately 51 3 Use a l phillips screwdriver to loosen and remove the screws from the left and right side of the Chassis ID module Figure 3 20 Loosen the Captive Screws 4 Pull the Chassis ID module from the bay 4 y pg Figure 3 21 Remove the Chassis ID Module 5 Do not allow the bay to remain empty as overheating may occur The Topspin 270 should never be run without the replacement module so you should immediately install the new Chassis ID module Refer to Installing a Replacement Chassis ID Module on page 52 52 Installing a Replacement Chassis ID Module 1 Remove the new Chassis ID module from the package 2 Ground yourself appropriately 3 Remove the old Chassis ID module if you have not already done so Refer to Removing a Chassis ID Module on page 50 if a Chassis ID module is still in place 4 Align the chassis ID module with the slot and push the module completely into the bay
40. cally Refer to the static example below a When the system has completed booting press lt Enter gt several times to display the CLI prompt Login b Enter the user name and password The default user name is super and the default password is super Login super Password super REPS Lia Z2 70 gt c Enter the enable command Topspin 270 gt enable Topspin 270 d Enter the configure command Topspin 270 configure Topspin 270 config e Enter the interface mgmt ethernet command Topspin 270 config interface mgmt ethernet f Enter the IP address of the management port followed by the netmask Topspin 270 config if mgmt ethernet ip address 10 10 0 22 255 255 255 0 g Set the default gateway for the management port For example Topspin 270 config if mgmt ethernet gateway 10 10 0 1 h Enable the management port by entering the no shutdown command Topspin 270 config if mgmt ethernet no shutdown i Save the configuration to preserve it between reboots Topspin 270 config if mgmt ethernet exit Topspin 270 config exit Topspin 270 copy running config startup config j Ifyou have two Fabric Controllers installed in Core slots you will receive a message asking if you want to save the configuration to the Standby Controller Answer Yes to save the configuration across both controllers Assigning an Address Dynamically a When the system has completed booting press l
41. diness e Redundant components operational status is ensured periodically during normal operation including the logic required to perform the transition from faulted primary to redundant component Detection of abnormal status is reported 21 Installing the Topspin 270 System This chapter describes how to install and manage the Topspin 270 system hardware Prepare the Site on page 21 Configure Basic Connectivity on page 28 Mount the Topspin 270 Chassis in a Rack on page 22 Connect Network Devices on page 30 Manage the System on page 31 Prepare the Site This section provides information that you need to safely and successfully prepare your environment for your Topspin 270 Read this section carefully before you install your device Read the Cautionary Statements Refer to the FCC Statement on page vii and the Safety Information on page vii Prepare the Physical Environment for the Topspin 270 System Make sure you have the right cables and sufficient ventilation Unpack the Topspin 270 package Prepare a management workstation such as a PC running a terminal program and a straight through M F DB 9 serial cable included 22 Mount the Topspin 270 Chassis in a Rack This section describes how to install the Topspin 270 chassis in an equipment rack The following procedure is the standard installation using your own rack shelf Topspin also has an optional shelf that is app
42. e Interface modules with 4x transmission capacity ib12port4xTX e Fabric controller modules installed in a Core slot The 12x refers to the internal chip speed controllerFabric12x e Fabric controller modules installed in a Node slot The 12x refers to the internal chip speed fabric12x e Management Interface modules mgmt O e Chassis ID modules idModule Note the Current Status column which should read up The current status refers to the Up or Down state of the card Note the Operational State column The possible states are as follows unknown 80 normal bootFailed tooHot booting checkingBootImage wrongBootImage rebooting standby recoveryImage 9 Click the radio button next to the card that you want to monitor 10 Click the Properties button to view details about the card or to reset the card Reset a card to return it to the default state The cards cannot be shut down because they are architected so that they are always in service once they are inserted into the chassis Monitor the InfiniBand Ports 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Log into the chassis The Device Rear View and the Device Front View appear 4 In the left navigation tree select the Chassis gt Cards The list of cards appear 5 Click the radio button next to the LIM card whose ports you want to view 6 Inthe drop down menu above the card list selec
43. er Supplies A Fans 5 Sensors Device Front View A Backplane Management Ports 4 Maintenance 4 InfiniBand Help Figure 5 11 Chassis Manager 270 Device Front View 4 View the information at the top of the window to quickly check the status color and the number of modules installed The numbers associated with the Card field indicate the slot numbers where modules are installed The numbers associated with the Power Supplies and Fans fields indicate the ID for the module e If only one number appears next to any of the modules then only one item is installed The actual integer refers to the slot in which the power supply module is installed A field that read Power Supplies 1 2 would indicate that a power supply was installed in both possible slots e The color of the integer indicates the status of the power supply A green number indicates that the power supply is Up A red number indicates that the power supply has failed 5 Use the visual representation of the LEDs to monitor the status of the cards ports power supplies and fans e A green LED indicates that a hardware component is installed in the slot and that the component is functioning without errors e A white LED indicates that is no component is installed in the slot e A grey LED indicates one of the following states e A component is installed but is unmanaged or unreachable 79 e In the case of
44. es Tab Data cached at 10 43 13 Interpreting the Fans Tab Use the Health gt Status gt Fans tab to view the operating status of the fans 5 Health Status Retesh My E E cose Het Figure 5 20 Element Manager Fans Tab Data cached at 10 43 16 4 Note the OperStatus field A status of up means that the fan is operating properly 5 Note the Speed field The integer in this field represents a percentage The percentage changes based on the ambient temperature of the unit and will increase as the temperature rises Interpreting the Sensors Tab Use the Health gt Status gt Sensors tab to view the operating status of the temperature sensor of the system 86 S Health Status normal normal normal sor DE E eee Figure 5 21 Element Manager Sensors Tab 6 row s 6 Note the OperStatus field A status of up indicates that the sensor is functioning properly 7 Note the Temperature field Each slot has an individual temperature sensor The Alarm and Shutdown temperatures vary by slot type Monitor the System With the CLI For a complete list of CLI commands refer to the CLI Reference Guide Use the following CLI commands to monitor the Power Supplies the fans and the sensors Monitor the Power Supplies e show power supply Example Topspin 270 gt show power supply Power supply Information ps type oper status WELL gatiloem
45. ew fabric controller 44 Installing a Fabric Controller Remove the new Fabric Controller module from the packaging Ground yourself appropriately Remove the bezel cover if you have not already done so Refer to Removing a Fabric Controller on page 42 if a Fabric Controller module is still installed Remove the blanking panel if no fabric controller was previously installed Align the new Fabric Controller module with the bay into which you want to install it Gently push the Fabric Controller module into the bay as far as you can Dt Oy Oe we pa When the module is nearly completely installed press the levers in toward the Fabric Controller until they are flat against the module 9 Check the LEDs on the Fabric Controller module and the chassis to verify the status Refer to Fabric Controller LEDs on page 70 10 Tighten the captive screws on both ends of the Fabric Controller module Figure 3 11 Tight the Captive Screws 11 Refer to Upgrading the Controller Card on page 44 Upgrading the Controller Card It is likely that the image version on the new Fabric Controller card will not match the firmware on the rest of the chassis After the new Fabric Controller module has been installed upgrade the firmware on the chassis and the new Fabric Controller will be upgraded automatically The version of the firmware is verified during the upgrade process If a module does not need to be upgraded the modu
46. exists and click the Delete button Repeat for all Server Switches you wish to upgrade 3 Copy the new image to the Server Switch a b C gt Select Maintenance gt File Management Click the Import button File Type Image Select Remote FTP Server or Local File to navigate to the file Import File Copy a file to the systern File Type image v Copy From Remote FTP Server Local File salfile Topspin270 TopspinOS 2 2 0 buildS31 img Copy To File Name on System Topspin270 TopspinOS 2 2 0 build531 img Figure 4 1 Copy the img File to Server Switch e Click the Copy button and wait for the message that indicates the copy was successful f Click Ok 4 Install the new image 55 Note all cards in the chassis must have an oper status of up for the upgrade to be effective on all components a Select Maintenance gt File Management b Click the new img file and click the Install button c Click Yes when prompted d Repeat for all Server Switches you wish to upgrade File Management You can import a file from an external FTP server export a file to an FTP server install an image in the system or delete a file from the system Current Files on Systern Slot ID File Name File Type Size Date Topspin270 TopspinOS 2 2 0 build531 img image 10657535 Thu Jan 27 01 07 34 11 TopspinOS 2 2 Ofuild530 i 6684672 Wed Jan 26 01 07 1
47. he InfiniBand port LEDs are located to the right of each InfiniBand port if you are facing the LIM card in the back of the chassis SS So oo Fo Fo oe IB Port LEDs Figure 5 9 InfiniBand Port LEDs Interpreting the InfiniBand Port LEDs Table 5 15 Interpreting the InfiniBand Port Status LED Top Logical LED Bottom Traffic LED Indication off off No link signal detected off solid on Physical link OK No logical link established solid on solid on Physical link OK Logical link established blinking solid on Physical and logical link OK Traffic active Monitoring the System with Chassis Manager For information regarding launching the Chassis Manager GUI refer to the Topspin 270 Quick Start Guide or the Chassis Manager User Guide Monitor the Device 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Press the Enter key 77 3 The Device Rear View and the Device Front View appear SS TOPSPIN X 10 3 108 4 Device View ag Chassis 10 3 108 4 gt Device View A Cards Ports amp Power Supplies amp Fans Device Rear View eS Sensors ee wp wee eee eee eee ae Ry a Management Ports aA Maintenance H 4 InfiniBand Help Figure 5 10 Chassis Manager 270 Device Rear View SS TOPSPIN stem Name X 10 3 108 4 nunnan 34 Chassis bl oldie S 8 mii sh Ors sart Pow
48. ifies the card on which your CLI session runs In the command output an asterisk identifies the controller card to which you are connected In the example that follows the CLI session applies to the controller card in slot 11 the active card as identified by the asterisk next to the slot number line 10 12 Example i Topspin 270 show card De 35 4 5 Card Information Ge es admin oper admin oper oper os slot type type status status code Q SRS SSS SS SSS SSS SS SS SS SS SS SS SS SS SS 5 SS SS SSS SS SSS SS SSS SSS SS SSS SSS OR 11 controllerFabricl2x controllerFabric1l2x up up normal ae 12 controllerFabric1l2x controllerFabric1l2x up up standby The oper code field of the command output identifies the controller card as the active card normal or the hot standby card standby By default when you power on a Topspin 270 the card in slot 11 becomes active and the card in slot 12 becomes the standby The master and standby controllers automatically synchronize state and configuration information InfiniBand Connectivity The Topspin 270 is an enterprise class managed 960Gb S 6U InfiniBand IB Switch that is capable of supporting up to 96 4X ports In addition the Topspin 270 is designed to support the 12x InfiniBand ports when they become available Flexibility The architecture of the Topspin 270 offers flexibility in that the switch fabric modules are generic and will assume the role of fabric controlle
49. ild531 Image One TopspinOS 2 2 O ouild531 Image Two CPU Description PPC 440GP Rey C Rev 4 129 pvr 4012 0481 IB Firmware Revision 0004031 fes Figure 4 4 Upgraded Current Image 8 Repeat for additional switches in the fabric Upgrading a Topspin 270 with the Chassis Manager For the procedure needed to upgrade a Topspin 270 chassis using the Chassis Manager GUI refer to the Chassis Manager User Guide Upgrading an Individual Controller Card When upgrading a single Controller card for example when replacing an old fabric Controller card with a new one it is recommended that you use the CLI if you need to change the image on the new card You can use the Element Manager or Chassis Manager to upgrade downgrade one Controller card to match the image of another However the GUIs do not support the ability to reboot the standby Controller Therefore you must physically remove and reinsert the standby Controller card to activate the image 57 Upgrading a Controller Card with the CLI For the commands needed to upgrade a single Controller card using the CLI refer to the CLI Reference Guide 1 Check the version of your Controller card with the show card inventory command In the example below only the Controller card in slot 12 needs to be upgraded Example Topspin 270 show card inventory Card Resource Inventory Information S1LOc 1Cl used memory free memory used disk space free disk sp
50. indicates a failure Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example Run the identity test on Power Supply 1 config mode gt diag card 15 gt test led gt gt start start flashing gt stop stop flashing Management Interface Status LEDs Table 5 8 Interpreting the Management Interface Module Status LEDs see Figure 5 6 Green over Yellow off No module power or LED failure Green solid on The Management Interface module is running with no errors detected 173 Table 5 8 Interpreting the Management Interface Module Status LEDs see Figure 5 6 Fan Tray Module Status LEDs Color State Indication Green off No power to the Management Interface module LED failure or yellow LED is ON Yellow solid on Failure of the Management Interface module Operator attention required The label indicates a failure Yellow blinking Identify a Management Interface module Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example For example to run the identity
51. iniBand User Guide 1 Launch the Element Manager 2 Select Health gt Status The Health Status window opens 5 Health Status Power Supplies Fans Sensors Up Time 2h 30rm 28s 10 01 2003 10 37 39 Power Fans Ya Sensors Yi Refresh Close Help Figure 5 18 Element Manager Summary Tab Interpreting the Summary Tab 3 Use the Summary tab to view the status of the Power Fans and Temperature Sensors at once Interpreting the Power Field e A green check next to the Power summary indicates that at least one power source is connected and functioning properly e A red check mark indicates that power supply AC is disconnected Interpreting the Fans field e A green check next to the Fans summary indicates at least one fan is present and functioning properly e A red check indicates a fan failure Interpreting the Sensors field e A green check next to the Sensors summary indicates that the system temperature is at an acceptable level e A red check for Sensor is a warning of high temperature 85 Interpreting the Power Supplies Tab Use the Health gt Status gt Power Supplies tab to view the operating status of the power supplies Health Status Summary Fans Sensors Pou Type OperStatus Utilization Voltage ProductSerialNum PeaSerialNum PcaAssemblyNum FruNum lac up n a 12 n a Inja Inja n a Retesh M E E ciose Het Figure 5 19 Element Manager Power Suppli
52. ious steps you should already have these items configured If your device meets these requirements proceed to Launch Chassis Manager on page 32 Otherwise to prepare your device perform the following steps NOTE Consult your network administrator for an IP address subnet mask and gateway address before you begin this process 1 Ifyou have not yet configured an IP address for the Management port refer to Configure Basic Connectivity on page 28 2 Enter the telnet enable command in global config mode to start telnet services on your device You may need to use the exit command to return to global config mode Example Topspin 270 gt enable Topspin 270 configure Topspin 270 config telnet enable 3 Enter the ip http server command to enable HTTP services on your device HTTPs is also supported Example Topspin 270 config ip http server Launch Chassis Manager To launch Chassis Manager perform the following steps 1 Launch your Web browser Chassis Manager supports the following browsers e Microsoft Internet Explorer version 6 e Netscape Navigator version 6 e Mozilla version 1 4 33 2 Type the Management IP address of your Server Switch in the address field of your browser and press Enter You configured the IP address in Step f File Edit View Favorites Tools Help a Q pack Q x E JO Search she Favorites meda O Ors A lu D 3 Address 10 2 182 44 vE Links Search fo
53. ithin the data center and they do not block any ventilation inlets or outlets e Rack mount the Topspin chassis according to the rack manufacturer recommendations Contact Information Table Ill 1 Customer Contact Information Address Topspin Communications Inc 515 Ellis St Mountain View CA 94043 Telephone Corporate Headquarters 866 TOPSPIN or 650 316 3300 Fax 650 316 3269 Telephone Technical Support 800 499 1473 Email Technical Support support topspin com Web site Support http support topspin com 11 Features Overview The Topspin 270 Switched Computing System provides data center managers with a high performance low latency interconnect e System Architecture on page 11 e About the Topspin 270 Chassis on page 13 e Administrative Features on page 20 System Architecture Hot Standby On the Topspin 270 the inactive controller card remains on hot standby so that it can assume control in the event that the active controller card fails over The hot standby feature lets the Server Switch fail over without rebooting or resetting any cards in the chassis You can access the hot standby controller card with a connection to the serial console port A CLI session to the standby controller provides read only access to the controller You can view but not configure the card Because the hot standby features keeps both controller cards up the show card command ident
54. known MTU 0 Last Changed On none Figure 5 14 Chassis Manager Port Properties Dialog Box Monitor the Power Supplies 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Log into the chassis 3 The Device Rear View and the Device Front View appear 4 View the Power Supply status at the top of the window to quickly check the status color and the number of modules installed If only one number appears then only one power supply is installed The actual integer refers to the slot in which the power supply module is installed A field that read Power Supplies 1 2 would indicate that a power supply was installed in both possible slots e The color of the integer indicates the status of the power supply A green number indicates that the power supply is Up A red number indicates that the power supply has failed 5 Click on Power Supplies at the top of the window to view details 82 The Power Supplies view appears SS Topspin em Name Topspin 270 X 10 3 108 4 Power Supplies a Chassis 10 3 108 4 gt Health gt Power Supplies amp Cards Ports amp Power Supplies ec PSID Type Current Status Utilization Voltage S ee 2 ac up 0 48 A Backplane 1 Management Ports 4 Maintenance Ht InfiniBand Help Data Refreshed At Friday April 23 2004 1 22 11 PM Figure 5 15 Chassis M
55. label indicates a failure Master Management Connection Status LED The Master Management Connection Status LED is a single LED that indicates whether that particular Management Interface module is connected to the system s active master controller Refer to Figure 5 5 to view the location of this LED Table 5 11 Interpreting the Master Management Connection LEDs see Figure 5 6 Color State Indication Green off Management module is not connected to the active master controller or is connected to the system s standby master controller Green solid on Management module is connected to the system s active master controller Ethernet Management Port RJ 45 connector LEDs Refer to Figure 5 5 to view the location of this LED Table 5 12 Interpreting the Ethernet Management Port LEDs see Figure 5 6 Location State Indication Left port LED off Port is not connected or logical link is down Left port LED on Port is connected and logical link is up Right port LED off No traffic is moving on the Ethernet Management port link Logical link may or may not be Up Right port LED on Traffic is moving on the Ethernet Management port link Chassis ID LEDs Figure 5 7 Chassis ID LEDs The Chassis ID Status LED indicates the status of this module Table 5 13 Interpreting the Chassis ID Status LEDs see Figure 5 7 75 LIM Status LEDs Location Color State Indicati
56. le Refer to Installing a New Power Supply Module on page 38 if you are installing a new Power Supply module Installing a New Power Supply Module Caution Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin 270 chassis To insert a power supply unit 1 2 3 Remove the new power supply from the package Ground yourself appropriately Insert the Topspin provided power cord into the receptacle if one is not already connected and plug it into a power source The power cord receptacles are located on the back of the chassis Each power cord receptacle controls the power supply that is closest to it on the same end of the chassis Remove the bezel cover from the front of the switch if you have not already done so Refer to Removing a Power Supply Module on page 36 if a power supply is still in the bay Remove the blanking panel from the power supply bay with a 1 phillips screwdriver if it is still in place 39 7 Insert the new power supply unit into the open slot until it is fully seated You may need to push the unit with your thumbs to get it completely into the bay Figure 3 4 Inserting Power Supply Module 8 Secure the captive screws with a 1 phillips screwdriver 9 Check the LEDs on the power supply to verify the status of the module Refer to Power Supply LEDs on page 67 Fan Tray Modules About the Fan Tray M
57. le will be skipped and move on to the next module The image can be upgraded with the following methods e Upgrade the Image with the CLT on page 45 e Upgrade the Image with the Chassis Manager on page 45 e Upgrade the Image with the Element Manager on page 45 45 Upgrade the Image with the CLI To upgrade the image of the Controller card with the Command Line Interface refer to Upgrading a Controller Card with the CLI on page 57 and the following document e CLI Reference Guide Upgrade the Image with the Chassis Manager To upgrade the system image with the Chassis Manager web GUL refer to the following document e Chassis Manager User Guide Upgrade the Image with the Element Manager To upgrade the system image with the optionally licensed Element Manager GUI refer to the following documents e Element Manager User Guide Line Interface Management Modules LIM About LIM Cards Refer to Line Interface Modules LIM on page 19 for specific information about the LIM modules About a Failed LIM Module If you believe a line interface module has failed contact technical support Refer to LIM Status LEDs on page 75 and Monitor the Cards on page 79 for monitoring information Locating a Failed LIM Module The Line Interface Management modules are located on the back of the chassis Refer to Figure 1 2 for a diagram of the chassis Removing a LIM Card 1 Ground yourself appropriately
58. m a diagnostic self test on a card 1 Enter the diag card command You cannot stop the card test once it has begun Example Topspin 270 gt enable Topspin 270 config Topspin 270 config diag card 11 Topspin 270 config diag card 11 test self test Topspin 270 config diag card 11 start Topspin 270 config diag card 11 exit Topspin 270 config exit 2 Exit the test to view the progress of the test The test takes approximately 5 8 minutes Example of test in progress Topspin 270 show diagnostic card 11 Diagnostic Tests For Carde test self test sior E iil iterations all AGIELOM 8 Sialic result In progress percentage completed 0 result string Example of completed test Topspin 270 config diag card 11 exit Topspin 270 config exit Topspin 270 show diagnostic card 11 Diagnostic Tests For Cards test alor tel Lreracions ac PS tion SSE percentage completed result string system self test IIL start Success 100 Cardi rkest Final report PASSI HDA Please reboot 92 TOPSPIN Thu Mar Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu fae aL Wie ab Fri Iie ak fae aL Eprat eL Ie aL Eras Event OLEOLE 1 3 Use the mo
59. n the system Operator attention required 67 Table 5 1 Interpreting the System Wide LED see Figure 5 1 Color State Indication Yellow blinking System identify This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run identity test on Power Supply 1 config mode gt diag power supply 1 gt test led gt gt start start flashing gt stop stop flashing Power Supply LEDs Locating the Power Supply LEDs The Power Supply LEDs are located in the outside of each Power Supply module if you are facing the front of the chassis with the bezel cover removed Figure 5 2 Power Supply LEDs 68 In addition a Power Supply Identify LED is located on the outside of the chassis next to the corresponding power supply This LED allows the user to initiate a command that causes the Identify LED to blink which assists in locating a particular Field Replaceable Unit Figure 5 3 Power Supply Identify LED Replicas of the Power Supply LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 71 Interpreting the Power Supply LEDs Table 5 2 Interpreting the Power Supply LEDs see Figure 5 2 Color State Indication Green and off No power to the power supply or LED failure Yellow Green solid on Power Supply running wi
60. ntroller module has failed contact technical support Refer to Fabric Controller LEDs on page 70 and Monitor the Cards on page 79 for monitoring information Locating a Failed Fabric Controller The fabric controllers are located on the front of the chassis behind the bezel cover Refer to Figure 1 1 for a diagram of the chassis The slots are labelled on the chassis Considerations Before Removing a Fabric Controller Card Before removing a fabric controller card from either a node or a core slot of the Topspin 270 keep the following in mind 1 The fabric controllers cards in the core slots are paired to the Management Interface modules on the other side of the chassis e Fabric controller card in slot 11 is paired to the Management Interface module in slot 15 e Fabric controller card in slot 12 is paired to Management Interface module in slot 16 Therefore before removing a fabric controller from a core slot make sure that the second pairing of Fabric Controller Management Interface module is intact 2 Removing the fabric controller in the core slot that is currently acting as master will cause a fail over to the standby pair of fabric controller Management Interface modules It takes up to 3 seconds for the standby fabric controller to become a new master a Ifyou are removing a fabric controller card from a node slot the card will be located in slots 9 10 13 and 14 marked on the chassis b Ifyou are rem
61. oOo000 doood Ooo0oD OOOO000 R Kis OOoO000 doood OOO000 ajaj n n aja OOO0000 m ajn auja eres ee Fabric Controller Module Node JOO i Node Slot 7 Slots Aee mim jaja n n a Fabric Controller Module OoOoO00D Node S ot 2 oooo000 saanuna SAggoo Fabric Controller Module Core JOOO Core Slot 1 Slots 79090 55 5 5 a Fabric Controller Module isinin nin 50o oa Core Slot 2 oooooo P m ujajn nla Fabric Controller Module oooo00 Node siot 3 Node 2000 1 Slots R a 5 Fabric Controller Module omoooo Node Slot 4 ooo0000 E l j oo0000 oOo0000 m ajn n nln Figure 1 1 Front Chassis View without Bezel 14 POWER PLUG 2 MANAGEMENT I O ee MODULES E DOOD OOO ODOODOO OOo g CHASSIS ID MAIN AIR EXHAUST 4x COPPER LINE INTERFACE MODULE N ggoggg OO0O000 OOO0000 OOO000 OOO0U00 OOO000 Oo00000 gogoggg ggogogg ogoogoog gogogo oggooogg CE CECECECECI pgpoogogoogg ggogooggo DOo000 DOO000 OOo0O000 OOO0D0o oOooo000 HOODOO ggogogoggg OOoO000 Figure 1 2 Back Chassis View 15 Chassis Slot Numbering Slot numbers are printed on the chassis for easy identification Power Supply ID 1 Fan ID 1 Pply Power Supply ID 2 Fan ID 2 oooo0o0o gogogaga oOoooo0o oo00000 oooo0oo00 ooooo00 ooooo0o0 ooooo0o ooooo00 ooooo00 OOoOo0o00 Slot 9 o ogogooogog OOUUOOU ggogogogg Ooooog Slot 10 oOooogg ala a aja a
62. odules Refer to Fan Trays on page 17 for more information about the fan tray modules About a Failed Fan Tray If you believe a fan module has failed contact technical support Refer to Fan LEDs on page 69 and Interpreting the Fans Tab on page 85 for monitoring information Locating the Fan Tray Both fan modules are located on the front of the chassis When facing the front of the switch with the bezel cover removed the fan modules are located in the left and right receptacles of the Topspin 270 chassis The slots are labelled on the chassis Refer to Figure 1 1 for a diagram of the chassis Removing a Fan Module Note Do not remove Fan module without first loosening the screws with a phillips screwdriver 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 3 Remove the bezel cover from the front of the switch if you have not already done so 40 4 Usea l phillips screw driver to unscrew the captive screws Figure 3 5 Loosening Captive Screws 5 Carefully begin to pull the fan tray 1 out of the bay Do not pull the fan tray completely from the bay ae The fan tray does not have guards over the blades You must wait until the fan blades have stopped moving before completely removing the fan tray from the bay or risk inj
63. on Green Yellow off No module power or LED failure Green solid on Chassis ID module is running with no errors detected Green off No power to the Chassis ID module LED failure or yellow LED is On Yellow solid on Failure of the Chassis ID module Operator attention required The label indicates a failure Yellow blinking Identify a Chassis ID module Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user Refer to the Command Line Reference Guide for more information The LIM Status LED is located at the far left of each LIM card if you are facing the back of the chassis Refer to Figure 5 5 for a complete diagram of the back panel of the chassis LIM Status Indicator Figure 5 8 LIM Status Indicator LEDs Interpreting the LIM Status LEDs Table 5 14 Interpreting the LIM Status LED Color State Indication Green and off No module power or LED failure Yellow Green solid on Line Interface module is running with no errors detected Green off No power to the Line Interface module LED failure or yellow LED is On Yellow solid on Failure of the Line Interface module Operator attention required The label indicates a failure InfiniBand Port LEDs The InfiniBand LED represents the logical link and the logical link activity This LED can be initiated manually by using the diag LED test 76 Location T
64. opooooooooooonpnonooooooonooon oooo0oocoo0pooocopnooocoocooocooconoooooooonpoooconooocnnoooconooococoooo0oo0o0on HE E i e i e ha hhh heh hh hhh hi Figure 2 8 Power Jack Locations 7 Plug the other end of each AC power cable into a 90 264VAC power outlet operating at 47 63Hz Attach the chassis only to approved power sources as indicated by the electrical ratings label If you are unsure of the correct power source to use contact your support personnel or your local power company The chassis automatically starts and boots You can watch the running status via the serial console Use the correct external power source 8 Check the LEDs on the front of the Topspin 270 system When the system first powers up it performs a power on self test POST Refer to System Wide Status LED on page 66 You will know that the system has completed the boot process when you see a login prompt Configure Basic Connectivity 1 Configure an IP address for the Ethernet Management port It is necessary to configure connectivity to the out of box Ethernet management port in order to use Telnet the Web GUI or the Java GUI The system is not pre configured when it ships from the factory IP addresses can be assigned statically which is the default or dynamically assigned using DHCP e Assigning an Address Statically on page 29 e Assigning an Address Dynamically on page 29 29 Assigning an Address Stati
65. ors detected Yellow solid on Operator intervention required Failure detected within the fan tray Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run the identity test on fan 1 config mode gt diag fan 1 gt test led gt gt start start flashing gt stop stop flashing 70 Fabric Controller LEDs Each Fabric Module has two sets of LED indicators The two LEDs green or yellow are the module status indicators Table 5 5 e The single LED green only indicates whether of not the module is the active system master Table 5 6 Location Refer to Figure 5 1 to view the front panel of the chassis Module Active System Maste Status P gt lt Figure 5 4 Locate Fabric Controller LEDs Interpreting the Fabric Controller LEDs Table 5 5 Interpreting the Fabric Controller s Module Status LEDs see Figure 5 1 and Figure 5 4 Color State Indication Green over off No system power or LED failure Yellow Green solid on System running with no errors detected Green off No power to the fabric controller LED failure or yellow LED is On Yellow off No failure on the fabric controller Yellow solid on Module error detected Operator intervention
66. ort4xTX 79 idModule 79 inactive controller card 11 InfiniBand port LEDs 75 L launching Chassis Manager 32 LEDs 71 74 chassis ID 74 chassis manager 78 ethernet management port 74 fabric controller 70 fans 69 InfiniBand ports 75 LIM 75 line interface module 75 management interface module 71 master management connection 74 power supply 67 73 system status 66 LIM about 19 LEDs 75 line interface modules about 19 M management interface module LEDs fans 73 general 71 master connection 74 mgmt interface 72 power supply LEDs 73 system status 72 mgmt O 79 more command 92 MTU 81 N node slot about 18 O oper status 81 operational state 79 P port admin status 81 oper status 81 port details 81 port LEDs 75 port properties 81 port RJ 45 LEDs 74 power off card 80 power supplies about 16 power supply LEDs 67 prepare your switch 32 prerequisites 32 R rack locator 89 reload 92 reset card 80 reset controller card 59 S safety information vii setup 32 show card 11 show diag chassis 93 show diag fru error 96 98 show diag post 94 show power supply 97 slot numbers 15 speed port 81 starting Chassis Manager 32 switch configuration 32 switch fabric end nodes about the fabric controller module 18 switch fabric spine about the fabric controller module 18 system controller about the fabric controller module 18 system wide status LED 66 T temperature monitor with chassis manager 83 V view port properties 81
67. ou do not already know the number a Enter the show fan command Example Topspin 270 gt enable Topspin 270 show fan Fan Information fan oper status speed 3 i up 74 2 up 70 23 up 89 4 up 87 2 Enter the diag fan lt id gt command Example Topspin 270 config Topspin 270 config diag fan 1 Topspin 270 config diag fan 1 test self test Topspin 270 config diag fan 1 start Topspin 270 config diag fan 1 exit Topspin 270 config exit Topspin 270 show diag fan 95 Diagnostic Tests For Fan module type fan module number ik test self test iterations 3 iL action start result in progress percentage completed 70 result string Self Test Current report Topspin 270 Lid SIPFOCIPSS Sic 5 6 3 Use the show diag fan command to show the progress of the test Example Topspin 270 config diag fan 1 exit Topspin 270 config exit Topspin 270 show diag fan Diagnostic Tests For Fan module type fan module number i test self test ICSicaicLoms s il action start result in progress percentage completed 70 result string Self Test Current report Topspin 270 Ta DOG SSS o c 96 4 Use the show diag fru error command to display any errors that appear in the result string Example Topspin 270 show diag fru error Bie rare ea card 2 card 7 card 8 cardio card 1 card 1 carraig card
68. oving a fabric controller card from a core slot the card will be located in slots 11 or 12 Removing a Fabric Controller 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 3 Remove the bezel cover from the front of the switch 4 Determine whether the Fabric Controller you want to remove is installed in a Core slot or a Node slot Refer to Figure 1 1 for a diagram of the chassis If you are removing a Fabric Controller from one of the core slots make sure that the redundant core Fabric Controller is functional Removal of the Fabric Controller currently acting as master will cause a fail over It takes up to 3 seconds for the standby fabric controller to become a new master 43 5 Usea l phillips screwdriver to unscrew the captive screws from both ends of the Fabric Controller card Figure 3 9 Loosen Captive Screws Before Removing 6 Pull the levers on both sides of the card toward you The card will be released Figure 3 10 Opening Fabric Card Levers 7 Use the levers to pull the card toward you and remove it from the bay 8 Place a blanking panel over the open bay No bay should ever be run without a blanking panel or a module installed as overheating may occur or Refer to Installing a Fabric Controller on page 44 to install a n
69. pply to install immediately or a blanking panel to place over the open bay as overheating may occur 36 In most cases vital information can be recovered from the failed module If you have the appropriate support agreement you may want to return the unit for diagnosis Refer to Vital Product Data Storage on page 20 Locating the Power Supply Module Both power supply units are located on the front of the chassis When facing the front of the switch with the bezel cover removed the power modules are located in the top center bays of the Topspin 270 chassis Refer to Figure 1 1 for a diagram of the chassis Removing a Power Supply Module Caution Do not remove a Power Supply module without first loosening the screws with a phillips screw driver 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove a failed power supply unless you have a replacement or a blanking panel to install in its place as overheating may occur 3 Make sure you have a 1 phillips screwdriver to disengage the fasteners Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin 270 chassis 4 Remove the bezel cover from the front of the switch 5 Locate the power supply that you want to remove Refer to Power Supply LEDs on page 67 If you have two power supply modules installed you can remove one of
70. pspin 270 Power Supply Errors Error Number Description 2 Power Supply Fan Fail Error 3 Power Supply I2C Communication Error 102 Specifications and Compliance Certifications This chapter details the Topspin 270 specifications and compliance certifications Chassis and Management Interface Table 7 1 Chassis and Management General Specifications 103 Operating Temperature Ranges 0 to 40C Operating Temperature 0 to 42C Non Operating Temperature 40 to 70C Operating Temperature Gradient 20C max per 60 minutes Operating Altitude 0 to 10 000 ft Non Operating Altitude 0 to 40 000 ft Operating Humidity 8 to 80 non condensing Non Operating Humidity 5 to 90 RH at 65C for 24 hrs Non condensing Operating Humidity Gradient 10 maximum per 60 min Operating Shock 5G max 1ms half sine wave 10G max 5ms half sine wave Non Operating Shock 10G max 11ms half sine wave Operating Vibration Sinusoidal 0 25G max 3 200Hz 15min Non Operating Vibration Sinusoidal 0 50G max 3 200Hz 15min Operating Vibration random 1 02Grms 3 axis bottom top left right front back 104 Table 7 1 Chassis and Management General Specifications Operating Temperature Ranges 0 to 40C Non Operating Vibration random 2 09Grms 3 axis bottom top left right front back Max Operating Inclination 15 degrees Electrical Specifications
71. r P Web Search z RAI alealed 2 Popups blocked rete 2301 Figure 2 10 Web Browser Window vuele A log in window opens Figure 2 11 displays the log in window Connect to 10 3 106 83 A Topspin Web Management Interface User name super Password COLET Cl Remember my password Figure 2 11 Chassis Manager Log In Window 3 Enter your Server Switch user name and password in the log in window and click the OK button Chassis Manager loads in your browser window SS TOPSPIN X 10 3 108 4 Device View a Chassis 10 3 108 4 gt Device View Cards amp Ports Refresh m Power Supplies Fans Device Rear View M Sensors E Backplane SER ag Management Ports s s Maintenance InfiniBand Help 10 12 4 2 3 4 5 6 7 8 Figure 2 12 Chassis Manager Opening 34 Optional Install the Element Manager GUI For more information regarding the Element Manager tasks refer to the InfiniBand User Guide 1 Check that you have sufficient system resources You will need e 32 MB free RAM e 50 MB disk space 50MB additional temporary space during installation e 300 Mhz processor e 800x600 screen resolution 16bit color depth Insert the Topspin CD ROM into the CD ROM drive of a server that has network access to a Topspin management port The GUI can be found in the following location on the CD ROM e Linux install_linux_ia64 bin e Linux install_lin
72. r a Fibre Channel switch you must add a Topspin device with the optional Ethernet or Fibre Channel gateway such as the Topspin 90 or Topspin 360 to your fabric Connecting Multiple IB Switches Two InfiniBand systems such as a Topspin 120 and a Topspin 270 can be connected by an InfiniBand cable inter switch link ISL System You can manage the InfiniBand server switch system using the following methods e Command Line Interface CLI a text based interface accessible through a direct serial connection Telnet over IP or SSH over IP 32 e Chassis Manager GUI A web based graphic user interface e Element Manager GUI An optionally available graphic interface installed on a workstation and accessible over IP Refer to the nfiniBand User Guide the CLI Reference Guide or the Chassis Manager User Guide for more information about managing the InfiniBand systems Optional Launch the Chassis Manager Chassis Manager is a web based GUI that can be used to manage a single InfiniBand chassis Prepare Your Device To launch Chassis Manager on your Topspin 270 you must e Configure an IP address on the Ethernet management port if you have not already done so e Configure an IP gateway on the Ethernet management port if you have not already done so e Optional Enable Telnet services should be enabled by default e Optional Enable HTTP or HTTPs services HTTP should be enabled by default If you followed the prev
73. r or fabric node depending on the slot into which they are installed This increases the ease of replacing components because only one type of fabric card has to be stocked and will become a controller or node depending on what is needed Hi Availability Hardware The Topspin 270 features hot swappable and redundant e Power Supply Modules e Fan Trays e Line Interface Modules redundant connection from HCA e Fabric Controller Modules in redundant Core or Node slots e Management Interface module The Topspin 270 features hot swappable e Chassis ID Ports No failure on any single InfiniBand port will result in interruptions in service for any of the other ports 13 Fabric For redundancy InfiniBand Host Channel Adapters HCAs can be dual connected to a redundant pair of Topspin 270s In an InfiniBand fabric that includes more than one Topspin 270 if the subnet manager on the Topspin 270 that is acting as the master fails another subnet manager will take over within seconds All necessary state information is kept in sync Non Blocking Architecture The Topspin 270 provides non blocking switch element architecture with full bi directional bandwidth for the switch chassis About the Topspin 270 Chassis An overview of the Topspin 270 hardware features are described in this section Diagram of the Chassis POWER SUPPLIES FAN FAN oO
74. rd that is being monitored 88 89 Hardware Diagnostic Tests This chapter describes how to run diagnostic tests on the Topspin 270 system hardware e About Diagnostic Tests on page 89 e Displaying Hardware Errors on page 99 About Diagnostic Tests Hardware diagnostic tests can be performed through the CLI or the Element Manager GUI Refer to the Element Manager User Guide for instruction on running tests in with the Java GUI A Power On Self Test POST is run automatically on various components at power on The following examples refer to running tests and displaying test with the CLI For complete diagnostic command information refer to the Command Line Reference Guide LED Tests LED tests allow you to blink a particular LED to more easily identify a specific component in a rack Self Tests The Topspin 270 provides the following diagnostic tests e Card Test page 90 e Chassis Test page 92 e Fan Test page 94 e Interface Test e Power Supply page 96 e Rack Locator 90 Run Card Tests References to Cards refers to fabric controllers management I O modules node cards and the chassis ID module Fans and power supplies are not included in the Card tests they can be tested through the standard chassis test page 93 or as individual components The following tests are available to locate and diagnose one or more cards in the chassis e LED e Self test page 90 Run a Card Self Test To perfor
75. re file system file name command to display the log file and view the detailed results of the diag test Example of log results 70 more syslog hwif_log 4 UOsSIS49 ZOOS POSWE SM ZROMS BANS OED 4 10 57 49 2004 POST FPGA PASSED A ILOgS S4o ZOO4s IWOSWs SUMM IVS Sap ASSO Ron s 151 Ome O10 Ate Splish Ce AtS SED ATTOR S 200 POST FAN PASSED 4 10 57 54 200 Cared Srartto 3 Carel 1S Staicaimeg WS 4 10 57 54 2004 Anafa2Init a2update set to IGNORE 4 10 58 16 2004 Anafa2 POST firmware check PASSED 4 UPSsOlsss ZOO4As POSWS SMEZIROMS E PASO ED ARIES OMN gt ome 70 045 ie POS N RRCA PASSED A ADs OiLsSS ZOO4s WOSWs SUIMIMAIPS ieYNSSTaID 4 iMeOless6 ZOO4s POSWs ININCS IAS SED AIRS O 101m O10 45 See Sule T EAN RASS ED 4 193023800 2004s Cered Sica 3 Carci 1S Stating wo 4 19 02 00 2004 Anafa2Init a2update set to IGNORE AES S08 26 2004 POs Sh HE PROM PASS HD 4 19203256 2004 POST FPGA PASSED 4 IDsOSsea7 ZOO4s POSS SUMMONS LYAS SIRI 4 iGeOSsses ZOOS POSWs IVINS RAG SIaID 4 19 04 02 2004 POST FAN PASSED A U9s04302 2004 carc Sic Carci 1S Sicelicieiime US 4 19 04 02 2004 Anafa2Init a2update set to IGNORE 4 19 04 23 2004 An
76. ribution thereof by or for the U S Government shall be governed solely by the terms of the license agreement and shall be prohibited except to the extent expressly permitted by the terms of that agreement Any technical data provided that is not covered by the above provisions shall be deemed technical data commercial items pursuant to DFAR section 227 7015 a or equivalent Any use modification reproduction release performance display or disclosure of such technical data shall be governed by the terms of DFAR section 227 7015 b or equivalent This documentation may include technical inaccuracies or typographical errors and is subject to correction and other revision without notice TOPSPIN PROVIDES THIS DOCUMENTATION AS IS AND WITHOUT ANY WARRANTIES OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states or jurisdictions do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you Copyright 2004 2005 Topspin Communications Inc All rights reserved Topspin is a registered trademark and the Topspin logo TopspinOS and Topspin Switched Computing System are trademarks of Topspin Communications Inc Other company product or service names are the property of their respective owners Document Version 2 2 0 Update 1 Part Number OL 7725 01 Product ID 140265
77. ropriate for pre racking and shipping a unit in the rack If you have purchased this shelf unit separately refer to the documentation that is provided in the shelf kit package Requirements In addition to the accessories provided with the server switch you should have 2 phillips screwdriver Rack with 6Us of available space Shelf installed in the rack not provided The shelf must support the weight of the chassis which can be up to 106 pounds with all possible components installed Screws appropriate to your rack which are used to secure the rack handles to the rack posts Three people are recommended to lift the switch into the rack however the procedure can be performed by two people Note The Topspin 270 chassis weighs up to 106 pounds fully loaded Rack Mount Installation To mount the Topspin 270 chassis in a rack 1 2 3 Remove the CD ROM parts bag and documentation from the box Remove the packaging from around the chassis Install a shelf for the chassis if you have not already done so Verify that the shelf can support the weight of the chassis which can be up to 106 pounds with all possible components installed Figure 2 1 Install Appropriate Rack in Shelf 23 Determine the direction that the switch will be installed and justified in the rack The direction that the switch will be justified will determine which set of rack ears that need to be removed Use a 2 phillips screwdriver to remove
78. s free disk space 71488 kbytes last image s6urce 3 WejpssjimOS 2 2 0 iouLld4as primary image source TopspinOS 2 2 0 bui1d487 image TopspinOS 2 2 0 build487 cpu descr PPC 440GP Rev C Rev 4 129 pvr 4012 0481 fpga fLirmware rev ae ib firmware rev 0004031fe8 3 Check to see if you already have a img file that can be installed Delete the img file if it not the one you need to install Use the delete s ot number jimage file name command Example Topspin 270 dir image Existing Boot Images on System slot date created size file name iL Taw Jam 27 01 07 36 2005 10657935 WoospimZ27 O TopsSi100S 2 2 0 oULLES Si q e ii mau Jam 27 OlgZIsO2 2005 CG6SagyZ MepspimMOS 2 52 0 owiilcs3il Topspin 270 4 Copy the image to the switch if you do not have the correct img file copy ftp lt remote login gt lt remote passwd gt lt remote system gt lt filename gt img lt slot number of the controller gt image lt filename gt img Example Topspin 270 copy ftp bob mypassword 10 0 0 5 Topspin 360 TopspinOS 2 2 0 build531 img IMACS 8 topspin 60RA 1 0 owLIcA 7 ase LCGOS OLMAOS 2Z I OAlg ua Ika 7 ain 59 5 Activate the image After downloading the image file to the chassis controller it must be installed to become active The install command installs the specified image file into the system a Inthe privileged execute mode enter inst
79. s test standard Topspin 270 config diag chassis start Topspin 270 config diag chassis 2 Enter the show diag chassis command to show the status of the diag test Example Topspin 270 config diag chassis exit Topspin 270 config exit Topspin 270 show diag chassis DIACGMOSeELe Tests For Chassis module type module number chassis test iterations OSE LOI AGCLOMm result percentage completed tandard S none STALE success 100 result string Standard System Test Completed Final Failed 0 Total 1 Topspin 270 TEOLL Passed 1 94 3 Enter the show diag post command to show any errors on individual components Example Topspin 270 show diag post Post Scaicus reuU S Loc post status error codes camai passed 05030 0 card 2 passed 0302050 card 7 passed 0030 00 carde passed 00300 card 9 passed 050300 card 10 passed 00 050 euecl 411 passed 050300 card 12 passed 020300 canditi passed 05030 0 card 14 passed 0202050 Saree lky passed 0503 0 0 camai passed ORTOP ORO fan 1 passed 0s 030 0 fan 2 passed 00 050 fan 3 passed 020300 fan 4 passed 020300 power supply 1 passed 0 0 0 0 power supply 2 passed 0 0 0 0 Run Fan Tests The following tests are available to locate and diagnose fan modules LED e Self Test page 94 Run a Self Test on a Fan 1 Identify the number of the fan you want to diagnose if y
80. ssis Refer to Figure 1 2 fora diagram of the chassis The slots are labelled on the chassis Removing a Management Interface Module 1 Verify that you have a method of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 2 Determine if the Management Interface module that you are removing is currently connected to the master fabric controller Note The removal of the Master Management Interface module will trigger a fail over of the Fabric Controller module Switch traffic will be disrupted If the system is left without a Management Interface card the switch will become inoperative If the Master LED is lit then the module is connected to the master fabric controller module and a fail over will occur when you remove the management interface module The system will fail to the redundant management interface module and the redundant fabric controller module 3 Ground yourself appropriately 4 Unplug any Ethernet management or serial cables that are connected to the ports of the Management Interface module 5 Usea l phillips screwdriver to unscrew the Management Interface module 6 Use the handle to pull the module from the bay B86 4 4 424A GGG Ah GF Sf ee Sess Figure 3 19 Example of the Management Interface Module 50 7 Ifthe Master Management Interface module was removed the standby
81. t 15 e The Fabric Interface Module in the Core slot 12 is connected to the Management I O module in slot 16 Line Interface Modules LIM Refer to Installing a LIM Module on page 48 for installation instructions The Topspin 270 has hot swappable Line Interface Modules LIMs which are installed in the rear of the chassis this is opposite of the fabric modules see Figure 1 2 on page 14 The LIMs provide the physical InfiniBand port interfaces and are 4x over copper The number of LIMs in your Topspin 270 chassis depends on your particular configuration Chassis ID Module The chassis ID module is hot swappable and is located on the rear of the chassis see Figure 1 2 on page 14 Figure 1 6 Chassis ID Module How the Chassis ID is Used Upon initial boot up of the Topspin 270 the Chassis ID is read by the system and the unique information is populated to the rest of the chassis The Chassis ID module contains model and serial number information and logs certain error information Refer to Vital Product Data Storage on page 20 About Replacing a Chassis ID Module In the event that a Chassis ID module must be replaced the replacement module will arrive with your previous serial number and GUID When a Chassis ID modules is installed the new Chassis ID is detected This allows your chassis to maintain the same identification and GUID Rack Alarm The Rack Alarm also referred to as the Rack Locator fea
82. t Enter gt several times to display the CLI prompt Login 30 b Enter the user name and password The default user name is super and the default password is super Login super Password super Rosi Lia Z 70 gt c Enter the enable command Topspin 270 gt enable Topspin 270 d Enter the configure command Topspin 270 configure Topspin 270 config e Enter the interface mgmt ethernet command Topspin 270 config interface mgmt ethernet f Enter the option dhcp command The chosen method will be persistent across module reboots Topspin 270 config if mgmt ethernet option dhcp g Enable the management port by entering the no shutdown command Topspin 270 config if mgmt ethernet no shutdown h Save the configuration to preserve it between reboots Topspin 270 config if mgmt ethernet exit Topspin 270 config exit Topspin 270 copy running config startup config If you have two Fabric Controllers installed in Core slots you will receive a message asking if you want to save the configuration to the Standby Controller i Answer Yes to save the configuration across both controllers Connect Network Devices This section describes how to connect the InfiniBand system to other network devices InfiniBand devices can be connected to InfiniBand enabled servers About Connectors The Topspin 270 device supports the following types of connectors e Management ports e Serial console port male D
83. t Show Ports The list of IB ports for the selected LIM card will be displayed The Ports view appears X 10 3 108 4 Ports a Chassis 10 3 108 4 gt Chassis gt Ports amp Cards Ports Filter Card 5 a Sensors 3 Backplane a a Donar ee La E a Management Ports O ib4xFX down SAs Maintenance O 52 52 ib4xFX up down 0 a mi On S S 5 3 ib4xFX up down 0 O 5 4 5 4 ib4xFX up down 0 OTOS 5 5 ib4xFX up down 0 On ro 5 6 ib4xFX up down 0 On ou a7 ib4xFX up down 0 On pis 5 8 ib4xFX up down 0 SF 5 9 ib4xFX up down 0 On Sa S ib4xFX up down 0 OR Sahl pst ib4xFX up down 0 Jae pale ib4xFX up down 0 Data Refreshed At Friday April 23 2004 1 03 31 PM Figure 5 13 Chassis Manager Ports Window 81 7 View the Admin Status and the Oper Status of the ports The Admin Status refers to the state that the admin has chosen the port to be The Oper Status refers to the actual state of the port For example an admin may have selected a state of Up for a port but the Oper Status would display down if a cable had not been connected to the port 8 Click the radio button next to a port whose details you want to view 9 Click the Properties button The Port Properties dialog box appears Port Properties Port 5 3 Name 5 3 Type ibAxF Xx Admin Status up O down Oper Status down Auto Negotiation Supported true Auto Negotiation Enable Set Port Speed 2500M 10G 30G Current Speed un
84. tes and hardware errors 1 Use the more syslog hwif_log command to locate the topspin log hwif_log Example of output without errors Topspin 270 gt enable Topspin 270 more syslog Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar Thu Mar lt output Sia Baie 2 57 49 2 57 34 Soule 9 O 2004 2004 2004 2004 200 5 ONEA SOF 200 Sikh eas 200 358 le sO1lsss SOLE ss gO1L255 3015 56 102010 310200 9702500 truncated gt A fey ey flay fe ey fey ay T ret R ree pe ee gle Me KE Ke Koy Kep KG 1S GS Ei CO Goa feet fy let H H H 2004 2004 2004 2004 2004 2004 2004 2004 hwif_log EOST SHI2INOM S ERENS S PROS INIEVNS IVS Sab OSES SiUIMIMOCIES IeYAVS SNe POSS IRM IVS SED OSS IVAN S 1274S SED Gaiecl SicsweieUyjonx Anafa2In Anafa2 POST BROSIS POSTE BOSHE BOSHE BOSIB card 5 Anafa2Iin sitar SE EPROM IN IPG S SU RE MMI Ge FAN ca a2update set to IGNORE firmware check PASSE PASSI PASSI aes IPAANSHS PASS PASS IPE WYO 5 3 ait a2update set to IGNORE s eal eal 99 card is starting up ED E EA D D G ED E D card is starting up G You can also
85. th no detected errors Green off No power to the power supply LED failure or DC output failure Yellow off Power Supply running with no detected errors Yellow solid on Operator intervention required Failure detected within the Power Supply Table 5 3 Interpreting the Power Supply Identify LED Color Yellow State blinking Fan LEDs Location Indication Identify Assists in identifying a particular Field Replaceable Unit on the chassis see Figure 5 3 This state is initiated by the user This LED status is initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run identity test on fan 1 config mode gt power supply 1 gt test led gt gt start start flashing gt stop stop flashing 69 The Fan LEDs are located at the top of each Fan tray module Refer to Front Panel LEDs on page 66 In addition replicas of the Fan Tray LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 71 Interpreting the Fan LEDs Table 5 4 Interpreting the Fan LEDs see Figure 5 1 Color State Indication Green and off No power to the fan tray or LED failure Yellow Green solid on Fan tray running with no detected errors Green off No power to the fan tray LED failure or yellow LED is ON Yellow off Fan tray running with no err
86. the fabric controller cards the card that is currently on standby not the master will be grey The master fabric controller will be green e A red LED indicates that an error has been detected Monitor the Cards 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser Press the Enter key The Device Rear View and the Device Front View appear Click open the Chassis file in the left navigation tree The individual components appear beneath the Chassis menu Click Cards in the left navigation tree The Cards view appears X 10 3 108 4 E i Chassis lt g a Fans Sensors Backplane Management Ports S Maintenance InfiniBand Help m Power Supplies R B E Cards 10 3 108 4 gt Chassis gt Cards ECC Current Status Operational State Boot Stage Boot Status ib12port4xTX up normal z 2 ib12port4xTX up normal jE ib12port4xTX up normal 10 fabric12x up normal 11 controllerFabric12x up normal 13 fabric12x up normal O 15 mgmtlo up normal O 16 mgmtlo up normal 17 idModule up normal Data Refreshed At Wednesday April 21 2004 12 30 59 PM Figure 5 12 Chassis Manager Cards Window You can monitor the following types of cards none none none done done done none none none none none none success success success none none none InfiniBand 12 port Lin
87. tory tab and view the version name in the Current Image Source field IB Fabric Core Card 11 10 3 108 6 Inventory Used Memory 17672 KB Free Memory 110088 KB Used Disk Space 30879 KB Free Disk Space 71500 KB Current Image Source TopspinOS 2 2 Ofouild531 Image Source for Next Reboot TopspinOS 2 2 0fuild531 Image One TopspinOS 2 2 O0 build531 Image Two CPU Description PPC 440GP Rey C Rev 4 129 pvr 4012 0481 IB Firmware Revision 0004031 fed Figure 4 9 Upgraded Current Image Card Card 63 Upgrading Node Card s with the Chassis Manager For the procedure needed to upgrade node cards using the Chassis Manager GUI refer to the Chassis Manager User Guide 64 65 Monitoring the Topspin 270 System This chapter describes how to install and manage the Topspin 270 system hardware e Interpreting the Front Panel LEDs on page 65 e Interpreting the Back Panel LEDs on page 71 e Monitoring the System with Chassis Manager on page 76 e Monitoring the System with Element Manager on page 84 e Monitor the System With the CLI on page 86 Interpreting the Front Panel LEDs The front of the Topspin 270 chassis has the following types of LED indicators e System Wide Status LED on page 66 e Power Supply LEDs on page 67 e Fan LEDs on page 69 e Fabric Controller LEDs on page 70 66
88. ture is supported with release 2 2 0 and later 20 Administrative Features Real Time Clock The Topspin 270 maintains correct time regardless of power conditions or connectivity Latency The Topspin 270 has port to port latency of less than 600ns Non Volatile Memory The memory supports up to e three stored system images not including recovery image e one week of log files at normal verbosity and one day of log files at maximum verbosity Vital Product Data Storage Vital Product Data VPD is stored in non volatile memory in the Chassis ID module refer to Chassis ID Module on page 19 and the Fabric Controller that is installed in the Core slot refer to Fabric Controller Modules on page 18 and is available electronically Some VPD info can be accessed via CLI the Java GUI or the Web GUI The following Vital Product Data can be recovered by returning a Field Replaceable Unit to the original manufacturer Note This requires the appropriate support contract e Power on hours e Manufacturing part number e Serial number e Final test date e Card ID e Failure code e Failure date e Operation status e Failure log e OEM part number Diagnostics Refer to About Diagnostic Tests on page 89 for more detailed information The following tests are run to determine operational status e Power On Self Test POST is performed on all system components during power on to determine operational rea
89. ule on page 38 for installation instructions The Topspin 270 has two AC DC bulk power supply modules Only one power supply in either of the two slots is required to power the system However a blanking panel is required for any unused bay 17 e The power supplies are 1U 48V 1200W e The supplies provide regulated 48V DC to all other modules in the system e Each power supply has self contained fans for cooling Fan Trays Refer to Installing a Fan Module on page 41 for installation instructions The fan trays are redundant hot swappable cooling units e Only one fan tray in either of the two slots is required to cool the system However a blanking panel is required for any unused bay e The replacement of any one fan tray does not disrupt the operation of the device in any way and can be successfully completed without having to remove the device from a rack or disconnect any cables The fan modules use DC Voltage regulators to control fan speed and contain a hardware fan control circuit which controls and monitors the fan speed e The fan module will default to full speed after a power on or reset condition e If one the fans fails the redundant fan will go to full speed if one is available A CAUTION The fan modules do not have guards Do not remove a fan module completely from the bay until the blades have stopped moving approximately 15 seconds Management Interface Modules See Figure 1
90. urself appropriately Remove the blanking panel if a LIM module was not currently in place Align the LIM module and slide it into the bay as far as possible The module will not go into the bay completely without closing the levers When the LIM module is nearly installed in the bay press the levers in toward each other Figure 3 17 Closing the LIM Module Levers Check the LIM status LED to determine that the module is functioning properly Refer to LIM Status LEDs on page 75 Tighten the captive screws on both ends of the LIM card Nig Bas Figure 3 18 Tighten the Captive Screws 49 9 Connect the InfiniBand cables to the ports a To plug in an InfiniBand cable push the connector into the interface until you hear feel a click Check the InfiniBand port LEDs to determine the status of the port Refer to InfiniBand Port LEDs on page 75 Management Interface Modules About Management Interface Modules Refer to Management Interface Modules on page 17 for additional information About a Failed Management Interface Module If you believe a Management Interface module has failed contact technical support Management Interface modules are hot swappable Refer to Management Interface Module LEDs on page 71 and Monitor the Cards on page 79 for monitoring information Determining Which Management Interface Module has Failed The Management Interface modules are located on the back of the cha
91. ury 6 Wait 15 seconds to allow the fan blades to stop moving a Figure 3 6 Partially Removing Fan Tray 7 Pull the fan tray completely out of the bay 8 Install the blanking panel in place of the fan module Only one fan module is required to run the switch however the device should never be run without either a blanking panel or module in any bay as overheating may occur 41 or Install a new fan module Refer to Installing a Fan Module on page 41 if you are installing a new fan module Installing a Fan Module Remove the new fan module from the package Ground yourself appropriately Remove the bezel cover from the front of the switch if you have not already done so Refer to Removing a Fan Module on page 39 if a module is still in place Remove the blanking panel if no fan module was in place Oy Ou eee Align the fan module with the bay and slide the module into the bay Figure 3 7 Pushing Fan Module into Slot 7 Push the module until it is firmly in place 8 Use the 1 phillips screw driver to tighten the captive screws Figure 3 8 Tightening Captive Screws 42 9 Check the LEDs on the fan module to verify the status Refer to Fan LEDs on page 69 Fabric Controllers About the Fabric Controllers Refer to Fabric Controller Modules on page 18 for more specific information about this module About a Failed Fabric Controller If you believe a fabric co
92. uuo hhh oo0000 o00000000000 EEHEEHE pe eee Figure 2 7 Recommended Management Configuration Power on the Chassis Use only the power cable provided with your InfiniBand system 4 Remove the power cords from the shipping package One power cord UL rated 12 Amps 125 VAC or greater is provided by default If you have a high availability unit you will have a second power cord 28 5 Inspect the power cord and determine if it provides the proper plug and is appropriately certified for use with your electrical system Discard the cord if it is inappropriate for your national electrical system and obtain the proper cord as required by your national electrical codes or ordinances Grounding is supplied by the ground prong on the 3 prong power plug Do not attach a separate ground cable Do not use adapter plugs Do not remove the ground prong from the cable Ensure the ground connection on the power supply is correct and functioning before applying power to the chassis 6 Insert the power cords to the power jacks on the rear of the chassis If you only have a single power cord insert it into the left receptacle Refer toFigure 2 8 for power jack locations oJ ocoooooooooocooooooooooocooooocooooocooooooooa0o og DPEN lt si I a soo osccee Mn oococooocooocoooocoooooooococooooocooooooooooooooooooo0o000O0 oooo0o0 oooo0o00 SS Se So oS eo os eee ooocooocoopnococoooooooooonoo
93. ux_x86 bin e Windows install exe e Solaris install_solaris_sparc bin Run the installation wizard a On Windows double click install exe b On Solaris and Linux systems enter sh install_yourOS bin c Follow the on screen instructions d Enter location and preference information as requested by the installation program 6 Start the Element Manager 35 Managing Individual Components This chapter describes how to install the Field Replaceable Units FRUs in the Topspin 270 system About the Field Replaceable Units The following Field Replaceable Units FRUs are a part of the Topspin 270 chassis e Power Supply Modules on page 35 e Fan Tray Modules on page 39 e Fabric Controllers on page 42 e Line Interface Management Modules LIM on page 45 e Management Interface Modules on page 49 e Chassis ID Modules on page 50 Power Supply Modules About the Power Supply Modules The Topspin 270 power supply units are hot swappable Only one power module is necessary to power the unit however two power supplies are recommended for redundancy You can add a second module without powering off the chassis If you have two power supply units installed you can remove one of them without removing power from the chassis About a Failed Power Supply If you believe a power supply module has failed contact technical support Do not remove a failed power supply unless you have a new power su
94. viede 22 Reg irements aseron tse erased hacia esis Witla E a la T d ea upstate sped atiauae mgs anede Ronse 22 Rack Mount Installation i iei anrea aa Samco Aa e a EA aE es eda Sra a AAEE EE 22 T stall Optional HA Components ac oe septinetsvca ote hace EE a EER EEE A E E 25 iii Connect Management Devices i1 0ssepitscs dias eitesrcod tied Aue bcos Antara i A eo eas ana Aas cis 26 Attach a Serial Console Cable to a PC or Terminal ccccccecccceseeeceesceeeseceeceeeeeaeenseeeeeeenseees 26 Connect an Ethernet Cable to the Ethernet Management Port 00 cceceeceesseeeteceeeeeeeeeeseeeeaeenes 27 Recommended Management Configuration s si 0 ois cst entsusy od Seuss nai aa lees oadave nas 27 Power on the Chassis gh cies ae e a a a a eva ge wis A ae a AE Eo 27 Configure Basic Connectivity vices css etsstds ssekay seas teas asatextecattde Soyo bdeseeid o REE E i EAER 28 Assigning an Address Statically yocs 24 snes Polcesaledeaedguacd ia iari a E A seta toes Bien 29 Assigning an Address Dynan Cally sie scscce cs asucccsvectusssoeeeevvs da seaeaavavelaliasabdonatedtbeninea nage tsdeseiaaskes 29 Connect Network DEVICES siera aes a eh Li E hace le a hi ae NO ag ta cea gras 30 About COMES OLS sok nsec ast aaccnscetcase dst E h e e a E neseneceet eas 30 Connecting InfiniBand Devices 2 serrata suis saci dee estonia carne ah ae oe bac cuar ate tere Mit 0d 31 Ma ge the Systemi aad 23 fel a a ea eubadeplakeinnchawe siti dasvazedsaabacas a E T EAT
95. view POST error codes with the show diagnostic post command This command displays the results of each POST that executed at power on Example Topspin 270 show diagnostic post EO Siar mca lls power supply 1 OSC SS caicus failed passed passed passed error codes 100 To view FRU errors on your Topspin 270 enter the show diagnostic fru error command Example Topspin 270 show diagnostic fru error ie WIS ie ie Lieu SILO PAUS SARRO cancel CL FRU POWER CONTROLLER CONFIG ERR fan 1 none fan 2 none power supply 1 none Error Types The following types of hardware errors are logged for the Topspin 270 baseboard Table 6 1 Topspin 270 Baseboard Errors Error Number Description 1 Summit Power Controller Configuration Error 2 12C Communication Error 3 Serial EEPROM Error 4 DiskOnChip Error 5 Memory Error 6 March SSO Error 7 Realtime Clock Error 8 Fan Error 9 FPGA Error 10 Summit High Temp Error 12 Summit High Current Error 14 Summit Voltage Out of Range Error 16 Firmware Error on the switch chip 17 Bus error on switch chip 18 ID Module Error 19 Management Module Error The following types of hardware errors are logged for the Topspin 270 power supply Table 6 2 Topspin 270 Power Supply Errors Error Number Description S Power Supply DC Fail Error 101 Table 6 2 To
96. voltage i AC up n a 48 2 AC down n a 48 Power supply Seeprom precoouce pca pca EU ps serial number serial number number number al 2 Monitor the Fans e show fan An oper status of up means that the blower is operating correctly within the fan unit 87 Note the Speed field The integer in this field represents a percentage The percentage changes based on the ambient temperature of the unit and will increase as the temperature rises Example Topspin 270 gt show fan Fan Information fan oper status speed 1 up 76 2 up V7 3 up 74 4 up V7 Fan Seeprom product pca pca EU fan serial number serial number number number 2A D 3 4 Monitor the Sensors e show sensor Example Topspin 270 show sensor Sensor Information sensor oper status oper code temperature c alarm temp c shutdown temp c 10 1 up normal 35 73 85 EII up normal Salt 15 85 12 1 up normal Zo 75 85 LS 1 up normal Cull 75 35 15 1 up normal 38 70 80 16 1 up normal 37 70 80 Note the Temperature field The temperature is displayed in Celsius The system s maximum external ambient temperature is 30 degrees C 0 10 000 ft Acceptable internal temperature ranges are 30 degrees C above external ambient plus 1 degree for every 1 000 ft above sea level The acceptable temperatures for the Topspin 270 varies depending on the ca

Cisco Systems SFS 7008 Server User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents