Home

Sun Microsystems 6900 Outdoor Storage User Manual

Contents

1. TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array 5 2 S 5 3 z p 3 E 5 S F E 8 i B lt lt volume State Red Y 1 Open a Telnet session Change to the affected Sun StorEdge T3 array 2 Verify the status of the LUNs with vol_mode or vol_stat Drive Status Messages 0 Drive mounted 2 Drive present 3 Drive is spun up 4 Drive is disabled 5 Drive has been replaced 7 Invalid system area on drive 9 Drive not present D Drive disabled is being reconstructed S Drive substituted power State Red Y The Sun StorEdge T3 1 Check the power Change array has reported that a supply and cables power cooling unit has 2 Replace PCU if been disabled necessary A PCU failure can happen due to 1 Power loss 2 The PCU fails 3 The power switch is disrupted enclosure Statistics Displays statistics about the Sun StorEdge T3 array enclosure Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 105 Sun Proprietary Confidential Internal Use Only 106 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 9 Troubleshooting Virtualization Engine Devices This chapter describes how to troubleshoot the virtualization engine component of a Sun StorEdge 6900 series system This chapter contains the following sections a About the Virtualization Engine on page 107 a Virtu
2. TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array E g 5 z gt B ii B lt a lt power temp Alarm The power temperature is normal sysvolslice Alarm Yellow Y The vol slice feature is This option is disabled possible in Sun StorEdge by default To activate T3 array firmware the feature type version 2 1 and above sys_enable_volslice_on This option enables from the Sun StorEdge volume slicing up to 16 T3 array command line LUN per single Sun StorEdge T3 array or partner group This feature also enables LUN masking HBA zoning features disk port Alarm Red Y The Sun StorEdge T3 1 Open a Telnet session array has reported that to the affected Sun one port of a dual ported StorEdge T3 array disk has failed 2 Verify disk state in fru stat fru list and vol stat 96 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array 2 z z z c E 5 S 3 E 8 i B lt lt interface Alarm Red Y The Sun StorEdge T3 Open a Telnet session loopcard cable array has reported that a to the affected Sun loopcard is in a failed StorEdge T3 array state Verify the loopcard state with fru stat Possible Drive Status Verify the matching Messages firmware with the other
3. Alternate Master Drive MPDrive 1 T3ES Master 0A 1P Il Alternate Master 1A 0P FIGURE 2 4 Path Failure Before the Second Tier of Switches 14 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only The virtualization engine recognizes the primary active and secondary passive pathing for the LUNs and routes the I O to the primary controller unless there is a path failure to the primary path In that case the virtualization engine initiates a LUN failover and routes the I O through the secondary path which in turn goes through the interconnect cables Refer to FIGURE 2 5 which illustrates a path failure where I O is routed through both HBAs A Host with HBA 0 and HBA 1 UNO 10G LI LUNO 10G Active MPDrived Active MPDrive0 LUN1 10G Active MPDrivel LUN1 10G Active MPDrivel Switch SAN Database Virtualization gt Engine 2 Virtualization Engine 1 lt gt Storage O and Virtualization Engine Communications Traffic Logical LUNO 500G Multipath Drive Passive Master MPDrive 0 LUNO 500 G Active Master LUN1 500G A Active Logical Alternate Master Multipath Drive LUN1 500G Passive Altern
4. Change in port statistics on 1 Check the Topology GUI statistics switch diag156 swlb for any link errors ip 192 168 0 31 2 Quiesce I O on the link 3 Run linktest on the link The switch has reported a to isolate the failing change in an error counter FRU This could indicate a failing component in the link chassis Alarm Yellow Y chassis fan 1 status None fan changed from OK system_ Alarm Yellow Y The uptime of the switch 1 Check to see if the switch reboot was less than the previous has been reset uptime of the switch This 2 Check the power going could indicate that the to the switch switch has been reset either by a user or by the loss of power chassis Alarm Yellow chassis power 1 status None power changed from OK This event monitors changes in the status of the chassis power supply as reported by the SANbox chassis status chassis Alarm Yellow chassis temp 1 status None temp changed from OK This event monitors changes in the status of the chassis temperature supply as reported by SANbox chassis status 78 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE7 1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches Continued g gt 5 5 Se nL 5 g g sfsg S gt ga sep 2 c so Bee ae ce S g J amp eore5 aes 8
5. Rear Fault LED Power Plug Status Port LED i FC Port Host Side FC Port Device Side l AAE e Ethernet Port 0 Link Activity LED Speed LED Rear Status LED The Ethernet port LEDs indicate the speed activity and validity of the link shown in TABLE 9 3 TABLE 9 3 Speed Activity and Validity of the Link LED Color State Speed Amber Solid on Off Link Activity Green Solid on Blink Description The link is 100Base TX The link is 10Base T A valid link is established Operations including data activity are normal 112 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FC Link Error Status Report The virtualization engine s host side and device side interfaces provide statistical data for the counts listed in TABLE 9 4 TABLE 9 4 Virtualization Engine Statistical Data Count Type Description Link failure count The number of times the virtualization engine s frame manager detects a nonoperational state or other failure of N port initialization protocol Loss of The number of times that the virtualization engine detects a loss in synchronization synchronization count Loss of signal count The number of times that the virtualization engine s frame manager detects a loss of signal Primitive sequence The number of times that the virtualization engine s frame manager protocol error detects N port protocol e
6. t3o0fdg Diagnostic Red The t30fdg 1M test Test failed t3test Diagnostic Red The t3test 1M test Test failed t3volverify Diagnostic Red The t3volverify 1M Test test failed 100 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Component Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array Event Type Severity Action Description Action enclosure controller disk interface loopcard Discovery Topology Topology Topology The Storage Automated Diagnostic Environment discovered a new Sun StorEdge T3 array Discovery events occur the first time the Storage Automated Diagnostic Environment probes a storage device The Discovery event creates a detailed description of the device monitored and sends it using any active notifier such as the SRS Net Connect provider service or email A new controller as identified by its serial number has been installed on the Sun StorEdge T3 array A new disk as identified by its serial number has been installed on the Sun StorEdge T3 array A new loopcard as identified by its serial number has been installed on the Sun StorEdge T3 array power enclosure enclosure Topology Location Change QuiesceEnd A new PCU has been installed on the Sun StorEdge T3 array The l
7. command line examples and QLogic s SANblade Manager Chapter 4 discusses Ethernet hub troubleshooting Information associated with the 3Com Ethernet hubs is limited in this guide however because 3Com does not allow duplication of its information Chapter 5 provides Fibre Channel FC link diagrams and troubleshooting procedures XV Sun Proprietary Confidential Internal Use Only Chapter 6 provides information on host device troubleshooting Chapter 7 provides information on troubleshooting a Sun StorEdge Network FC switch 8 and switch 16 switch device Chapter 8 describes how to troubleshoot the Sun StorEdge T3 array devices Also included in this chapter is information about the Explorer Data Collection Utility Chapter 9 provides detailed information for troubleshooting the virtualization engines Chapter 10 describes how to troubleshoot using Microsoft Windows 2000 It also explains how to launch the Sun StorEdge T3 Array Failover Driver GUI and interpret the multipath configurator Chapter 11 provides an example of fault isolation It begins with how to discover an error and shows the user steps that are necessary for resolution Appendix A provides virtualization engine references including Service Request Numbers SRNs and Simple Network Management Protocol SNMP Reference an SRN SNMP single point of failure table and port communication and service code tables Appendix B provides a list of SUNWsecfg 1M erro
8. dev rdsk c4t2B00006022004186d0s2 Status Port A O K Vendor SUN Product ID SESSOL1 WWN Node 2a00006022004186 WWN Port A 2b00006022004186 Revision 080E Serial Num Unsupported Unformatted capacity 56320 000 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c4t2B00006022004186d0s2 devices pci l1lf 4000 pci 2 SUNW qlc 5 fpe0 0 ssd w2b00006022004186 0 c raw Chapter 9 Troubleshooting Virtualization Engine Devices 115 Sun Proprietary Confidential Internal Use Only Displaying the VLUN Serial Number v To Display Devices That are Not Sun StorEdge Traffic Manager MPxIO Enabled 1 Use the format e command 2 Type the number of the disk on which you are working at the format prompt 3 Type inquiry at the scsi prompt 4 Find the VLUN serial number in the Inquiry displayed list format e c4t2B00006022004186d0 format gt scsi scsi gt inquiry Inquiry 00 00 03 12 2b 00 00 02 53 55 4e 20 20 20 20 20 ita eH SUN 53 45 53 53 30 31 2020 20 20 20 20 20 20 20 20 SESSO1 30 38 30 45 62 57 33 4b 30 30 31 48 30 30 30 080EbW3K001H000 Vendor SUN Product SESSO1 Revision 080E Removable media no Device type 0 From this screen note that the VLUN number is 62 57 33 4b 30 30 31 48 beginning with the fifth pair of numbers on the third line up to and including the twelfth pair 116 Sun StorEdge 3900 a
9. e If necessary import the SAN zone configuration 54 The cabling configuration is e Check the cabling unauthorized 57 Too many HBAs are attempting e Check the cabling to log in 60 The node mapping table was e No action required cleared using SW2 62 SW2 settings are incorrect e Correct the SW2 setting e Cycle the virtualization engine power 126 Too many virtualization engines e Remove the extra in the SAN virtualization engine e Cycle the virtualization engine power 130 The connection between e Correct the problem virtualization engines is down e Cycle the power on the follower virtualization engine Virtualization Engine References 161 Sun Proprietary Confidential Internal Use Only TABLE A 5 Virtualization Engine Service Codes 400 599 Device Side Interface Driver Errors Service Code Number 409 434 Cause of Error The FC device side type code is invalid Cannot continue due to many elastic store errors Elastic store errors result from a clock mismatch between transmitter and receiver and indicate an unreliable link This error can also occur if a device in the SAN loses power unexpectedly Recommended Corrective Action e Cycle the power e If the problem persists replace the virtualization engine e Check for the faulty component and replace it e Cycle the power on the faulty virtualization engine 162 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide Mar
10. par des fournisseurs de Sun Des parties de ce produit pourront tre d riv es des syst mes Berkeley BSD licenci s par l Universit de Californie UNIX est une marque d pos e aux Etats Unis et dans d autres pays et licenci e exclusivement par X Open Company Ltd Sun Sun Microsystems le logo Sun AnswerBook2 Sun StorEdge StorTools docs sun com Sun Enterprise Sun Fire SunOS Netra SunSolve et Solaris sont des marques de fabrique ou des marques d pos es ou marques de service de Sun Microsystems Inc aux Etats Unis et dans d autres pays Toutes les marques SPARC sont utilis es sous licence et sont des marques de fabrique ou des marques d pos es de SPARC International Inc aux Etats Unis et dans d autres pays Les produits portant les marques SPARC sont bas s sur une architecture d velopp e par Sun Microsystems Inc Toutes les marques SPARC sont utilis es sous licence et sont des marques de fabrique ou des marques d pos es de SPARC International Inc aux Etats Unis et dans d autres pays Les produits protant les marques SPARC sont bas s sur une architecture d velopp e par Sun Microsystems Inc L interface d utilisation graphique OPEN LOOK et Sun a t d velopp e par Sun Microsystems Inc pour ses utilisateurs et licenci s Sun reconnait les efforts de pionniers de Xerox pour la recherche et le d veloppment du concept des interfaces d utilisation visuelle ou graphique pour l industrie de informatiq
11. 120 LEDs 110 map viewing 118 power LED codes 111 primary pathing options 16 reading LED service and diagnostic codes 111 references 155 retrieving service information 108 service codes 108 160 162 service request numbers 108 SRN and SNMP single points of failure 159 troubleshooting 107 VLUN serial number displaying 116 W warning levels 25 Windows 2000 troubleshooting 137 Windows NT configurations 7 worldwide name WWN how to find 18 WWN see worldwide name 18 Index 183 Sun Proprietary Confidential Internal Use Only Z zone modifications 74 Index 184 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only
12. Chapter 5 Troubleshooting the Fibre Channel FC Links 59 Sun Proprietary Confidential Internal Use Only Troubleshooting the A4 or B4 FC Link The A4 or B4 link is the FC link from the switch to the Sun StorEdge T3 array If a problem occurs with the A4 or B4 FC link m Ina Sun StorEdge 3900 series system the Sun StorEdge T3 array will fail over m Ina Sun StorEdge 6900 series system no Sun StorEdge T3 array will fail over but an error with the FC link can cause a path to go offline FIGURE 5 11 and FIGURE 5 12 are examples of A4 or B4 Link Notification Events Site FSDE LAB Broomfield CO Source diag xxxxx xxx com Severity Warning Category Message DeviceId message diag xxxxx xxx com EventType LogEvent driver MPXIO_offline EventTime 01 29 2002 14 28 06 Found 2 driver MPXIO_offline warning s in logfile var adm messages on diag xxxxx xxx com id 80e4aa60 lt snip gt Site FSDE LAB Broomfield CO Source diag xxxxx xxx com Severity Warning Category Message DeviceId message diag xxxxx xxx com EventType LogEvent driver Fabric_Warning EventTime 01 29 2002 14 28 06 Found 1 driver Fabric_Warning warning s in logfile var adm messages on diag xxxxx xxx com id 80e4aa60 INFORMATION Fabric warning lt snip gt status of hba devices pci a 2000 pci 2 SUNW qlc 5 fp 0 0 devctl on diag xxxxx xxx com changed from CONNECTED to NOT CONNECTED INFORMATION monitors changes in the
13. KKKKKKKKKKKK Port Status KKKKKKKKKKKK Port Port Type Admin State Oper State Status Loop Mode 1 F_Port online online logged in 2 TL_Port online offline Not logged in 3 Th Port online offline Not logged in 4 TL Port online offline Not logged in 5 ITL Port online offline Not logged in 6 TL_Port online offline Not logged in 7 T P6rt online online logged in 8 T_Port online online logged in KKKKKKKKKKKK Name Server KKKKKKKKKKKK Port Address Type PortWWN Node WWN FC 4 Types 01 10C000 N 2900006022004195 2800006022004195 SCSI_FCP Here port 2 on sw2a is offline If required ports are offline then check the GBICs and cables If a Sun StorEdge T3 array switch port is offline then login to the T3 array and look at the status of the controllers and the port list as follows t3b0 lt 1l gt fru stat ulcl CTLR STATUS STATE ROLE PARTNER TEMP ulctr ready disabled 7 t3b0 lt 2 gt fru stat uzel CTLR STATUS STATE ROLE PARTNER TEMP u2ctr ready enabled master 27 0 E3b0e 83 sport List port targetid addr_type status host wwn ulpl 0 hard offline sun 50020 2300006dfa u2pl 1 hard online sun 50020 230000725b 130 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 3 After corrective action has been successfully completed run the following command creatediskpools n t3b0 The SEcfglog file should display the follow
14. Remote Services SRS Net Connect service or email enclosure Location Location of switch rasd2 Change swb0 ip xxx 0 0 40 was changed 80 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE7 1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches Continued gt 5 3 Ce o je 5 g 5 sees 5 z B gese s S L 2 a g rs 2 8 lt a fees ie port State port 1 in SWITCH Change diag185 ip xxx 20 67 185 is now Available status state changed from offline to online The port on the switch is now available port State Red Y port 1 in SWITCH 1 Verify cables GBICs and Change diag185 connections along the FC ip xxx 20 67 185 is path now Not Available status Check the Storage state changed from online to Automated Diagnostic offline Environment SAN Topology GUI to identify A port on the switch has failing segment of the logged out of the Fabric data path connection and has gone Verify the correct FC offline switch configuration enclosure Statistics Statistics about switch d2 swb1 ipxxx 0 0 41 10002000007a609 Chapter 7 Troubleshooting Switches 81 Sun Proprietary Confidential Internal Use Only TABLE 7 2 Component TABLE 7 2 lists the switch events for Sun StorEdge network FC switch 8 and switch 16 2 Gbit switches St
15. Switch Port 2 Pattern 0x25252525 Port 2 passed all tests on Switch switchtest Stopped successfully All Storage Automated Diagnostic Environment diagnostic tests are located in opt SUNWstade Diags bin Refer to the Storage Automated Diagnostic Environment User s Guide for a complete list of tests subtests options and restrictions 28 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Monitoring Sun StorEdge T3 and T3 Arrays Using the Explorer Data Collection Utility The Explorer Data Collection Utility script is included on the Storage Service Processor in the export packages directory The Explorer Data Collection Utility is not installed by default but can be installed during rack setup Customer specific site information can be entered at that time To find out more about the Explorer Data Collection Utility you can access the web site with the following URL http webhome eng mdeSW Project Explorer html v To Install the Explorer Data Collection Utility on the Storage Service Processor 1 Type cd export packages pkgadd d SUNWexplo 2 When you are prompted for site specific information during the installation process you can optionally click Return to accept the blank defaults Caution Do not accept automatic emailing of the Explorer Data Collection Utility output unless the Storage Ser
16. Tue Jan 29 16 14 01 MST 2002 savevemap v1 EXIT When savevemap ve pair EXIT is displayed the savevemap 1M process has successfully exited 10 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Sun StorEdge 6900 Series Multipathing Example This Sun StorEdge 6900 series multipathing example contains the following elements m One Sun StorEdge T3 array partner group a Two total LUNs m One 500 Gbyte RAID5 LUN per partner group See FIGURE 2 1 for a logical view of the Sun StorEdge 6900 series g _ Host with HBA 0 and HBA 1 LUNO 10G Active MPDrive LUN1 10G Active MPDrivel LUNO 10G Active MPDrive LUN1 10G rive Switch Switch SAN Virtualization Database Virtualization Engine e gt lt gt Engine 1 2 d Mas y Storage I O and Virtualization Engine Communications Traffic Switch Switch ii Logical Multipath t Drive LUNO 500G MPDrive 0 mene Passive Master f Active Master l l LUN1 500G LUNI 500G Active Alternate Logical Multipath Passive Alternate Master Drive Master MPDrive 1 T3ES Master 0A 1P Alternate Master 1A 0P FIGURE 2 1 Sun StorEdge 6900 Series Logical View Chapter 2 General Troubleshooting Procedures 11 Sun Proprietary Confident
17. When you are troubleshooting the T1 or T2 data path note the following a Two T port links provide redundancy m If one of the two links is lost no Sun StorEdge T3 array LUN failover occurs and no pathing failures are detected m If both T port links fail a Sun StorEdge T3 array LUN failover occurs as one of the virtualization engines takes control of the I O operations One of the Sun StorEdge T3 array LUNs fail over as all I O is routed to the controlling virtualization engine m The host detects a pathing failure in its multipathing software 88 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Notification Events FIGURE 8 1 shows a typical port failure event Site Lab 3286 DSQA1 Broomfield Source diag xxxxx xXxx com Severity Error Actionable Category Switch DevicelId switch 100000c0dd00b682 EventType StateChangeEvent M port 8 EventTime 01 30 2002 11 17 22 port 8 in SWITCH diag209 sw2a ip 192 168 0 32 is now Not Available status state changed from Online to Offline INFORMATION A port on the switch has logged out of the fabric and gone offline PROBABLE CAUSE 1 Verify cables GBICs and connections along Fibre Channel path 2 Check Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segment of the data path 3 Verify correct FC switch configuration Site Lab 3286 DSQA1
18. called with options dev 2 192 168 0 30 0 switchtest Started Testing port 2 Using ip_addr 192 168 0 30 fcaddr 0x0 to access this port Chassis Status for Device Switch Power OK Temp OK 23 0c Fan 1 OK Fan 2 OK 02 06 02 15 09 45 diag Storage Automated Diagnostic Environment MSGID 4001 switchtest WARNING switchO Maximum transfer size for a FABRIC port is 200 Changing transfer size 2000 to 200 Testing Device Switch Port 2 Pattern 0x7e7e7e7e Testing Device Switch Port 2 Pattern Oxlelelele Note The Storage Automated Diagnostic Environment automatically resets the transfer size if it notes that it is about to test a switch to the HBA connection This is done both in the Storage Automated Diagnostic Environment GUI and from the command line interface CLI Chapter 5 Troubleshooting the Fibre Channel FC Links 47 Sun Proprietary Confidential Internal Use Only 48 10 To Isolate the Al or B1 FC Link To isolate the Al or B1 link which is the FC link from the HBA to the switch follow these steps Quiesce the I O on the A1 or B1 FC link path Run switchtest 1M or qlctest 1M to test the entire link Break the connection by uncabling the link Insert a loopback connector into the switch port Rerun switchtest a If switchtest fails replace the gigabit interface converter GBIC and rerun switchtest b If switchtest fails again replace the switch Insert a loopback connector
19. gt virtualization engine 2 gt switch gt master controller gt backend loop to alternate master secondary route from HBA 1 Chapter 2 General Troubleshooting Procedures 13 Sun Proprietary Confidential Internal Use Only The host using multipathing software is presented with two primary active paths for each LUN allowing the host to route I O through either or both HBAs If a path failure occurs before the second tier of Sun StorEdge network FC switch 8 and switch 16 switches one of the paths is disabled but the other path continues sending I O as it normally would and takes over the entire load Refer to FIGURE 2 4 which illustrates a path failure before the second tier of switches No Sun StorEdge T3 array failure is noted because of the redundant path by way of the Sun StorEdge network FC switch 8 and switch 16 switch T ports Host with HBA 0 and HBA 1 UNO 10G L LUNO 10G Active MPDrive0 Active MPDrive LUN1 10G Active MPDrive Switch LUN1 10G Active MPDrivel Switch SAN Database Virtualization gt Engine 2 Masking _ Storage I O and Virtualization Engine Communications Traffic Switch Switch Logical Multipath LUNO 500G Passive Master 1 LUNO 500G Active Master Drive MPDrive 0 LUN1 500G Active l LUN1 500G Logical Multipath Passive Alternate Master
20. virtualization engine 111 diagnostic tests running from command line 27 examples 27 Index 179 Sun Proprietary Confidential Internal Use Only DMP enabled paths returning to production 22 documentation organization XV shell prompts XVII using UNIX commands XVI dynamic multipathing DMP 20 E error discovery 4 error messages other SUNWsecfg 175 Sun StorEdge network FC switch 168 Sun StorEdge T3 array 171 virtualization engine 164 error status checking Fibre Channel link manually 113 error status report Fibre Channel link 113 Ethernet hubs 3Com related documentation 35 troubleshooting 35 ethernet hubs related documentation 35 event grid for host 67 for Sun StorEdge T3 array 95 for virtualization engine 132 sorting criteria 25 switch 77 event grid criteria 25 Explorer Data Collection Utility 4 29 requirements before running 30 F failback virtualization engine 120 failback operations 16 failover operations 16 fault isolation examples 147 Fibre Channel link A1 or B1 data host verification 45 A2 to B2 host side verification 51 A3 or B3 host side verification 56 check error status manually 113 FRU tests for A2 or B2 link 52 FRU tests for A3 or B3 link 57 troubleshooting 37 troubleshooting A4 B4 link 60 used for PFA 2 verifying A2 to B2 52 verifying A3 or B3 link 57 verifying data host 62 field replaceable units isolating 5 testing 5 FRU tests available
21. 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 3 sanblade Manager Connect Configure Events Alarms Refresh o x HBA Information Device List Statistics NVRAM Settings Link Status Utilities Diag amp Host smi 2gs5m2bdhom Host smi 2gs5m2bdhom Node Name 20 00 00 E0 8B 02 65 17 wm Adapter 2200 Adapter 0 2200 PortName 21 00 00 E0 8B 02 65 17 ww Adapter 2200 Port ID 10 41 00 General Information Serial Number 837061 Driver Version 8 1 3 N2K IP BIOS Version 1 76 Firmware Version 2 01 38 ogi Ti Simplify Adapter 2200 FIGURE 3 3 Qlogic SANblade Manager HBA Driver and Firmware Versions Chapter 3 Sun Proprietary Confidential Internal Use Only Troubleshooting Tools 33 34 QLogic SANblade Manager is also useful for viewing a primitive topology and a LUN listing E SANblade Manager lol x ile Host View Help of a glee Connect Configure Events Alarms Refresh Simplity Diagnostics amp Host smi 2gs5m2bdhom Host smi 2gs5m2bdhom NodeName 20 00 00 E0 8B 02 65 17 9 Samana Adapter 0 2200 PortName 21 00 00 E0 8B 02 65 17 amp Device 50 02 0F 2 Pati my Adapter 2200 0 00 E 3 rTest Configuration j Data Pattern 55 01010101 s Number of test s 1 10 000 N A Customized Ge
22. 6 Quiesce the I O along the path to be tested using one of the following methods a For installations using VERITAS Dynamic Multi Pathing DMP disable vxdmpadm 1M a For installations using the Sun StorEdge Traffic Manager MPxIO software unconfigure the Fabric device m Refer to To Quiesce the I O on page 17 m Halt the application 7 Test and isolate field replaceable units FRUs using the following tools m Storage Automated Diagnostic Environment diagnostic tests this might require a loopback cable for isolation m Sun StorEdge T3 array tests including t 3test 1M t 30fdg 1M and t3volverify 1M which can be found in the Storage Automated Diagnostic Environment User s Guide Chapter 2 General Troubleshooting Procedures 5 Sun Proprietary Confidential Internal Use Only Note These tests isolate the problem to a FRU that must be replaced Follow the instructions in the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide and the Sun StorEdge 3900 and 6900 Series 2 0 Installation Guide for proper FRU replacement procedures 8 Verify the fix using the following tools m Storage Automated Diagnostic Environment GUI Topology View and Diagnostic Tests mw var adm messages on the data host 9 Return the path to service with one of the following methods a Use the multipathing software Restart the application Host Side Troubleshooting Host side troubleshooting refers to the m
23. Address 50020 2300006443 1 Class secondary State ONLINE Note This type of error may also cause the device to show up as unusable in cfgadm as shown in CODE EXAMPLE 5 8 Chapter 5 Troubleshooting the Fibre Channel FC Links 63 Sun Proprietary Confidential Internal Use Only CODE EXAMPLE 5 8 Failed Path Marked Unusable cfgadm al Ap_Id Type Receptacle Occupant Condition ac0O bank0 memory connected configured ok acO bankl memory empty unconfigured unknown el scsi bus connected configured unknown c16 scsi bus connected unconfigured unknown c18 scsi bus connected unconfigured unknown c19 scsi bus connected unconfigured unknown cl dsk cl1t6d0 CD ROM connected configured unknown c20 fc private connected unconfigured unknown c21 fc fabric connected configured unknown c21 50020 2300006355 disk connected configured unusable FRU Tests Available for the A4 or B4 FC Link Segment m The switchtest can only be run from the Storage Service Processor m The linktest can isolate the switch and the GBIC on the switch It cannot isolate the cable or the Sun StorEdge T3 array controller v To Isolate the A4 or B4 FC Link To isolate the A4 or B4 link which is the FC link from the switch to the Sun StorEdge T3 array follow these steps 1 Quiesce the I O on the A4 or B4 FC link path 2 Run linktest 1M from the Storage Automated Diagnostic Environment GUI to isolate suspected failing components Alte
24. Confidential Internal Use Only 2 Predictive Failure Analysis PFA Capabilities The Storage Automated Diagnostic Environment software provides the health and monitoring functions for the Sun StorEdge 3900 and 6900 series systems This software provides the following predictive failure analysis PFA capabilities a FC links Fibre Channel FC links are monitored at all end points using the Fibre Channel Extended Link Service FC ELS link counters When link errors surpass the threshold values an alert is sent This enables Sun trained personnel to replace components that are experiencing high transient fault levels before a hard fault occurs Enclosure status Many devices like the Sun StorEdge FC switch 8 and switch 16 switch and the Sun StorEdge T3 array cause the Storage Automated Diagnostic Environment alerts to be sent if the temperature thresholds are exceeded This enables Sun trained personnel to address the problem before the component and enclosure fails a Single Point of Failure SPOF notification Storage Automated Diagnostic Environment notification for path failures and failovers that is Sun StorEdge Traffic Manager software failover can be considered a PFA method since Sun trained personnel are notified and can repair the primary path This eliminates the time of exposure to SPOF and helps to preserve customer availability during the repair process PFA is not always effective in detecting or isola
25. Engine Back Panel 112 Virtualization Engine Event Grid 132 Launching the Sun StorEdge T3 Array Failover Driver 138 Sun StorEdge T3 Array Failover Driver Versions 2 0 0 123 and 2 1 0 104 139 Healthy Sun StorEdge 3900 series system shown using Multipath Configurator 140 Sun StorEdge 3900 series system with a LUN failover shown using Multipath Configurator 141 Multipath Configurator Array Properties 141 Multipath Configurator LUN Properties Detail 142 Sun StorEdge T3 Array Failover Driver CLI Output for the Sun StorEdge 3900 Series 143 Sun StorEdge T3 Array Failover Driver CLI Example Output for the Sun StorEdge 6900 Series 144 Alerts Display Using the Storage Automated Diagnostic Environment 147 Drilling Down for Sun StorEdge T3 Array Failover Driver Fault Detail 148 Fault Confirmation Using QLogic SunBlade 149 Diagnostics Using QLogic SunBlade 150 Storage Automated Diagnostic Environment Test from Topology 151 Storage Automated Diagnostic Environment Test from Topology Pull Down Menu 152 Storage Automated Diagnostic Environment Test from Topology Test Detail 152 List of Figures X Sun Proprietary Confidential Internal Use Only FIGURE 11 8 Successful Switch Test Results 153 FIGURE 11 9 Multipath Recovery using the Sun StorEdge T3 Array Multipath Configurator 154 FIGURE 11 10 Recovered Paths 154 List of Figures XI Sun Proprietary Confidential Internal Use Only XII Sun StorEdge 3900 and 6900 Series 2 0 Troubleshootin
26. HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE primary STATE up_active 2 CONTROLLER ID 0 DESC Sun Microsystems 69XX Array Controller FIGURE 10 8 Sun StorEdge T3 Array Failover Driver CLI Example Output for the Sun StorEdge 6900 Series 144 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Component Device LUN TABLE 10 1 lists some of the codes and descriptions for CLI output for a Sun StorEdge 6910 series system TABLE 10 1 Tips for Interpreting Sun StorEdge 6910 Series CLI Output Output Code FW_REV WWN NAME WWN PATH TYPE Description Firmware revision level of the virtualization engine The worldwide name of the Master virtualization engine of the partner group Microsoft Windows 2000 Device letter e The first 16 digits correspond to the Master virtualization engine WWN from the Device section e The last 16 digits are the VLUN serial number You can crosscheck the WWN using e The SUNWsecfg virtualization engine maps e The Storage Automated Diagnostic Environment s device monitoring section click on virtualization engine to view details The individual physical paths to the HBAs All paths in a 6910 configuration should be Primary Chapter 10 Troubleshooting Using Microsoft Windows 2000 145 Sun Proprietary Confidential Internal Use Only 146 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 20
27. Microsoft Windows 2000 Event Properties System Log 26 Qlogic SANblade Manager HBA Driver and Firmware Versions 33 QLogic SANblade Manager Diagnostics 34 Sun StorEdge 3900 Series FC Link Diagram 39 Sun StorEdge 6900 Series FC Link Diagram 41 Data Host Notification of Intermittent Problems 43 Data Host Notification of Severe Link Error 43 Storage Service Processor Notification 44 A2 or B2 FC Link Host Side Event 49 A2 or B2 FC Link Storage Service Processor Side Event 50 A3 or B3 FC Link Host Side Event 54 A3 or B3 FC Link Storage Service Processor Side Event 55 A3 or B3 FC Link Storage Service Processor Side Event 55 List of Figures IX Sun Proprietary Confidential Internal Use Only GURE 5 11 GURE 5 12 GURE 6 1 GURE 7 1 GURE 8 1 GURE 8 2 GURE 8 3 GURE 8 4 GURE 8 5 GURE 9 1 GURE 9 2 GURE 9 3 GURE 10 1 GURE 10 2 GURE 10 3 GURE 10 4 GURE 10 5 GURE 10 6 GURE 10 7 GURE 10 8 GURE 11 1 GURE 11 2 GURE 11 3 GURE 11 4 GURE 11 5 GURE 11 6 GURE 11 7 A4 or B4 FC Link Data Host Notification 60 Storage Service Processor Side Notification 61 Sample Host Event Grid 68 Switch Event Grid 77 Storage Service Processor Event 89 Virtualization Engine Alert 90 Manage Configuration Files Menu 92 Example Link Test Text Output from the Storage Automated Diagnostic Environment 93 Sun StorEdge T3 Array Event Grid 95 Virtualization Engine Front Panel LEDs 111 Virtualization
28. Suggested Corrective Action 2 Gbit switches modifyswitch 2 Gbit switches saveswitch 2 Gbit switches 2 Gbit switches have a zone configuration with a zoneset and zone s Each zone then has port or WWN members 1 Gbit switches had numbered hard zones only checkswitch e The current configuration on 1 Select View Logs or see LOGFILE switch does not match the defined for more details configuration 2 Rerun setupswitch on the e One of the predefined static switch specified switch configuration parameters that can be overridden for special configurations such as NT connect or cascaded switches is set incorrectly checkswitch The other back end switch is not the Run the following command on the same type as this switch Firmware other back end switch to upgrade it should be upgraded or downgraded setswitchflash s switch2 f so the two switches match usr opt SUNWsmgr2 firmware S ANbox1 16040233 fl1s checkswitch No active zone set found 1 Attempt to activate an existing zone set 2 opt SUNWsecfg flib sanbox2 x switchip get_zoneset_list 3 From that list select the zoneset you want to be active It should be named something similar to hostname_swla_zset 4 opt SUNWsecfg flib sanbox2 x switchip activate_zoneset zoneset 5 If you are still having problems rerun setupswitch 6 Rerun the initial command you attempted to run restoreswitch Map file format or version is invalid fo
29. Syne Count Loss of Signal Count Protocol Error Count Invalid Word Count Invalid CRC Count O 0 OoOo I00001 Device Side FC Vital Statistics Link Failure Count 0 Loss of Sync Count 0 Loss of Signal Count 0 Protocol Error Count 0 Invalid Word Count 139 Invalid CRC Count 0 I00002 Host Side FC Vital Statistics Link Failure Count 0 Loss of Sync Count 0 Loss of Signal Count 0 Protocol Error Count Invalid Word Count Invalid CRC Count oro I00002 Device Side FC Vital Statistics Link Failure Count 0 Loss of Sync Count 0 Loss of Signal Count 0 Protocol Error Count 0 Invalid Word Count 135 Invalid CRC Count 0 diag xxxxx xxx com root Note v1 represents the first virtualization engine pair 114 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Note The Serial Loop IntraConnect SLIC daemon must be running for the svstat 1M d v1 command to work Translating Host Device Names You can translate host device names to VLUN disk pool and physical Sun StorEdge T3 array LUNs The luxadm output for a host device shown in CODE EXAMPLE 9 2 does not include the unique VLUN serial number that is needed to identify this LUN The procedure to obtain the VLUN serial number is detailed next CODE EXAMPLE 9 2 luxadm Output for a Host Device usr sbin luxadm display dev rdsk c4t2B00006022004186d0s2 DEVICE PROPERTIES for disk
30. The User Service Utility Menu is displayed Type 9 to clear the SAN database a A successful command displays the message SAN database has been cleared m An unsuccessful command results in the service code 051 If this occurs repeat Steps 1 through 3 m Ifthe command continues to fail replace the virtualization engine To reconnect the virtualization engine s device side FC cables type setveport v virtualization engine name c Type B to reboot the virtualization engine Chapter 9 Troubleshooting Virtualization Engine Devices 125 Sun Proprietary Confidential Internal Use Only Restarting the sl1icd Daemon Follow this procedure to restart the slicd daemon if the SLIC daemon becomes unresponsive or if a message similar to the following is displayed connect Connection refusedor Socket error encountered v To Restart the slicd Daemon 1 Check whether the slicd daemon is running ps ef grep slicd 2 Use the ipcs 1 command to check for any message queues shared memory or semaphores still in use ipcs IPC status from lt running system gt as of Wed Feb 20 12 48 30 MST 2002 T ID KEY MODE OWNER GROUP Message Queues Shared Memory m 0 0x50000483 rw r r root root m 301 0x5555aa8a rw root other m 302 0x5555aaaa rw root other m 303 0x5555aaba rw root other m 4 Ox7cc a 7 root root Semaphores s 196608 0x5555aa9a ra root other s 196609 0
31. Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Other Error Messages TABLE B 4 Source of Error Message Other SUNWsecfg Error Messages Cause of Error Message Suggested Corrective Action Appendix B Common to all If the Sun StorEdge 3900 or 6900 Set the BOXTYPE variable as components series has more than two failures follows for example both virtualization BOXTYPE 6910 export BOXTYPE engines and two switches are down the getcabinet tool might not determine the correct cabinet type In this example the getcabinet command might determine the device to be a Sun StorEdge 3900 series when in reality it is a Sun StorEdge 6900 series checkdefaultconfig e Could not determine the Sun To use the command line interface StorEdge system type CLI set the BOXTYPE e Multiple components might be environment variable to one of the down and the getcabinet seven values command could not determine the Sun StorEdge series type 3910 For example BOXTYPE 3910 3960 6910 or 6960 export BOXTYPE listavailable e The component is unavailable It If no other commands are running is either not found or the and you believe the configuration configuration lock is set lock might be set in error run the e The components are down they removelocks command do not respond to a ping e Another SUNWsecfg command is running and is updating the configuration ps ef setdefaultconfig The s
32. When you insert a loopback connector in to the T port no green light appears to indicate a proper insertion However the test will run and be valid a If only one of the links has failed and the I O is traveling over the remaining link I O is automatically routed over the repaired link by the switch after the failed link is replaced and recabled No manual intervention is required m If both links have failed and a LUN failover has occurred you must manually run a failbackt3path command to return the paths to their optimal state after you repair and recable the links wv To Isolate the T1 or T2 Data Path 1 Run linktest from the Storage Automated Diagnostic Environment for a guided isolation procedure 2 After replacing the failed FRU run failbackt3path if needed 94 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Sun StorEdge T3 Array Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort Sun StorEdge T3 array events by component category or event type The Storage Automated Diagnostic Environment GUI displays an event grid that describes an event and its severity and tells what if any action should be taken Refer to the Storage Automated Diagnostic Environment User s Guide for more information v To Use the Sun StorEdge T3 Array Event Grid 1 From the Storage Automated Diagnostic Environment Help menu click
33. arrays with 1 xx firmware e No response received from t3_name Aborting operation volslice is not enabled on this Sun StorEdge T3 array e Error while opening the Sun StorEdge T3 array e Cannot open after resetting the Sun StorEdge T3 array 1 Refer to the T3 default custom configuration table in the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide 2 Use showt3 n t3_name to display the present configuration 3 Check the Sun StorEdge T3 firmware version it should be version 2 01 00 or higher Upgrade if required 1 Check the Sun StorEdge T3 network connection 2 Check with Ping t3_name command to determine if the Sun StorEdge T3 array is operating Check the T3 firmware 2 01 00 or higher to make sure volume slicing is allowed with this version of firmware 1 Check the Sun StorEdge T3 master and alternate master network connection 2 Check with Ping t3_name command to determine if the Sun StorEdge T3 array is operating present Unable to check configuration checkt3config The vol init command is being 1 Check whether any other secfg executed by another user Additional utility is running vol commands cannot run 2 If an secfg utility is running allow it to finish checkt3config An error occurred while the Check whether any other checkt3config command was secfg T3 or native Sun StorEdge checking the process list causing the T3 array commands are
34. being t3_name to abort executed on that particular Sun StorEdge T3 array checkt3config Snapshot configuration files are not 1 Verify that the snapshot files are saved and have read permissions in the opt SUNWsecfg etc t3name directory 2 If the snapshot files are not available create them using the savet 3config command 172 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE B 3 Source of Error Message Sun StorEdge T3 Array Error Messages Continued Cause of Error Message Suggested Corrective Action WWN wwn from group checkt 3mount e The 1lun status reported a bad or 1 Run the showt3 n command to nonexistent LUN verify that the requested LUN e While checking the configuration exists on the Sun StorEdge T3 using the showt3 n command array operations abort 2 Confirm that the Sun StorEdge T3 array configuration matches standard configurations createt 3group User specified LUN 1un does not Create the required slice or LUN exist on the Sun StorEdge T3 array before you set permissions using the createt3slice command createt3group Unable to set permissions on the Refer to the Sun StorEdge T3 and T3 LUN 1un to perm for new group array documentation createt3group Error while resetting permissions on Refer to the Sun StorEdge T3 and T3 LUN 1un to NONE for group array documentation Sgroup delfromt3group Error d
35. failover 141 N notification Storage Service Processor 44 used in PFA 2 notification events A1 or B1 43 A2 or B2 49 A3 or B3 54 A4 or B4 60 T1 or T2 89 P parity error PCI bus 160 paths how to unconfigure 17 returning to production 19 PCI bus parity error 160 port state change in A1 or B1 link 44 Predictive Failure Analysis 2 problem determination 4 isolation 38 Q QLogic SANBlade Manager HBA driver versions 33 QLogic SANblade Manager diagnostics 34 quiescing I O 59 R retrieving diagnostic codes 108 service information 108 service request numbers 108 SAN 4 1 Index 181 Sun Proprietary Confidential Internal Use Only switches 74 SAN database manually clearing 123 manually restoring 123 resetting 124 service codes interpreting 111 overview 108 retrieving 108 virtualization engine 111 156 service processor troubleshooting 6 service request numbers for virtualization engine 155 retrieving 109 virtualization engine 108 setswitchflash to upgrade switches 74 settings configuration 7 SLIC daemon communication with virtualization engine 108 killing and restarting 126 statistical data FC link errors 113 status virtualization engine 113 Storage Automated Diagnostic Environment example topology 24 used to troubleshoot 23 Storage Service Processor messages 4 notification 44 running SanSurfer from GUI 5 verifying 92 Storage Service Processor side
36. for A1 or B1 FC link 46 available for A2 or B2 FC link 52 available for A3 or B3 FC link 57 H HBA monitoring using QLogic SANBlade Manager 32 health functions for Sun StorEdge 3900 and 6900 series 2 host replacing See Event Grid 71 see event grid 67 host bus adapters see HBA 32 host devices troubleshooting 67 verifying 51 host side troubleshooting 6 host device names translating 115 l I O manually halting 17 quiescing 17 suspending 18 59 Index 180 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only installations Sun StorEdge Traffic Manager 5 VERITAS VxDMP 5 isolating Al or B1 FC link 48 A2 or B2 FC link 52 A3 or B3 link 58 isolating FRUs 5 isolation procedures Al or B1 FC link 48 for A2 B2 link 52 L LED service and diagnostic codes reading virtualization engine 111 LEDs Ethernet port 112 ethernet port 111 power status 111 virtualization engine 110 link error example of severe data host error 43 lock file clearing 10 log files displaying 109 loss of communication events 3 luxadm 1M used to display information 21 verifying functionality 4 Microsoft Windows 2000 troubleshooting 137 viewing system errors 26 Microsoft Windows NT configurations 7 monitoring functions for Sun StorEdge 3900 and 6900 Series 2 multipath configurator array properties 141 healthy configuration 140 with LUN
37. into the HBA Run glctest a If the qlctest test fails replace the HBA b If the qlctest test passes replace the cable Recable the entire link Run switchtest or qlctest to validate the fix Put the path back into production Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Troubleshooting the A2 or B2 FC Link The A2 or B2 link is the FC link from the first switch to the virtualization engine This link exists in the Sun StorEdge 6900 Series only An error with the FC link can cause a path to go offline FIGURE 5 6 and FIGURE 5 7 are examples of A2 or B2 Link Notification Events From root Tue Jan 8 18 39 48 2002 Date Tue 8 Jan 2002 18 39 47 0700 MST Message Id lt 200201090139 g091d1g07015 diag xxxxx xxx com gt From Storage Automated Diagnostic Environment Agent Subject Message from diag xxxxx xxx com 2 0 B2 002 Content Length 2742 You requested the following events be forwarded to you from diag xxxxx xxx com Site FSDE LAB Broomfield CO Source diag226 xxxxx xxx com Severity Normal Category Message Key message diag xxxxx xXxxX Ccom EventType LogEvent driver Fabric_Warning EventTime 01 08 2002 17 34 47 Found 1 driver Fabric_Warning warning s in logfile var adm messages on diag xxxxx xxx com id 80fee746 Info Fabric warning Jan 8 17 34 36 WWN 2b000060220041f4 diag xxxxx xxx com fp ID 5
38. output of luxadm e port Found path to 20 HBA ports devices sbus 2 0 SUNW socal d 10000 0 NOT CONNECTED FIGURE 5 11 A4 or B4 FC Link Data Host Notification 60 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Site FSDE LAB Broomfield CO Source diag Severity Warning Category Switch DeviceId switch 100000c0dd0061bb EventType LogEvent MessageLog EventTime 01 29 2002 14 25 05 Change in Port Statistics on switch diag swlb ip 192 168 0 31 Port 1 Received 16289 InvalidTxWds in 0 mins value 365972 Site FSDE LAB Broomfield CO Source diag Severity Warning Category T3message DeviceId t3message 83060c0c EventType LogEvent MessageLog EventTime 01 29 2002 14 25 06 Warning s found in logfile var adm messages t3 on diag id 83060c0c Jan 29 14 12 58 t3b0 ISR1 2 W u2ctr ISP2100 2 Received LOOP DOWN async event Jan 29 14 13 32 t3b0 MNXT 1 W ulctr starting lun 1 failover Site FSDE LAB Broomfield CO Source diag Severity Warning Category T3message DeviceId t3message 83060c0c EventType LogEvent MessageLog EventTime 01 29 2002 14 11 14 Warning s found in logfile var adm messages t3 on diag id 83060c0c Jan 29 14 05 18 t3b0 ISR1 1 W u2d4 SVD_PATH_FAILOVER path_id 0 Jan 29 14 05 18 t3b0 ISR1 1 W u2d5 SVD_PATH_FAILOVER path_id 0 Jan 29 14 05 18 t3b0 ISR1 1 W
39. series system the Sun StorEdge T3 array will fail over m Ina Sun StorEdge 6900 series system no Sun StorEdge T3 array will fail over but an error with the FC link can cause a path to go offline 42 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FIGURE 5 3 FIGURE 5 4 and FIGURE 5 5 are examples of A1 or B1 link notification events Site FSDE LAB Broomfield CO Source diag xxxxx xxx com Severity Normal Category Message Key message diag xxxxx xxx com EventType LogEvent driver LOOP_OFFLINE EventTime 01 08 2002 14 34 45 Found 1 driver LOOP_OFFLINE error s in logfile var adm messages on diag xxxxx xxx com id 80fee746 info Loop Offline Jan 8 14 34 25 WWN Received 2 Loop Offline message s threshold is 1 in 5mins Last Message diag xxxxx xxx com qlc ID 686697 kern info NOTICE Qlogic qlc 0 Loop OFFLINE FIGURE 5 3 Data Host Notification of Intermittent Problems Site FSDE LAB Broomfield CO Source diag xxXexx xxx com Severity Normal Category Message Key message diag xxxxx xXxxX Com EventType LogEvent driver MPXIO_offline EventTime 01 08 2002 14 48 02 Found 2 driver MPXIO_offline warning s in logfile var adm messages on diag xxxxx xxx com id 80fee746 Jan 8 14 47 07 WWN 2b000060220041 f9 diag xxxxx xxx com mpxio ID 779286 kern info scsi_vhci ssd g29000060220041 96
40. shown in CODE EXAMPLE 2 1 m Run the checkswitch 1M checkt3config 1M checkve 1M checkvemap 1M scripts from opt SUNWsecfg bin to check the settings on the Sun StorEdge network FC switch 8 and switch 16 switches the Sun StorEdge T3 array and the virtualization engine The scripts check the default configuration files in the opt SUNWsecfg etc directory and compare the current live settings to those of the defaults Any differences are marked with a FAIL Note For cluster configurations and systems that are attached to Microsoft Windows NT the default configurations may not match the current installed configuration Be aware of this when running the verification scripts Certain items may be flagged as FAIL in these special circumstances Chapter 2 General Troubleshooting Procedures 7 Sun Proprietary Confidential Internal Use Only CODE EXAMPLE 2 1 checkdefaultconfig 1M Output opt SUNWsecfg checkdefaultconfig Checking all accessible components hecking switch swla witch swla PASSED hecking switch swlb witch swlb PASSED hecking switch sw2a witch sw2a PASSED hecking switch sw2b Switch sw2b PASSED Please enter the Sun StorEdge T3 array password OU GOAO D O Checking T3 t3b0 Checking 2 t3b0 Config ra t i n sss ss Checking command ver PASS Checking command vol stat PASS Checking command port list PASS Checking command port listmap PASS Checking command sys lis
41. start the daemon 2 Reset or power off the virtualization engine if the problem persists 1 Set the VEPASSWD environment variable with the proper value 2 Try to login again 164 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE B 1 Virtualization Engine Error Messages Continued Source of Error Message Cause of Error Message Suggested Corrective Action Common to virtualization engine Common to virtualization engine After resetting the virtualization engine the VENAME is unreachable The hardware might be faulty e The device side operating mode is not set properly e The device side UID reporting scheme is not set properly e The host side operating mode is not set properly e The host side LUN mapping mode is not set properly e The host side Command Queue Depth is not set properly e The host side UID distinguish is not set properly e The IP is not set properly e The subnet mask is not set properly e The default gateway is not set properly e The server port number is not set properly e The host WWN Authentications are not set properly e The host IP Authentications are not set properly e The Other VEHOST IP is not set properly Check the IP address and netmask that has been assigned to the virtualization engine hardware Be aware that the machine takes approximately 30 seconds to boot a
42. the Event Grid link 2 Select the criteria from the Storage Automated Diagnostic Environment event grid like the one shown in FIGURE 8 5 Monitor Maintenance Diagnose amp Sun rage Automated Diagnostic Environment halal ag 0 06 010 diag176 central sun com Help Event Grid Help Help Page Select a Category Component EventType and type GO to limit the report Click on the Columns headers to change the sort Event Grid EventType Architecture Diagnostics Diag Strategy Utilities Release Notes power temp User s Guide pd Abbreviations disk port Copyrights Info Action The state of disk u1d1 Port1State on T3 300 changed from OK to failed nterface loopcard cab hanged from OK to failed Info Action The state of power ulpcul BatState on iag213 ip xxx 20 67 213 is Fault power battery iag213 ip xxx 20 67 213 is Fault power output Anfo Action The state of power u1pcu1 PowOutput on power temp fag213 ip xxx 20 67 213 is Fault Info Action Errors s found in logfile enclosure enclosure ost 2001 10 26 12 21 04 enclosure Info Auditing a new T3 called ras d2 t3b1 FIGURE 8 5 Sun StorEdge T3 Array Event Grid Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 95 Sun Proprietary Confidential Internal Use Only TABLE 8 1 lists all of the events for the Sun StorEdge T3 array
43. the same hard zone as the cascaded switch in a SAN environment or the user has run the modifyswitch 1M command on a Sun StorEdge 3900 Series system Chapter 7 Troubleshooting Switches Sun Proprietary Confidential Internal Use Only 85 Note If multiple systems are connected to a switch the switch settings might not match the default settings 86 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 8 Troubleshooting the Sun StorEdge T3 Array Devices The Sun StorEdge T3 array is a high performance modular scalable storage device that contains an internal RAID controller and disk drives with FC connectivity to the data host In the Sun StorEdge 3900 and 6900 series the Sun StorEdge T3 array is used as a building block configured in various ways to provide a storage solution optimized to the host application The array is sometimes called a controller unit which refers to the internal RAID controller on the controller card Arrays without the controller card are called expansion units When connected to a controller unit the expansion unit enables the user to increase storage capacity This chapter contains the following sections a Troubleshooting the T1 or T2 Data Path on page 88 m Sun StorEdge T3 Array Event Grid on page 95 87 Sun Proprietary Confidential Internal Use Only Troubleshooting the T1 or T2 Data Path
44. the switch is now available port State Red Y A port on switch2 has 1 Verify cables GBICs and Change logged out of the Fabric connections along the FC connection and has gone path offline Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segment of the data path Verify the correct FC switch configuration enclosure Statistics Statistics about switch d2 swb1 ipxxx 0 0 41 10002000007a609 84 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only setupswitch Exit Values TABLE 0 1 lists the setupswitch exit values The associated messages are logged in the var adm log SEcfglog file TABLE 0 1 setupswitch Exit Values Severity Level Message Type Message Meaning 0 INFO ERROR WARNING WARNING WARNING All switch settings are properly set The switch setting matches the default configuration Errors occurred while you tried to set the proper switch settings The switch setting does not match the default configuration or any valid alternatives Errors occurred while you tried to set the proper switch settings The ports did not self configure properly A cable connection might not be working properly T ports self configure that is the configuration tool cannot control the configuration from F ports when they are cabled properly Specificall
45. using virtualization engine failback Refer to To Failback the Virtualization Engine on page 120 18 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Note To confirm that a failover is occurring open a Telnet session to the Sun StorEdge T3 array and check the output of port listmap Another but slower method is to run the runsecfg script and verify the virtualization engine maps by polling them against a live system occur on the data host and a brief suspension of I O will occur f Caution During the failover small computer systems interface SCSI errors will v To Put the c2 Path Back into Production 1 Type cfgadm c configure c2 2b000060220041f4 2 Verify that I O has resumed on all paths Chapter 2 General Troubleshooting Procedures 19 Sun Proprietary Confidential Internal Use Only v To View the Dynamic Multi Pathing DMP Properties 1 Type Device type hostid disk group flags pubpaths version iosize public private update headers configs logs config config log CTLR NAME devicetag privpaths vxdisk list Disk_1 Disk_1 Disk_1 sliced diag xxxxx xxx COM name t3dg02 id 1010283311 1163 diag xxxxx xxx com name t3dg id 1010283312 1166 diag xxxxx xxx com online ready private autoconfig nohotuse autoimport imported block dev vx dmp Disk_1s4 char dev vx
46. 0 SUNW qlc 2 fp 0 0 devctl CONNECTED devices pcit6 4000 SUNW qlc 3 fp 0 0 devctl CONNECTED usr sbin luxadm display dev rdsk c6t29000060220041F96257354230303052d0s2 DEVICE PROPERTIES for disk dev rdsk c6t29000060220041F96257354230303052d0s2 devices scsi_vhci ssd g29000060220041f96257354230303052 c raw Controller devices pcit6 4000 SUNW qlc 3 fp 0 0 Device Address 2b6000060220041 9 0 Class primary State OFFLINE Controller devices pcit6 4000 SUNW qlc 2 fp 0 0 Device Address 2b6000060220041f4 0 Class primary State ONLINE 56 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CODE EXAMPLE 5 6 DMP Error Message Jul 8 18 26 38 diag xxxxx xxx com vxdmp ID 619769 kern notice NOTICE dmp Path failure on 118 0x1f8 Jul 8 18 26 38 diag xxxxx xxx com vxdmp ID 997040 kern notice NOTICE vxvm vxdmp disabled path 118 0x1f8 belonging to the dmpnode 231 0xd0 Verifying the Storage Service Processor Side You can check the A3 or B3 FC link using the Storage Automated Diagnostic Environment s Test from Topology functionality The Storage Automated Diagnostic Environment s implementation of diagnostic tests verifies the operation of user selected components Using the Topology view you can select specific tests subtests and test options Refer to the Storage Automated Diagnostic Environment User s Guide for more information F
47. 0 Series Multipathing Example 11 Multipathing Options in the Sun StorEdge 6900 Series 16 Manually Halting the I O 17 v ToQuiesce theI O 17 v To Unconfigure the c2 Path 17 Suspending the I O 18 v To Put the c2 Path Back into Production 19 v To View the Dynamic Multi Pathing DMP Properties 20 v To Put the DMP Enabled Paths Back into Production 22 Troubleshooting Tools 23 Storage Automated Diagnostic Environment 2 2 23 Example Topology 24 Generating Component Specific Event Grids 25 v To Customize an Event Report 25 Microsoft Windows 2000 System Errors 26 Command Line Test Examples 27 qlctest 1M 27 switchtest 1M 28 Monitoring Sun StorEdge T3 and T3 Arrays Using the Explorer Data Collection Utility 29 v To Install the Explorer Data Collection Utility on the Storage Service Processor 29 Monitoring Host Bus Adapters HBAs Using QLogic SANblade Manager 32 Troubleshooting Ethernet Hubs 35 Troubleshooting the Fibre Channel FC Links 37 FC Links 38 FC Link Diagrams 39 Contents IV Sun Proprietary Confidential Internal Use Only Troubleshooting the Al or B1 FC Link 42 Verifying the Data Host 45 FRU Tests Available for the Al or B1 FC Link Segment 46 v ToTsolate the A1 or B1 FC Link 48 Troubleshooting the A2 or B2 FC Link 49 Verifying the Data Host 51 Verifying the A2 or B2 FC Link 52 FRU Tests Available for the A2 or B2 FC Link Segment 52 v___ To Isolate the A2 or B2 FC Link 52 Troubleshooting the A3 or B3 FC Link 54 Verifyin
48. 0 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 4 Compare the healthy Sun StorEdge 3900 series system to a system that has experienced a LUN failover A system that has experienced a LUN failover has a broken line connecting the HBA to the storage as shown in FIGURE 10 4 amp Multipath Configurator Driver HBA Path Array Help smi 2gs5m2bdhom 60020F20000003D50000000000000000 60020F20000003D50000000000000000 Fibre Channel Adaptor ae ONE data path Array 2 LUN Count 1 FIGURE 10 4 Sun StorEdge 3900 series system with a LUN failover shown using Multipath Configurator 5 To further check the affected Sun StorEdge T3 array a Right click the Sun StorEdge T3 array in the failed path b Select Array Properties amp Array Properties Device Sun Microsystems T3 Disk Array WWN 60020F20000003DS50000000000000000 Serial Number 00163874 Firmware Level 0201 LUN F 0 M Details Contains the master controller unit o FIGURE 10 5 Multipath Configurator Array Properties Chapter 10 Troubleshooting Using Microsoft Windows 2000 141 Sun Proprietary Confidential Internal Use Only c To view details about the Sun StorEdge T3 Array paths click the Details button The Multipath Configurator LUN Properties detail window is displayed LUN Properties 3 xj Volume F LUN 0 Primary Path Unknown A
49. 00 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Power LED Codes The virtualization engine LEDs are shown in FIGURE 9 1 VIRTUALIZATION ENGINE STATUS LED POWER LED oo o FAULT LED A FIGURE 9 1 Virtualization Engine Front Panel LEDs Interpreting LED Service and Diagnostic Codes The Status LED communicates the status of the virtualization engine in decimal numbers Each decimal number is represented by a number of blinks followed by a medium duration period two seconds of no LED display TABLE 9 2 lists the status LED code descriptions TABLE 9 2 LED Diagnostic Codes Code LED Blink Pattern 0 Fast 1 Once 2 Twice with one second between blinks 3 Three times with one second between blinks 10 Ten times with one second between blinks The blink code repeats continuously with a four second off interval between code sequences Chapter 9 Troubleshooting Virtualization Engine Devices 111 Sun Proprietary Confidential Internal Use Only Back Panel Features The back panel of the virtualization engine contains the Sun StorEdge network FC switch 8 or switch 16 switches a socket for the AC power input and various data ports and LEDs Power Switch Serial Port gt o E FC Port Host Side FC Port Device Side FIGURE 9 2 Virtualization Engine Back Panel Ethernet Port LEDs
50. 0220041f9 diag xxxxx xxx com fp ID 517869 kern warning WARNING fp 1 N_x Port with D_ID 104000 PWWN 2b000060220041 f9 disappeared from fabric FIGURE 5 8 A3 or B3 FC Link Host Side Event 54 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Site FSDE LAB Broomfield CO Source diag xxxxx xXxx com Severity Normal Category Switch Key switch 100000c0dd0057bd EventType StateChangeEvent M port 1 EventTime 01 08 2002 18 28 38 port 1 in SWITCH diag swla ip 192 168 0 30 is now Not Available status state changed from Online to Offline Info A port on the switch has logged out of the fabric and gone offline Action 1 Verify cables GBICs and connections along FC path 2 Check Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segment of the data path 3 Verify correct FC switch configuration FIGURE 5 9 A3 or B3 FC Link Storage Service Processor Side Event Site FSDE LAB Broomfield CO Source diag xxxxx xXxx com Severity Normal Category Switch Key switch 100000c0dd00cbfe EventType StateChangeEvent M port 1 EventTime 01 08 2002 18 28 40 port 1 in SWITCH diag sw2a ip 192 168 0 32 is now Not Available status state changed from Online to Offline Info A port on the switch has logged out of the fabric and gone offline Action 1 Verify ca
51. 03 Sun Proprietary Confidential Internal Use Only CHAPTER 1 1 Example of Fault Isolation In the following example a fault was injected into a running Sun StorEdge 3900 series system to show a troubleshooting flow 1 Discover the Error One of the best ways to discover errors is by using the Storage Automated Diagnostic Environment monitoring system The Storage Automated Diagnostic Environment should be configured to email alerts and events to a local System Administrator In FIGURE 11 1 the alert was displayed using the Storage Automated Diagnostic Environment GUI Storage Automated Diagnostic Environment admin Monitor i Diagnose Manage Report Monitor Devices Monitor Topology Monitor Log Monitor Devices Help J Monitor Devices Help Summ Alerts Log Report Graph Search Host All Hosts Go r crash3 048 2002 07 23 12 483 22 14 19 01 port d in SWITCH crash3 011 ip anged from Online to offline crash3 Switch crash3 026 2002 07 23 12 49 1 2002 07 23 12 49 11 21 03 TW E estination port 4 on crash3 rash3 central sun com 2002 07 23 12 43 51 Delete Alerts diag209 jaqg209 sw2a RY 2002 07 23 05 18 01 diag209 _ iag209 t3b0 D 2002 07 23 05 18 3 FIGURE 11 1 Alerts Display Using the Storage Automated Diagnostic Environment 147 Sun Proprietary Confidential Internal Use Only In this configuration Port 2 is shown to have gone o
52. 17869 kern warning WARNING fp 0 N_x Port with D_ID 108000 PWWN 2b000060220041f4 disappeared from fabric lt snip gt multipath status degraded path pci 6 4000 SUNW qlc 2 fp 0 0 fp0 to target address 2b000060220041f4 1 is offline Jan 8 17 34 55 WWN 2b000060220041f4 diag xxxxx xxxX com mpxio ID 779286 kern info scsi_vhci ssd g29000060220041f96257354230303052 ssd18 multipath status degraded path pci 6 4000 SUNW qlc 2 fp 0 0 fp0 to target address 2b000060220041f4 0 is offline FIGURE 5 6 A2 or B2 FC Link Host Side Event Chapter 5 Troubleshooting the Fibre Channel FC Links 49 Sun Proprietary Confidential Internal Use Only 50 Site FSDE LAB Broomfield CO Source diag xxxxx xXxx com Severity Normal Category Switch Key switch 100000c0dd0061bb EventType StateChangeEvent X port 1 EventTime 01 08 2002 17 38 32 port 1 in SWITCH diag swlb ip 192 168 0 31 is now Unknown status state changed from Online to Admin Site FSDE LAB Broomfield CO Source diag xxxxx xXxx com Severity Normal Category San Key switch 100000c0dd0061bb 1 EventType LinkEvent ITW switch ve EventTime 01 08 2002 17 39 47 ITW ERROR 765 in 11 mins Origin port 1 on switch swlb 192 168 0 31 Destination port 1 on ve diag vlb 29000060220041f4 Info An invalid transmission word ITW was detected between two components This could indicate a potential proble
53. 257354230303053 ssd19 multipath status degraded path pci 6 4000 SUNW glc 3 fp 0 0 fpl to target address 2b000060220041f9 1 is offline Jan 8 14 47 07 WWN 2b000060220041 f9 diag xxxxx xxx com mpxio ID 779286 kern info scsi_vhci ssd g29000060220041 96257354230303052 ssd18 multipath status degraded path pci 6 4000 SUNW glc 3 fp 0 0 fpl to target address 2b000060220041f9 0 is offline FIGURE 5 4 Data Host Notification of Severe Link Error Chapter 5 Troubleshooting the Fibre Channel FC Links 43 Sun Proprietary Confidential Internal Use Only Site FSDE LAB Broomfield CO Source diag xxxxx xXxx com Severity Normal Category Switch Key switch 100000c0dd0057bd EventType StateChangeEvent X port 6 EventTime 01 08 2002 14 54 20 port 6 in SWITCH diag swla ip 192 168 0 30 is now Unknown status state changed from Online to Admin FIGURE 5 5 Storage Service Processor Notification Note An A1 or B1 FC link error can cause a port in swla or sw1b to change state 44 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Verifying the Data Host The following example shows an error in the Al or B1 FC link which can cause a path to go offline in the multipathing software CODE EXAMPLE 5 1 luxadm 1M Display usr sbin luxadm display dev rdsk c6t29000060220041F96257354230303052d0s2 DEVI
54. 9 Port Communication Numbers on page 160 Virtualization Engine Service Codes on page 160 SRN Reference TABLE A 1 provides an explanation of SRNs for the virtualization engine Sun Proprietary Confidential Internal Use Only 155 TABLE A 1 SRN Reference SRN Description Corrective Action 1xxxx The SCSI Request Sense command has If too many check conditions are returned reported the condition of the disk drive where check the link status xxxx is the Unit Error Code in Sense Data bytes 20 to 21 70000 The SAN configuration has changed No action is needed 70001 The rebuild process has started No action is needed 70002 The rebuild completed without error No action is needed 70003 The drive copying information cannot be read If a spare drive is available use it to from the primary drive replace the failed drive If no spare is available replace the failed drive with a new drive 70004 If the initiator is master then its follower has If a spare drive is available use it to detected a write error on a member within a replace the failed drive If no spare is mirror drive available replace the failed drive with a new drive 70005 If the initiator is master then it has detected a If a spare drive is available use it to write error on a member within a mirror drive replace the failed drive If no spare is available replace the failed drive with a new drive 70006 Communication between the virtua
55. AnswerBook2 Sun StorEdge StorTools docs sun com Sun Enterprise Sun Fire SunOS Netra SunSolve and Solaris are trademarks registered trademarks or service marks of Sun Microsystems Inc in the U S and other countries All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc in the U S and other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems Inc All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc in the U S and in other countries Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems Inc The OPEN LOOK and Sun Graphical User Interface was developed by Sun Microsystems Inc for its users and licensees Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry Sun holds a non exclusive license from Xerox to the Xerox Graphical User Interface which license also covers Sun s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun s written license agreements Netscape Navigator is a trademark or registered trademark of Netscape Communications Corporation in the United States and other countries U S Government Rights Commercial use Government users are subject to the Sun Microsystems Inc standard licen
56. Broomfield Source diag XXXXX XXX COom Severity Warning Category Switch DeviceId switch 100000c0dd00b682 EventType LogEvent MessageLog EventTime 01 30 2002 11 17 22 Change in Port Statistics on switch diag209 sw2a ip 192 168 0 32 Port 8 Received 9746 InvalidTxWds in 0 mins value 9805 FIGURE 8 1 Storage Service Processor Event Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 89 Sun Proprietary Confidential Internal Use Only If both T ports go offline you might see a message like the following The virtualization engine event is alerting the LUN failover Site Lab 3286 DSQA1 Broomfield Source diag xxxxx xxx com Severity Warning Actionable Category Ve DeviceId ve 6257335A 30303142 EventType AlarmEvent volume EventTime 01 30 2002 11 49 05 Volume T49152 on diag209 vla changed from 6257335A 30303142 active 50020F23 00006DFA passive to 6257335A 30303142 active 50020F23 00006DFA passive 50020F23 0000725B INFORMATION This event occurs when the virtualization engine has detected a change in status for a Multipath Drive or VLUN usually meaning a pathing problem to a Sun StorEdge T3 array controller for changes in Active Passive paths 1 Check Sun StorEdge T3 array for current LUN ownership port listmap 2 Use mpdrive failback if needed to fail LUNs back to correct the controller if needed Site Lab 3286 DSQA1 Broomfield Source diag xxxxx
57. CE PROPERTIES for disk dev rdsk c6t29000060220041F96257354230303052d0s2 Status Port A Status Port B Vendor Product ID WWN Node WWN Port A WWN Port B Revision Serial Num Unformatted capacity Write Cache Read Cache Minimum prefetch Maximum prefetch Device Type Path s Controller Device Address Class State Controller Device Address O K O K SUN SESSO1 2a000060220041f4 2b000060220041 f4 2b000060220041f9 080c Unsupported 102400 000 MBytes Enabled Enabled 0x0 0x0 Disk device dev rdsk c6t29000060220041F96257354230303052d0s2 devices scsi_vhci ssd g29000060220041 96257354230303052 c raw devices pci 6 4000 SUNW glc 3 fp 0 0 2b6000060220041f9 0 primary OFFLINE devices pci 6 4000 SUNW glc 2 fp 0 0 2b6000060220041f4 0 Sun Proprietary Confidential Internal Use Only Class primary State ONLINE Chapter 5 Troubleshooting the Fibre Channel FC Links 45 An error in the Al or B1 FC link can also cause a device to enter the unusable state in cf gadm a1 as shown in CODE EXAMPLE 5 2 CODE EXAMPLE 5 2 cfgadm al Display usr sbin cfgadm al Ap_Id Type Receptacle Occupant Condition c0 scsi bus connected configured unknown c0 dsk c0t0d0 disk connected configured unknown c0 dsk c0t1d0 disk connected configured unknown onl scsi bus connected configured unknown cl dsk clt6 d0 CD ROM connected configured unknown c2 fc fabric connected configure
58. DEVICE PROPERTIES for disk dev rdsk c6t29000060220041F96257354230303052d0s2 Status Port A O K Status Port B O K Vendor SUN Product ID SESSO1 WWN Node 2a000060220041f4 WWN Port A 2b000060220041f4 WWN Port B 2b000060220041 f9 Revision 080C Serial Num Unsupported Unformatted capacity 102400 000 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c6t29000060220041F96257354230303052d0s2 devices scsi_vhci ssd g29000060220041f96257354230303052 c raw Controller devices pci 6 4000 SUNW glc 2 fp 0 0 Device Address 2b000060220041f4 0 Class primary State ONLINE Controller devices pci 6 4000 SUNW glc 3 fp 0 0 Device Address 2b6000060220041f9 0 Class primary State ONLINE 16 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Note that in the Class and State fields the virtualization engines are presented as two primary ONLINE devices The current Sun StorEdge Traffic Manager software design does not enable you to manually halt the I O that is you cannot perform a failover to the secondary path when only primary devices are present Manually Halting the I O As an alternative to using the Sun StorEdge Traffic Manager MPxIO software you can manually halt the I O using one of two methods m Quiesce the I O m Unconfigure the c2 pat
59. Gb McData Intrepid Director Switch Guide to Documentation Sun StorEdge SAN 4 1 Release Notes SuperStack 3 Baseline Hub 12 Port TP User Guide SuperStack 3 Baseline Hub 24 Port TP User Guide Part Number 816 5254 816 5252 816 5253 816 5257 816 5256 816 4771 816 4768 816 0774 816 4769 816 4770 806 7979 816 3142 816 4470 816 4469 806 5513 816 5285 816 4472 817 0061 817 0056 817 0057 817 0062 817 0063 817 0071 3C16440A 3C16441A XVIII Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Product Title Part Number SANbox 8 16 e SANbox 8 16 Segmented Loop Fibre Channel Switch Management 875 3060 Segmented Loop FC User s Manual Switch e SANbox 8 Segmented Loop Fibre Channel Switch Installer s User s 875 1881 Manual e SANbox 16 Segmented Loop Fibre Channel Switch Installer s User s 875 3059 Manual Expansion cabinet e Sun StorEdge Expansion Cabinet Installation and Service Manual 805 3067 Storage Server Processor Sun V100 Server User s Guide 806 5980 e Netra X1 Server User s Guide 806 5980 e Netra X1 Server Hard Disk Drive Installation Guide 806 7670 Preface XIX Sun Proprietary Confidential Internal Use Only Accessing Sun Documentation Online You can view print or purchase a broad selection of Sun documentation including localized versions at http www sun com documentation Sun We
60. Host DeviceId host diag xxxxx xxx com EventType AlarmEvent P hba EventTime 01 30 2002 11 50 10 status of hba devices pci 1f 4000 pci 2 SUNW qlc 5 fp 0 0 devctl on diag xxxxx xxx com changed from NOT CONNECTED to CONNECTED INFORMATION monitors changes in the output of luxadm e port Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices Sun Proprietary Confidential Internal Use Only 91 v To Verify the Storage Service Processor 1 Run the Sun StorEdge T3 array port 1istmap command to see the failover event t3b0 lt 1 gt port listmap port targetid addr_type lun volume owner access ulpl 0 hard 0 voll ul primary ulpl 0 hard 1 vol2 ul failover u2pl 1 hard 0 voll ul failover u2pl 1 hard 1 vol2 ul primary 2 Compare the virtualization engine configuration to a saved configuration by running runsecfg 1M 3 Choose Verify Virtualization Engine Map The output is from the diff 1 command which shows the lines that have been added changed or deleted Notice that the active Sun StorEdge T3 array controller WWN for one of the Sun StorEdge T3 arrays has changed indicating it is using its alternate path MANAGE CONFIGURATION FILES MENU 1 Display Virtualization Engine Map 2 Save Virtualization Engine Map 3 Verify Virtualization Engine Map 4 Help 5 Return Select configuration option above gt 3 Verifying Virtualization Engine map for vl ERROR virtualization engine map f
61. Host with HBA 0 and HBA 1 LUNO 10G Active MPDrive0 LUNO 10G Active MPDrive0 LUNI 10G Active MPDrivel LUN1 10G Active MPDrivel Switch L 4 lt A Switch SAN il Virtualization Database Virtualization Engine 1 lt gt lt gt Engine 2 Storage I O and Virtualization Engine Communications Traffic Switch Switch f Logical Multipath Drive LUNO 500G Passive Master s MPDrive 0 LUNO 500G Active Master LUN1 500G Active Alternate Master LUNI 500G Passive Logical Multipath Drive Alternate Master S MPDrive1 T3ES Master 0A 1P I Alternate Master 1A OP FIGURE 2 3 Primary Data Paths to the Master Sun StorEdge T3 Array To access the LUN on the alternate master the Sun StorEdge T3 array I O could travel From HBA O gt switch gt virtualization engine 1 gt switch gt alternate master controller primary route from HBA 0 From HBA O gt switch gt virtualization engine 1 gt switch gt switch gt master controller gt backend loop to alternate master secondary route from HBA 0 From HBA 1 gt switch gt virtualization engine 2 gt switch gt switch gt alternate master controller primary route from HBA 1 From HBA 1 gt switch
62. If errors persist manually power The switch took longer than two cycle the switch minutes to reset after a configuration change setupswitch Could not set chassis ID on switch 1 Check the switch chassis IDs of all switch to cid switches in the SAN 2 Verify that each ID is unique This occurs only ina SAN 3 After the chassis IDs have been environment with cascaded switches established override the switch chassis IDs with the following command setupswitch s Sswitch_name i Sunique_chassis_id v 170 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Sun StorEdge T3 Array Partner Group Error Messages Caution Running restoret 3config 1M or modifyt3config 1M destroys all data on the Sun StorEdge T3 array TABLEB 3 Sun StorEdge T3 Array Error Messages Source of Error Message Common to Sun StorEdge T3 array Cause of Error Message match the reference standard configurations e This particular configuration is not a standard supported type Suggested Corrective Action e The current configuration does not 1 Check the current Sun StorEdge T3 array configuration with the showt3 n lt t3 gt command 2 Verify whether the configuration is corrupted or has changed 3 Refer to the raid cfg files in opt SUNWsecfg etc to determine if the configuration commands are set up and functioning properly Common to S
63. N failback commands Chapter 9 Troubleshooting Virtualization Engine Devices 119 Sun Proprietary Confidential Internal Use Only v To Failback the Virtualization Engine In the event of a Sun StorEdge T3 array LUN failover the virtualization engine will route all I O through the failover port on the Sun StorEdge T3 array After you isolate and check the cause of the failover the virtualization engine continues to send I O through the failover path To restore the I O to the primary path and fail the LUN back to its original controller use the following procedure 1 Verify that the T3 array active path needs to be restored by viewing a live snapshot of the virtualization engine map as shown in Viewing the Virtualization Engine Map on page 118 If there has been a failover the Multipath Drive Summary will show the same Sun StorEdge T3 array active path WWN for all LUNs associated with one Sun StorEdge T3 array as shown in CODE EXAMPLE 9 3 CODE EXAMPLE 9 3 Multipath Drive Summary Disk pool MP Drive T3 Active Controller Serial Target Path WWN Number t3b00 T49152 50020F230000725B 60020F2000006DFA t3b01 T49153 50020F230000725B 60020F2000006DFA 2 If the Sun StorEdge T3 array LUNS have failed over run the command found in CODE EXAMPLE 9 5 for that specific Sun StorEdge T3 array Note The Sun StorEdge T3 array name is the same as the disk pool name but with the last digit equal to the Sun StorEdge T3 a
64. Netscape IE Output Options Test List agi 56 central sun com Device 1 2 192 168 0 30 0x0 Ended 2002 06 20 12 51 44 RC 0 switchtest called with options dev 2 192 168 0 30 0x01 xfersize 2000 iterations 1000001 i userpattem 0x7e7e7e7el selectpattern critical switchtest Started i Testing port 2 Using ip_addr 192 168 0 30 fcaddr 0x0 to access this port Chassis Status for Device Switch Power OK Temp OK 24 0c Fan 1 OK Fan 2 OK Testing Device Switch Port 2 Pattern Ox7e7e7e7e Testing Device Switch Port 2 Pattern Oxlelelele Testing Device Switch Port 2 Pattern Oxflfififi Testing Device Switch Port 2 Pattern OxbSbSbSbS Testing Device Switch Port 2 Pattern Oxtadadada Testing Device Switch Port 2 Pattern 0x78787878 Testing Device Switch Port 2 Patten Oxe7e7e7e7 Testing Device Switch Port 2 Pattern 0xaa55aa55 Testing Device Switch Port 2 Pattern Ox7 7 7f 7f Testing Device Switch Port 2 Pattern OxOf0 0 0f Testing Device Switch Port 2 Pattern Ox00f00f Testing Device Switch Port 2 Pattern 0x25252525 Port 2 passed all tests on Switch FIGURE 11 8 Successful Switch Test Results On this pass the test was successful This indicates that the problem was most likely the switch side GBIC which was replaced 6 Recover the problem with the GBIC or the switch a Recable the link be
65. RU Tests Available for the A3 or B3 FC Link Segment m The linktest is not available Both the switch and the GBIC are tested using the switchtest test The switchtest test a Can be used only in conjunction with the loopback connector a Cannot be cabled to the virtualization engine while switchtest runs a No virtualization engine tests are available at this time Chapter 5 Troubleshooting the Fibre Channel FC Links 57 Sun Proprietary Confidential Internal Use Only v To Isolate the A3 or B3 FC Link To isolate the A3 or B3 link which is the FC link from the virtualization engine to the back end switch follow these steps Note The A3 or B3 FC link exists in a Sun StorEdge 6900 series only 1 Quiesce the I O on the A3 or B3 FC link path refer to Quiescing the I O on the A3 or B3 Link on page 59 2 Break the connection by uncabling the link 3 Insert the loopback connector in to the switch port 4 Run switchtest a If the test fails replace the GBIC and rerun switchtest b If the test fails again replace the switch 5 If the switch or the GBIC shows no errors replace the remaining components in the following order a Replace the virtualization engine side GBIC recable the link and monitor the link for errors b Replace the cable recable the link and monitor the link for errors c Replace the virtualization engine restore the virtualization engine settings recable the link and monitor the lin
66. SEcfglog to see if any errors are indicated In the following example creatediskpools 1M for t3b0 indicates a missing Sun StorEdge T3 array path H hu May 30 17 35 19 MDT 2002 creatediskpools t3b0 ENTER opt SUNWsecfg bin creatediskpools n t3b0 H hu May 30 17 35 19 MDT 2002 checkslicd vl ENTER opt SUNWsecfg bin heckslicd n vl Q Thu May 30 17 35 21 MDT 2002 checkslicd vl EXIT There are no eligible drives to create MultiPath drive automatically Thu May 30 17 35 32 MDT 2002 creatediskpools ERROR No mpdrives found on virtualization engine pair vl Thu May 30 17 35 32 MDT 2002 creatediskpools INFO verify all T3 array LUNS have 2 paths to the virtualization engine pair then re run creatediskpools Thu May 30 17 35 32 MDT 2002 creatediskpools Failed to create at least one disk pool Thu May 30 17 35 33 MDT 2002 creatediskpools t3b0 EXIT Chapter 9 Troubleshooting Virtualization Engine Devices 129 Sun Proprietary Confidential Internal Use Only 2 Run the showswitch 1M command for sw2a and sw2b Refer to the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Manual to see to which switch ports the Sun StorEdge T3 array and virtualization engine should be attached In this example the Sun StorEdge T3 array t 3b0 should be attached to port 2 of sw2a and sw2b and the virtualization engine should be attached to port 1 All ports should be online showswitch s sw2a
67. SPOF single point of failure SRN Service Request Number SRS Sun Remote Services SSP Storage Service Processor SVE storage virtualization engine TCP IP transport control protocol internet protocol VLUN virtual LUN WWN worldwide name Abbreviations and Acronyms 178 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Index NUMERICS 2 Gbit switch error messages 168 3Com Ethernet hubs 35 A Al or B1 link verifying 45 A2 or B2 link isolating 52 Storage Service Processor Side Event 50 verifying 51 A2 B2 link FRU test availability 50 A3 or B3 link FRU test availability 57 host side event 54 isolating 58 Storage Service Processor side event 55 verifying 57 A4 or B4 link FRU tests available 64 isolation of 64 Storage Service Processor side notification 61 troubleshooting 60 verifying data host 62 verifying Sun StorEdge 3900 series 62 verifying Sun StorEdge 6900 series 62 C c2 path returning to production 19 unconfiguring 17 cfgadm verifying functionality 4 checkdefaultconfig verifying functionality 4 command line test example qlctest 1M 27 switchtest 1M 28 communication loss event 3 configuration settings 23 verifying 7 creatediskpools 1M failure diagnosing 129 D data host Fibre Channel link 45 verifying Sun StorEdge 3900 series 62 verifying Sun StorEdge 6900 series 62 database corrupt 160 diagnostic codes
68. ST 2002 checkt3config Mon Jan 7 18 07 51 PST 2002 checkt3config 18 07 51 PST 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 8 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config 18 07 51 PST 2002 checkt3config t3b0 I t3b0 t3b0 t3b0 I t3b0 t3b0 t3b0 t3b0 t3b0 t3b0 I t3b0 t3b0 I t3b0 t3b0 t3b0 t3b0 I t3b0 t3b0 I t3b0 t3b0 t3b0 I FO FO FO FO FO F O FO FO FO FO FO FO INFO FO FO FO FO FO FO Ecfglog file for the 16k auto blocksize cache mirror auto mp_support rd_ahead recon_rate rw off sys memsize 32 cache memsize 16k auto off blocksize cache mirror rw off mp_support rd_ahead recon_rate sys memsize 32 cache memsize med med In this example the mirror setting in the Sun StorEdge T3 array system settings is off The saved configuration setting for this parameter which is the default setting should be auto 3 Fix the FAIL condition and then verify the settings again Checking Sun Proprietary Confidential Internal Use Only t3b0 Checking Checking Checking Checking Checking opt SUNWsecfg bin checkt3config n t3b0 Configurations lt command ver vol stat port dist comman
69. SUNW qlc 0 30000 fp 0 0 devctl on iag245 central sun com changed from NOT CONNECTED to ONNECTED Info status of hba devices sbus 9 0 SUNW qlc 0 30000 fp 0 0 devctl on iag245 central sun com changed from CONNECTED to NOT ONNECTED Info The state of lun T300 c14t50020F2300003EESd0s2 statusA on iag245 central sun com changed from O K to ERROR target t3 diag244 t3b0 90 0 0 40 iag245 central sun com changed from O K to ERROR ifptest diag240 on host failed qictest diag240 on host failed Info New Patch and Package Information generated Page 1 of 1 8 events Sev Severity of the event warning gt Error gt Down Action This event is Actionable and will be sent to RSS SRS SubComp SubComponent FIGURE 6 1 Sample Host Event Grid 68 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 6 1 lists all the host events in the Storage Automated Diagnostic Environment TABLE6 1 Storage Automated Diagnostic Environment Event Grid for the Host 5 Z JS E 5 z s lg Z 3 2 a d lt S o a 5 8 i a E HBA Alarm Yellow The status of hba Monitors changes in the devices sbus 9 0 output of the SUNW qlc 0 30000 luxadm e port fp 0 0 devctl on diag xXxxxx xxx com The status changed from not connected to connected HBA Alarm Red Y The status of hba
70. ache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c20t2B000060220041F4d0s2 devices pci a 2000 pci 2 SUNW qlc 4 fp 0 0 ssd w2b000060220041f4 0 c raw luxadm display dev rdsk c23t2B000060220041F9d0s2 DEVICE PROPERTIES for disk dev rdsk c23t2B000060220041F9d0s2 Status Port A O K Vendor SUN Product ID SESSO1 WWN Node 2a000060220041f9 WWN Port A 2b000060220041f9 Revision 080c Serial Num Unsupported Unformatted capacity 102400 000 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c23t2B000060220041F9d0s2 devices pci e 2000 pci 2 SUNW glc 4 fp 0 0 ssd w2b000060220041f9 0 c raw Chapter 2 General Troubleshooting Procedures 21 Sun Proprietary Confidential Internal Use Only v To Put the DMP Enabled Paths Back into Production 1 Type vxdmpadm enable ctlr lt cn gt 2 Verify that the path has been reenabled by typing vxdmpadm listctlr all 22 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 3 Troubleshooting Tools This chapter contains the following information related to tools used to troubleshoot the Sun StorEdge 3900 or 6900 series components Storage Automated Diagnostic Environment 2 2 on page 23 Microsoft Windows 2000 Syste
71. alization Engine Diagnostics on page 108 a Virtualization Engine LEDs on page 110 a Translating Host Device Names on page 115 a Virtualization Engine Event Grid on page 132 About the Virtualization Engine The virtualization engine supports the multipathing functionality of the Sun StorEdge T3 array Each virtualization engine has physical access to all underlying Sun StorEdge T3 arrays and controls access to half of the Sun StorEdge T3 arrays The virtualization engine has the ability to assume control of all arrays in the event of component failure The configuration is maintained between virtualization engine pairs through redundant T port connections by way of a pair of Sun StorEdge network FC switch 8 or switch 16 switches 107 Sun Proprietary Confidential Internal Use Only Virtualization Engine Diagnostics The virtualization engine monitors the following components m Virtualization engine router m Sun StorEdge T3 array a Cabling between the router and the storage Service Request Numbers SRNs SRNs are used to inform the user of storage subsystem activities Service and Diagnostic Codes The virtualization engine s service and diagnostic codes inform the user of subsystem activities The codes are presented as a light emitting diode LED readout See Appendix A for the table of codes and related appropriate actions to take In some cases you might not be able to receive SRNs because of
72. ar 22a oe aa Test Increment 1 10 000 125 g 4 10 Woo Woo t 1 Wan Ban foo h i Hoo f di A ali h K Test continuously i v Stop on error rLoopback Test Results Test Status CRC Error Disparity Error Frame length error Success 0 0 0 Cee 55 55 55 55 55 55 55 55 Stop ae Loopback Test ReadWrite Buffer Test Adapter 2200 FIGURE 3 4 QLogic SANblade Manager Diagnostics Note Differing HBA manufacturer s may bundle different features with their tools The information in this guide is written with the assumption of Qlogic software usage Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 4 Troubleshooting Ethernet Hubs The Sun StorEdge 3900 and 6900 series uses an Ethernet hub as the backbone for the internal service network The allocation of Ethernet ports is as follows m One for the Storage Service Processor per subsystem m One for each FC switch a One for each virtualization engine a Two for each Sun StorEdge T3 array partner group m One for the Ethernet hub that is installed on the second Sun StorEdge Expansion Cabinet in the Sun StorEdge 3960 and 6960 series systems Note Information about LED status lights power information and front panel settings can be found in the 3Com document SuperStack 3 Baseline Hub 12 Port TP User Guide or SuperStack 3 Baseline Hub 24 Port TP User Gu
73. ate Maste MPDrive 1 T3ES Master 0A 1P II Alternate Master 1A 0P FIGURE 2 5 Path Failure I O Routed Through Both HBAs In the event of a path failure after the second tier of Sun StorEdge network FC switch 8 and switch 16 switches or in the event that both T ports fail between the switches the virtualization engine forces a LUN failover of the affected Sun StorEdge T3 array and routes all I O to its secondary path From the host side nothing has changed all I O is routed through both HBAs refer to FIGURE 2 5 Chapter 2 General Troubleshooting Procedures 15 Sun Proprietary Confidential Internal Use Only Multipathing Options in the Sun StorEdge 6900 Series The presence of the virtualization engine makes multipathing in a Sun StorEdge 6900 series environment challenging Unlike Sun StorEdge T3 array and Sun StorEdge network FC switch 8 and switch 16 switch installations which present primary and secondary pathing options the virtualization engines present only primary pathing options to the data host The virtualization engines handle all failover and failback operations and mask those operations from the multipathing software on the data host The following example illustrates a Sun StorEdge Traffic Manager MPxIO software problem on a Sun StorEdge 6900 series system usr sbin luxadm display dev rdsk c6t29000060220041F96257354230303052d0s2
74. ate of the power in 1 Open a Telnet session the Sun StorEdge T3 to the affected Sun array power cooling unit StorEdge T3 array is not optimal 2 Verify power cooling unit state in fru stat 3 Replace power cooling unit if necessary power temp Alarm Red Y The state of the 1 Open a Telnet session temperature in the Sun to the affected Sun StorEdge T3 array StorEdge T3 array power cooling unit is 2 Verify that the power either too high or is cooling unit state is in unknown fru stat 3 Replace the power cooling unit if necessary log Alarm Red Y This event includes all Check the messages important errors found file for appropriate action time_diff Alarm Yellow Y Fix the date and time on the Sun StorEdge T3 array using the date command The date and time should be the same as the monitoring host 98 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array E 2 S gt a c T c E 5 S 3 E ra B lt a 2 enclosure Audit Auditing a new Sun StorEdge T3 array Audits occur every week The Storage Automated Diagnostic Environment sends a detailed description of the enclosure to the Sun Network Storage Command Center NSCC ib Comm_ Communication regained Established InBand ib oob Comm_ Co
75. ation has been recovered between the No action is needed two virtualization engines Appendix A Virtualization Engine References 157 Sun Proprietary Confidential Internal Use Only TABLE A 1 SRN Reference SRN Description Corrective Action 71001 This is a generic error code for the SLIC It 1 Check the condition of the signifies communication problems between the virtualization engine virtualization engine and the daemon 2 Check the cabling between the virtualization engine and daemon Error halt mode also forces this service request server number 71002 The SLIC was busy Check the condition of the virtualization engine Error halt mode also forces this service request Check the cabling between the number virtualization engine and the daemon server 71003 The SLIC master was unreachable Check conditions of the virtualization engines in the SAN 71010 The status of the SLIC daemon has changed No action is needed 72000 The primary and secondary SLIC daemon No action is needed connection is active 72001 The virtualization engine failed to read the SAN No action is needed drive configuration 72002 The virtualization engine failed to lock on to the No action is needed SLIC daemon 72003 The virtualization engine failed to read the SAN No action is needed SignOn information 72004 The virtualization engine failed to read the zone No action is needed configuration 72005 The virtualization engi
76. bles GBICs and connections along FC path 2 Check Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segment of the data path 3 Verify correct FC switch configuration FIGURE 5 10 A3 or B3 FC Link Storage Service Processor Side Event Chapter 5 Troubleshooting the Fibre Channel FC Links 55 Sun Proprietary Confidential Internal Use Only Verifying the Data Host An error in the A3 or B3 FC link results in a device being listed as in an unusable state in cf gadm but no HBAs are listed as in the unconnected state in luxadm output The multipathing software will note an offline path CODE EXAMPLE 5 5 Devices in the Connected State cfgadm al Ap_Id Type Receptacle Occupant Condition c0 scsi bus connected configured unknown c0 dsk c0t0d0 disk connected configured unknown c0 dsk c0t1d0 disk connected configured unknown ag scsi bus connected configured unknown c1 dsk c1t6d0 CD ROM connected configured unknown C2 fc fabric connected configured unknown c2 210100e08b23fa25 unknown connected unconfigured unknown c2 2b000060220041f4 disk connected configured unknown 3 fc fabric connected configured unknown c3 2b000060220041f9 disk connected configured unusable c3 210100e08b230926 unknown connected unconfigured unknown c4 fc private connected unconfigured unknown c5 fc connected unconfigured unknown usr sbin luxadm e port Found path to 2 HBA ports devices pcit6 400
77. bleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only m You can now run opt SUNWexplo bin explorer for information about the Storage Service Processor operating system the Sun StorEdge network FC switch 8 or switch 16 switch and Sun StorEdge T3 array information that you can use for troubleshooting purposes a Atar gzip file is put in the opt SUNWexplo output tar gzip file directory You can send the tar gzip file to Sun Solution Center for evaluation m The Sun StorEdge network FC switch 8 and switch 16 switch information is placed in the san directory of the tar file m Sun StorEdge T3 array information is placed in the disk s t3 directory Chapter 3 Troubleshooting Tools 31 Sun Proprietary Confidential Internal Use Only Monitoring Host Bus Adapters HBAs Using QLogic SANblade Manager The most effective way to retrieve HBA status and information is by using the HBA manufacturer s utility such as the Qlogic SANblade Manager software provided by Qlogic for their HBAs This software is freely downloadable from Qlogic s website http www qlogic com Note Other manufacturer s utilities such as LightPulse s Emulex are needed for other HBA s such as Emulex HBAs Use the Qlogic SANblade Manager to extract information about HBA Driver versions m Firmware versions a A primitive topology view a ALUN listing m Diagnostics on the HBA 32 Sun StorEdge 3900 and 6900 Series
78. ces when gathering system information from the customer Chapter 10 Troubleshooting Using Microsoft Windows 2000 139 Sun Proprietary Confidential Internal Use Only v To Use the Sun StorEdge T3 Array Failover Driver GUI Note The Sun StorEdge T3 Array Failover Driver GUI is limited to the Sun StorEdge 3900 series systems You must use the CLI for the Sun StorEdge 6900 series systems 1 Make sure the Sun StorEdge T3 Array failover driver is loaded From the Microsoft Windows 2000 Advanced Server GUI click Administrative Tools gt Computer Management gt Software Environment 2 Ensure the Jafo driver is in a Running and OK state 3 Launch the Sun StorEdge T3 Array Failover Driver GUI using instructions found in Launching the Sun StorEdge T3 Array Failover Driver GUI on page 138 The Multipath Configurator window is displayed A healthy Sun StorEdge 3900 series system has a solid line connecting the HBA to the storage as shown in FIGURE 10 3 Multipath Configurator E Sfl x Driver HBA Path Array Help 5 smi 2gsSm2bdhom 60020F20000003D50000000000000000 Fibre Channel Adapter Array 1 LUN Count 1 data path ae ea acre A 60020F20000003D50000000000000000 Array 2 LUN Count 1 data path FIGURE 10 3 Healthy Sun StorEdge 3900 series system shown using Multipath Configurator Note Note the connection between the two arrays indicating that the back end loop is being used 14
79. ch Refer to the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide to determine which switch ports are used for each component Run the showswitch 1M command for sw2a and sw2b Look at the output sections Port Status and Name Server to see if the ports are online The output will look like that in CODE EXAMPLE 9 6 if there are no problems Chapter 9 Troubleshooting Virtualization Engine Devices 121 Sun Proprietary Confidential Internal Use Only 122 CODE EXAMPLE 9 6 showswitch s sw2a KKKKKKKKKKKK Port Status KKKKKKKKKKKK Error Free Online Switch Ports Port Port Type Admin State Oper State Status Loop Mode 1 F_Port online online logged in 2 TL_Port online online logged in Target Devices 1 Address 0x02 Oxef Proxy AL_PA Public Address World Wide Name E8 0010C000 2900006022004195 E4 00110000 2900006022004186 3 TL_Port online offline Not logged in 4 Lh Pore online offline Not logged in 5 TL_Port online offline Not logged in 6 TL_Port online offline Not logged in 7 T_Port online online logged in 8 T_Port online online logged in KKKKKKKKK Name Server KKKKKKKKKKKK Port Address Type PortWWN Node WWN FC 4 Types 01 10C000 N 02 LOCLEF NL 2900006022004195 50020f2300006dfa 2800006022004195 SCSI_FCP 50020 2000006dfa SCSI_FCP Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 6 If either port 1 or por
80. ch 2003 Sun Proprietary Confidential Internal Use Only APPENDIX B Configuration Utility Error Messages The Sun StorEdge 3900 and 6900 Series Reference Manual lists and defines the command utilities that configure the various components of the Sun StorEdge 3900 and 6900 series storage systems If you encounter errors with the command line utilities refer to the recommendations for corrective action in this appendix The error messages are broken out into the following sections a Virtualization Engine Error Messages on page 164 TABLE B 1 lists SUNWsecfg error messages specific to the virtualization engine m Switch Error Messages on page 168 TABLE B 2 lists SUNWsecfg error messages specific to the Sun StorEdge network FC switch 8 and switch 16 switches m Sun StorEdge T3 Array Partner Group Error Messages on page 171 TABLE B 3 lists SUNWsecfg error messages specific to the Sun StorEdge T3 array m Other Error Messages on page 175 TABLE B 4 lists miscellaneous SUNWsecfg error messages common to all components 163 Sun Proprietary Confidential Internal Use Only Virtualization Engine Error Messages TABLE B 1 Source of Error Message Virtualization Engine Error Messages Cause of Error Message Suggested Corrective Action Common to virtualization engine Invalid virtualization engine pair name or the virtualization engine is unavailable This is usually because the savevemap co
81. communication errors If this occurs you must read the virtualization engine LEDs to determine the problem Retrieving Service Information You can retrieve service information from one of two sources m CLI Interface m Error Log Analysis Commands Both of these sources are described in the following sections CLI Interface The Serial Loop Intraconnect SLIC daemon which runs on the Storage Service Processor communicates with the virtualization engine The SLIC daemon periodically polls the virtualization engine for all subsystem errors and topology changes It then passes this information in the form of a SRN to the Error Log file 108 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Error Log Analysis Commands v To Display the Log Files and Retrieve SRNs Type opt svengine sduc sreadlog Errors that need action are returned in the following format TimeStamp nnn Txxxxx uuuuuuUuU SRN mmmmm TimeStamp nnn Txxxxx uuuuuuUuU SRN mmmmm TimeStamp Nnnn TXXXXX UUUUUUUU SRN mmmmm A description of the errors follows Item Description TimeStamp The time and date when the error occurred nnn The name of the virtualization engine pair v1 or v2 Txxxxx The LUN where the error occurred uuuuuuuu The unique ID of the drive or the virtualization engine router SRN mmmmm The SRN defined in numerical order Refer to Virtualization Engine Ref
82. d command command command sys list Chapter 2 port listmap PASS PASS PASS PASS PASS General Troubleshooting Procedures 9 Clearing the Lock File If you interrupt any of the Configuration Utility scripts by typing Control c for example a lock file might remain in the opt SUNWsecfg etc directory causing subsequent commands to fail Use the following procedure to clear the lock file v To Clear the Lock File 1 Type the following command opt SUNWsecfg bin removelocks usage removelocks t s v where t remove all T3 related lock files s remove all switch related lock files v remove all virtualization engine related lock files opt SUNWsecfg bin removelocks v Note After making any change to the virtualization engine configuration the script saves a new copy of the virtualization engine map This may take a minimum of two minutes during which time no additional virtualization engine changes are accepted If a process such as savevemap 1M is running you cannot remove the lock file using the removelocks 1M command This process causes a component to be unavailable 2 Monitor the var adm log SEcfglog file to see when the savevemap 1M process successfully exits CODE EXAMPLE 2 2 savevemap 1M Output Tue Jan 29 16 12 34 MST 2002 savevemap vl ENTER Tue Jan 29 16 12 34 MST 2002 checkslicd vl ENTER Tue Jan 29 16 12 42 MST 2002 checkslicd vl EXIT
83. d switch 16 switch to Sun StorEdge T3 array link T3 alternate master A4 T3 Master FIGURE 5 1 Sun StorEdge 3900 Series FC Link Diagram Chapter 5 Troubleshooting the Fibre Channel FC Links 39 Sun Proprietary Confidential Internal Use Only 40 TABLE 5 2 and FIGURE 5 2 shows the basic components and the FC links for a Sun StorEdge 6900 series system TABLE 5 2 Link Al to B1 A2 to B2 A3 to B3 A4 to B4 T1 to T2 Ax to Bx FC Links Provides FC Link Between These Components HBA to Sun StorEdge network FC switch 8 and switch 16 switch link Sun StorEdge network FC switch 8 and switch 16 switch to virtualization engine link on the host side Sun StorEdge network FC switch 8 and switch 16 switch to the virtualization engine link on the device side Sun StorEdge network FC switch 8 and switch 16 switch to Sun StorEdge T3 array link T port switch to switch link Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FIGURE 5 2 Sun StorEdge 6900 Series FC Link Diagram Chapter 5 Troubleshooting the Fibre Channel FC Links 41 Sun Proprietary Confidential Internal Use Only Troubleshooting the Al or B1 FC Link The A1 or B1 link is the FC link from the HBA to the switch What happens when a FC link fails depends on the system If a problem occurs with the A1 or B1 FC link m Ina Sun StorEdge 3900
84. d unknown c2 210100e08b23fa25 unknown connected unconfigured unknown c2 2b000060220041f4 disk connected configured unknown 3 fc fabric connected configured unknown c3 2b000060220041f9 disk connected configured unusable c4 fc private connected unconfigured unknown eke fg connected unconfigured unknown FRU Tests Available for the A1 or B1 FC Link Segment The following FRU tests are available for the A1 or B1 FC link segment All diagnostics are located in opt SUNWstade Diags bin Refer to the man pages for more details a HBA qlictest 1M a Available only if the Storage Automated Diagnostic Environment is installed on a data host Causes HBA to go offline and online during tests m Switch switchtest 1M Can be run while the link is still cabled and online connected to HBA a Can be run only from the Storage Service Processor a The dev option to switchtest is in the following format Port IP Address FCAddress The FCAddress can be set to 0x0 Note If you are testing an A1 or B1 FC link that is connected to an HBA you must specify a payload of 200 bytes or less This is a limitation in the HBA application specific integrated circuit ASIC 46 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CODE EXAMPLE 5 3 switchtest 1M Called With Options opt SUNWstade Diags bin switchtest v o dev 2 192 168 0 30 0 switchtest
85. e Monitors changes in the devices sbus 9 0 output of the luxadm e SUNW qlc 0 30000 port fp 0 0 devctl on e Finds the path to 20 diag xxxxx xxx com HBA ports The status changed from connected to not connected LUN Alarm Red Y The state of The luxadm display 300 1UN t300 c14t50020F2 reported a change in the 300003EE5d0s2 status port status of one of its A on paths The Storage diag xxxxx xxx com Automated Diagnostic The status changed from Environment tries to find OK to error the enclosure target t3 diag244 t3b0 corresponding to this path 90 0 0 40 by reviewing its database of Sun StorEdge T3 arrays and virtualization engines LUN Alarm Red Y The state of The luxadm display VE LUN VE c14t50020F230 reported a change in the O003EE5d0s2 statusA port status of one of its on diag xxxxx xxx com paths The Storage Automated Diagnostic The Status changed from Environment tries to find OK to error the enclosure target ve diag244 corresponding to this path ve0 90 0 0 40 by reviewing its database of Sun StorEdge T3 arrays and virtualization engines Chapter 6 Troubleshooting Host Devices Sun Proprietary Confidential Internal Use Only 69 TABLE6 1 Storage Automated Diagnostic Environment Event Grid for the Host Continued 5 a 8 E 8 e 5 a z 3 2 S 2 S 8 i ifptest Diagnostic Red Y ifptest diag240 on the Check Test Manager for Test host failed failure details
86. e SAN 4 1 Release Field Troubleshooting Guide This document covers the Sun StorEdge network FC switch 8 and switch 16 switch and the interconnections HBA GBIC and cables on either side of the switch The Sun StorEdge SAN 4 1 Release Field Troubleshooting Guide also includes an appendix on the Brocade Silkworm switch troubleshooting 76 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Using the Switch Event Grid The Storage Automated Diagnostic Environment Switch Event Grid enables you to sort switch events by component category or event type The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severity of the event tells whether action is required provides a description of the event and gives the recommended action Refer to the Storage Automated Diagnostic Environment User s Guide for more information v To Use the Switch Event Grid 1 From the Storage Automated Diagnostic Environment Help menu select the Event Grid link 2 FIGURE 7 1 shows the Switch Event Grid from which you can select related criteria for the event you are troubleshooting Maintenance Monitor Diagnose Report Utilities FE amp Sun Storage Automated Diagnostic Environment e PEANT j 4 SAAd 2 0 06 010 diag176 central sun com Help Site Help Event Grid Help Help Page Select a Category Component EventType and type GO
87. e The restore physical and logical data failed e The restore zone data failed e The virtualization engine is unable to properly configure the virtualization engine host vehost e The virtualization engine cannot continue the configuration of other components The setupve 1M command failed Appendix B 1 Check the status of both virtualization engines 2 If an error condition exists refer to Appendix A for corrective action 3 Run the restorevemap command again 1 Check the status of the virtualization engine and reset if necessary 2 Run the setdefaultconfig command again 1 Run setupve n ve_hostname v verbose mode 2 Check the errors 3 Run checkve n ve_hostname You can continue to configure VLUNs and zones only if both of these commands work Configuration Utility Error Messages Sun Proprietary Confidential Internal Use Only 167 Switch Error Messages TABLE B 2 Source of Error Message Sun StorEdge Network FC Switch Error Messages Cause of Error Message Suggested Corrective Action Common to all Sun StorEdge network FC switches The Sun StorEdge system type entered cab_type does not match the system type discovered boxtype Either call the command with the f force option to force the series type or do not specify the cabinet type no c option Common to all Sun StorEdge network FC switches Common to all Sun StorEdge ne
88. eleting the world wide name Refer to the Sun StorEdge T3 and T3 array documentation enablevolslicing enablevolslicing Error checking the Sun StorEdge T3 enable volume slicing status Cannot enable volume slicing The Sun StorEdge T3 array firmware does not support this feature 1 Check the Sun StorEdge T3 array firmware level and verify it is 2 01 00 or higher 2 Check if volume slicing is supported and enabled on the Sun StorEdge T3 array 1 Check the Sun StorEdge T3 array firmware level and verify it is 2 01 00 or higher 2 Upgrade the firmware if required size compare command was executing e The Sun StorEdge T3 array block size parameter is different from the snapshot file The Sun StorEdge T3 array may have been reconfigured modifyt3config e The lock file clear waiting period 1 Check to see if any secfg T3 expired commands are being executed e The creatediskpools command 2 If the commands are executing aborted wait for them to complete 3 Run creatediskpools n t3name restoret3config e An error occurred while the block Run the restoret3config command Appendix B Configuration Utility Error Messages 173 Sun Proprietary Confidential Internal Use Only TABLE B 3 Source of Error Message Sun StorEdge T3 Array Error Messages Continued Cause of Error Message Suggested Corrective Action restoret3config e SLUN configuration fai
89. erences on page 155 for the SRN codes 1 Txxxxx can represent a physical or a logical LUN Chapter9 Troubleshooting Virtualization Engine Devices 109 Sun Proprietary Confidential Internal Use Only Example 2002 2002 2002 2002 2002 2002 v To Clear the Log Type Jan Jan Jan Jan Jan Jan WWW WW Ww 103 LOR STO i10 10 10 K3 13 L73 17 322 225 05 31 10 37 26 54 v1 v1 vl v1 v1 vl opt svengine sduc sreadlog d v1 29000060 220041F9 29000060 220041F9 29000060 220041F9 29000060 220041F9 29000060 220041F9 29000060 220041F9 SRN 70030 SRN 70030 SRN 70030 SRN 70030 SRN 70030 SRN 70030 opt svengine sduc sclrlog Virtualization Engine LEDs TABLE 9 1 describes the LEDs on the back of the virtualization engine TABLE 9 1 Virtualization Engine LEDs LED Color State Description Power Green Solid on The virtualization engine is powered on Status Green Solid on This is the normal operating mode Blink service code The number of blinks indicate a decimal number that corresponds to a diagnostic code Fault Amber Solid on Serious problem Decipher the blinking of the Status LED to determine the diagnostic code After you have determined the diagnostic code look up the decimal number of the code in Appendix A 1 The Status LED blinks a service code when the Fault LED is solid on 110 Sun StorEdge 3900 and 69
90. essages and errors that the data host detects Usually these messages appear in the var adm messages file 6 Storage Service Processor Side Troubleshooting Storage Service Processor side troubleshooting refers to messages alerts and errors that the Storage Automated Diagnostic Environment detects while running on the Storage Service Processor You can find these messages by monitoring the following Sun StorEdge 3900 series and Sun StorEdge 6900 series components m Sun StorEdge network FC switch 8 and switch 16 switches a Virtualization engine m Sun StorEdge T3 array Combining the host side messages and errors and the Storage Service Processor side messages alerts and errors into a meaningful context is essential for proper troubleshooting Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Verifying the Configuration Settings During the course of troubleshooting you might need to verify configuration settings on the various components in the Sun StorEdge 3900 or 6900 series To Verify Configuration Settings Run one of the following scripts a Run the runsecfg 1M script and select the various Verify menu selections for the Sun StorEdge T3 arrays the Sun StorEdge network FC switch 8 and switch 16 switches and the virtualization engine components m Run the checkdefaultconfig 1M script to check all accessible components The output is
91. f4 foc c3 210100eE08b230926 c3 2b000060220041f9 c4 cS cfgadm c unconfigure c2 2b000060220041f4 Type scsi bus disk disk scsi bus CD ROM fc fabric unknown disk fc fabric unknown disk fc private to Receptacle connected connected connected connected connected connected connected connected connected connected connected connected connected Occupant configured configured configured configured configured unconfigured unconfigured unconfigured configured unconfigured configured unconfigured unconfigured Condition unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown unknown 4 Verify that the I O has halted Disabling the path halts the I O only up to the A3 to B3 link see FIGURE 5 8 I O continues to move over the T1 and T2 data paths as well as the A4 to B4 links to the Sun StorEdge T3 array Suspending the I O Use one of the following methods to suspend the I O while the failover occurs m Stop all customer applications that are accessing the Sun StorEdge T3 array a Manually pull the link from the Sun StorEdge T3 array to the switch and wait for a Sun StorEdge T3 array logical unit number LUN failover a After the failover occurs replace the cable and proceed with the testing and FRU isolation a After the testing and any FRU replacement are finished return the Controller state back to the default by
92. ffline Port 2 is a Microsoft Windows 2000 host to switch connection Since the Storage Automated Diagnostic Environment does not have visibility to a Microsoft Windows 2000 host use the Sun StorEdge T3 Array Failover Driver utility the Multipath Configurator and the HBA utility to troubleshoot the host side 2 Check the Sun StorEdge T3 Array Failover Driver The next set of diagrams show the fault as displayed by the driver and the results of drilling down for more details Multipath Configurator lolx Driver HBA Path Array Help 60020F20000003D50000000000000000 B smi 2gsSm2bdhom Fibre Channel Adapter Array LUN Count 1 60020F20000003D50000000000000000 data path Multipath Configurator Driver HBA Path Array Help smi 2gsSm2bdhom 60020F20000003D50000000000000000 Device Sun Microsystems T3 Disk Array WWN 60020F20000003Ds0000000000000000 Serial Number 00163874 Firmware Level 0201 LUN Fo oetans Contains the master controller unit Volume F LUN 0 Primary Path Unknown Alternate Path active enabled nm aoe Array 1 A solid line connecting the HBA to the storage represents a healthy system Array 2 A dotted line connecting the HBA to the storage represents a LUN failover For more information about the affected Sun StorEdge T3 arra LUN right click on the affecte Sun StorEdge T3 array in
93. fline Concentrating on the switch and fixing that failure can help bring the ports and HBAs back online Sun Proprietary Confidential Internal Use Only 1 Discover the error by checking one or more of the following messages or files Storage Automated Diagnostic Environment alerts or email messages a var adm messages a Sun StorEdge T3 array syslog file Storage Service Processor messages a var adm messages t3 messages a var adm log SEcfglog file 2 Determine the extent of the problem by using one or more of the following methods Review the Storage Automated Diagnostic Environment topology view Using the Storage Automated Diagnostic Environment revision checking functionality determine whether the package or patch is installed Verify the functionality using one of the following tools a checkdefaultconfig 1M m cfgadm al output a luxadm 1M output Review the multipathing status using the Sun StorEdge Traffic Manager MPxIO software or vxdmp 1M command 3 Check the status of a Sun StorEdge T3 array by using one or more of the following methods Review the Storage Automated Diagnostic Environment device monitoring reports Run the checkt 3config 1M and showt 3 1M commands which check and display the Sun StorEdge T3 array configuration Manually open a Telnet session to the Sun StorEdge T3 array Review the luxadm 1M display output Review the LED status on the Sun StorEdge T3 array Review t
94. fter a reset 1 Log in to the virtualization engine and verify that the device host and network settings are correct 2 Make sure the virtualization engine hardware is not in ERROR 50 mode 3 If required power cycle the virtualization engine hardware or disable the host side switch port 4 Run the setupve n ve_name command and enable the switch port checkslicd checkslicd The virtualization engine cannot establish communication with the S vepair The virtualization engine cannot establish communication with the virtualization engine pair S vepair initiator Sinitiator Appendix B Run startslicd n S vepair 1 Determine the host name associated with initiator by using the showvemap n vepair f command output 2 Run the command resetve n vename Configuration Utility Error Messages 165 Sun Proprietary Confidential Internal Use Only TABLE B 1 Virtualization Engine Error Messages Continued Source of Error Message Cause of Error Message Suggested Corrective Action checkvemap Cannot establish communication 1 Run the checkvemap command with vepair again 2 If this fails check the status of both virtualization engines 3 If there is an error condition see Appendix A for corrective action createvezone An invalid WWN Swwn is on the The WWN that was specified has a vepair initiator Sinit or the SLIC zone and or an HBA alias has virtualization engine is una
95. g Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 1 1 TABLE 3 1 TABLE 5 1 TABLE 5 2 TABLE 6 1 TABLE 7 1 TABLE 7 2 TABLE 0 1 TABLE 8 1 TABLE 9 1 TABLE 9 2 TABLE 9 3 TABLE 9 4 TABLE 9 5 TABLE 10 1 TABLE A 1 TABLE A 2 TABLE A 3 TABLE A 4 List of Tables Sun StorEdge 3900 and 6900 Series Configurations 1 Event Grid Sorting Criteria 25 FC Links 38 Ax to Bx FC Links 40 Storage Automated Diagnostic Environment Event Grid for the Host 69 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches 78 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches 82 setupswitch Exit Values 85 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array 96 Virtualization Engine LEDs 110 LED Diagnostic Codes 111 Speed Activity and Validity of the Link 112 Virtualization Engine Statistical Data 113 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine 133 Tips for Interpreting Sun StorEdge 6910 Series CLI Output 145 SRN Reference 156 SRN SNMP Single Point of Failure Table 159 Port CommunicationNumbers 160 Virtualization Engine Service Codes 0 399 Host Side Interface Driver Errors 160 List of Tables XIN Sun Proprietary Confidential Internal Use Only TABLE A 5 TABLE B 1 TABLE B 2 TABLE B 3 TABLE B 4 Virtualization Engine Service Codes 400 599 Device Side Interface Dri
96. g the Data Host 56 Verifying the Storage Service Processor Side 57 FRU Tests Available for the A3 or B3 FC Link Segment 57 v__ To Isolate the A3 or B3 FC Link 58 Quiescing the I O on the A3 or B3 Link 59 Suspending the I O on the A3 to B3 Link 59 Troubleshooting the A4 or B4 FC Link 60 Verifying the Data Host 62 Sun StorEdge 3900 Series 62 Sun StorEdge 6900 Series 62 FRU Tests Available for the A4 or B4 FC Link Segment 64 v ToTsolate the A4 or B4 FC Link 64 Troubleshooting Host Devices 67 Using the Host Event Grid 67 v To Access the Host Event Grid 67 Replacing the Master Alternate Master and Slave Monitoring Host 71 v To Replace the Master Host 71 Contents V Sun Proprietary Confidential Internal Use Only v To Replace the Alternate Master or Slave Monitoring Host 72 7 Troubleshooting Switches 73 About the Switches 73 Zone Modifications 74 Switchless Configurations 75 v Diagnosing and Troubleshooting Switch Hardware Problems 75 Using the Switch Event Grid 77 v To Use the Switch Event Grid 77 setupswitch Exit Values 85 8 Troubleshooting the Sun StorEdge T3 Array Devices 87 Troubleshooting the T1 or T2 Data Path 88 Notification Events 89 v To Verify the Storage Service Processor 92 FRU Tests Available for the T1 or T2 Data Path FRU 93 v ToTsolate the T1 or T2 Data Path 94 Sun StorEdge T3 Array Event Grid 95 v To Use the Sun StorEdge T3 Array Event Grid 95 9 Troubleshooting Virtualization Engine Devices 107 Abou
97. h These methods are explained in the following sections To Quiesce the I O Determine the path you want to disable Type cfgadm c unconfigure device To Unconfigure the c2 Path Type cfgadm al Ap_Id Type Receptacle to scsi bus connected c0 dsk c0t0d0 disk connected c0 dsk c0t1id0o disk connected cl scsi bus connected cl dsk cl1t6d0 CD ROM connected c2 fc fabric connected c2 210100e08b23fa25 unknown connected c2 2b000060220041f4 disk connected c3 fc fabric connected c3 210100e08b230926 unknown connected c3 2b000060220041f9 disk connected c4 fc private connected cS ie connected Chapter 2 Occupant configured configured configured configured configured configured unconfigured configured configured unconfigured configured unconfigured unconfigured Sun Proprietary Confidential Internal Use Only Condition un un un un un un un un un un un un un KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN KNOWN General Troubleshooting Procedures 17 2 Using the Storage Automated Diagnostic Environment GUI Topology determine which virtualization engine is in the path you need to disable 3 Use the worldwide name WWN of the virtualization engine that is in the unconfigure command as follows cfgadm al Ap_Id c0 c0 dsk c0t0d0 c0 dsk c0t1d0 el cl dsk c1t6d0 C2 c2 210100e08b23fa25 c2 2b000060220041
98. hapter 9 Troubleshooting Virtualization Engine Devices 123 Sun Proprietary Confidential Internal Use Only v To Reset the SAN Database on Both Virtualization Engines 1 Type xresetsandb n vepair xrestorevemap n vepair You do not need to manually open a Telnet session to the virtualization engines unless an ERROR HALT 50 state is detected Although you might need to power cycle the virtualization engine first attempt to reset the virtualization engines using the following steps 2 To disable the switch ports associated with the vehostname type opt SUNWsecfg flib setveport n vehostname d 3 Open a Telnet session into vehostname and clear the SAN database by entering 9 at the prompt 4 Select Q to exit the telnet session 5 To enable the switch ports associated with the vehostname type opt SUNWsecfg flib setveport n vehostname e 6 To reset the virtualization engine and force it to synchronize with its partner virtualization engine type resetve n vehostname 124 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only To Reset the SAN Database on a Single Virtualization Engine To disconnect the virtualization engine s device side FC cables type setveport v virtualization engine name d Open a Telnet session to the virtualization engine specified in Step 1 Enter the password
99. he Explorer Data Collection Utility output which is located on the Storage Service Processor 4 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 4 Check the status of the Sun StorEdge network FC switch 8 and switch 16 switches using the following tools m Review the Storage Automated Diagnostic Environment device monitoring reports m Run the checkswitch 1M and showswitch 1M commands which check and display the Sun StorEdge FC switch configurations m Review the online and offline LED status codes and POST error codes which can be found in the Sun StorEdge SAN 4 0 and SAN 4 1 Release Installation Guide m Review the Explorer Data Collection Utility output which is located on the Storage Service Processor m Refer to the SANsurfer GUI which supports the Sun StorEdge 4 0 Release or the SANbox Manager which supports the Sun StorEdge 4 1 Release Note To run the SANsurfer GUI or SANbox Manager from the Storage Service Processor you must export X Display 5 Check the status of the virtualization engine using one or more of the following methods m Review the Storage Automated Diagnostic Environment device monitoring reports m Run the checkve 1M checkvemap 1M and showvemap 1M commands which check and display the virtualization host and LUN configurations m Refer to the LED status blink codes Virtualization Engine LEDs on page 110
100. ial Internal Use Only Currently one 10 Gbyte VLUN is created from each physical LUN for a total of two VLUNs The Sun StorEdge 6900 series has four possible physical paths to each Sun StorEdge T3 array volume LUN Refer to FIGURE 2 2 which illustrates primary data paths to the alternate master and FIGURE 2 3 which illustrates the primary data paths to the master Sun StorEdge T3 array Host with HBA 0 and HBA 1 LUNO 10G LUNO 10G Active MPDrive 0 Active MPDrive 0 LUNI 10G LUNI 10G Active MPDrive 1 Active MPDrive 1 Switch Rego Switch SAN Database Virtualization lt gt Engine 2 Virtualization Engine 1 lt gt Storage I O and Virtualization Engine Communications Traffic Switch al LUNO 500G Active Master Switch f Logical Multipath Drive _ MPDrive 0 oa LUNO 500G Passive Master LUNI 500G Passive Alternate Master LUNI 500G ctive Alternate Master Logical Multipath Drive MPDrive 1 T3ES Master 0A 1P LI Alternate Master 1A OP FIGURE 2 2 Primary Data Paths to the Alternate Master 12 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only
101. ial number 6257334B30303148 Chapter 9 Troubleshooting Virtualization Engine Devices 117 Sun Proprietary Confidential Internal Use Only Viewing the Virtualization Engine Map The virtualization engine map is stored on the Storage Service Processor 1 To view the virtualization engine map type opt SUNWsecfg showvemap n vl f VIRTUAL LUN SUMMARY Disk pool VLUN Serial MP Drive VLUN VLUN Size SLIC Zones Number Target Target Name GB t3b00 6257334F30304148 T49152 T16384 VDRV000 55 0 t3b00 6257334F30304149 T49152 T16385 VDRVOO1 55 0 DISK POOL SUMMARY Disk pool RAID MP Drive Size Largest Free Total Free Number of Target GB Block GB Space GB VLUNs t3b00 5 T49152 477 367 367 2 t3b01 5 T49153 477 477 477 0 MULTIPATH DRIVE SUMMARY Disk pool MP Drive T3 Active Controller Serial Target Path WWN Number t3b00 T49152 50020F2300006DFA 60020F2000006DFA t3b01 T49153 50020F230000725B 60020F2000006DFA VIRTUALIZATION ENGINE SUMMARY Initiator UID VE Host Online Revision Number of SLIC Zones 100001 2900006022004195 vila Yes O81 0 100002 2900006022004186 vib Yes 08 17 0 ZONE SUMMARY Zone Name HBA WWN HBA Name Initiator Online Number of VLUNs Undefined 210000E08B033401 Undefined I100001 Yes 0 Undefined 210000EO8BO26COF Undefined 100002 Yes 0 Note This example uses the virtualization engine map file which could include old information 118 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide Ma
102. ide available at http www 3com com For repair and replacement procedures refer to the Sun StorEdge 3900 and 6900 Series Reference and Service Guide 35 Sun Proprietary Confidential Internal Use Only 36 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 5 Troubleshooting the Fibre Channel FC Links FC links diagnose Sun StorEdge network FC components in a SAN or a direct attached storage DAS environment 1inktest 1M which tests the health of the FC links is available only from the Test from Topology view of the Storage Automated Diagnostic Environment GUI Note linktest tests both ends of the link segment and enters a guided isolation when a fault is detected Faults can be detected in one of two ways when linktest sends an alert on a bad or intermittent link or when a red link appears on the topology graph indicating a failure This chapter contains the following sections a FC Links on page 38 a Troubleshooting the A1 or B1 FC Link on page 42 a Troubleshooting the A2 or B2 FC Link on page 49 a Troubleshooting the A3 or B3 FC Link on page 54 a Troubleshooting the A4 or B4 FC Link on page 60 37 Sun Proprietary Confidential Internal Use Only FC Links The following sections provide troubleshooting information for the basic components and FC links listed in TABLE 5 1 TABLE 5 1 FC L
103. idential Internal Use Only TABLE 9 5 Component oob slicd Storage Automated Diagnostic Environment Event Grid for Virtualization Engine Continued EventType Comm_ Lost Severity Down Information The virtualization engine failed to execute slicd command Required Action 2 Check the status of the slicd daemon Check the power on the virtualization engine Make sure the virtualization engine is booted correctly Verify that the TCP IP settings on the virtualization engine are correct Check the T3 message log for failover conditions in the Sun StorEdge T3 array Replace the virtualization engine if necessary oob command 134 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Comm_ Lost Down Invalid command or slicd daemon problem Check the status of the slicd daemon Check the power on the virtualization engine Make sure the virtualization engine is booted correctly Verify that the TCP IP settings on the virtualization engine are correct Check the T3 message log for failover conditions in the Sun StorEdge T3 array Replace the virtualization engine if necessary Sun Proprietary Confidential Internal Use Only TABLE9 5 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine Continued Component Required EventType Severity Information Action ve_diag Diagnostic Red The ve_diag test on
104. ing message New disk pool New disk pool Thu May 30 17 bin creatediskpools n t3b0 Thu May 30 17 checkslicd n vl Thu May 30 17 MultiPath found T00000 and T00002 MultiPath found T00001 and T00003 Automatic MultiPath Drive created successfully Thu May 30 17 Thu May 30 17 Thu May 30 17 40 23 MDT 2002 creatediskpools t3b0 ENTER opt SUNWsecfg 40 24 MDT 2002 checkslicd vl ENTER opt SUNWsecfg bin 40 28 MDT 2002 checkslicd vl EXIT 40 58 MDT 2002 creatediskpools mpdrive T49152 is t3b00 name is t3b00 41 17 MDT 2002 creatediskpools mpdrive T49153 is t3b01 name is t3b01 41 30 MDT 2002 creatediskpools t3b0 EXIT Chapter 9 Troubleshooting Virtualization Engine Devices 131 Sun Proprietary Confidential Internal Use Only Virtualization Engine Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort virtualization engine events by component category or event type The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severity of the event tells whether action is required provides a description of the event and lists the recommended action Refer to the Storage Automated Diagnostic Environment User s Guide Help section for more information v To Use the Virtualization Engine Event Grid From the Storage Automated Diagnostic Environment Help menu select the Event Grid link FIGURE 9 3 shows the Vir
105. inks Link Provides FC Link Between These Components Al to B1 Data host swla and sw1b A2 swla and vla B2 swlb and vlb A3 via and sw2a B3 vib and sw2b A4 Master Sun StorEdge T3 array and the A path switch B4 Alternate master Sun StorEdge T3 array and the B path switch T1 to T2 sw2a and sw2b Sun StorEdge 6900 1 1 Series only By using the Storage Automated Diagnostic Environment you should be able to isolate the problem to one particular segment of the configuration Note The information found in this section is based on the assumption that the Storage Automated Diagnostic Environment is running on the data host and that it is configured to monitor host errors The following diagrams provide troubleshooting information for the basic components and FC links specific to the Sun StorEdge 3900 1 1 series shown in FIGURE 5 1 and the Sun StorEdge 6900 1 1 series shown in FIGURE 5 2 Note An actual Sun StorEdge 3900 or 6900 series configuration could have more Sun StorEdge T3 arrays than are shown in FIGURE 5 1 and FIGURE 5 2 38 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FC Link Diagrams FIGURE 5 1 shows the basic components and the FC links for a Sun StorEdge 3900 series system A1 to B1 HBA to Sun StorEdge network FC switch 8 and switch 16 switch link m A4 to B4 Sun StorEdge network FC switch 8 an
106. k for errors Note The procedures for restoring virtualization engine settings are in the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide 6 Return the path to production 58 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Quiescing the I O on the A3 or B3 Link 1 Determine the path you want to disable 2 Disable the path by typing the following usr bin vxdmpadm disable ctlr lt cn gt 3 Verify that the path is disabled usr bin vxdmpadm listctlr all Steps 1 and 2 halt I O only up to the A3 to B3 link I O continues to move over the T1 and T2 paths as well as the A4 to B4 links to the Sun StorEdge T3 array Suspending the I O on the A3 to B3 Link Use one of the following methods to suspend I O while the failover occurs a Stop all customer applications that are accessing the Sun StorEdge T3 array a Manually pull the link from the Sun StorEdge T3 array to the switch and wait for a Sun StorEdge T3 array LUN failover a After the failover occurs replace the cable and proceed with testing and FRU isolation After testing is complete and any FRU replacement is finished return the controller state back to the default by using the virtualization engine failback command Caution This action will cause SCSI errors on the data host and a brief suspension of I O while the failover occurs
107. l Use Only test examples command line 27 qlctest 1M 27 switchtest 1M 28 testing FRUs 5 tests how to run 5 Sun StorEdge T3 arrays 5 thresholds used in PFA 2 tools troubleshooting 23 troubleshooting broad steps 3 check status of Sun StorEdge T3 array 4 check status of the Sun StorEdge network FC switch 8 and switch 16 switch 5 check status of the virtualization engine 5 determine extent of the problem 4 discovering the error 4 Ethernet hubs 35 event grid tool 95 general procedures 3 host side 6 quiesce IO 5 Storage Service Processor side 6 Sun StorEdge FC switch 8 and switch 16 switches 73 Sun StorEdge T3 array 87 test and isolate FRUs 5 tools and resources available 3 virtualization engine 107 V verifying A2 or B2 FC links 52 A4 or B4 FC link 62 cfgadm al output 4 checkdefaultconfig 4 configuration settings 7 data host 45 failover luxadm display 63 host side 51 luxadm output 4 operation of user selected components 57 storage service processor 92 storage service processor side 57 Veritas DMP installations 5 used in troubleshooting 20 Veritas DMP error message for A3 or B3 link 57 viewing virtualization engine map 118 virtualization engine backpanel 112 checking status 5 clearing log files 108 description of 107 diagnostic codes 108 diagnostics 108 displaying log files 109 error messages 164 Ethernet port LEDs 112 event grid 132 failback
108. lcomes Your Comments Sun is interested in improving its documentation and welcomes your comments and suggestions You can email your comments to Sun at docfeedback sun com Please include the part number 816 5255 of your document in the subject line of your email XX Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 1 Introduction The Sun StorEdge 3900 and 6900 series storage subsystems are complete preconfigured storage solutions The configurations for each of the storage subsystems are shown in TABLE 1 1 TABLE 1 1 Sun StorEdge 3900 and 6900 Series Configurations Additional Array Partner Groups Sun StorEdge Supported Sun StorEdge T3 Array with Optional Fibre Channel Partner Additional Switches Groups Expansion Virtualization Series System Supported Supported Cabinet Engine Sun StorEdge Sun StorEdge Two 8 port One to four N A N A 3900 series 3910 system switches 3900SL Sun StorEdge Two 16 port One to four One to five 3960 system switches Sun StorEdge Sun StorEdge Four 8 port One to three One to four One virtualization 6900 series 6910 system switches engine pair 6910SL3 Sun StorEdge Four 16 port One to three One to four Two virtualization 6960SL 6960 system switches engine pairs 1 1 Gbit or 2 Gbit switches 2 3900SL No switches 3 6910SL and 6960SL No front end switches two back end switches Sun Proprietary
109. led to 1 Check the Sun StorEdge T3 restore configuration with the showt 3 e The force option tried n t3_name command unsuccessfully to reinitialize 2 Refer to the Sun StorEdge T3 and T3 documentation restoret3config e SLUN configuration is not found in 1 Check for snapshot files in the the Srestore_file opt SUNWsecfg etc t3_name e Cannot restore LUN directory 2 If the snapshot files are not found use the modifyt3config command to configure the Sun StorEdge T3 array rmt 3group An error occurred while removing Refer to the Sun StorEdge T3 and T3 Group array documentation rmt3slice An error failed to remove slice Refer to the Sun StorEdge T3 and T3 slicename array documentation rmt3slice An error failed to remove slice from 1 Check the volume status using the volume volume checkt 3mount or showt3 command 2 If unmounted use the restoret3config command to mount savet3config While checking the configuration the 1 If the configuration is different Sun StorEdge T3 array from standard Sun StorEdge T3 configuration was not saved array configuration run the showt 3 n t3_name command to check the Sun StorEdge T3 array configuration 2 Use the modifyt3config command to reconfigure the device sett3lunperm LUN 1un does not exist on the Sun 1 Create a Sun StorEdge T3 array StorEdge T3 array slice 2 Before setting permissions use the createt3slice command 174 Sun StorEdge 3900 and 6900 Series 2 0
110. list of the supported switches visit the http www sun com web site Direct attachment to the StorEdge 3900 and 6900 Series arrays with 1 Gbit or 2 Gbit HBAs require no changes Before making any changes to the Sun StorEdge 3900 or 6900 series you must have a Sun StorEdge SAN 4 1 infrastructure already in place and functional This includes at a minimum m A Solaris host on the SAN management network loaded with SANbox2 Manager m Sun StorEdge 2 Gbit 16 port switch network configured in desired topology ring star mesh or cascade with healthy ISL links Diagnosing and Troubleshooting Switch Hardware Problems Note Whereas 1 Gbit switch port numbers are numbered starting with 1 one 2 Gbit switch port numbers are numbered starting with 0 zero To compare the current configuration to the default configuration type checkswitch s switch v To compare the current switch configuration to the most recently saved map file type checkswitch s switch p v To display the current switch configuration type showswitch s switch Chapter 7 Troubleshooting Switches 75 Sun Proprietary Confidential Internal Use Only 4 To restore the configuration from the saved map file back to the default switch configuration type restoreswitch s switch For detailed diagnostic and troubleshooting procedures for the Sun StorEdge network FC switch 8 and switch 16 switch hardware refer to the Sun StorEdg
111. lization Update the firmware engines has failed 70007 The primary drive cannot write to the drive If a spare drive is available use it to being built replace the failed drive If no spare is available replace the failed drive with a new drive 70008 If the initiator is master then its slave has If a spare drive is available use it to detected a read error on a member within a replace the failed drive If no spare is mirror drive available replace the failed drive with a new drive 70009 If the initiator is master then it has detected a If a spare drive is available use it to read error on a member within a mirror drive replace the failed drive If no spare is available replace the failed drive with a new drive 70010 The CleanUp configuration table is completed No action is needed 70020 The SAN physical configuration has changed If the change was unintentional check the condition of the drives 156 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE A 1 SRN Reference SRN Description Corrective Action 70021 The drive is offline If the change was unintentional check the condition of the drives 70022 The virtualization engine is offline If the change was unintentional check the condition of the drives 70023 The drive is unresponsive Check the condition of drives 70024 For the Sun StorEdge T3 array pack
112. loopcard Value Description Reenable the loopcard 0 Drive mounted if possible enable u 2 Drive present encid 112 3 Drive is spun up Replace the loopcard 4 Drive is disabled if necessary 5 Drive has been Reenable the disk if replaced possible 7 Invalid system area on Replace the disk if drive necessary 9 Drive not present D Drive disabled drive is being reconstructed S Drive substituted power battery Alarm Red Y The state of the batteries Open a Telnet session in the Sun StorEdge T3 to the affected Sun array is not optimal StorEdge T3 array Run refresh s to Possible causes are verify the battery e The voltage level on state the power supply and Replace the battery if the battery have moved necessary out of acceptable thresholds e The internal power cooling unit PCU temperature has exceeded acceptable thresholds e A PCU fan has failed Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 97 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array z g 5 5 3 z a a c c 5 g 3 B lt lt power fan Alarm Red Y The state of a fan on the 1 Open a Telnet session Sun StorEdge T3 array to the affected Sun is not optimal StorEdge T3 array 2 Verify the fan state with fru stat 3 Replace the power cooling unit if necessary power output Alarm Red Y The st
113. lternate Path active enabled OK FIGURE 10 6 Multipath Configurator LUN Properties Detail Note From this example note the Primary Path is Unknown and the Alternate Path is currently in use v To Use the Sun StorEdge T3 Array Failover Driver Command Line Interface CLI Use the jafo_nutil exe interface which is available with Sun StorEdge T3 Array Failover Driver version 2 1 and later to gather information about a The WWN of monitored Sun StorEdge T3 array partner groups a The WWN of individual LUNs m Device paths a LUN to drive letter mapping m The status for primary paths secondary paths standby paths active paths In addition you can use the jafo_nutil exe interface to perform failback operations in recovery scenarios Although the Sun StorEdge T3 Array Failover Driver GUI is limited to the Sun StorEdge 3900 series systems you can use the CLI for both the Sun StorEdge 3900 series systems and the Sun StorEdge 6900 series systems 142 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FIGURE 10 7 displays example ouput for a Sun StorEdge 3900 series system from the jafo_nutil exe interface E Program Files Sun Microsystems T3 Storedge Multiplatform Driver gt jafo_nutil exe HBA WWN 00000000000000000000000000000000 NAME Device ScsiPort5 DESC QLogic QLA2200 PCI Fibre Channel Adapter DRIVER ql2200 FW_REV Can t obtain f
114. lure Replace the controller as indicated by the NVRAM failure code Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 103 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Component Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array Event Type Severity Action Description Action disk State Change Red lt The Sun StorEdge T3 array has reported that a disk has failed 1 Open a Telnet session to the affected Sun StorEdge T3 array 2 Verify the disk state with vol_stat fru_stat and fru_list Drive Status Messages 0 Drive mounted 2 Drive present 3 Drive is spun up 4 Drive is disabled 5 Drive has been replaced 7 Invalid system area on drive 9 Drive not present D Drive disabled is being reconstructed S Drive substituted 3 Replace the disk if necessary interface loopcard State Change Red The Sun StorEdge T3 array has indicated that the loopcard is no longer in an optimal state 104 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 1 Open a Telnet session to the affected Sun StorEdge T3 array 2 Verify loopcard state with fru stat 3 Verify matching firmware with other loopcard 4 Reenable the loopcard if possible with enable u encid 1121 5 Replace the loopcard if necessary
115. m Cause Likely Causes are GBIC FC Cable and device optical connections Action To isolate further please run the Storage Automated Diagnostic Environment tests associated with this link segment FIGURE 5 7 A2 or B2 FC Link Storage Service Processor Side Event Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Verifying the Data Host An error in the A2 or B2 FC link can result in a device being listed as in an unusable state in cfgadm but no HBAs being listed in the unconnected state in the luxadm output The multipathing software will note an offline path as shown in CODE EXAMPLE 5 4 CODE EXAMPLE 5 4 cfgadm al usr sbin cfgadm al Ap_Id Type Receptacle Occupant Condition c0 scsi bus connected configured unknown usr sbin luxadm e port Found path to 2 HBA ports devices pci 6 4000 SUNW qlc 2 fp 0 0 devetl CONNECTED devices pci 6 4000 SUNW qlc 3 fp 0 0 devctl CONNECTED usr sbin luxadm display dev rdsk c6t29000060220041F96257354230303052d0s2 DEVICE PROPERTIES for disk dev rdsk c6t29000060220041F96257354230303052d0s2 Status Port A ORs Status Port B O K Vendor SUN Product ID SESSOL WWN Node 2a000060220041f 9 WWN Port A 2b000060220041f 9 WWN Port B 2b000060220041f4 Revision 080C Serial Num Unsupported Unformatted capacity 102400 000 MBytes Write Cache Enabled Read Cache Enabled Minimu
116. m Errors on page 26 Command Line Test Examples on page 27 Monitoring Sun StorEdge T3 and T3 Arrays Using the Explorer Data Collection Utility on page 29 a Monitoring Host Bus Adapters HBAs Using QLogic SANblade Manager on page 32 Storage Automated Diagnostic Environment 2 2 Check the internal status of the Sun StorEdge 3900 or 6900 series systems using the Storage Automated Diagnostic Environment utility version 2 2 The Storage Automated Diagnostic Environment is installed on every Storage Service Processor that ships with the unit All that is needed is web browser access to the Storage Service Processor In non Sun host configurations such as Microsoft Windows 2000 the Storage Automated Diagnostic Environment will be able to monitor the internals of the storage unit switches virtualization engines and the Sun StorEdge T3 arrays but will not be able to completely monitor the host to storage unit link the HBA to switch Certain conditions will be noted by Storage Automated Diagnostic Environment however such as a port going offline or increasing Fibre Channel errors on the port 23 Sun Proprietary Confidential Internal Use Only Example Topology In the Storage Automated Diagnostic Environment topology shown in FIGURE 3 1 the internel components of a Sun StorEdge 3910 system are shown There is also a Solaris host diag221 and the Storage Service Processor diag156 in the view What is missi
117. m prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c6t29000060220041F96257354230303052d0s2 devices scsi_vhci ssd g29000060220041 96257354230303052 c raw Controller devices pci 6 4000 SUNW qlc 3 fp 0 0 Device Address 2b000060220041f9 0 Class primary State ONLINE Controller devices pci 6 4000 SUNW qlc 2 fp 0 0 Device Address 2b000060220041f4 0 Class primary State OFFLINE Note You can find procedures for restoring virtualization engine settings in the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide Chapter 5 Troubleshooting the Fibre Channel FC Links 51 Sun Proprietary Confidential Internal Use Only Verifying the A2 or B2 FC Link You can check the A2 or B2 FC link using the Storage Automated Diagnostic Environment Diagnose Test from Topology functionality The Storage Automated Diagnostic Environment s implementation of diagnostic tests verifies the operation of user selected components Using the Topology view you can select specific tests subtests and test options FRU Tests Available for the A2 or B2 FC Link Segment m The linktest is not available Both the switch and the GBIC are tested using the switchtest test The switchtest test a Can be used only in conjunction with the loopback connector a Cannot be cabled to the virtualization engine while switchtest runs a No virtualization engine tests are available v To Isolate the A2 o
118. mmand is running Runps ef grep savevemap or listavailable v which returns the status of individual virtualization engines to confirm that the configuration locks are set Common to virtualization engine Common to virtualization engine No virtualization engine pairs were found or the virtualization engine pairs are offline This is usually due to the savevemap command running The virtualization engine was unable to obtain a lock on vepair Another virtualization engine command is updating the configuration Runps ef grep savevemap or listavailable v which returns the status of individual virtualization engines to confirm that the configuration locks are set 1 Run listavailable v which returns the status of individual virtualization engines 2 Check for the lock file directly by using 1s la opt SUNWsecfg etc look for vl1 lock or v2 1lock 3 If the lock is set in error use the removelocks v command to clear Common to virtualization engine Common to virtualization engine The virtualization engine was unable to start sLicd on vepair so it cannot execute the command The login failed A password is required to log in to the virtualization engine The utility uses the VEPASSWD environment variable to login The environment variable VEPASSWD might be set to an incorrect value 1 Run startslicd and then showlogs e 50 to determine why startslicd could not
119. mmunication regained Established oob OutOfBand ib Comm _Lost Down Y Since InBand ib Verify luxadm with monitoring is established the command line using luxadm the luxadm probe monitoring may not be luxadm display activated for a particular Verify cables GBICs Sun StorEdge T3 array and connections along the data path Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify the failing segment of the data path Verify the correct FC switch configuration if applicable Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 99 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array 5 2 S A a z E 5 S 3 E rf B lt a 2 oob Comm _Lost Down Y OutOfBand oob means 1 Check the Ethernet that the Sun StorEdge connectivity to the T3 array failed to affected Sun StorEdge answer to a ping or failed T3 array to return its tokens 2 Verify that the Sun StorEdge T3 array is This OutOfBand problem booted correctly can be caused by a very 3 Verify the correct slow network or because TCP IP settings on the Ethernet connection the Sun StorEdge T3 to this Sun StorEdge T3 array array was lost 4 Increase the http timeout 5 Ping timeout in Utilities gt System gt System gt Timeouts The current default timeouts are 10 seconds for ping and 60 seconds for http tokens
120. n the 1 Check the Sun StorEdge T3 virtualization engine has array for current LUN detected a change in status ownership for a multipath drive or a 2 Use the SUNWsecfg utility on VLUN This usually the Storage Service Processor indicates a pathing problem to fail LUNs back to the to a Sun StorEdge T3 array correct controller if needed controller such as changes in active and passive paths volume_add Alarm Yellow A new VLUN was added to None the configuration volume_ Alarm Yellow A VLUN was deleted from None delete the configuration enclosure Alarm log Yellow Port statistics on None virtualization engine via changed enclosure Audit Automatic weekly audits None send a detailed description of the enclosure to the Sun Network Storage Command Center NSCC oob Comm_ Communication regained OutofBand Established with virtualization engine vla oob ping Comm_ Down Ethernet connectivity to the 1 Check power to the Lost virtualization engine has virtualization engine been lost 2 Check Ethernet connectivity to the virtualization engine 3 Check the status of the slicd daemon 4 Make sure the virtualization engine is booted correctly 5 Verify the correct TCP IP settings on the virtualization engine 6 Replace the virtualization engine if necessary 7 Run ipcs 1 and ipcrm 1 to clean up old semaphore Chapter 9 Troubleshooting Virtualization Engine Devices 133 Sun Proprietary Conf
121. nd 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only To Display Sun StorEdge Traffic Manager MPxIO Enabled Devices If the devices support the Sun StorEdge Traffic Manager software you can use this shortcut Type usr sbin luxadm display dev rdsk c6t29000060220041956257334B30303148d0s2 DEVICE PROPERTIES for disk dev rdsk c6t29000060220041956257334B30303148d0s2 Status Port A O K Status Port B O K Vendor SUN Product ID SESSO1 WWN Node 2a00006022004195 WWN Port A 2b00006022004195 WWN Port B 2b00006022004186 Revision 080E Serial Num Unsupported Unformatted capacity 56320 000 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c6t29000060220041956257334B30303148d0s2 devices scsi_vhci ssd g29000060220041956257334b30303148 c raw Controller devices pci lf 4000 SUNW qlc 4 fpe0 0 Device Address 2b600006022004195 0 Class primary State ONLINE Controller devices pci lf 4000 pci 2 SUNW qlc 5 fpe0 0 Device Address 2b00006022004186 0 Class primary State ONLINE The dev rdsk cntn represents the Global Unique Identifier of the device It is 32 bits long m The first 16 bits correspond to the WWN of the master virtualization engine router m The remaining 16 bits are the VLUN serial number a Virtualization engine WWN 2900006022004195 a VLUN ser
122. ndows 2000 138 Launching the Sun StorEdge T3 Array Failover Driver GUI 138 Checking the Version of the Sun StorEdge T3 Array Failover Driver 139 Contents VII Sun Proprietary Confidential Internal Use Only Vill v To Use the Sun StorEdge T3 Array Failover Driver GUI v To Use the Sun StorEdge T3 Array Failover Driver Command Line Interface CLI 142 11 Example of Fault Isolation 147 A Virtualization Engine References 155 SRN Reference 155 SRN SNMP Single Point of Failure Descriptions 159 Port Communication Numbers 160 Virtualization Engine Service Codes 160 B Configuration Utility Error Messages 163 Virtualization Engine Error Messages 164 Switch Error Messages 168 Sun StorEdge T3 Array Partner Group Error Messages 171 Other Error Messages 175 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only GURE 2 1 GURE 2 2 GURE 2 3 GURE 2 4 GURE 2 5 GURE 3 1 GURE 3 2 GURE 3 3 GURE 3 4 GURE 5 1 GURE 5 2 GURE 5 3 GURE 5 4 GURE 5 5 GURE 5 6 GURE 5 7 GURE 5 8 GURE 5 9 GURE 5 10 List of Figures Sun StorEdge 6900 Series Logical View 11 Primary Data Paths to the Alternate Master 12 Primary Data Paths to the Master Sun StorEdge T3 Array 13 Path Failure Before the Second Tier of Switches 14 Path Failure l O Routed Through Both HBAs 15 Storage Automated Diagnostic Environment Example Topology 24
123. ne failed to check for No action is needed SAN changes 72006 The virtualization engine failed to read the SAN No action is needed event log 72007 The SLIC daemon connection is down Wait 1 to 5 minutes for the backup daemon to come up If it doesn t check the network connection for virtualization engine halt or hardware failure 158 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only SRN SNMP Single Point of Failure Descriptions TABLE A 2 provides Simple Network Management Protocol SNMP descriptions associated Service Request Numbers SRNs and recommendations for corrective action TABLE A 2 SRN SNMP Single Point of Failure Table SRN SNMP Description 70020 e The SAN topology has changed 70021 e The Global SAN configuration has 70030 changed 70050 e The SAN configuration has changed e A physical device is missing 70025 The IP of the partner s virtualization engine is not reachable 72000 e The SAN topology has changed 72007 e The Global SAN configuration has 70020 changed 70021 e The SAN configuration has 70022 changed 70025 The IP of the partner virtualization engine is not reachable e A physical device is missing e A SLIC virtualization engine is missing e A SLIC daemon connection is inactive e The virtualization engine failed to check for SAN changes or a daemon error occurred e A sec
124. ng is the Microsoft Windows 2000 host which is also connected amp Sun Storage Automated Diagnostic Environment microsystems f admin A Diagnose Mana Report Monitor Devices Monitor Topology Monitor Log Hosts Host crash3 Filter None al Home Help Logout ROOT 2 1 B3 004 Help Clear Search Topology Help M bradster TOPOs MASTER Brocade b7 104 25 Switch1 8 coy Bar DE switch b 104 26 100000cOdd006fb3 06 23 09 04 12 Regained Communication OutOfBan J Save XY Graphics On Layout Horizontal Links Onf After having selected a topology several functions are available in the graphical view 1 Clicking in the middle of an object or ina specific component of an object will display the status at the bottom of the page 2 Right clicking the background will present a zoom menu 3 Right clicking on an object will present a menu of functions available for that object 4 Right clicking on a link will present a menu of functions available for that link 5 Holding the Shift button while selecting objects allows to select multiple objects This can be used to move multiple objects at the same time 6 After resizing the windows the GO button must be selected to refresh the topology view FIGURE 3 1 Storage Automated Diagnostic Environment Example Topology 24 Sun StorEdge 3900 and 6900 Series 2 0 Troublesh
125. ocation of a Sun StorEdge T3 array has been changed Quiesce has ended on a Sun StorEdge T3 array Sun Proprietary Confidential Internal Use Only Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 101 TABLE 8 1 Component Event Type Severity Action Description Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array Action enclosure controller disk interface loopcard QuiesceStart Topology Topology Topology Red Red Red Quiesce has started on a Sun StorEdge T3 array The Sun StorEdge T3 array has reported that a controller was removed from the chassis The Sun StorEdge T3 array has reported a disk has been removed from the chassis The Sun StorEdge T3 array has reported that a loopcard has been removed from the chassis Replace the controller within the 30 minute power shutdown timeframe Replace the disk within the 30 minute power shutdown timeframe Replace the loopcard within the 30 minute power shutdown timeframe power Topology Red The Sun StorEdge T3 array has reported that a power cooling unit PCU has been removed from the chassis Replace the PCU within the 30 minute shutdown timeframe controller State Change The status of the controller has changed from disabled to ready enabled disk State Change The status of the disk has changed fr
126. offer diagnostics a best guess effort will have to suffice Storage Automated Diagnostic Environment cannot test HBAs on Microsoft Windows 2000 hosts at this time Chapter 11 Example of Fault Isolation 149 Sun Proprietary Confidential Internal Use Only sisanblade Manager E lol xj File Host View Help O agoe pogie Connect Configure Events Alarms Refresh Simplity HBA Diagnostics E R ERTO TCE amp Host smi 2gs5m2bdhom Host smi 2gs5m2bdhom Node Name 20 00 00 E0 8B 02 65 17 y PEA 1 Adapter 0 2200 PortName 21 00 00 E0 8B 02 65 17 GA Device 50 02 0F my Adapter 2200 Port ID 00 00 EF Test Configuration Data Pattern 55 01010101 ad Number of test s 1 10 000 N A Customized XX XX XX XX XX XX XX XX Test Increment 1 10 000 125 somizad Ge xn re o E i ha y Test continuously v Stop on error Loopback Test Results Test Status CRC Error Disparity Error Frame length error J Success ae Se U 0 i EfLoopback test smi 2gssmandndni IE 55 55 55 55 55 55 55 55 Stop ri Loopback Test Readavrite Buffer Test Adapter 2200 150 FIGURE 11 4 Diagnostics Using QLogic SunBlade In this example the HBA to switch cable was removed temporarily and a loopback connector was inserted into the HBA The Qloge SANblade LoopBack diagnostics were then run The HBA passed the tests Note The next components tha
127. om fault disabled to ready enabled interface loopcard volume State Change State Change The Sun StorEdge T3 array has reported that a loopcard has been replaced or brought back online The status of the LUN in a Sun StorEdge T3 array has changed from unmounted to mounted and is now available 102 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 8 1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3 Array o lt a z 3 5 5 5 5 S 3 3 3 8 rf Co lt A lt power State The status of the PCU Change has changed from ready disable to ready enable controller State Change disk State Change interface State loopcard Change volume State Change power State The Sun StorEdge T3 Change array has reported that a LUN has changed state controller State Red Y The Sun StorEdge T3 Open a Telnet session Change array has reported that a to the affected Sun power cooling unit has StorEdge T3 array been disabled Verify the controller state with fru_stat and sys_stat Re enable the controller if possible enable u Run logger dmprstlog from a serial port session on the affected controller The output from logger will only go the the syslog facility Review the syslog on the master controller to determine the cause of fai
128. on engine to the backend switch The A3 or B3 FC link exists in a Sun StorEdge 6900 Series only An error with the FC link can cause a path to go offline FIGURE 5 8 FIGURE 5 9 and FIGURE 5 10 are examples of A3 or B3 link notification events Site FSDE LAB Broomfield CO Source diag xxxxx xxx com Severity Normal Category Message Key message diag xxxxx xxx com EventType LogEvent driver MPXIO_offline EventTime 01 08 2002 18 25 18 Found 2 driver MPXIO_offline warning s in logfile var adm messages on diag xxxxx xxx com id 80fee746 Jan 8 18 24 24 WWN 2b000060220041f9 diag xxxxx xxx com mpxio ID 779286 kern info scsi_vhci ssd g29000060220041 96257354230303053 ssd19 multipath status degraded path pci 6 4000 SUNW qlc 3 fp 0 0 fpl to target address 2b000060220041f9 1 is offline Jan 8 18 24 24 WWN 2b000060220041f9 diag xxxxx xxx com mpxio ID 779286 kern info scsi_vhci ssd g29000060220041 96257354230303052 ssd18 multipath status degraded path pci 6 4000 SUNW qlc 3 fp 0 0 fpl to target address 2b000060220041f9 0 is offline Site FSDE LAB Broomfield CO Source diag xxxxx xxx com Severity Normal Category Message Key message diag xxxxx xxx com EventType LogEvent driver Fabric_Warning EventTime 01 08 2002 18 25 18 Found 1 driver Fabric_Warning warning s in logfile var adm messages on diag xxxxx xxx com id 80fee746 Info Fabric warning Jan 8 18 24 04 WWN 2b00006
129. ondary daemon connection is active 70030 70050 Sun StorEdge T3 array LUN failover Sun StorEdge T3 array LUN failback Corrective Action e Check the SAN cabling and connections between the Sun StorEdge T3 array andthe virtualization engine e Perform Sun StorEdge T3 array failback if necessary Check the Ethernet cabling and connections e Check cabling and connections between the virtualization engines e Cycle power on failed virtualization engine if the fault LED flashes e Perform Sun StorEdge T3 array failback if necessary e Enable VERITAS path e Check the SLIC virtualization engine SRN after Corrective Action 70020 70030 70051 None 70020 70021 70022 70024 70030 70050 Appendix A Virtualization Engine References 159 Sun Proprietary Confidential Internal Use Only Port Communication Numbers TABLE A 3 Port CommunicationNumbers Port Port Port Number Daemon Management programs 20000 Daemon Daemon 20001 Daemon Virtualization engine 25000 Virtualization engine Virtualization engine 25001 Virtualization Engine Service Codes TABLE A 4 lists the service code numbers for errors that occur on the virtualization engine along with recommendations for corrective action TABLE A 4 Virtualization Engine Service Codes 0 399 Host Side Interface Driver Errors Service Code Number Cause of Error Recommended Correc
130. ooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Generating Component Specific Event Grids The Storage Automated Diagnostic Environment generates component specific event grids that describe the severity of an event tell whether action is required provide a description of the event and recommended action Refer to Chapters 5 through 9 of this troubleshooting guide for component specific event grids v To Customize an Event Report 1 Choose the Event Grid link on the the Storage Automated Diagnostic Environment Help menu 2 Select the criteria from the Storage Automated Diagnostic Environment event grid like the one shown in in TABLE 3 1 TABLE 3 1 Event Grid Sorting Criteria Category Component e All default e All e Sun StorEdge default A3500FC array e Backplane e Sun StorEdge A5000 e Controller array e Disk e Agent e Interface e Host e LUN e Message e Port e Sun Switch e Power e Sun StorEdge T3 array e Tape e Virtualization engine Event Type Agent Deinstall Agent Install Alarm FC Alternate Master Audit Communication Established Communication Lost Discovery Heartbeat Insert Component Location Change Patch Info Quiesce End Quiesce Start Removal Remove Component State Change from offline to online State Change from online to offline Statistics Backup Severity Action Yes This 1 event is FFE actionable critical error andis
131. or vl has changed 18c18 lt t3b01 5 T49153 116 7 0 7 50020F230000725B 1 gt t3b01 5 T49153 116 7 0 7 50020F2300006DFA 1 28c28 lt t3b01 T49153 50020F230000725B 60020F2000006DFA gt t3b01 T49153 50020F2300006DFA 60020F2000006DFA 31037 lt 100002 2900006022004186 vib Yes 08 14 0 gt 100002 2900006022004186 Unknown No Unknown 0 46d45 lt Undefined 210000EO8B026COF 100002 Yes 0 checkvemap virtualization engine map vl verification complete FAIL FIGURE 8 3 Manage Configuration Files Menu 92 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only FRU Tests Available for the T1 or T2 Data Path FRU Running the tests from the Storage Automated Diagnostic Environment GUI guides you in discovering the failed FRU Refer to Chapter 5 of the Storage Automated Diagnostic Environment User s Guide for instructions on how to run tests m Run the switchtest to test the switches m Run the linktest to test the T1 or T2 connections After a test has completed its run an email message similar to the message in FIGURE 8 4 is sent to the specified email recipient running on diag xxxxx xxx com linktest started on FC interconnect switch to switch switchtest started on switch 100000c0Odd00b682 port 8 Estimated test time 14 minute s 01 30 02 11 21 26 diag209 Storage Automated Diagnostic Environment MSGID 6013 switchtest FATAL switch0O Device Switch Po
132. orEdge T3 array Failover Driver software before connecting the host to the switches 137 Sun Proprietary Confidential Internal Use Only Troubleshooting Tasks Using Microsoft Windows 2000 Launching the Sun StorEdge T3 Array Failover Driver GUI From the Microsoft Windows 2000 Advanced Server GUI click Programs gt T3 StorEdge Configurator gt Configurator FIGURE 10 1 Launching the Sun StorEdge T3 Array Failover Driver 138 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Checking the Version of the Sun StorEdge T3 Array Failover Driver From the Microsoft Windows 2000 Advanced Server GUI click Help gt About The About Multipath Configurator window is displayed About Multipath Configurator X About Multipath Configurator amp Sun microsystems amp Sun microsystems MULTIPATH CONFIGURATOR MULTIPATH CONFIGURATOR lt a p_a Suy Sun FIGURE 10 2 Sun StorEdge T3 Array Failover Driver Versions 2 0 0 123 and 2 1 0 104 Note In FIGURE 10 2 the example on the left shows build number 2 0 130 comprised of driver version 2 0 0 123 and application version 2 0 0 125 The same build number might have a different driver version and application version The example on the right shows build number 2 0 130 comprised of driver version 2 1 0 104 and application version 2 1 0 104 Be aware of these possible version differen
133. orage Automated Diagnostic Environment Event Grid for 2 GBit Switches EventType Severity is exactly Description Note Text within as it appears on quotation marks the Event Grid Action Required chassis fan chassis board chassis power system_ reboot enclosure Alarm Alarm Alarm Alarm Audit Yellow Yellow Yellow Yellow lt Action lt chassis fan 1 status changed from OK The uptime of the switch was less than the previous uptime of the switch This could indicate that the switch has been reset either by a user or by the loss of power chassis power 1 status changed from OK This event monitors changes in the status of the chassis power supply as reported by the SANbox chassis status Switch swla was rezoned This event reports changes in the zoning of a switch Auditing a new switch called ras d2 swb1 ip xxx 0 0 41 10002000007a609 None 1 Check to see if the switch has been reset 2 Check the power going to the switch None oob Comm_ Established Communication regained with swla Lp xxx 20 67 213 82 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 7 2 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches Continued gt 5 5 for 5 v 5
134. place the Master Host on page 71 a To Replace the Alternate Master or Slave Monitoring Host on page 72 Using the Host Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort host events by component category or event type The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severity of the event tells whether action is required provides a description of the event and gives the recommended action Refer to the Storage Automated Diagnostic Environment User s Guide for more information To Access the Host Event Grid From the Storage Automated Diagnostic Environment Help menu choose the Event Grid link FIGURE 6 1 shows the Host Event Grid from which you can select related criteria for the event you are troubleshooting 67 Sun Proprietary Confidential Internal Use Only amp SUN Storage Automated Diagnostic Environment i rt Utiliti 2 0 06 010 diag176 central sun com Help WY Vun Help Event Grid Help Help Page Select a Category Component EventType and type GO to limit the report Click on the Columns headers to change the Event Grid sort Check ReportFormat to displ Click Info Action to Review Event Grid pdf Category EventType ReportFormat Architecture Diagnostics Diag Strategy Utilities Release Notes All host a Info status of hba devices sbus 9 0
135. qlctest Diagnostic Red qlctest diag240 on the Check Test Manager for Test host failed failure details socal Diagnostic Red socaltest diag240 on Check Test Manager for test Test the host failed failure details enclosure PatchInfo New patch and package Send changes to the output information were of generated showrev p and pkginfo enclosure backup The Agent was backed up Backs up the configuration file of the Agent disk_ Alarm Yellow Y Detected that Remove unused files and capacity var opt SUNWstade is directories to free up space at or above 98 capacity Use a larger disk for by typing var opt SUNWstade usr sbin df k var opt SUNWstade disk_ Alarm Detected that No action is required capacity_ var opt SUNWstade is okay now below 98 capacity by typing usr sbin df k var opt SUNWstade 70 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Replacing the Master Alternate Master and Slave Monitoring Host The following procedures are a high level overview of the procedures that are detailed in the Storage Automated Diagnostic Environment User s Guide Follow these procedures when replacing a master alternate master or slave monitoring host Note The procedures for replacing the master host are different from the procedures for replacing an alternate master or slave monitoring host To Replace
136. r B2 FC Link To isolate the A2 or B2 link which is the FC link from the first switch to the virtualization engine only in the Sun StorEdge 6900 Series follow these steps Note The A2 or B2 FC link exists in a Sun StorEdge 6900 series only 1 Quiesce the I O on the A2 or B2 FC link path 2 Break the connection by uncabling the link 3 Insert the loopback connector in to the switch port 4 Run switchtest a If the test fails replace the GBIC and rerun switchtest b If the test fails again replace the switch 52 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 5 If the switch and the GBIC show no errors replace the remaining components in the following order a Replace the virtualization engine side GBIC recable the link and monitor the link for errors b Replace the cable recable the link and monitor the link for errors c Replace the virtualization engine restore the virtualization engine settings recable the link and monitor the link for errors Note The procedures for restoring virtualization engine settings are in the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide 6 Return the path to production Chapter 5 Troubleshooting the Fibre Channel FC Links 53 Sun Proprietary Confidential Internal Use Only Troubleshooting the A3 or B3 FC Link The A3 or B3 link is the FC link from the virtualizati
137. r messages and recommendations for corrective action Using UNIX Commands This document may not contain information on basic UNIX commands and procedures such as shutting down the system booting the system and configuring devices See one or more of the following documents for this information m Solaris Handbook for Sun Peripherals a AnswerBook2 online documentation for the Solaris operating environment a Other software documentation that you received with your system XVI Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Typographic Conventions Typeface AaBbCc123 AaBbCc123 AaBbCc123 Meaning The names of commands files and directories on screen computer output What you type when contrasted with on screen computer output Book titles new words or terms words to be emphasized Command line variable replace with a real name or value Examples Edit your login file Use 1s a to list all files o You have mail S su Password Read Chapter 6 in the User s Guide These are called class options You must be superuser to do this To delete a file type rm filename Shell Prompts Shell C shell C shell superuser Bourne shell and Korn shell Bourne shell and Korn shell superuser Prompt machine name machine name Preface XVII Sun Proprietary Confidential Internal U
138. r switch type found User must have upgraded or changed out the switch with a different type and did not use the SUNWsecfg commands to reconfigure Appendix B 1 cp opt SUNWsecfg etc switch map opt SUNWsecfg etc switch save 2 Run saveswitch s switch 3 Manually edit the configurable items in the opt SUNWsecfg etc switch map file to valid values that equal the values in switch save file 4 Rerun restoreswitch s switch Configuration Utility Error Messages 169 Sun Proprietary Confidential Internal Use Only TABLE B 2 Sun StorEdge Network FC Switch Error Messages Continued Source of Error Message Cause of Error Message Suggested Corrective Action setswitchflash Invalid flash file flashfile You might be attempting to download Check the number of ports on switch a flash file for an 8 port switch to a 16 Sswitch port switch Check showswitch s switch and look for number of ports Ensure that this matches the second and third characters of the flash file name setswitchflash switch timed out after reset 1 Wait several minutes The switch took longer than two 2 Run ping switch minutes to reset after a configuration 3 If errors persist manually power change cycle the switch The switch might not be set for rarp or rarp is not working correctly setupswitch Switch switch timed out after 1 Wait several minutes reset 2 Run ping switch 3
139. ration the virtualization engine pairs handle the failover In addition the multipathing software notes a path failure on the data host the Sun StorEdge Traffic Manager or DMP software takes the entire path that was connected to the failed switch offline and the Inter Switch Link ISL ports on the surviving switch go offline as well 62 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only To verify that the failover luxadm display can be used the failed path is marked offline as shown in CODE EXAMPLE 5 7 CODE EXAMPLE 5 7 Failed Path Marked Offline usr sbin luxadm display dev rdsk c26t60020F200000644 gt DEVICE PROPERTIES for disk dev rdsk c26t60020F20000064433C3352A60003E82Fd0s2 Status Port A O K Status Port B O K Vendor SUN Product ID T300 WWN Node 50020 2000006443 WWN Port A 50020 2300006355 WWN Port B 50020 2300006443 Revision 0118 Serial Num Unsupported Unformatted capacity 488642 000 MBytes Write Cache Enabled Read Cache Enabled Minimum prefetch 0x0 Maximum prefetch 0x0 Device Type Disk device Path s dev rdsk c26t60020F20000064433C3352A60003E82Fd0s2 devices scsi_vhci ssd g60020 20000064433c3352a60003e82f c raw Controller devices pci a 2000 pcit 2 SUNW qlc 5 fp 0 0 Device Address 50020 2300006355 1 Class primary State OFFLINE Controller devices pcile 2000 pcit2 SUNW glc 5 fp 0 0 Device
140. rch 2003 Sun Proprietary Confidential Internal Use Only 2 Optionally open a Telnet session to the virtualization engine and run the runsecfg utility to poll a live snapshot of the virtualization engine map Refer to To Failback the Virtualization Engine on page 120 for instructions about how to open a Telnet session Determining the virtualization engine pairs on the system MAIN MENU SUN StorEdge 6910 SYSTEM CONFIGURATION TOOL 1 T3 Configuration Utility 2 Switch Configuration Utility 3 Virtualization Engine Configuration Utility 4 View Logs 5 View Errors 6 Exit Select option above gt 3 VIRTUALIZATION ENGINE MAIN MENU 1 anage VLUNs 2 anage Virtualization Engine Zones 3 Manage Configuration Files 4 Manage Virtualization Engine Hosts 5 Help 6 Return Select option above gt 3 MANAGE CONFIGURATION FILES MENU 1 Display Virtualization Engine Map 2 Save Virtualization Engine Map 3 Verify Virtualization Engine Map 4 Help 5 Return Select configuration option above gt 1 Do you want to poll the live system time consuming or view the file l f 1 From the virtualization engine map output you can match the VLUN serial number to the VLUN name VDRV000 the disk pool t3b00 and the multipath MP drive target T49152 This information can also help you find the controller serial number 60020F2000006DFA which you need to perform Sun StorEdge T3 array LU
141. rdmp Disk_1s4 block dev vx dmp Disk_1s3 char dev vx rdmp Disk_1s3 252 min 512 bytes max 2048 blocks slice 4 offset 0 len 209698816 slice 3 offset 1 len 4095 time 1010434311 seqno 0 6 0 248 count 1 len 3004 count 1 len 455 Defined regions priv 000017 000247 000231 copy 01 offset 000000 enabled priv 000249 003021 002773 copy 01 offset 000231 enabled priv 003022 003476 000455 copy 01 offset 000000 enabled Multipathing information numpaths 2 c20t2B000060220041F4d0s2 state enabled c23t2B000060220041F9d0s2 state enabled vxdmpadm listctlr all ENCLR TYPE STATE ENCLR NAME OTHER_DISKS ENABLED OTHER_DISKS SENA ENABLED SENAO SENA ENABLED SENAO Disk ENABLED Disk Disk ENABLED Disk The vxdisk output includes two physical paths to the LUN m c20t2B000060220041F4d0s2 m c23t2B000060220041F9d0s2 Both of these paths are currently enabled with DMP 20 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 2 Use the luxadm 1M command to display further information about the underlying LUN usr sbin luxadm display dev rdsk c20t2B000060220041F4d0s2 DEVICE PROPERTIES for disk dev rdsk c20t2B000060220041F4d0s2 Status Port A O K Vendor SUN Product ID SESSO1 WWN Node 2a000060220041f4 WWN Port A 2b000060220041 f4 Revision 080C Serial Num Unsupported Unformatted capacity 102400 000 MBytes Write Cache Enabled Read C
142. rnatively follow these steps 1 Quiesce the I O on the A4 or B4 FC link path 2 Run switchtest 1M to test the entire link re create the problem 3 Break the connection by uncabling the link 4 Insert the loopback connector in to the switch port 64 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Rerun switchtest a If switchtest fails replace the GBIC and rerun switchtest b If the test fails again replace the switch If switchtest passes assume that the suspect components are the cable and the Sun StorEdge T3 array controller a Replace the cable b Rerun switchtest If the test fails again replace the Sun StorEdge T3 array controller Return the path to production Return the Sun StorEdge T3 array LUNs to the correct controllers if a failover occurred Determine if failovers occur using the luxadm failover or failbackt3path commands Chapter 5 Troubleshooting the Fibre Channel FC Links 65 Sun Proprietary Confidential Internal Use Only 66 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 6 Troubleshooting Host Devices This chapter describes how to troubleshoot components associated with a Sun StorEdge 3900 or 6900 series host This chapter contains the following sections a To Access the Host Event Grid on page 67 m To Re
143. rom OS HBA WWN 00000000000000000000000000000000 NAME Device ScsiPort4 DESC QLogic QLA2200 PCI Fibre Channel Adapter DRIVER ql2200 FW_REV Can t obtain from OS DEVICE VENDOR Sun Microsystems T3 Disk Array FW_REV 0201 SERIAL 00163874 WWN 60020 20000003d50000000000000000 FO_CAPABLE true MASTER true LUN NAME G WWN 60020f20000003d53cf7c0f500028022 GOOD_PATHS 2 STATE up 1 PATH NAME 5 0 0 0 HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE secondary STATE up_standby 3 PATH NAME 4 0 0 0 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE primary STATE up_active 2 CONTROLLER ID 0 DESC Sun T3 Disk Array Controller DEVICE VENDOR Sun Microsystems T3 Disk Array FW_REV 0201 SERIAL 00524894 WWN 60020 20000003d50000000000000000 FO_CAPABLE true MASTER false LUN AME H WWN 60020f20000003d53cf7c4640008025e GOOD_PATHS 2 STATE up 1 PATH NAME 5 0 0 5 HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE primary STATE up_active 2 PATH NAME 4 0 0 5 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE secondary STATE up_standby 3 CONTROLLER ID 0 DESC Sun T3 Disk Array Controller FIGURE 10 7 Sun StorEdge T3 Array Failover Driver CLI Output for the Sun StorEdge 3900 Series Chapter 10 Troubleshooting Using Microsoft Windows 2000 143 Sun Proprietary Confidential Internal Use Only FIGURE 10 8 displays example ouput for a Sun StorEdge 6900 series system from the jafo_nutil exe interface E Program Files Sun Microsystems T3 Storedge Multipla
144. rr 8 E a 2ezLgsS ie chassis Alarm Yellow Switch swla was rezoned zone This event reports changes in the zoning of a switch enclosure Audit Auditing a new switch called ras d2 swb1 ip xxx 0 0 41 10002000007a609 oob Comm_ Communication regained Established with swla Lp xxx 20 67 213 oob Comm_ Down Y Lost communication with 1 Check Ethernet Lost swla connectivity to the ip xxx 20 67 213 switch 2 Verify that the switch is Ethernet connectivity to the booted correctly with no switch has been lost POST errors 3 Verify that the switch Test Mode is set for normal operations 4 Verify the TCP IP settings on switch by way of Forced PROM Mode access 5 Replace switch if needed switch Diagnostic Red Check Test Manager for test Test failure details Chapter 7 Troubleshooting Switches 79 Sun Proprietary Confidential Internal Use Only TABLE7 1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches Continued Component EventType Severity Action Description Note Text within quotation marks is exactly as it appears on the Event Grid Action Required enclosure Discovery Discovered a new switch called ras d2 swb1 ip xxx 0 0 41 10002000007a609 Discovery events occur the very first time the agent probes a storage device It creates a detailed description of the device monitored and sends it using any active notifier such as the Sun
145. rray LUN number removed as shown in CODE EXAMPLE 9 4 For example the LUNs in disk pools t3b00 and t3b01 are named t3b0 on the Sun StorEdge T3 array device CODE EXAMPLE 9 4 Sun StorEdge T3 array and Disk Pool Name opt SUWNsecfg bin failbackt3path n t3b0 120 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only a If no failures occur the command exits with no output b If failures occur you might see one of the following messages CODE EXAMPLE 9 5 Sun StorEdge T3 Array Failure Codes opt SUWNsecfg bin failbackt3path n t3b0 MultiPath failback command failed Returned Result opt SUWNsecfg bin failbackt3path n t3b0 MultiPath failback command failed Returned Result The message return code 513 indicates that the Sun StorEdge T3 array did not require a failback The message return code 586 indicates that the Sun StorEdge T3 array failback could not be completed because the primary path could not be reached If you encounter the return code 586 check the switches sw2a and sw2b and make sure the ports associated with the Sun StorEdge T3 array and virtualization engines are online In this example a t3b0 should be plugged in to port 2 on a 1 Gbit switch of both sw2a and sw2b port 1 on a 2 Gbit switch b The virtualization engine should be plugged in to port 1 on a 1 Gbit switch of the same two switches port 0 on a 2 Gbit swit
146. rrors Invalid transmission The number of times that the virtualization engine s 8 bit and 10 bit word decoder does not detect a valid 10 bit code Invalid cyclic The number of times that the virtualization engine receives frames redundancy code with a defective CRC and a valid EOF A valid EOF includes EOFn CRC count EOFt or EOFdti To Check the FC Link Error Status Manually The Storage Automated Diagnostic Environment which runs on the Storage Service Processor monitors the FC link status of the virtualization engine The virtualization engine must be power cycled to reset the counters Therefore you should manually check the accumulation of errors during a fixed period of time To check the status manually follow these steps Use the svstat command to take a reading as shown in CODE EXAMPLE 9 1 A status report for the host side and device side ports is displayed Within the next few minutes take another reading The number of new errors that occurred within that time frame represents the number of link errors Chapter 9 Troubleshooting Virtualization Engine Devices 113 Sun Proprietary Confidential Internal Use Only Note If the t 30fdg 1M is running while you perform these steps the following error message is displayed Daemon error check the SLIC router CODE EXAMPLE 9 1 FC Link Error Status Example opt svengine sduc svstat d vl I00001 Host Side FC Vital Statistics Link Failure Count 0 Loss of
147. rt 8 is Offline switchtest failed Remove FC Cable from switch 100000c0Odd00b682 port 8 Insert FC loopback cable into switch 100000c0dd00b682 port 8 Continue Isolation switchtest started on switch 100000c0dd00b682 port 8 Estimated test time 14 minute s 01 30 02 11 22 11 diag209 Storage Automated Diagnostic Environment MSGID 6013 switchtest FATAL switchO Device Switch Port 8 is Offline switchtest failed Remove FC loopback cable from switch 100000c0dd00b682 port 8 Insert a NEW FC GBIC into switch 100000c0dd00b682 port 8 Insert FC loopback cable into switch 100000c0dd00b682 port 8 Continue Isolation switchtest started on switch 100000c0dd00b682 port 8 Estimated test time 14 minute s 01 30 02 11 25 12 diag209 Storage Automated Diagnostic Environment MSGID 4001 switchtest WARNING switch0O Maximum transfer size for a FABRIC port is 200 Changing transfer size 2000 to 200 switchtest completed successfully Remove FC loopback cable from switch 100000c0dd00b682 port 8 Restore ORIGINAL FC Cable into switch 100000cOdd00b682 port 8 Suspect ORIGINAL FC GBIC in switch 100000c0dd00b682 port 8 Retest to verify FRU replacement linktest completed on FC interconnect switch to switch FIGURE 8 4 Example Link Test Text Output from the Storage Automated Diagnostic Environment Chapter 8 Troubleshooting the Sun StorEdge T3 Array Devices 93 Sun Proprietary Confidential Internal Use Only a
148. s qlctest 1M and switchtest 1M are provided as examples qlctest 1M The qlctest 1M test comprises several subtests that test the functions of the Sun StorEdge PCI dual Fibre Channel FC host adapter board This board is an HBA that has diagnostic support This diagnostic test is not scalable CODE EXAMPLE 3 1 qlctest 1M opt SUNWstade Diags bin qlctest v o dev devices pci 6 4000 SUNW qlc 3 fp 0 0 devct1 run_connect Yes mbox Disable ilb Disable ilb_10 Disable elb Enable qlctest called with options dev devices pci 6 4000 SUNW qlc 3 fp 0 0 devct1 run_connect Yes mbox Disable ilb Disable ilb_10 Disablel el b Enable qlctest Started Program Version is 4 0 1 Testing qlcO device at devices pci 6 4000 SUNW qlc 3 fp 0 0 devetl QLC Adapter Chip Revision 1 Risc Revision 3 Frame Buffer Revision 1029 Riscrom Revision 4 Driver Revision 5 a 2 1 15 Running ECHO command test with pattern Ox7e7e7e7e Running ECHO command test with pattern Oxlelelele Running ECHO command test with pattern Oxf1f1f1f1 Running ECHO command test with pattern 0x4a4a4a4a Running ECHO command test with pattern 0x78787878 Running ECHO command test with pattern 0x25252525 FCODE revision is ISP2200 FC AL Host Adapter Driver 1 12 01 01 16 Firmware revision is 2 1 7f Running CHECKSUM check Running diag selftest qlctest Stopped successfully Chapter 3 Troubleshooting Tool
149. s 27 Sun Proprietary Confidential Internal Use Only switchtest 1M switchtest 1M diagnoses the Sun StorEdge network FC switch 8 and switch 16 switch devices The switchtest process also provides command line access to switch diagnostics switchtest supports testing on local and remote switches switchtest runs the port diagnostic on connected switch ports While switchtest is running the switch ports monitor the port statistics and check the chassis status CODE EXAMPLE 3 2 switchtest 1M opt SUNWstade Diags bin switchtest v o dev 2 192 168 0 30 0x0 xfersize 200 switchtest called with options dev 2 192 168 0 30 0x0 xfersize 200 switchtest Started Testing port 2 Using ip_addr 192 168 0 30 fcaddr 0x0 to access this port Chassis Status for Device Switch Power OK Temp OK 23 0c Fan 1 OK Fan 2 OK Testing Device Switch Port 2 Pattern 0x7e7e7e7e Testing Device Switch Port 2 Pattern Oxlelelele Testing Device Switch Port 2 Pattern Oxf1f1f1f1 Testing Device Switch Port 2 Pattern 0xb5b5b5b5 Testing Device Switch Port 2 Pattern 0x4a4a4a4a Testing Device Switch Port 2 Pattern 0x78787878 Testing Device Switch Port 2 Pattern 0xe7e7e7e7 Testing Device Switch Port 2 Pattern 0xaa55aa55 Testing Device Switch Port 2 Pattern 0x7f7f7f7f Testing Device Switch Port 2 Pattern O0x0f0f0f0f Testing Device Switch Port 2 Pattern O0x00ff00ff Testing Device
150. sEgg9 e 2 es EStdae 5 A c J age c S g J Sere 5 oe a 8 lt a fees ie oob Comm_ Down Y Lost communication with 1 Check Ethernet Lost swla connectivity to the ip xxx 20 67 213 switch 2 Verify that the switch is Ethernet connectivity to the booted correctly with no switch has been lost POST errors 3 Verify that the switch Test Mode is set for normal operations 4 Verify the TCP IP settings on switch by way of Forced PROM Mode access 5 Replace switch if needed switch2 Diagnostic Red Check Test Manager for test Test failure details enclosure Discovery Discovered a new switch called ras d2 swb1 ip xxx 0 0 41 10002000007a609 Discovery events occur the very first time the agent probes a storage device It creates a detailed description of the device monitored and sends it using any active notifier such as the Sun Remote Services SRS Net Connect service or email enclosure Location Location of switch rasd2 Change swb0 ip xxx 0 0 40 was changed Chapter 7 Troubleshooting Switches Sun Proprietary Confidential Internal Use Only TABLE 7 2 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches Continued 25 z g cb ees O S L 2 a g rs r 2 rf 8 lt a ffs ac port State port 1 in SWITCH Change diag185 ip xxx 20 67 185 is now Available status state changed from offline to online The port on
151. sS amp Sun microsystems Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide Sun Microsystems Inc 4150 Network Circle Santa Clara CA 95054 U S A 650 960 1300 Part No 816 5255 12 March 2003 Revision A Send comments about this document to docfeedback sun com Copyright 2003 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 U S A All rights reserved Sun Microsystems Inc has intellectual property rights relating to technology embodied in the product that is described in this document In particular and without limitation these intellectual property rights may include one or more of the U S patents listed at ttp www sun com patents and one or more additional patents or pending patent applications in the U S and in other countries This document and the product to which it pertains are distributed under licenses restricting their use copying distribution and decompilation No part of the product or of this document may be reproduced in any form by any means without prior written authorization of Sun and its licensors if any Third party software including font technology is copyrighted and licensed from Sun suppliers Parts of the s produet may be derived from Berkeley BSD systems licensed from the University of California UNIX is a registered trademark in the U S and in other countries exclusively licensed through X Open Company Ltd Sun Sun Microsystems the Sun logo
152. se Only Related Documentation Product Late breaking News Sun StorEdge 3900 and 6900 series information Sun StorEdge T3 and T3 array Diagnostics Sun StorEdge SAN 4 0 1 Gb switches Sun StorEdge SAN 4 1 2 Gb switches 3Com Ethernet hubs Title Sun StorEdge 3900 and 6900 Series 2 0 Release Notes Sun StorEdge 3900 and 6900 Series 2 0 Installation Guide Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide Sun StorEdge 3900 and 6900 Series 2 0 Regulatory and Safety Compliance Manual M e e anual Sun StorEdge 3900 and 6900 Series 2 0 Site Prep Guide Sun StorEdge T3 Array Release Notes Sun StorEdge T3 Array Start Here Sun StorEdge T3 and T3 Array Regulatory and Safety Compliance Sun StorEdge T3 Array Installation and Configuration Manual Sun StorEdge T3 Array Administrator s Guide Sun StorEdge T3 Array Cabinet Installation Guide Storage Automated Diagnostics Environment User s Guide Sun StorEdge SAN 4 0 Release Guide to Documentation Sun StorEdge SAN 4 0 Release Installation Guide Sun StorEdge SAN 4 0 Release Configuration Guide Sun StorEdge Network 2 Gb FC Switch 16 FRU Installation Sun StorEdge SAN 4 0 Release Notes Sun StorEdge SAN 4 1 Release Guide to Documentation Sun StorEdge SAN 4 1 Release Installation Guide Sun StorEdge SAN 4 1 Release Configuration Guide Sun StorEdge SAN 4 1 2 Gb Brocade Silkworm Fabric Switch Guide to Documentation Sun StorEdge SAN 4 1 2
153. se agreement and applicable provisions of the FAR and its supplements DOCUMENTATION IS PROVIDED AS IS AND ALL EXPRESS OR IMPLIED CONDITIONS REPRESENTATIONS AND WARRANTIES INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR NON INFRINGEMENT ARE DISCLAIMED EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID Copyright 2003 Sun Microsystems Inc 4150 Network Circle Santa Clara California 95054 Etats Unis Tous droits r serv s Sun Microsystems Inc a les droits de propri t intellectuels relatants a la technologie incorpor e dans le produit qui est d crit dans ce document En particulier et sans la limitation ces droits de propri t intellectuels peuvent inclure un ou plus des brevets am ricains num r s a http www sun com patents et un ou les brevets plus suppl mentaires ou les applications de brevet en attente dans les Etats Unis et dans les autres pays Ce produit ou document est prot g par un copyright et distribu avec des licences qui en restreignent l utilisation la copie la distribution et la d compilation Aucune partie de ce produit ou document ne peut tre reproduite sous aucune forme parquelque moyen que ce soit sans l autorisation pr alable et crite de Sun et de ses bailleurs de licence s il y ena Le logiciel d tenu par des tiers et quicomprend la technologie relative aux polices de caract res est prot g par un copyright et licenci
154. sed to move multiple objects at the same time switch diag156 sw1a 100000c0dd0057 bd port 2 06 20 12 26 54 port 2 in diag156 swla ip 192 168 0 30 is n lt E XY Layout Save Layout aor BE Horizontal tip ittpitiiag2e T iT654 9GO GUI Review welcome A FIGURE 11 5 Storage Automated Diagnostic Environment Test from Topology Chapter 11 Example of Fault Isolation 151 Sun Proprietary Confidential Internal Use Only Host san MASTER Filter SE Series diag 56 swla switch 100000cOdd0057bd S Test diag156 5w1a switch p2 Current Tests diag156 sw1a switch p2 Report on diag156 sw1a switch p2 Discman on diag156 sw 1a switch p2 Alerts diag156 sw1a switch p2 switch diag156 sw1a 100000cOdd0057 bd port 2 06 20 12 26 54 port 2 in SWITCH diag156 sw 1a ip 192 168 0 30 is nc Topology Help After having selected a topology several functions are available in the graphical view 1 Clicking in the middle of an object or in a specific component of an object will display the status at the bottom of the page 2 Right clicking the background will present a zoom menu 3 Right clicking on an object will present a menu of functions available for that object 4 Right clicking on a link will present a menu of functions available for that link Holding the Shift button while selecting objects allows to select multiple objects This can be
155. sentito the RSS SRS A providers alert warning No This event is nonactionable system down Chapter 3 Troubleshooting Tools 25 Sun Proprietary Confidential Internal Use Only Microsoft Windows 2000 System Errors You can view Microsoft Windows 2000 errors through the Event Properties System Log The types of errors that would indicate a Sun StorEdge T3 Array Failover Driver issue have the Source Jafo An example is shown in FIGURE 3 2 You should also look for other events such as any HBA driver related events qla2200 for example or disk related events Event Date 7 15 2002 Source Jafo t Time 11 48 Category None Type Error EventID 3 User N A Computer DELL 2 Description JAFO Path state changed Old State UP ACTIVE New state DOWN Path ID 4 0 0 121 Reason Code Path now down Data C Bytes Words 0000 0000000f 00520005 00000000 coog0oos 0010 00000000 00000000 00000000 00000000 0020 00000000 00000000 FIGURE 3 2 Microsoft Windows 2000 Event Properties System Log 26 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Command Line Test Examples To run a single Sun StorEdge diagnostic test from the command line rather than through the Storage Automated Diagnostic Environment interface you must log in to the appropriate host or slave for testing the components The following two test
156. t FAIL lt Failure Noted Checking T3 t3b2 Checking t3b2 Configuration Checking command ver PASS Checking command vol stat PASS Checking command port list PASS Checking command port listmap PASS Checking command sys list PASS lt snip gt Checking Virtualization Engine Pair Parameters vla vla configuration check passed Checking Virtualization Engine Pair Parameters vlb vilb configuration check passed Checking Virtualization Engine Pair Configuration vl checkvemap virtualization engine map vl verification complete PASS 8 Sun StorEdge 3900 and 6900 2 0 Series Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 2 If anything is marked FAIL check the var adm 1log S details of the failure Jan 7 Jan 7 Jan 7 Mon Mon Mon Jan Jan Jan N NNN Jan Bytes Mon Jan 7 on Jan 7 on Jan 7 CURRENT CONFIGURATION 256 MBytes 2002 checkt3config on Jan 7 18 07 51 PST 2002 checkt3config SAVED CONF IGURATION on Jan 7 18 07 51 PST 2002 checkt3config on Jan 7 18 07 51 PST 2002 checkt3config on Jan 7 18 07 51 PST 2002 checkt3config on Jan 7 18 07 51 PST 2002 checkt3config Mon Jan 7 18 07 51 PST 2002 checkt3config on Jan 7 18 07 51 PST 2002 checkt3config Mon Jan 7 18 07 51 PST 2002 checkt3config Bytes on Jan 7 18 07 51 PST 2002 checkt3config 256 MBytes on Jan 7 18 07 51 P
157. t 2 is offline check the GBICs and cables 7 If a Sun StorEdge T3 array switch port is offline log in to the Sun StorEdge T3 array and look at the status of the controllers and the port list as shown in CODE EXAMPLE 9 7 CODE EXAMPLE 9 7 Status of Sun StorEdge T3 Array Controllers and Port List t3b0 lt 1l gt fru stat ulcl CTLR STATUS STATE ROLE PARTNER TEMP ulctr ready enabled master u2Zctr 28 0 t3b0 lt 2 gt fru stat u2c1 CTLR STATUS STATE ROLE PARTNER TEMP u2Zctr ready enabled alternate master ulctr 27 60 t3b0 lt 3 gt port list port targetid addr_type status host wwn ulpl 0 hard online sun 50020f2300006dfa u2p1 1 hard online sun 50020 230000725b 8 If either controller is in a disabled state or if either port is offline refer to the Sun StorEdge T3 Installation and Configuration Guide for corrective action 9 After the problem has been corrected repeat Step 2 Manually Clearing and Restoring the SAN Database It is occasionally necessary to manually clear and restore the SAN database on the virtualization engines Caution This procedure clears the SAN database and removes the configuration of the disk pools multipath drives zoning and VLUNs After you perform this procedure you must restore the virtualization map to the virtualization engine pair using restorevemap 1M This requires a valid copy of the vi san or v2 san files located in the opt WUNWsecfg etc vn map directory C
158. t User s Guide 2 In the Maintain Hosts window from the Existing Hosts list select the host to be replaced and click Delete 3 Install the new host Refer to Chapter 2 of the Storage Automated Diagnostic Environment User s Guide for detailed instructions for the next four steps 4 Install the SUNWstade package on the new host 5 Run opt SUNWstade bin ras_install 6 Configure the host as a slave 7 Choose Maintenance gt General Maintenance gt Maintain Hosts Refer to the maintenance section in Chapter 3 of the Storage Automated Diagnostic User s Guide for detailed instructions 8 In the Maintain Hosts window select the new host 9 Configure the options as needed 10 Choose Maintenance gt Topology Maintenance gt Topology Snapshot a In the Topology Snapshot window select the new host b Click the Create and Retrieve Selected Topologies button c Click the Merge and Push Master Topology button Note Any time you replace a master alternate master or slave monitoring host you must recover the configuration using the procedures described in this section This is especially important when the Storage Service Processor is replaced as a FRU whether the Storage Service Processor is the master or the slave 72 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 7 Troubleshooting Switches This chapter describes ho
159. t can be isolated are the switch side GBIC and the Sun StorEdge network FC switch itself For these components you can launch the tests using the Storage Automated Diagnostic Environment Diagnose gt Tests gt Test From Topology functionality Again temporarily remove the cable from the switch port in question insert a loopback connector plug and run the switch port diagnostics The first run will test the switch side GBIC as well as the Sun StorEdge network FC switch Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only In the examples shown in FIGURE 11 5 FIGURE 11 6 and FIGURE 11 7 Port 2 on Switch diag156 swla was marked with a Red icon indicating a problem Note All tests were run with the default values Host san MASTER Filter None Topology Help After having selected a topology several diag156 functions are available in the graphical view 1 Clicking in the middle of an object or in a specific component of an object will display the SE Series status at the bottom of the page 2 Right clicking the background will present a zoom menu 3 Right clicking on an object will present a menu of functions available for that object 4 Right clicking on a link will present a menu of functions available for that link 5 Holding the Shift button while selecting objects allows to select multiple objects This can be u
160. t compatible firmware Caution Use caution when upgrading back end switches to the 2 Gbit compatible firmware Use only the set switchflash command which performs the upgrade and creates the zone configuration in a controlled manner refer to the Sun StorEdge 3900 and 6900 Series 2 0 Reference and Service Guide for the procedures Zone Modifications You should not modify the shared zone set on the back end switches doing so can cause an error Error State 50 on the virtualization engine If you determine however that you must modify the shared zone set follow these steps 1 Offline the T ports interswitch links 2 Offline the virtualization engine ports 3 Modify the zone on one switch while the other switch continues to run 4 Online the T ports interswitch links 5 Allow the zone database to merge 6 Online the virtualization engine ports You can use the sanbox2 1M command to offline the ports For example opt SUNWsecfg flib sanbox2 x switch ip addr port state offline By default m T ports are 671415 m Virtualization engine ports are 0 8 74 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Switchless Configurations In a switchless configuration Sun StorEdge 3900SL 6910SL or 6960SL series system you can upgrade the switches that are connected to the Solaris server to the Sun StorEdge SAN 4 1 Release firmware For a
161. t the Virtualization Engine 107 Virtualization Engine Diagnostics 108 Service Request Numbers SRNs 108 Service and Diagnostic Codes 108 Retrieving Service Information 108 CLI Interface 108 Error Log Analysis Commands 109 v To Display the Log Files and Retrieve SRNs 109 VI Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 10 v ToClear the Log 110 Virtualization Engine LEDs 110 Power LED Codes 111 Interpreting LED Service and Diagnostic Codes 111 Back Panel Features 112 Ethernet Port LEDs 112 FC Link Error Status Report 113 v To Check the FC Link Error Status Manually 113 Translating Host Device Names 115 Displaying the VLUN Serial Number 116 v To Display Devices That are Not Sun StorEdge Traffic Manager MPxIO Enabled 116 v To Display Sun StorEdge Traffic Manager MPxIO Enabled Devices 117 Viewing the Virtualization Engine Map 118 v To Failback the Virtualization Engine 120 Manually Clearing and Restoring the SAN Database 123 v To Reset the SAN Database on Both Virtualization Engines 124 v To Reset the SAN Database on a Single Virtualization Engine 125 Restarting the slicd Daemon 126 v To Restart the slicd Daemon 126 Diagnosing a creatediskpools 1M Failure 129 Virtualization Engine Event Grid 132 v To Use the Virtualization Engine Event Grid 132 Troubleshooting Using Microsoft Windows 2000 137 General Notes 137 Troubleshooting Tasks Using Microsoft Wi
162. tform Driver gt jafo_nutil HBA WWN 00000000000000000000000000000000 NAME Device ScsiPort4 DESC QLogic QLA2200 PCI Fibre Channel Adapter DRIVER ql2200 FW_REV Can t obtain from OS HBA WWN 00000000000000000000000000000000 NAME Device ScsiPort5 DESC QLogic QLA2200 PCI Fibre Channel Adapter DRIVER q12200 FW_REV Can t obtain from OS DEVICE VENDOR Sun Microsystems 69XX Storage Subsystem FW_REV 0811 SERIAL bW3TO001lw WWN 290000602200418 0000000000000000 FO_CAPABLE true MASTER true LUN NAME J WWN 290000602200418f6257335430303177 GOOD_PATHS 2 STATE up 1 PATH NAME 4 0 0 0 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE primary STATE up_active 2 PATH NAME 5 0 0 0 HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE primary STATE up_active 2 LUN NAME K WWN 290000602200418 f6257335430303178 GOOD_PATHS 2 STATE up 1 PATH NAME 4 0 0 1 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE primary STATE up_active 2 PATH NAME 5 0 0 1 HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE primary STATE up_active 2 LU NAME G WWN 290000602200418f 6257335430303179 GOOD_PATHS 2 STATE up 1 PATH NAME 4 0 0 2 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE primary STATE up_active 2 PATH NAME 5 0 0 2 HBA_NAME Device ScsiPort5 TARGET 0 0 0 TYPE primary STATE up_active 2 LU NAME H WWN 290000602200418f 625733543030317a GOOD_PATHS 2 STATE up 1 PATH NAME 4 0 0 3 HBA_NAME Device ScsiPort4 TARGET 0 0 0 TYPE primary STATE up_active 2 PATH NAME 5 0 0 3
163. the failed path and click Array Properties From the Array Properties window click Details and OK The LUN Properties window is displayed FIGURE 11 2 Drilling Down for Sun StorEdge T3 Array Failover Driver Fault Detail 148 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only The primary path to Drive F failed The alternate path is currently handling all of the I O 3 Check the HBA Using the HBA utility Qlogic SANblade in this example confirm the fault E ioixi File Host View Help Connect Configure Events Alarms Refresh h U Simplify Information Device List Statistics NVRAM Settings Link Status Utilities Diagnostics Host smi 2gs6m2bdhom Host smi 2gsSm2bdhom NodeName 20 00 00 E0 88 02 65 17 mH Adapter 2200 Adapter 0 2200 PortName 21 00 00 E0 88 02 65 17 ad PortID 10 41 00 General Information Serial Number B37061 Driver Version 8 1 3 WV2K IP BIOS Version 1 76 Firmware Version 2 01 38 FIGURE 11 3 Fault Confirmation Using QLogic SunBlade 4 Isolate the components in the path The components in the path are the HBA the cable the switch side GBIC and the Sun StorEdge network FC switch itself To isolate all components use a combination of the Storage Automated Diagnostic Environment and the HBA utility QLogic SunBlade Note If no HBA utility is present or if the utilities do not
164. the Master Host Refer to Chapter 2 of the Storage Automated Diagnostic Environment User s Guide for detailed instructions for the next four steps Install the SUNWstade package on a new master host Run opt SUNWstade bin ras_install on the new master host Configure the host as the master host Connect to the master server s GUI at http lt servername gt 7654 Choose System Utilities gt Recover Config Refer to Chapter 3 of the Storage Automated Diagnostic Environment User s Guide for detailed instructions a In the Recover Config window enter the IP address of any alternate master or slave monitoring host All hosts keep a copy of the configuration b Make sure the checkboxes for Recover config and Reset slave to this master are checked c Click Recover Choose Maintenance gt General Maintenance a Ensure that all host and device settings are recovered correctly b Refer to Chapter 3 of the Storage Automated Diagnostic Environment User s Guide for detailed instructions Chapter 6 Troubleshooting Host Devices 71 Sun Proprietary Confidential Internal Use Only 7 Choose Maintenance gt General Maintenance gt Start Stop Agent to start the agent on the master host v To Replace the Alternate Master or Slave Monitoring Host 1 Choose Maintenance gt General Maintenance gt Maintain Hosts Refer to the maintenance section in Chapter 3 of the Storage Automated Diagnostic Environmen
165. the master No action is needed virtualization engine has detected the partner virtualization engine s IP Address 70025 For Sun StorEdge T3 array pack The master Check the Ethernet connection between virtualization engine is unable to detect the the two virtualization engines partner virtualization engine s IP address 70030 The SAN configuration was changed by the SAN No action is needed Builder 70040 The zoning configuration of the host has No action is needed changed 70050 A multipath drive failover occurred Check the multipath drive 70051 A multipath drive failback occurred No action is needed 70098 Instant copy degraded If no spare is available replace the failed drive with a new drive 70099 Degrade because the drive has disappeared Reinsert the missing drive or replace it with a drive of equal or greater capacity 7009A A mirror drive was written to causing it to enter Reinsert the missing drive or replace it the read degrade state with a drive of equal or greater capacity 7009B A drive entered the write degrade state Reinsert the drive if good or replace it if it is defective 7009C During a rebuild the last primary drive failed 1 Backup the drive data This is a very rare multipoint failure 2 Destroy the mirror drive where the failure has occurred 3 Format the drives using mode 14 4 Create a new mirror drive 5 Reassign the old SCSI ID and LUN to the new mirror drive 6 Restore the data 71000 Communic
166. ting failures The remainder of this document provides guidelines that you can use to troubleshoot problems that occur in supported components of the Sun StorEdge 3900 and 6900 series Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 2 General Troubleshooting Procedures This chapter contains the following sections a High Level Troubleshooting Tasks on page 3 a Host Side Troubleshooting on page 6 a Storage Service Processor Side Troubleshooting on page 6 m Verifying the Configuration Settings on page 7 m Sun StorEdge 6900 Series Multipathing Example on page 11 a Multipathing Options in the Sun StorEdge 6900 Series on page 16 High Level Troubleshooting Tasks This section lists the high level steps you can take to isolate and troubleshoot problems in the Sun StorEdge 3900 and 6900 series It offers a methodical approach and lists the tools and resources available at each step Note A single problem can cause various errors throughout the storage area network SAN A good practice is to begin by investigating the devices that have experienced Loss of Communication events in the Storage Automated Diagnostic Environment These errors usually indicate more serious problems A Loss of Communication error on a switch for example could cause multiple ports and host bus adapters HBAs to go of
167. tive Action 005 A PCI bus parity error has e Replace the virtualization occurred engine 24 The attempt to report one error e Cycle power to the resulted in another error virtualization engine 40 The database is corrupt e Clear the SAN database e Cycle power to the virtualization engine e Import the SAN zone configuration 41 The database is corrupt e Clear the SAN database e Cycle power to the virtualization engine e Import the SAN zone configuration 42 The zone mapping database is e Import the SAN zone corrupt configuration 160 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE A 4 Virtualization Engine Service Codes Continued 0 399 Host Side Interface Driver Errors Appendix A 050 An attempt to write a value into e Clear the SAN database nonvolatile storage failed e Cycle power to the perhaps because a hardware virtualization engine failure or one of the databases stored in Flash memory could not accept the entry being added 051 The virtualization engine cannot e Replace the virtualization erase Flash memory engine 53 The cabling configuration is e Check the cabling Ensure the unauthorized server and switch connect to the host side and the storage connects to the device side of the virtualization engine If necessary clear the SAN database e If necessary cycle the virtualization engine power
168. to limit the report Click on the Columns headers to change the sort Event Grid Event Grid pdf Check ReportFormat to display a Report format Click Info Action to Review ReportFormat Info Action Change in Port Statistics on switc iag156 sw1b ip 192 168 0 31 Info Switch sw1a was rezoned new zones Auditing a new switch called ras 0002000007a609 chassis zone enclosure Comm_Lost ip xxx 20 67 213 DiagnosticTest g 0002000007a609 Info Discovered a new switch called ras d2 swb1 switchtest enclosure Discovery Location of switch rasd2 sw bO ip xxx 0 0 40 was changed enclosure LocationC hange StateChange now Available ee ened from Offline to Online StateChange enclosure Statistics Page 1 of 1 13 events a Sev Severity of the event warning gt Error gt Down Action This event is Actionable and will be sent to RSS SRS SubComp SubComponent FIGURE 7 1 Switch Event Grid Chapter 7 Troubleshooting Switches 77 Sun Proprietary Confidential Internal Use Only TABLE 7 1 lists the switch events for Sun StorEdge network FC switch 8 and switch 16 1 Gbit switches TABLE7 1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches 2 gt 5 kej z g EBES oJ Q 5 j 3 seers F i 8B lt a 2fsres lt a port Log Yellow Y
169. tualization engines with firmware revision 8 14 or earlier For virtualization engines with firmware revision 8 17 or later you can determine error conditions with the following steps a Open a Telnet session into v1_hostname b Display the Vital Product Data VPD by entering 1 at the prompt The last line of the output displays any error codes as shown in the following example Chapter 9 Troubleshooting Virtualization Engine Devices 127 Sun Proprietary Confidential Internal Use Only 128 Loader Revision Unique ID Unit Serial Number PCB Number AAC address DIP SW1 00000000 76543210 Error None Product Type FC FC 3 SVE H FC FC 3 router H Firmware Revision Vicom release Apr 11 2002 17 49 16 2 02 42 00000060 2200418A 00250339 00166425 0 60 22 3 D1 E3 DIP SW2 00000011 76543210 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 8 017 Official Release 1 down 0 up Diagnosing a creatediskpools 1M Failure When modifying the Sun StorEdge T3 array configuration on a Sun StorEdge 6900 series the system should automatically create disk pools If the virtualization engine cannot find two paths to all Sun StorEdge T3 array LUNs however the multipath drives cannot be created If this happens the following procedure can help troubleshoot the problem Inspect the SUNWsecfg log file var adm log
170. tualization Engine Event Grid from which you can select related criteria for the event you are troubleshooting Home Help 2 1 B1 001 ccadieux central sun com General Reports Event Grid Help Select a Category Component EventType and type GO to limit the report Click on the Columns headers to change the sort Check ReportFormat to display a Report format Click Info Action to Review Event Grid pdf Architecture Diagnostics Diag Strategy Utilities Release Notes User s Guide pd Copyrights Abbreviations ReportFormat _ Info Volume E00012 on 1a changed mapping Info Auditing a Virtualization Engine called v1a Info Action Lost communication with VE v1a e_ diag diag240 on ve 1 ip xxx 20 67 213 failed Discovery Info Discovered a new Virtualization Engine called v1a 9 events Sey Severity of the event warning gt Error gt Down a Action This event is Actionable and will be sent to RSS SRS SubComp SubComponent FIGURE 9 3 Virtualization Engine Event Grid 132 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE 9 5 lists the Virtualization Engine Events TABLE9 5 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine Component Required EventType Severity Information Action volume Alarm Yellow This event occurs whe
171. tween the HBA and switch b Use the Sun StorEdge T3 Array Failover Driver GUI for the Sun StorEdge 3900 series system or the CLI for the 6900 series to recover the multipathing Chapter 11 Example of Fault Isolation 153 Sun Proprietary Confidential Internal Use Only 154 amp gt Multipath Configurator Driver HBA Array Help smi 2gs5 array 4 gt 60020F20000003D50000000000000000 Make primary paths HA 60020F20000003p Take primary paths Array 1 LUN Count 1 Array 2 LUN Count 1 FIGURE 11 9 Multipath Recovery using the Sun StorEdge T3 Array Multipath Configurator Note Storage Automated Diagnostic Environment should also post an event noting that the Port has gone back online The Multipath Configurator GUI should show both paths online and handling I O as illustrated in FIGURE 11 10 amp Multipath Configurator Driver HBA Path Array Help smi 2gs5m2bdhom 60020F20000003D50000000000000000 Fibre Channel Adapter Array 1 LUN Count 1 T data path z 60020F20000003D50000000000000000 Array 2 LUN Count 1 FIGURE 11 10 Recovered Paths Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only APPENDIX A Virtualization Engine References This appendix contains the following information SRN Reference on page 155 SRN SNMP Single Point of Failure Descriptions on page 15
172. twork FC switch commands Common to all Sun StorEdge network FC switch commands The switch is unable to obtain a lock on switch switch Another command is running Unable to determine switch type Interface may be down or type is unsupported The switch commands now have to be able to determine if the switch is a 1 Gbit or 2 Gbit switch and they were unable to obtain the flash revision on the switch for some reason Invalid login id and or password entered The user has set a login id and password on the 2Gbit switch 1 Check listavailable s to see if another switch command might be updating the configuration 2 If the switch in question does not appear check for the existence of the lock file directly by typing 1s la opt SUNWsecfg etc look for switch lock 3 If the lock is set in error use the removelocks s command to clear it Due to a non reentrant interface there is a single lock file for all switches Only one can be accessed at a time 1 Reset the switch 2 Rerun the appropriate switch command 1 Set the SWLOGIN and SWPASSWD environment variables to the correct switch login id and password 2 Re run the switch command 168 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE B 2 Source of Error Message Cause of Error Message Sun StorEdge Network FC Switch Error Messages Continued
173. u2d6 SVD_PATH_FAILOVER path_id 0 Jan 29 14 05 18 t3b0 ISR1 1 W u2d7 SVD_PATH_FAILOVER path_id 0 Jan 29 14 05 18 t3b0 ISR1 1 W u2d8 SVD_PATH_FAILOVER path_id 0 Jan 29 14 05 18 t3b0 ISR1 1 W u2d9 SVD_PATH_FAILOVER path_id 0 FIGURE 5 12 Storage Service Processor Side Notification Chapter 5 Troubleshooting the Fibre Channel FC Links 61 Sun Proprietary Confidential Internal Use Only Verifying the Data Host A problem in the A4 or B4 FC Link appears differently on the data host depending on whether the array is a Sun StorEdge 3900 series or a Sun StorEdge 6900 series device Sun StorEdge 3900 Series In a Sun StorEdge 3900 series device the data host multipathing software is responsible for initiating the failover and reports it in var adm messages such as those reported by the Storage Automated Diagnostic Environment email notifications The luxadm failover command is used to fail the Sun StorEdge T3 array LUNs back to the proper configuration after the failing FRU is replaced This command is issued from the data host Sun StorEdge 6900 Series In a Sun StorEdge 6900 series device the virtualization engine pairs handle the failover and the failover is not noted on the data host All paths remain online and active The failbackt3path command is used and is issued from the Storage Service Processor Note In the event of a complete sw1b or sw2b failure in a Sun StorEdge 6900 series configu
174. ue Sun d tient une license non exclusive do Xerox sur l interface d utilisation graphique Xerox cette licence couvrant galement les licenci es de Sun qui mettent en place l interface d utilisation graphique OPEN LOOK et qui en outre se conforment aux licences crites de Sun Netscape Navigator est une marque de Netscape Communications Corporation aux Etats Unis et dans d autrespays LA DOCUMENTATION EST FOURNIE EN L ETAT ET TOUTES AUTRES CONDITIONS DECLARATIONS ET GARANTIES EXPRESSES OU TACITES SONT FORMELLEMENT EXCLUES DANS LA MESURE AUTORISEE PAR LA LOI APPLICABLE Y COMPRIS NOTAMMENT TOUTE GARANTIE IMPLICITE RELATIVE A LA QUALITE MARCHANDE A L APTITUDE A UNE UTILISATION PARTICULIERE OU A L ABSENCE DE CONTREFA ON Ory Please ga amp Recycle amp Adobe PostScript Contents Preface XV How This Book Is Organized XV Using UNIX Commands XVI Typographic Conventions XVII Shell Prompts XVII Related Documentation XVIII Accessing Sun Documentation Online XX Sun Welcomes Your Comments XX Introduction 1 Predictive Failure Analysis PFA Capabilities 2 General Troubleshooting Procedures 3 High Level Troubleshooting Tasks 3 Host Side Troubleshooting 6 Storage Service Processor Side Troubleshooting 6 Verifying the Configuration Settings 7 v To Verify Configuration Settings 7 Clearing the Lock File 10 v To Clear the Lock File 10 Contents IM Sun Proprietary Confidential Internal Use Only Sun StorEdge 690
175. un StorEdge T3 array Common to Sun StorEdge T3 array Common to Sun StorEdge T3 array e Could not mount volume volume e Slun config does not match e The LUN might have multiple drive failures or corrupted data or parity e No volumes exist on this Sun StorEdge T3 array e Svolume volume not found on this Sun StorEdge T3 array e The fru status is not ready or enabled e Operations on the Sun StorEdge Appendix B 1 Replace the failed FRUs 2 Restore the Sun StorEdge T3 array configuration with the restoret3config f n t3_name command Create and restore the Sun StorEdge T3 array LUNs using restoret3config 1M or modifyt3config 1M The disk controller or loop interface card in the Sun StorEdge T3 array might be faulty Replace the failed FRU and rerun the utility T3 array are being aborted Configuration Utility Error Messages 171 Sun Proprietary Confidential Internal Use Only TABLE B 3 Source of Error Message Sun StorEdge T3 Array Error Messages Continued Cause of Error Message Suggested Corrective Action Common to Sun StorEdge T3 array Common to Sun StorEdge T3 array Common to Sun StorEdge T3 array Common to Sun StorEdge T3 array e The Sun StorEdge T3 array is not of T3B type so it aborts operations e t3config utilities are supported only in the Sun StorEdge T3 array the t3config utilities are not supported on Sun StorEdge T3
176. used to move multiple objects at the same time FIGURE 11 6 Storage Automated Diagnostic Environment Test from Topology Pull Down Menu i 002 06 20 12 55 39 RC 1 switchtest called with options dev 2 192 168 0 30 0x01 xfersize 2000 iterations 1000001 serpattem 0x7e7e7e7el selectpattern critical switchtest Started Testing port 2 Using ip_addr 192 168 0 30 fcaddr 0x0 to access this port Chassis Status for Device Switch Power OK Temp OK 24 0c Fan 1 OK Fan 2 OK 96 2002 12 55 39 diag1 56 StorADE 2 0 MSGID 6013 switchtest FATAL switch0 Device Switch Port 2 is Offline Probable_Cause s lt Fibre Channel cable disconnected gt Bad GBIC or bad Fibre Channel cable gt lt Port offline gt lt Bad device connected to switch gt Recommended Actions lt Run link test to this port gt lt Check fibre channel connections gt lt Check port state gt Check devices connected to switch gt FIGURE 11 7 Storage Automated Diagnostic Environment Test from Topology Test Detail 152 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only The first run failed indicating a problem with either the GBIC or with the Sun StorEdge network FC switch To further isolate the problem a new GBIC was inserted into the port the loopback connector was re inserted and the same test was run a second time
177. vailable already been assigned 1 Ifa zone name is assigned run the For a WWN to be available for rmvezone command createvezone the WWN in the 2 If errors still exist run map file showvemap n sadapter alias d vepair ve_pairname must be Undefined r Sinitiator a zone n and the online status should be aie Yes 3 Run savemap n vepair createvlun Invalid disk pool diskpool on 1 Run the showvemap n vepair or disk pool is unavailable Svepair command to verify that the disk pool was created properly 2 If the disk pool is unavailable run creatediskpools n t3name 3 If that fails check the Sun StorEdge T3 array for unmounted volumes or path failures by running checkt 3config n t3name v createvlun Unable to execute command The 1 Run checkt3mount n associated Sun StorEdge T3 array t3name physical LUN t31lun for disk 1 ALL to see the mount status of pool diskpool might not be the volume mounted 2 For further information about problems with the underlying Sun StorEdge T3 array run checkt 3config n t3name v 166 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only TABLE B 1 Source of Error Message Virtualization Engine Error Messages Continued Cause of Error Message Suggested Corrective Action restorevemap setdefaultconfig setdefaultconfig e The import zone data failed
178. ve 1 Test failed veluntest Diagnostic Red The veluntest failed Test enclosure Discovery The discovery device found a new virtualization engine called via Discovery events occur the first time the agent probes a storage device and creates a detailed description of the device monitored The discovery device sends it using any active notifier such as NetConnect or email Chapter 9 Troubleshooting Virtualization Engine Devices Sun Proprietary Confidential Internal Use Only 135 136 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only CHAPTER 10 Troubleshooting Using Microsoft Windows 2000 General Notes m Use the Manufacturer s HBA Utilities to monitor and diagnose the HBAs The examples in this chapter use Qlogic s SANblade Manager utility a The Storage Automated Diagnostic Environment running on the Storage Service Processor is not able to monitor the host to switch link a The Storage Automated Diagnostic Environment running the Storage Service Processor is not able to execute switchtest 1M on switch ports with Microsoft Windows 2000 HBAs currently attached as F ports m The FRUs in the host to switch link can be isolated using the HBA utilities on the host and Storage Automated Diagnostic Environment s switchtest on the Storage Service Processor in conjunction with loopback connector plugs m Install the Sun St
179. ver Errors 162 Virtualization Engine Error Messages 164 Sun StorEdge Network FC Switch Error Messages 168 Sun StorEdge T3 Array Error Messages 171 Other SUNWsecfg Error Messages 175 XIV Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Preface The Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide provides guidelines for isolating problems in supported configurations of the Sun StorEdge 3900 and 6900 series For detailed configuration information refer to the Sun StorEdge 3900 and 6900 Series Reference Manual The scope of this troubleshooting guide is limited to information pertaining to the components of the Sun StorEdge 3900 and 6900 series including the Storage Service Processor Sun StorEdge 1 Gbit and 2 Gbit switches Sun StorEdge T3 arrays and the virtualization engines in the Sun StorEdge 6900 series This guide is written for Sun personnel who have been fully trained on all the components in the configuration How This Book Is Organized This book contains the following topics Chapter 1 introduces the Sun StorEdge 3900 and 6900 series storage subsystems Chapter 2 offers general troubleshooting guidelines such as manually halting the I O and returning paths to production Chapter 3 presents information about tools used to troubleshoot Tools include the Storage Automated Diagnostic Environment component specific event grids
180. verifying 57 Sun StorEdge 3900 and 6900 Series description of 1 related documentation XVIII Sun StorEdge 6900 Series I O routed through both HBAs 15 logical view 11 multipathing options 16 primary data paths to alternate master 12 primary data paths to Sun StorEdge T3 array 13 Sun StorEdge Network FC Switch 8 and Switch 16 switch diagnosis of 28 Sun StorEdge network FC switch 8 and switch 16 switch checking status 5 Sun StorEdge T3 array event grid 95 Explorer Data Collection Utility 29 LUN failover 18 reviewing LED status 4 status checking 4 syslog file 4 troubleshooting 87 Sun StorEdge T3 Array Failover Driver CLI output for Sun StorEdge 3900 series 143 CLI output for Sun StorEdge 6900 series 144 how to check version levels 139 launching 138 using the CLI 142 Sun StorEdge Traffic Manager alternatives to using 17 enabled devices 117 installations 5 problem on a Sun StorEdge 6900 Series 16 troubleshooting workarounds 17 svengine command 110 switch error messages 168 event grid 77 loss of communication error 3 pairing through SANSurfer GUI 73 switch diagnostics 28 75 switchless SL configurations 75 T T1 or T2 data path 88 notification events 89 T1 T2 data path FRU tests available 93 isolation procedures 94 test t3ofdg 5 t3test 5 t3volverify 5 Index 182 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Interna
181. vice Processor is set up to handle mail correctly Automatic Email Submission Would you like all explorer output to be sent to explorer database americas sun com at the completion of explorer when mail or e is specified y n n Chapter 3 Troubleshooting Tools 29 Sun Proprietary Confidential Internal Use Only 3 Before running the Explorer Data Collection Utility make sure that the switch and Sun StorEdge T3 array information is added to the proper opt SUNWexplo etc files Example Type switch information in the opt SUNWexplo etc saninput txt file Edit the file and add the switch information as shown in CODE EXAMPLE 3 3 CODE EXAMPLE 3 3 Editing Switch Information Using vi vi saninput txt Input file for extended data collection Format is SWITCH SWITCH TYPE PASSWORD LOGIN Valid switch types are ancor and brocade LOGIN is required for brocade switches the default is admin swla ancor swlb ancor sw2a ancor sw2b ancor wq 4 Type Sun StorEdge T3 array information in the opt SUNWexplo etc t3input txt file 5 Type the password for your specific site CODE EXAMPLE 3 4 Editing Sun StorEdge T3 Array Information Using vi vi t3input txt Input file for extended data collection Format is HOST PASSWORD t3b0 xxxx t3b2 xxxx t3b3 xxxx wq Note xxxx represents Sun StorEdge T3 array passwords 30 Sun StorEdge 3900 and 6900 Series 2 0 Trou
182. w to troubleshoot the 1 Gbit and 2 Gbit switch components associated with a Sun StorEdge 3900 or 6900 series system This chapter contains the following sections a About the Switches on page 73 m Using the Switch Event Grid on page 77 a setupswitch Exit Values on page 85 About the Switches The Sun StorEdge network FC switch 8 and switch 16 switches provide cable consolidation and increased connectivity for the internal data interconnection infrastructure The switches are paired to provide redundancy Two switches are used in each Sun StorEdge 3900 series and four switches are used in each Sun StorEdge 6900 series Each Sun StorEdge network FC switch 8 and switch 16 switch is connected by way of an Ethernet to the service network for management and service from the Storage Service Processor These switches can be monitored through the SANSurfer GUI for SAN Release 4 0 or the SANbox Manager for SAN Release 4 1 which is available on the Storage Service Processor You configure and modify the switches using the Configuration Utilities Caution Do not configure or modify the switches using any method other than the Configuration Utilities included in the SUNWsecfg package 73 Sun Proprietary Confidential Internal Use Only The Sun StorEdge network FC switches in a Sun StorEdge 3900 or 6900 configuration now support the Sun StorEdge SAN 4 1 Release You can upgrade the switches to support the 402xx 2 Gbi
183. x5555aa7a ra root other s 196610 0x5555aaba ra root other s 3 0x10e1 a root root Segments identified with 0x5555aa in the address are associated with slicd 3 Remove the segments by typing the following ipcrm m 301 m 302 m 303 s 196608 s 196609 s 196610 Refer to the ipcrm 1 man page for details The message queues and shared memory and semaphores have been removed 126 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only 4 To restart the slicd for the v1 virtualization engine type opt SUNWsecfg bin startslicd n v1 or v2 depending on configuration 5 Confirm that the s1icd daemon is running ps ef grep slicd root 16132 16130 0 11 45 00 0 00 slicd root 16135 16130 0 11 45 00 0 00 slicd root 16130 1 0 11 45 00 0 00 slicd root 16131 16130 0 11 45 00 0 00 slicd root 16189 15877 0 11 48 49 pts 1 0 00 grep slicd root 16143 16130 0 11 45 00 0 00 slicd If the slicd daemon is running it resets the virtualization engine If the process fails the sl1icd daemon changes the IP address to that of the second virtualization engine and attempts to restart the slicd process 6 If the second virtualization engine fails power cycle the virtualization engines and make sure they are not in an ERROR HALT 50 condition An ERROR HALT 50 condition requires that you visually inspect the vir
184. xxx com Severity Warning Category Message DeviceId message diag xxxxx xxx com EventType LogEvent driver SSD_WARN EventTime 01 30 2002 11 50 07 Found 1 driver SSD_WARN warning s in logfile var adm messages on diag xxxxx xxx com id 809f76b4 INFORMATION SSD warnings Jan 30 11 49 48 WWN Received 7 SSD Warning message s on ssd56 in 8 mins threshold is 5 in 24hours Last Message diag xxxxx xxx com scsi ID 243001 kern warning WARNING scsi_vhci ssd g29000060220041956257335a30303145 ssd56 continued on next page FIGURE 8 2 Virtualization Engine Alert 90 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only continued from previous page Site Lab 3286 DSQA1 Broomfield Source diag xxxxx xxx com Severity Warning Category Message DeviceId message diag xxxxx xxx com EventType LogEvent driver Fabric_Warning EventTime 01 30 2002 11 50 07 Found 1 driver Fabric_Warning warning s in logfile var adm messages on diag xxxxx xxx com id 809f76b4 INFORMATION Fabric warning Jan 30 11 46 37 WWN 2b00006022004186 diag xxxxx xxx com fp ID 517869 kern warning WARNING fp 2 N_x Port with D_ID 108000 PWWN 2b00006022004186 reappeared in fabric in backup diag xxxxx xxx com Site Lab 3286 DSQA1 Broomfield Source diag xxxxx xxx com Severity Warning Actionable Category
185. y these are the ports on the back end switches in Sun StorEdge 6900 series configurations only The ports support the ISL connections The Flash code is different from the release level The switch Flash code does not match the current release version The Sun StorEdge network FC switch 8 and switch 16 switches periodically releases new versions of the switch Flash code and the new version will not match the default version The configuration is not set to the default but the differences are likely supported alternatives The default switch configurations were overridden with valid alternatives which are also supported by the SUNWsecfg configuration tools It should still be flagged as not the default The exit value can imply any of the following alternatives these messages are printed to the screen and to the Storage Automated Diagnostic Environment GUI e Some ports have been set to SL TL or F mode but should have been set using the setswitcht1 or setswitchf commands View and verify this nonstandard configuration setup as required using the showswitch command Refer to the Sun StorEdge 3900 and 6900 Series Version 1 1 Reference and Service Guide for detailed configuration information e The chassis ID on the switch is not set to the default value This could be caused by unique ID settings or by conflicts in a SAN environment e Ports are identified that are not in the default hard zone This could be because the port is set to
186. ystem could not determine the To use the command line interface Sun StorEdge system type CLI set the BOXTYPE environment variable to one of the four values For example BOXTYPE 3910 export BOXTYPE Configuration Utility Error Messages Sun Proprietary Confidential Internal Use Only 175 176 Sun StorEdge 3900 and 6900 Series 2 0 Troubleshooting Guide March 2003 Sun Proprietary Confidential Internal Use Only Abbreviations and Acronyms This list contains definitions for acronyms used in this troubleshooting guide ASIC application specific integrated circuit CLI command line interface CRC cyclic redundancy code DAS direct attached storage EOF end of file FC Fibre Channel FC ELS Fibre Channel Extended Link Service FRU field replaceable unit GBIC gigabit interface converter GUI graphical user interface HBA host bus adapter ISL inter switch link LED light emitting diode LUN logical unit number MAC media access control NSCC Network Storage Command Center PCU power cooling unit PDU power distribution unit Abbreviations and Acronyms 177 Sun Proprietary Confidential Internal Use Only PFA predictive failure analysis POST power on self test RAID redundant array of independent disks RARP reverse address resolution protocol RFE request for enhancement RSS Remote Storage Services SAN storage area network SCSI small computer system interface SLIC Serial Loop IntraConnect SNMP simple network management protocol

Download Pdf Manuals

Related Search

Related Contents

取扱説明書  Logix 420 IOM - Flowserve Corporation  クリーンモア取扱説明書 - ワイズプラント株式会社  Autokamera Handbuch Vico-TF2+  MANUALE DI VOLO  MOOR INSTRUMENTS LIMITED Moorsoft for Windows for moorLAB  Verbatim DVD+R Double Layer Wide Inkjet Printable 8x  

Copyright © All rights reserved. Failed to retrieve file