Sun Microsystems 6900 manual

Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054 U .S.A. 650-960-1300 Send comments about this document to: docfeedback@sun.com Sun StorEdge ™ 3900 and 6900 Ser ies 2.0 T roub leshooting Guide P ar t No. 816-5255-12 March 2003, Revision A[...]

Please Recycle Copyright 2003 Sun Microsystems, Inc., 4150 Network Cir cle, Santa Clara, California 95054, U.S.A. All rights reserved. Sun Microsystems, Inc. has intellectual pr operty rights relating to technology embodied in the product that is described in this document. In particular , and without limitation, these intellectual property rights [...]

Contents III Sun Proprietary/Confidential: Internal Use Only Contents Preface XV How This Book Is Organized XV Using UNIX Commands XVI Typographic Conventions XVII Shell Prompts XVII Related Documentation XVIII Accessing Sun Documentation Online XX Sun Welcomes Your Comments XX 1. Introduction 1 Predictive Failur e Analysis (PF A) Capabilities 2 2.[...]

Contents IV Sun Proprietary/Confidential: Internal Use Only Sun StorEdge 6900 Series Multipathing Example 1 1 Multipathing Options in the Sun StorEdge 6900 Series 16 Manually Halting the I/O 17 ▼ T o Quiesce the I/O 17 ▼ T o Unconfigure the c2 Path 17 Suspending the I/O 18 ▼ T o Put the c2 Path Back into Production 19 ▼ T o V iew the Dynami[...]

Contents V Sun Proprietary/Confidential: Internal Use Only T roubleshooting the A1 or B1 FC Link 42 V erifying the Data Host 45 FRU T ests A vailable for the A1 or B1 FC Link Segment 46 ▼ T o Isolate the A1 or B1 FC Link 48 T roubleshooting the A2 or B2 FC Link 49 V erifying the Data Host 51 V erifying the A2 or B2 FC Link 52 FRU T ests A vailabl[...]

VI Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Replace the Alternate Master or Slave Monitoring Host 72 7. T roubleshooting Switches 73 About the Switches 73 Zone Modifications 74 Switchless Configurations 75 ▼ Diagnosing and T roubleshooting Switch Hardwar e P[...]

Contents VII Sun Proprietary/Confidential: Internal Use Only ▼ T o Clear the Log 1 10 V irtualization Engine LEDs 1 10 Power LED Codes 1 1 1 Interpreting LED Service and Diagnostic Codes 1 1 1 Back Panel Features 1 12 Ethernet Port LEDs 1 12 FC Link Error Status Report 1 13 ▼ T o Check the FC Link Error Status Manually 1 13 T ranslating Host-De[...]

VIII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Use the Sun StorEdge T3+ Array Failover Driver GUI 140 ▼ T o Use the Sun StorEdge T3+ Array Failover Driver Command Line Interface (CLI) 142 1 1. Example of Fault Isolation 147 A. V irtualization Engine Reference[...]

List of Figures IX Sun Proprietary/Confidential: Internal Use Only List of Figur es FIGURE 2-1 Sun StorEdge 6900 Series Logical View 11 FIGURE 2-2 Primary Data Paths to the Alternate Master 12 FIGURE 2-3 Primary Data Paths to the Master Sun StorEdge T3+ Array 13 FIGURE 2-4 Path Failure—Before the Second Tier of Switches 14 FIGURE 2-5 Path Failure[...]

List of Figures X Sun Proprietary/Confidential: Internal Use Only FIGURE 5-11 A4 or B4 FC Link Data-Host Notification 60 FIGURE 5-12 Storage Service Processor-Side Notification 61 FIGURE 6-1 Sample Host Event Grid 68 FIGURE 7-1 Switch Event Grid 77 FIGURE 8-1 Storage Service Processor Event 89 FIGURE 8-2 Virtualization Engine Alert 90 FIGURE 8-3 Ma[...]

List of Figures XI Sun Proprietary/Confidential: Internal Use Only FIGURE 11-8 Successful Switch Test Results 153 FIGURE 11-9 Multipath Recovery using the Sun StorEdge T3+ Array Multipath Configurator 154 FIGURE 11-10 Recovered Paths 154[...]

XII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

List of Tables XIII Sun Proprietary/Confidential: Internal Use Only List of T ables T ABLE 1-1 Sun StorEdge 3900 and 6900 Series Configurations 1 T ABLE 3-1 Event Grid Sorting Criteria 25 T ABLE 5-1 FC Links 38 T ABLE 5-2 Ax to Bx FC Links. 40 T ABLE 6-1 Storage Automated Diagnostic Environment Event Grid for the Host 69 T ABLE 7-1 Storage Automate[...]

XIV Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE A-5 Virtualization Engine Service Codes —400-599 Device-Side Interface Driver Errors 162 T ABLE B-1 Virtualization Engine Error Messages 164 T ABLE B-2 Sun StorEdge Network FC Switch Error Messages 168 T ABLE B-3 S[...]

XV Sun Proprietary/Confidential: Internal Use Only Pr eface The Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide provides guidelines for isolating problems in supported configurations of the Sun StorEdge TM 3900 and 6900 series. For detailed configuration information, refer to the Sun StorEdge 3900 and 6900 Series Reference Manual . The[...]

XVI Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Chapter 6 provides information on host device troubleshooting. Chapter 7 provides information on troubleshooting a Sun StorEdge Network FC switch-8 and switch-16 switch device. Chapter 8 describes how to troubleshoot the Su[...]

Preface XVII Sun Proprietary/Confidential: Internal Use Only T ypographic Conventions Shell Pr ompts T ypeface Meaning Examples AaBbCc123 The names of commands, files, and directories; on-scr een computer output Edit your .login file. Use ls -a to list all files. % You have mail . AaBbCc123 What you type, when contrasted with on-screen computer out[...]

XVIII Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Related Documentation Product Title Part Number Late-breaking News • Sun StorEdge 3900 and 6900 Series 2.0 Release Notes 816-5254 Sun StorEdge 3900 and 6900 series information • Sun StorEdge 3900 and 6900 Series 2.0 I[...]

Preface XIX Sun Proprietary/Confidential: Internal Use Only SANbox-8/16 Segmented Loop FC Switch • SANbox-8/16 Segmented Loop Fibre Channel Switch Management User ’ s Manual • SANbox-8 Segmented Loop Fibre Channel Switch Installer’ s/User ’ s Manual • SANbox-16 Segmented Loop Fibre Channel Switch Installer’ s/User ’ s Manual 875-306[...]

XX Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Accessing Sun Documentation Online Y ou can view , print, or purchase a broad selection of Sun documentation, including localized versions, at: http://www.sun.com/documentation Sun W elcomes Y our Comments Sun is interested [...]

1 CHAPTER Sun Proprietary/Confidential: Internal Use Only 1 Intr oduction The Sun StorEdge 3900 and 6900 series storage subsystems are complete preconf igured storage solutions. The configurations for each of the storage subsystems are shown in T ABLE 1-1 . T ABLE 1-1 Sun StorEdge 3900 and 6900 Series Configurations Series System Sun StorEdge Fibre[...]

2 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Pr edictive Failur e Analysis (PF A) Capabilities The Storage Automated Diagnostic Environment software pr ovides the health and monitoring functions for the Sun StorEdge 3900 and 6900 series systems. This software pr ovides[...]

3 CHAPTER Sun Proprietary/Confidential: Internal Use Only 2 General T r oubleshooting Pr ocedur es This chapter contains the following sections: ■ “High-Level T roubleshooting T asks” on page 3 ■ “Host-Side T roubleshooting” on page 6 ■ “Storage Service Processor -Side T roubleshooting” on page 6 ■ “V erifying the Configuratio[...]

4 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 1. Discover the error by checking one or more of the following messages or f iles: ■ Storage Automated Diagnostic Environment alerts or email messages ■ /var/adm/messages ■ Sun StorEdge T3+ array syslog file ■ Storag[...]

Chapter 2 General T roubleshooting Procedures 5 Sun Proprietary/Confidential: Internal Use Only 4. Check the status of the Sun StorEdge network FC switch-8 and switch-16 switches using the following tools: ■ Review the Storage Automated Diagnostic Environment device monitoring reports. ■ Run the checkswitch (1M) and showswitch (1M) commands, wh[...]

6 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Note – These tests isolate the problem to a FRU that must be r eplaced. Follow the instructions in the Sun StorEdge 3900 and 6900 Series 2.0 Reference and Service Guide and the Sun StorEdge 3900 and 6900 Series 2.0 Install[...]

Chapter 2 General T roubleshooting Procedures 7 Sun Proprietary/Confidential: Internal Use Only V erifying the Conf iguration Settings During the course of troubleshooting, you might need to verify conf iguration settings on the various components in the Sun StorEdge 3900 or 6900 series. ▼ T o V erify Configuration Settings 1. Run one of the foll[...]

8 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only CODE EXAMPLE 2-1 checkdefaultconfig (1M) Output # /opt/SUNWsecfg/checkdefaultconfig Checking all accessible components..... Checking switch: sw1a Switch sw1a - PASSED Checking switch: sw1b Switch sw1b - PASSED Checking switc[...]

Chapter 2 General T roubleshooting Procedures 9 Sun Proprietary/Confidential: Internal Use Only 2. If anything is marked F AIL, check the /var/adm/log/SEcfglog f ile for the details of the failure. In this example, the mirror setting in the Sun StorEdge T3+ array system settings is “off.” The saved conf iguration setting for this parameter , wh[...]

10 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Clearing the Lock File If you interrupt any of the Configuration Utility scripts (by typing Control-C , for example), a lock file might r emain in the /opt/SUNWsecfg/etc directory , causing subsequent commands to fail. Use [...]

Chapter 2 General T roubleshooting Procedures 11 Sun Proprietary/Confidential: Internal Use Only Sun StorEdge 6900 Series Multipathing Example This Sun StorEdge 6900 series multipathing example contains the following elements: ■ One Sun StorEdge T3+ array partner group ■ T wo total LUNs ■ One 500-Gbyte RAID5 LUN per partner group See FIGURE 2[...]

12 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Currently , one 10-Gbyte VLUN is created fr om each physical LUN, for a total of two VLUNs. The Sun StorEdge 6900 series has four possible physical paths to each Sun StorEdge T3+ array volume (LUN). Refer to FIGURE 2-2 , wh[...]

Chapter 2 General T roubleshooting Procedures 13 Sun Proprietary/Confidential: Internal Use Only FIGURE 2-3 Primary Data Paths to the Master Sun StorEdge T3+ Array T o access the LUN on the alternate master , the Sun StorEdge T3+ array I/O could travel: ■ From HBA-0 -> switch -> virtualization engine(1) -> switch -> alternate master c[...]

14 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only The host, using multipathing software, is presented with two primary (active) paths for each LUN, allowing the host to route I/O thr ough either or both HBAs. If a path failure occurs befor e the second tier of Sun StorEdge[...]

Chapter 2 General T roubleshooting Procedures 15 Sun Proprietary/Confidential: Internal Use Only The virtualization engine recognizes the primary (active) and secondary (passive) pathing for the LUNs, and routes the I/O to the primary contr oller—unless there is a path failure to the primary path. In that case, the virtualization engine initiates[...]

16 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Multipathing Options in the Sun StorEdge 6900 Series The presence of the virtualization engine makes multipathing in a Sun StorEdge 6900 series environment challenging. Unlike Sun StorEdge T3+ array and Sun StorEdge network[...]

Chapter 2 General T roubleshooting Procedures 17 Sun Proprietary/Confidential: Internal Use Only Note that in the Class and State fields, the virtualization engines ar e presented as two primary ONLINE devices. The current Sun StorEdge T raffic Manager software design does not enable you to manually halt the I/O (that is, you cannot perform a failo[...]

18 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 2. Using the Storage Automated Diagnostic Environment GUI T opology , determine which virtualization engine is in the path you need to disable. 3. Use the worldwide name (WWN) of the virtualization engine that is in the unc[...]

Chapter 2 General T roubleshooting Procedures 19 Sun Proprietary/Confidential: Internal Use Only Note – T o confirm that a failover is occurring, open a T elnet session to the Sun StorEdge T3+ array and check the output of port listmap . Another , but slower , method is to run the runsecfg script and verify the virtualization engine maps by polli[...]

20 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o V iew the Dynamic Multi-Pathing (DMP) Pr operties 1. T ype: The vxdisk output includes two physical paths to the LUN: ■ c20t2B000060220041F4d0s2 ■ c23t2B000060220041F9d0s2 Both of these paths are curr ently enab[...]

Chapter 2 General T roubleshooting Procedures 21 Sun Proprietary/Confidential: Internal Use Only 2. Use the luxadm (1M) command to display further information about the underlying LUN. # /usr/sbin/luxadm display /dev/rdsk/c20t2B000060220041F4d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c20t2B000060220041F4d0s2 Status(Port A): O.K. Vendor: SUN Product[...]

22 Sun StorEdge 3900 and 6900 2.0 Series T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Put the DMP -Enabled Paths Back into Pr oduction 1. T ype: 2. V erify that the path has been reenabled by typing: # vxdmpadm enable ctlr=<c n > # vxdmpadm listctlr all[...]

23 CHAPTER Sun Proprietary/Confidential: Internal Use Only 3 T r oubleshooting T ools This chapter contains the following information related to tools used to troubleshoot the Sun StorEdge 3900 or 6900 series components. ■ “Storage Automated Diagnostic Environment 2.2” on page 23 ■ “Microsoft W indows 2000 System Errors” on page 26 ■ [...]

24 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Example T opology In the Storage Automated Diagnostic Environment topology shown in FIGURE 3-1 , the internel components of a Sun StorEdge 3910 system are shown. There is also a Solaris host (diag221) and the Storage Servic[...]

Chapter 3 T roubleshooting T ools 25 Sun Proprietary/Confidential: Internal Use Only Generating Component-Specif ic Event Grids The Storage Automated Diagnostic Environment generates component-specif ic event grids that describe the severity of an event, tell whether action is required, pr ovide a description of the event, and recommended action. R[...]

26 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Micr osoft W indows 2000 System Err ors Y ou can view Microsoft W indows 2000 errors thr ough the Event Properties System Log. The types of errors that would indicate a Sun StorEdge T3+ Array Failover Driver issue have the [...]

Chapter 3 T roubleshooting T ools 27 Sun Proprietary/Confidential: Internal Use Only Command Line T est Examples T o run a single Sun StorEdge diagnostic test from the command line rather than through the Storage Automated Diagnostic Environment interface, you must log in to the appropriate host or slave for testing the components. The following tw[...]

28 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only switchtest (1M) switchtest (1M) diagnoses the Sun StorEdge network FC switch-8 and switch-16 switch devices. The switchtest process also pr ovides command-line access to switch diagnostics. switchtest supports testing on lo[...]

Chapter 3 T roubleshooting T ools 29 Sun Proprietary/Confidential: Internal Use Only Monitoring Sun StorEdge T3 and T3+ Arrays Using the Explor er Data Collection Utility The Explorer Data Collection Utility script is included on the Storage Service Processor in the /export/packages directory . The Explorer Data Collection Utility is not installed [...]

30 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 3. Before running the Explorer Data Collection Utility , make sure that the switch and Sun StorEdge T3+ array information is added to the proper /opt/SUNWexplo/etc f iles. Example T ype switch information in the /opt/SUNWex[...]

Chapter 3 T roubleshooting T ools 31 Sun Proprietary/Confidential: Internal Use Only ■ Y ou can now run /opt/SUNWexplo/bin/explorer for information about the Storage Service Processor operating system, the Sun StorEdge network FC switch- 8 or switch-16 switch, and Sun StorEdge T3+ array information that you can use for troubleshooting purposes. ?[...]

32 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Monitoring Host Bus Adapters (HBAs) Using QLogic SANblade Manager The most effective way to r etrieve HBA status and information is by using the HBA manufacturer ’s utility , such as the Qlogic SANblade Manager software p[...]

Chapter 3 T roubleshooting T ools 33 Sun Proprietary/Confidential: Internal Use Only FIGURE 3-3 Qlogic SANblade Manager HBA Driver and Firmwar e V ersions[...]

34 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only QLogic SANblade Manager is also useful for viewing a primitive topology and a LUN listing. FIGURE 3-4 QLogic SANblade Manager Diagnostics Note – Differing HBA manufactur er ’s may bundle different featur es with their t[...]

35 CHAPTER Sun Proprietary/Confidential: Internal Use Only 4 T r oubleshooting Ethernet Hubs The Sun StorEdge 3900 and 6900 series uses an Ethernet hub as the backbone for the internal service network. The allocation of Ethernet ports is as follows: ■ One for the Storage Service Processor (per subsystem) ■ One for each FC switch ■ One for eac[...]

36 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

37 CHAPTER Sun Proprietary/Confidential: Internal Use Only 5 T r oubleshooting the Fibr e Channel (FC) Links FC links diagnose Sun StorEdge network FC components in a SAN or a direct attached storage (DAS) environment. linktest (1M), which tests the health of the FC links, is available only from the T est from T opology view of the Storage Automate[...]

38 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FC Links The following sections provide troubleshooting information for the basic components and FC links, listed in T ABLE 5-1 . By using the Storage Automated Diagnostic Environment, you should be able to isolate the prob[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 39 Sun Proprietary/Confidential: Internal Use Only FC Link Diagrams FIGURE 5-1 shows the basic components and the FC links for a Sun StorEdge 3900 series system: ■ A1 to B1 —HBA to Sun StorEdge network FC switch-8 and switch-16 switch link ■ A4 to B4 —Sun StorEdge network FC switch-8 a[...]

40 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE 5-2 and FIGURE 5-2 shows the basic components and the FC links for a Sun StorEdge 6900 series system: T ABLE 5-2 A x to B x FC Links. Link Pro vides FC Link Between These Components A1 to B1 HBA to Sun StorEdge netwo[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 41 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-2 Sun StorEdge 6900 Series FC Link Diagram HOST HBA-A HBA-B A1 B1 sw1a sw1b A2 B2 v1a v1b A3 B3 T1 T2 A4 sw2b sw2a T3+ alternate master T3+ Master B4[...]

42 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting the A1 or B1 FC Link The A1 or B1 link is the FC link from the HBA to the switch. What happens when a FC link fails depends on the system. If a problem occurs with the A1 or B1 FC link: ■ In a Sun StorEd[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 43 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-3 , FIGURE 5-4 , and FIGURE 5-5 are examples of A1 or B1 link notification events. FIGURE 5-3 Data Host Notif ication of Intermittent Problems FIGURE 5-4 Data Host Notif ication of Severe Link Err or Site : FSDE LAB Broomfield CO Sour[...]

44 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-5 Storage Service Pr ocessor Notification Note – An A1 or B1 FC link error can cause a port in sw1a or sw1b to change state. Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : [...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 45 Sun Proprietary/Confidential: Internal Use Only V erifying the Data Host The following example shows an error in the A1 or B1 FC link, which can cause a path to go off line in the multipathing software. CODE EXAMPLE 5-1 luxadm (1M) Display # /usr/sbin/luxadm display /dev/rdsk/c6t29000060220[...]

46 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only An error in the A1 or B1 FC link can also cause a device to enter the “unusable” state in cfgadm -al , as shown in CODE EXAMPLE 5-2 . CODE EXAMPLE 5-2 cfgadm -al Display FRU T ests A vailable for the A1 or B1 FC Link Se[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 47 Sun Proprietary/Confidential: Internal Use Only CODE EXAMPLE 5-3 switchtest (1M) Called W ith Options Note – The Storage Automated Diagnostic Environment automatically resets the transfer size if it notes that it is about to test a switch to the HBA connection. This is done both in the St[...]

48 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Isolate the A1 or B1 FC Link T o isolate the A1 or B1 link, which is the FC link from the HBA to the switch, follow these steps: 1. Quiesce the I/O on the A1 or B1 FC link path. 2. Run switchtest (1M) or qlctest (1M[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 49 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting the A2 or B2 FC Link The A2 or B2 link is the FC link from the f irst switch to the virtualization engine. This link exists in the Sun StorEdge 6900 Series only . An error with the FC link can cause a path to go offline. FIGU[...]

50 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-7 A2 or B2 FC Link Storage Service Processor -Side Event Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch Key: switch:100000c0dd0061bb EventType: StateChangeEvent.X.port[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 51 Sun Proprietary/Confidential: Internal Use Only V erifying the Data Host An error in the A2 or B2 FC link can result in a device being listed as in an “unusable” state in cfgadm , but no HBAs being listed in the “unconnected” state in the luxadm output. The multipathing software wil[...]

52 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V erifying the A2 or B2 FC Link Y ou can check the A2 or B2 FC link using the Storage Automated Diagnostic Environment, Diagnose—T est from T opology functionality . The Storage Automated Diagnostic Environment’s implem[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 53 Sun Proprietary/Confidential: Internal Use Only 5. If the switch and the GBIC show no errors, replace the remaining components in the following order: a. Replace the virtualization engine-side GBIC, recable the link, and monitor the link for errors. b. Replace the cable, recable the link, a[...]

54 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting the A3 or B3 FC Link The A3 or B3 link is the FC link from the virtualization engine to the backend switch. The A3 or B3 FC link exists in a Sun StorEdge 6900 Series only . An error with the FC link can ca[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 55 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-9 A3 or B3 FC Link Storage Service Processor -Side Event FIGURE 5-10 A3 or B3 FC Link Storage Service Processor-Side Event Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch Key: switch:10000[...]

56 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V erifying the Data Host An error in the A3 or B3 FC link r esults in a device being listed as in an “unusable” state in cfgadm , but no HBAs are listed as in the “unconnected” state in luxadm output. The multipathi[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 57 Sun Proprietary/Confidential: Internal Use Only CODE EXAMPLE 5-6 DMP Error Message V erifying the Storage Service Processor -Side Y ou can check the A3 or B3 FC link using the Storage Automated Diagnostic Environment’s T est from T opology functionality . The Storage Automated Diagnostic [...]

58 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Isolate the A3 or B3 FC Link T o isolate the A3 or B3 link, which is the FC link from the virtualization engine to the back-end switch, follow these steps: Note – The A3 or B3 FC link exists in a Sun StorEdge 6900[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 59 Sun Proprietary/Confidential: Internal Use Only Quiescing the I/O on the A3 or B3 Link 1. Determine the path you want to disable. 2. Disable the path by typing the following: 3. V erify that the path is disabled: Steps 1 and 2 halt I/O only up to the A3 to B3 link. I/O continues to move ove[...]

60 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting the A4 or B4 FC Link The A4 or B4 link is the FC link from the switch to the Sun StorEdge T3+ array . If a problem occurs with the A4 or B4 FC link: ■ In a Sun StorEdge 3900 series system, the Sun StorEd[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 61 Sun Proprietary/Confidential: Internal Use Only FIGURE 5-12 Storage Service Processor-Side Notif ication Site : FSDE LAB Broomfield CO Source : diag Severity : Warning Category : Switch DeviceId : switch:100000c0dd0061bb EventType: LogEvent.MessageLog EventTime: 01/29/2002 14:25:05 Change i[...]

62 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V erifying the Data Host A problem in the A4 or B4 FC Link appears dif ferently on the data host, depending on whether the array is a Sun StorEdge 3900 series or a Sun StorEdge 6900 series device. Sun StorEdge 3900 Series I[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 63 Sun Proprietary/Confidential: Internal Use Only T o verify that the failover luxadm display can be used, the failed path is marked “off line,” as shown in CODE EXAMPLE 5-7 . CODE EXAMPLE 5-7 Failed Path Marked Off line Note – This type of error may also cause the device to show up as [...]

64 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only CODE EXAMPLE 5-8 Failed Path Marked Unusable FRU T ests A vailable for the A4 or B4 FC Link Segment ■ The switchtest can only be run from the Storage Service Pr ocessor . ■ The linktest can isolate the switch and the GB[...]

Chapter 5 T roubleshooting the Fibre Channel (FC) Links 65 Sun Proprietary/Confidential: Internal Use Only 5. Rerun switchtest . a. If switchtest fails, replace the GBIC and rerun switchtest . b. If the test fails again, replace the switch. 6. If switchtest passes, assume that the suspect components are the cable and the Sun StorEdge T3+ array cont[...]

66 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

67 CHAPTER Sun Proprietary/Confidential: Internal Use Only 6 T r oubleshooting Host Devices This chapter describes how to tr oubleshoot components associated with a Sun StorEdge 3900 or 6900 series host. This chapter contains the following sections: ■ “T o Access the Host Event Grid” on page 67 ■ “T o Replace the Master Host” on page 71[...]

68 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 6-1 Sample Host Event Grid[...]

Chapter 6 T roubleshooting Host De vices 69 Sun Proprietary/Confidential: Internal Use Only T ABLE 6-1 lists all the host events in the Storage Automated Diagnostic Environment. T ABLE 6-1 Storage Automated Diagnostic Environment Event Grid for the Host Component EventT ype Severity Action Description Information HBA Alarm+ Y ellow The status of hb[...]

70 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Component EventT ype Severity Action Description Information ifptest Diagnostic T est- Red Y ifptest (diag240) on the host failed. Check T est Manager for failure details. qlctest Diagnostic T est- Red qlctest (diag240) on [...]

Chapter 6 T roubleshooting Host De vices 71 Sun Proprietary/Confidential: Internal Use Only Replacing the Master , Alternate Master , and Slave Monitoring Host The following procedur es are a high-level overview of the procedures that ar e detailed in the Storage Automated Diagnostic Environment User ’ s Guide . Follow these procedur es when repl[...]

72 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 7. Choose Maintenance -> General Maintenance -> Start/Stop Agent to start the agent on the master host. ▼ T o Replace the Alternate Master or Slave Monitoring Host 1. Choose Maintenance -> General Maintenance -&g[...]

73 CHAPTER Sun Proprietary/Confidential: Internal Use Only 7 T r oubleshooting Switches This chapter describes how to troubleshoot the 1 Gbit and 2 Gbit switch components associated with a Sun StorEdge 3900 or 6900 series system. This chapter contains the following sections: ■ “About the Switches” on page 73 ■ “Using the Switch Event Grid[...]

74 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only The Sun StorEdge network FC switches in a Sun StorEdge 3900 or 6900 configuration now support the Sun StorEdge SAN 4.1 Release. Y ou can upgrade the switches to support the 402 xx 2 Gbit-compatible firmware. Caution – Use[...]

Chapter 7 T roubleshooting Switches 75 Sun Proprietary/Confidential: Internal Use Only Switchless Conf igurations In a switchless configuration (Sun StorEdge 3900SL, 6910SL, or 6960SL series system) you can upgrade the switches that are connected to the Solaris server to the Sun StorEdge SAN 4.1 Release firmwar e. For a list of the supported switch[...]

76 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 4. T o restore the conf iguration from the saved map f ile back to the default switch conf iguration, type: For detailed diagnostic and troubleshooting procedur es for the Sun StorEdge network FC switch-8 and switch-16 swit[...]

Chapter 7 T roubleshooting Switches 77 Sun Proprietary/Confidential: Internal Use Only Using the Switch Event Grid The Storage Automated Diagnostic Environment Switch Event Grid enables you to sort switch events by component, category , or event type. The Storage Automated Diagnostic Environment GUI displays an event grid that describes the severit[...]

78 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE 7-1 lists the switch events for Sun StorEdge network FC switch-8 and switch- 16 1 Gbit switches. T ABLE 7-1 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches Component EventT ype Severity Action[...]

Chapter 7 T roubleshooting Switches 79 Sun Proprietary/Confidential: Internal Use Only chassis. zone Alarm Y ellow “Switch sw1a was rezoned” This event reports changes in the zoning of a switch. enclosure Audit “Auditing a new switch called ras d2-swb1 (ip=xxx.0.0.41) 10002000007a609” oob Comm_ Established “Communication regained with sw1[...]

80 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only enclosure Discovery “Discovered a new switch called ras d2-swb1 ( ip=xxx.0.0.41 ) 10002000007a609” Discovery events occur the very first time the agent probes a storage device. It creates a detailed description of the d[...]

Chapter 7 T roubleshooting Switches 81 Sun Proprietary/Confidential: Internal Use Only port State Change+ “port.1 in SWITCH diag185 ( ip= xxx.20.67.185 ) is now A vailable (status-state changed from of fline to online)” The port on the switch is now available. port State Change- Red Y “port.1 in SWITCH diag185 ( ip= xxx .20.67.185 )i s now No[...]

82 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE 7-2 lists the switch events for Sun StorEdge network FC switch-8 and switch- 16 2 Gbit switches. T ABLE 7-2 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches Component EventT ype Severity Action[...]

Chapter 7 T roubleshooting Switches 83 Sun Proprietary/Confidential: Internal Use Only oob Comm_ Lost Down Y “Lost communication with sw1a ( ip=xxx.20.67.213 )” Ethernet connectivity to the switch has been lost. 1. Check Ethernet connectivity to the switch. 2. V erify that the switch is booted correctly with no POST errors. 3. V erify that the [...]

84 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only port State Change+ “port.1 in SWITCH diag185 ( ip= xxx.20.67.185 ) is now A vailable (status-state changed from of fline to online)” The port on the switch is now available. port State Change- Red Y A port on switch2 ha[...]

Chapter 7 T roubleshooting Switches 85 Sun Proprietary/Confidential: Internal Use Only setupswitch Exit V alues T ABLE 0-1 lists the setupswitch exit values. The associated messages are logged in the /var/adm/log/SEcfglog file. T ABLE 0-1 setupswitch Exit V alues Severity Level Message T ype Message Meaning 0 INFO All switch settings are pr operly [...]

86 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Note – If multiple systems are connected to a switch, the switch settings might not match the default settings.[...]

87 CHAPTER Sun Proprietary/Confidential: Internal Use Only 8 T r oubleshooting the Sun StorEdge T3+ Array Devices The Sun StorEdge T3+ array is a high-performance, modular , scalable storage device that contains an internal RAID controller and disk drives with FC connectivity to the data host. In the Sun StorEdge 3900 and 6900 series, the Sun StorE[...]

88 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting the T1 or T2 Data Path When you are tr oubleshooting the T1 or T2 data path, note the following: ■ T wo T port links provide r edundancy . ■ If one of the two links is lost, no Sun StorEdge T3+ array L[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 89 Sun Proprietary/Confidential: Internal Use Only Notif ication Events FIGURE 8-1 shows a typical port failure event . FIGURE 8-1 Storage Service Processor Event Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Error (Actionable) Category : Switch DeviceId : s[...]

90 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only If both T ports go off line, you might see a message like the following. The virtualization engine event is alerting the LUN failover . FIGURE 8-2 V irtualization Engine Alert Site : Lab 3286 - DSQA1 Broomfield Source : dia[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 91 Sun Proprietary/Confidential: Internal Use Only ...continued from previous page... ---------------------------------------------------------------------- Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Warning Category : Message DeviceId : message:diag.xxxx[...]

92 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o V erify the Storage Service Processor 1. Run the Sun StorEdge T3+ array port listmap command to see the failover event. 2. Compare the virtualization engine conf iguration to a saved conf iguration by running runsec[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 93 Sun Proprietary/Confidential: Internal Use Only FRU T ests A vailable for the T1 or T2 Data Path FRU Running the tests from the Storage Automated Diagnostic Environment GUI guides you in discovering the failed FRU. Refer to Chapter 5 of the Storage Automated Diagnostic Environment U[...]

94 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ■ When you insert a loopback connector in to the T port, no green light appears to indicate a proper insertion. However , the test will run and be valid. ■ If only one of the links has failed and the I/O is traveling ov[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 95 Sun Proprietary/Confidential: Internal Use Only Sun StorEdge T3+ Array Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort Sun StorEdge T3+ array events by component, category , or event type. The Storage Automated Diagnostic Environment GUI displa[...]

96 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE 8-1 lists all of the events for the Sun StorEdge T3+ array . T ABLE 8-1 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3+ Array Component Event T ype Severity Action Description Action powe[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 97 Sun Proprietary/Confidential: Internal Use Only interface. loopcard.cable Alarm- Red Y The Sun StorEdge T3+ array has reported that a loopcard is in a failed state. Possible Drive Status Messages: V alue Description 0 Drive mounted 2 Drive present 3 Drive is spun up 4 Drive is disab[...]

98 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only power .fan Alarm- Red Y The state of a fan on the Sun StorEdge T3+ array is not optimal. 1. Open a T elnet session to the affected Sun StorEdge T3+ array . 2. V erify the fan state with fru stat. 3. Replace the power coolin[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 99 Sun Proprietary/Confidential: Internal Use Only enclosure Audit Auditing a new Sun StorEdge T3+ array Audits occur every week. The Storage Automated Diagnostic Environment sends a detailed description of the enclosure to the Sun Network Storage Command Center (NSCC). ib Comm_ Establ[...]

100 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only oob Comm_Lost Down Y OutOfBand (oob) means that the Sun StorEdge T3+ array failed to answer to a ping or failed to return its tokens. This OutOfBand problem can be caused by a very slow network, or because the Ethernet con[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 101 Sun Proprietary/Confidential: Internal Use Only enclosure Discovery The Storage Automated Diagnostic Environment discovered a new Sun StorEdge T3+ array Discovery events occur the first time the Storage Automated Diagnostic Environment pr obes a storage device. The Discovery event [...]

102 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only enclosure QuiesceStart Quiesce has started on a Sun StorEdge T3+ array . controller T opology- Red Y ’The Sun StorEdge T3+ array has reported that a controller was r emoved from the chassis. Replace the controller within[...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 103 Sun Proprietary/Confidential: Internal Use Only power State Change+ The status of the PCU has changed from r eady- disable to ready-enable. controller State Change+ disk State Change+ interface. loopcard State Change+ volume State Change+ power State Change+ The Sun StorEdge T3+ ar[...]

104 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only disk State Change- Red Y The Sun StorEdge T3+ array has reported that a disk has failed. 1. Open a T elnet session to the affected Sun StorEdge T3+ array . 2. V erify the disk state with vol_stat , fru_stat , and fru_list [...]

Chapter 8 T roubleshooting the Sun StorEdge T3+ Arra y De vices 105 Sun Proprietary/Confidential: Internal Use Only volume State Change- Red Y 1. Open a T elnet session to the affected Sun StorEdge T3+ array . 2. V erify the status of the LUNs with vol_mode or vol _stat . Drive Status Messages: 0 Drive mounted 2 Drive present 3 Drive is spun up 4 D[...]

106 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

107 CHAPTER Sun Proprietary/Confidential: Internal Use Only 9 T r oubleshooting V irtualization Engine Devices This chapter describes how to troubleshoot the virtualization engine component of a Sun StorEdge 6900 series system. This chapter contains the following sections: ■ “About the V irtualization Engine” on page 107 ■ “V irtualizatio[...]

108 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V irtualization Engine Diagnostics The virtualization engine monitors the following components: ■ V irtualization engine router ■ Sun StorEdge T3+ array ■ Cabling between the router and the storage Service Request Nu[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 109 Sun Proprietary/Confidential: Internal Use Only Err or Log Analysis Commands ▼ T o Display the Log Files and Retrieve SRNs ● T ype Errors that need action ar e returned in the following format: A description of the errors follows. # /opt/svengine/sduc/sreadlog TimeStamp: nnn :T xxxxx.[...]

110 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Example ▼ T o Clear the Log ● T ype V irtualization Engine LEDs T ABLE 9-1 describes the LEDs on the back of the virtualization engine. # /opt/svengine/sduc/sreadlog -d v1 2002:Jan:3:10:13:05:v1.29000060-220041F9.SRN=7[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 111 Sun Proprietary/Confidential: Internal Use Only Power LED Codes The virtualization engine LEDs are shown in FIGURE 9-1 . FIGURE 9-1 V irtualization Engine Front Panel LEDs Interpr eting LED Service and Diagnostic Codes The Status LED communicates the status of the virtualization engine in[...]

112 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Back Panel Featur es The back panel of the virtualization engine contains the Sun StorEdge network FC switch-8 or switch-16 switches, a socket for the AC power input, and various data ports and LEDs. FIGURE 9-2 V irtualiza[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 113 Sun Proprietary/Confidential: Internal Use Only FC Link Err or Status Report The virtualization engine’s host-side and device-side interfaces provide statistical data for the counts listed in T ABLE 9-4 . ▼ T o Check the FC Link Err or Status Manually The Storage Automated Diagnostic [...]

114 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Note – If the t3ofdg (1M) is running while you perform these steps, the following error message is displayed: Daemon error: check the SLIC router. CODE EXAMPLE 9-1 FC Link Error Status Example Note – v1 repr esents the[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 115 Sun Proprietary/Confidential: Internal Use Only Note – The Serial Loop IntraConnect ( SLIC ) daemon must be running for the svstat (1M) -d v1 command to work. T ranslating Host-Device Names Y ou can translate host-device names to VLUN, disk pool, and physical Sun StorEdge T3+ array LUNs[...]

116 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Displaying the VLUN Serial Number ▼ T o Display Devices That ar e Not Sun StorEdge T raf fic Manager (MP x IO)-Enabled 1. Use the format -e command. 2. T ype the number of the disk on which you are working at the format [...]

Chapter 9 T roubleshooting Virtualization Engine Devices 117 Sun Proprietary/Confidential: Internal Use Only ▼ T o Display Sun StorEdge T raf fic Manager (MPxIO)-Enabled Devices If the devices support the Sun StorEdge T raff ic Manager software, you can use this shortcut. ● T ype: The /dev/rdsk/c n t n repr esents the Global Unique Identifier o[...]

118 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V iewing the V irtualization Engine Map The virtualization engine map is stored on the Storage Service Processor . 1. T o view the virtualization engine map, type: Note – This example uses the virtualization engine map f[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 119 Sun Proprietary/Confidential: Internal Use Only 2. Optionally open a T elnet session to the virtualization engine and run the runsecfg utility to poll a live snapshot of the virtualization engine map. Refer to “T o Failback the V irtualization Engine” on page 120 for instructions abou[...]

120 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Failback the V irtualization Engine In the event of a Sun StorEdge T3+ array LUN failover , the virtualization engine will route all I/O thr ough the failover port on the Sun StorEdge T3+ array . After you isolate [...]

Chapter 9 T roubleshooting Virtualization Engine Devices 121 Sun Proprietary/Confidential: Internal Use Only a. If no failures occur , the command exits with no output. b. If failures occur , you might see one of the following messages: CODE EXAMPLE 9-5 Sun StorEdge T3+ Array Failure Codes The message return code 513 indicates that the Sun StorEdge[...]

122 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only CODE EXAMPLE 9-6 Error -Free Online Switch Ports # showswitch -s sw2a ... ************ Port Status ************ Port # Port Type Admin State Oper State Status Loop Mode ------ --------- ----------- ---------- ------ - ----[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 123 Sun Proprietary/Confidential: Internal Use Only 6. If either port 1 or port 2 is off line, check the GBICs and cables. 7. If a Sun StorEdge T3+ array switch port is off line, log in to the Sun StorEdge T3+ array and look at the status of the controllers and the port list, as shown in CODE[...]

124 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Reset the SAN Database on Both V irtualization Engines 1. T ype: Y ou do not need to manually open a T elnet session to the virtualization engines, unless an ERROR HAL T 50 state is detected. Although you might nee[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 125 Sun Proprietary/Confidential: Internal Use Only ▼ T o Reset the SAN Database on a Single V irtualization Engine 1. T o disconnect the virtualization engine’ s device-side FC cables, type: 2. Open a T elnet session to the virtualization engine specif ied in Step 1. 3. Enter the passwor[...]

126 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Restarting the slicd Daemon Follow this procedur e to restart the slicd daemon if the SLIC daemon becomes unresponsive, or if a message similar to the following is displayed: connect: Connection refused or Socket error enc[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 127 Sun Proprietary/Confidential: Internal Use Only 4. T o restart the slicd for the v1 virtualization engine, type : 5. Conf irm that the slicd daemon is running: If the slicd daemon is running, it resets the virtualization engine. If the pr ocess fails, the slicd daemon changes the IP addre[...]

128 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Product Type : FC-FC-3 SVE H FC-FC-3 router H Firmware Revision : 8.017 Official Release Vicom(release), Apr 11 2002 17:49:16 Loader Revision : 2.02.42 Unique ID : 00000060-2200418A Unit Serial Number : 00250339 PCB Number[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 129 Sun Proprietary/Confidential: Internal Use Only Diagnosing a creatediskpools (1M) Failur e When modifying the Sun StorEdge T3+ array configuration on a Sun StorEdge 6900 series, the system should automatically create disk pools. If the virtualization engine cannot find two paths to all Su[...]

130 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 2. Run the showswitch (1M) command for sw2a and sw2b . Refer to the Sun StorEdge 3900 and 6900 Series 2.0 Reference and Service Manual to see to which switch ports the Sun StorEdge T3+ array and virtualization engine shoul[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 131 Sun Proprietary/Confidential: Internal Use Only 3. After corrective action has been successfully completed, run the following command: The SEcfglog file should display the following message: # creatediskpools -n t3b0 Thu May 30 17:40:23 MDT 2002 creatediskpools: t3b0 ENTER: /opt/SUNWsecfg[...]

132 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V irtualization Engine Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort virtualization engine events by component, category , or event type. The Storage Automated Diagnostic Environmen[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 133 Sun Proprietary/Confidential: Internal Use Only T ABLE 9-5 lists the V irtualization Engine Events. T ABLE 9-5 Storage Automated Diagnostic Environment Event Grid for V irtualization Engine Component EventT ype Severity Information Required Action volume Alarm Y ellow This event occurs wh[...]

134 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only oob.slicd Comm_ Lost Down The virtualization engine failed to execute slicd command. 1. Check the status of the slicd daemon. 2. Check the power on the virtualization engine. 3. Make sure the virtualization engine is boote[...]

Chapter 9 T roubleshooting Virtualization Engine Devices 135 Sun Proprietary/Confidential: Internal Use Only ve_diag Diagnostic T est- Red The ve_diag test on ve-1 failed veluntest Diagnostic T est- Red The veluntest failed enclosure Discovery The discovery device found a new virtualization engine called v1a . Discovery events occur the first time [...]

136 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

137 CHAPTER Sun Proprietary/Confidential: Internal Use Only 10 T r oubleshooting Using Micr osoft W indows 2000 General Notes ■ Use the Manufacturer ’s HBA Utilities to monitor and diagnose the HBAs. The examples in this chapter use Qlogic’s SANblade Manager utility . ■ The Storage Automated Diagnostic Environment running on the Storage Ser[...]

138 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T r oubleshooting T asks Using Microsoft W indows 2000 Launching the Sun StorEdge T3+ Array Failover Driver GUI ● From the Microsoft W indows 2000 Advanced Server GUI, click Programs -> T3 StorEdge Conf igurator ->[...]

Chapter 10 T roubleshooting Using Microsoft Windo ws 2000 139 Sun Proprietary/Confidential: Internal Use Only Checking the V ersion of the Sun StorEdge T3+ Array Failover Driver ● From the Microsoft W indows 2000 Advanced Server GUI, click Help -> About. The About Multipath Configurator window is displayed. FIGURE 10-2 Sun StorEdge T3+ Array F[...]

140 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only ▼ T o Use the Sun StorEdge T3+ Array Failover Driver GUI Note – The Sun StorEdge T3+ Array Failover Driver GUI is limited to the Sun StorEdge 3900 series systems. Y ou must use the CLI for the Sun StorEdge 6900 series [...]

Chapter 10 T roubleshooting Using Microsoft Windo ws 2000 141 Sun Proprietary/Confidential: Internal Use Only 4. Compare the healthy Sun StorEdge 3900 series system to a system that has experienced a LUN failover . A system that has experienced a LUN failover has a broken line connecting the HBA to the storage, as shown in FIGURE 10-4 . FIGURE 10-4[...]

142 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only c. T o view details about the Sun StorEdge T3+ Array paths, click the Details button. The Multipath Configurator LUN Properties detail window is displayed. FIGURE 10-6 Multipath Configurator LUN Properties Detail Note – [...]

Chapter 10 T roubleshooting Using Microsoft Windo ws 2000 143 Sun Proprietary/Confidential: Internal Use Only FIGURE 10-7 displays example ouput for a Sun StorEdge 3900 series system from the jafo_nutil.exe interface. FIGURE 10-7 Sun StorEdge T3+ Array Failover Driver CLI Output for the Sun StorEdge 3900 Series # E:Program FilesSun MicrosystemsT[...]

144 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 10-8 displays example ouput for a Sun StorEdge 6900 series system from the jafo_nutil.exe interface. FIGURE 10-8 Sun StorEdge T3+ Array Failover Driver CLI Example Output for the Sun StorEdge 6900 Series # E:Progra[...]

Chapter 10 T roubleshooting Using Microsoft Windo ws 2000 145 Sun Proprietary/Confidential: Internal Use Only T ABLE 10-1 lists some of the codes and descriptions for CLI output for a Sun StorEdge 6910 series system. T ABLE 10-1 T ips for Interpreting Sun StorEdge 6910 Series CLI Output Component Output Code Description Device FW_REV Firmware r evi[...]

146 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

147 CHAPTER Sun Proprietary/Confidential: Internal Use Only 11 Example of Fault Isolation In the following example, a fault was injected into a running Sun StorEdge 3900 series system to show a troubleshooting flow . 1. Discover the Error One of the best ways to discover errors is by using the Storage Automated Diagnostic Environment monitoring sys[...]

148 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only In this configuration, Port 2 is shown to have gone of fline. Port 2 is a Microsoft W indows 2000 host-to-switch connection. Since the Storage Automated Diagnostic Environment does not have visibility to a Micr osoft W ind[...]

Chapter 11 Example of Fault Isolation 149 Sun Proprietary/Confidential: Internal Use Only The primary path to Drive F: failed. The alternate path is currently handling all of the I/O. 3. Check the HBA Using the HBA utility (Qlogic SANblade in this example), confirm the fault. FIGURE 11-3 Fault Confirmation Using QLogic SunBlade 4. Isolate the compo[...]

150 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 11-4 Diagnostics Using QLogic SunBlade In this example, the HBA-to-switch cable was removed temporarily and a loopback connector was inserted into the HBA. The Qlogc SANblade LoopBack diagnostics were then run. The [...]

Chapter 11 Example of Fault Isolation 151 Sun Proprietary/Confidential: Internal Use Only In the examples shown in FIGURE 1 1-5 , FIGURE 1 1-6 , and FIGURE 1 1-7 , Port 2 on Switch diag156-sw1a was marked with a "Red" icon, indicating a problem. Note – All tests were run with the default values. FIGURE 11-5 Storage Automated Diagnostic [...]

152 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 11-6 Storage Automated Diagnostic Environment T est from T opology Pull-Down Menu FIGURE 11-7 Storage Automated Diagnostic Environment T est from T opology T est Detail[...]

Chapter 11 Example of Fault Isolation 153 Sun Proprietary/Confidential: Internal Use Only The first run failed, indicating a problem with either the GBIC or with the Sun StorEdge network FC switch. T o further isolate the problem, a new GBIC was inserted into the port, the loopback connector was re-inserted, and the same test was run a second time.[...]

154 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only FIGURE 11-9 Multipath Recovery using the Sun StorEdge T3+ Array Multipath Configurator Note – Storage Automated Diagnostic Environment should also post an event noting that the Port has gone back online. The Multipath Co[...]

155 APPENDIX Sun Proprietary/Confidential: Internal Use Only A V irtualization Engine Refer ences This appendix contains the following information: ■ “SRN Reference” on page 155 ■ “SRN/SNMP Single Point-of-Failure Descriptions” on page 159 ■ “Port Communication Numbers” on page 160 ■ “V irtualization Engine Service Codes” on[...]

156 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE A-1 SRN Reference SRN Description Corrective Action 1xxxx The SCSI Request Sense command has reported the condition of the disk drive, wher e xxxx is the Unit Error Code in Sense Data bytes 20 to 21. If too many chec[...]

Appendix A Vir tualization Engine References 157 Sun Proprietary/Confidential: Internal Use Only 70021 The drive is offline. If the change was unintentional, check the condition of the drives. 70022 The virtualization engine is off line. If the change was unintentional, check the condition of the drives. 70023 The drive is unresponsive. Check the c[...]

158 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only 71001 This is a generic error code for the SLIC .I t signifies communication pr oblems between the virtualization engine and the daemon. Error halt mode also for ces this service request number . 1. Check the condition of t[...]

Appendix A Vir tualization Engine References 159 Sun Proprietary/Confidential: Internal Use Only SRN/SNMP Single Point-of-Failur e Descriptions T ABLE A-2 provides Simple Network Management Pr otocol (SNMP) descriptions, associated Service Request Numbers (SRNs), and recommendations for corr ective action. T ABLE A-2 SRN/SNMP Single Point-of-Failur[...]

160 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Port Communication Numbers V irtualization Engine Service Codes T ABLE A-4 lists the service code numbers for errors that occur on the virtualization engine, along with recommendations for corr ective action : T ABLE A-3 Po[...]

Appendix A Vir tualization Engine References 161 Sun Proprietary/Confidential: Internal Use Only 050 An attempt to write a value into nonvolatile storage failed, perhaps because a har dware failure, or one of the databases stored in Flash memory could not accept the entry being added. • Clear the SAN database. • Cycle power to the virtualizatio[...]

162 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only T ABLE A-5 V irtualization Engine Service Codes —400-599 Device-Side Interface Driver Errors Service Code Number Cause of Err or Recommended Corrective Action 409 The FC device-side type code is invalid. • Cycle the pow[...]

163 APPENDIX Sun Proprietary/Confidential: Internal Use Only B Conf iguration Utility Err or Messages The Sun StorEdge 3900 and 6900 Series Reference Manual lists and defines the command utilities that configur e the various components of the Sun StorEdge 3900 and 6900 series storage systems. If you encounter errors with the command line utilities,[...]

164 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only V irtualization Engine Err or Messages T ABLE B-1 V irtualization Engine Error Messages Source of Err or Message Cause of Error Message Sug gested Corrective Action Common to virtualization engine Invalid virtualization eng[...]

Appendix B Configuration Utility Error Messages 165 Sun Proprietary/Confidential: Internal Use Only Common to virtualization engine After resetting the virtualization engine, the $VENAME is unreachable. The hardwar e might be faulty . Check the IP address and netmask that has been assigned to the virtualization engine hardwar e. Be aware that the m[...]

166 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only checkvemap Cannot establish communication with ${vepair} 1. Run the checkvemap command again. 2. If this fails, check the status of both virtualization engines. 3. If there is an err or condition, see Appendix A for correct[...]

Appendix B Configuration Utility Error Messages 167 Sun Proprietary/Confidential: Internal Use Only restorevemap • The import zone data failed. • The restore physical and logical data failed. • The restore zone data failed. 1. Check the status of both virtualization engines. 2. If an error condition exists, refer to Appendix A for corr ective[...]

168 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Switch Err or Messages T ABLE B-2 Sun StorEdge Network FC Switch Error Messages Source of Err or Message Cause of Error Message Suggested Corrective Action Common to all Sun StorEdge network FC switches The Sun StorEdge sys[...]

Appendix B Configuration Utility Error Messages 169 Sun Proprietary/Confidential: Internal Use Only checkswitch • The current configuration on $switch does not match the defined configuration. • One of the predefined static switch configuration parameters that can be overridden for special configurations (such as NT connect or cascaded switches[...]

170 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only setswitchflash Invalid flash f ile $flashfile . Check the number of ports on switch $switch . Y ou might be attempting to download a flash f ile for an 8-port switch to a 16- port switch. Check showswitch -s $switch and loo[...]

Appendix B Configuration Utility Error Messages 171 Sun Proprietary/Confidential: Internal Use Only Sun StorEdge T3+ Array Partner Gr oup Err or Messages Caution – Running restoret3config (1M) or modifyt3config (1M) destroys all data on the Sun StorEdge T3+ array . T ABLE B-3 Sun StorEdge T3+ Array Error Messages Source of Err or Message Cause of[...]

172 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Common to Sun StorEdge T3+ array • The Sun StorEdge T3+ array is not of T3B type, so it aborts operations. • t3config utilities are supported only in the Sun StorEdge T3+ array; the t3config utilities are not supported [...]

Appendix B Configuration Utility Error Messages 173 Sun Proprietary/Confidential: Internal Use Only checkt3mount • The $lun status reported a bad or nonexistent LUN. • While checking the configuration using the showt3 -n command, operations abort. 1. Run the showt3 -n command to verify that the requested LUN exists on the Sun StorEdge T3+ array[...]

174 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only restoret3config • $LUN configuration failed to restor e. • The force option tried unsuccessfully to reinitialize. 1. Check the Sun StorEdge T3+ configuration with the showt3 -n t3_name command. 2. Refer to the Sun StorE[...]

Appendix B Configuration Utility Error Messages 175 Sun Proprietary/Confidential: Internal Use Only Other Err or Messages T ABLE B-4 Other SUNWsecfg Error Messages Source of Err or Message Cause of Error Message Suggested Corrective Action Common to all components If the Sun StorEdge 3900 or 6900 series has more than two failur es (for example, bot[...]

176 Sun StorEdge 3900 and 6900 Series 2.0 Troubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only[...]

Abbreviations and Acr onyms-177 Sun Proprietary/Confidential: Internal Use Only Abbr eviations and Acr onyms This list contains definitions for acr onyms used in this troubleshooting guide. ASIC application-specific integrated cir cuit CLI command-line interface CRC cyclic redundancy code DAS direct attached storage EOF end of file FC Fibre Channel[...]

Abbreviations and Acr onyms-178 Sun StorEdge 3900 and 6900 Ser ies 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only PF A predictive failur e analysis POST power on self test RAID redundant array of independent disks RARP reverse addr ess resolution protocol RFE request for enhancement RSS Remote Storage Serv[...]

Index 17 9 Sun Proprietary/Confidential: Internal Use Only Index NUMERICS 2 Gbit switch error messages, 168 3Com Ethernet hubs, 35 A A1 or B1 link verifying, 45 A2 or B2 link isolating, 52 Storage Service Processor Side Event, 50 verifying, 51 A2/B2 link FRU test availability, 5 0 A3 or B3 link FRU test availability, 5 7 host side event, 5 4 isolat[...]

Index 180 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only DMP-enabled paths returning to pr oduction, 22 documentation organization, XV shell prompts, XVII using UNIX commands, X VI dynamic multipathing (DMP), 20 E error discovery, 4 error messages other SUNWsecfg, 1 75 Sun[...]

Index 181 Sun Proprietary/Confidential: Internal Use Only installations Sun StorEdge T raffic Manager, 5 VERIT AS VxDMP, 5 isolating A1 or B1 FC link, 4 8 A2 or B2 FC link, 5 2 A3 or B3 link, 5 8 isolating FRUs, 5 isolation procedur es A1 or B1 FC link, 4 8 for A2/B2 link, 5 2 L LED service and diagnostic codes reading virtualization engine, 1 1 1[...]

Index 182 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only switches, 74 SAN database manually clearing, 1 23 manually restoring, 123 resetting, 124 service codes interpreting, 1 1 1 overview, 108 retrieving, 108 virtualization engine, 11 1 , 1 56 service processor tr oublesh[...]

Index 183 Sun Proprietary/Confidential: Internal Use Only test examples command line, 2 7 qlctest(1M), 27 switchtest(1M), 28 testing FRUs, 5 tests how to run, 5 Sun StorEdge T3+ arrays, 5 thresholds used in PF A, 2 tools troubleshooting, 23 troubleshooting broad steps, 3 check status of Sun StorEdge T3+ array, 4 check status of the Sun StorEdge net[...]

Index 184 Sun StorEdge 3900 and 6900 Series 2.0 T roubleshooting Guide • March 2003 Sun Proprietary/Confidential: Internal Use Only Z zone modifications, 74[...]