English dropdown-ico

Sun Microsystems StorEdge 3900 Series manual

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162

Go to page of

Similar user manuals

A good user manual

The rules should oblige the seller to give the purchaser an operating instrucion of Sun Microsystems StorEdge 3900 Series, along with an item. The lack of an instruction or false information given to customer shall constitute grounds to apply for a complaint because of nonconformity of goods with the contract. In accordance with the law, a customer can receive an instruction in non-paper form; lately graphic and electronic forms of the manuals, as well as instructional videos have been majorly used. A necessary precondition for this is the unmistakable, legible character of an instruction.

What is an instruction?

The term originates from the Latin word „instructio”, which means organizing. Therefore, in an instruction of Sun Microsystems StorEdge 3900 Series one could find a process description. An instruction's purpose is to teach, to ease the start-up and an item's use or performance of certain activities. An instruction is a compilation of information about an item/a service, it is a clue.

Unfortunately, only a few customers devote their time to read an instruction of Sun Microsystems StorEdge 3900 Series. A good user manual introduces us to a number of additional functionalities of the purchased item, and also helps us to avoid the formation of most of the defects.

What should a perfect user manual contain?

First and foremost, an user manual of Sun Microsystems StorEdge 3900 Series should contain:
- informations concerning technical data of Sun Microsystems StorEdge 3900 Series
- name of the manufacturer and a year of construction of the Sun Microsystems StorEdge 3900 Series item
- rules of operation, control and maintenance of the Sun Microsystems StorEdge 3900 Series item
- safety signs and mark certificates which confirm compatibility with appropriate standards

Why don't we read the manuals?

Usually it results from the lack of time and certainty about functionalities of purchased items. Unfortunately, networking and start-up of Sun Microsystems StorEdge 3900 Series alone are not enough. An instruction contains a number of clues concerning respective functionalities, safety rules, maintenance methods (what means should be used), eventual defects of Sun Microsystems StorEdge 3900 Series, and methods of problem resolution. Eventually, when one still can't find the answer to his problems, he will be directed to the Sun Microsystems service. Lately animated manuals and instructional videos are quite popular among customers. These kinds of user manuals are effective; they assure that a customer will familiarize himself with the whole material, and won't skip complicated, technical information of Sun Microsystems StorEdge 3900 Series.

Why one should read the manuals?

It is mostly in the manuals where we will find the details concerning construction and possibility of the Sun Microsystems StorEdge 3900 Series item, and its use of respective accessory, as well as information concerning all the functions and facilities.

After a successful purchase of an item one should find a moment and get to know with every part of an instruction. Currently the manuals are carefully prearranged and translated, so they could be fully understood by its users. The manuals will serve as an informational aid.

Table of contents for the manual

  • Page 1

    Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054 U .S.A. 650-960-1300 Send comments about this document to: docfeedback@sun.com Sun StorEdge ™ 3900 and 6900 Ser ies T roub leshooting Guide P ar t No. 816-4290-11 March 2002, Revision A[...]

  • Page 2

    Please Recycle Copyright 2002 Sun Microsystems, Inc., 4150 Network Cir cle, Santa Clara, CA 95054 U.S.A. All rights reserved. This product or document is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written autho[...]

  • Page 3

    Contents iii For Internal Use Only Contents 1. Introduction 1 Predictive Failur e Analysis Capabilities 2 2. General T roubleshooting Procedures 3 T roubleshooting Overview T asks 3 Multipathing Options in the Sun StorEdge 6900 Series 7 Alternatives to Sun StorEdge T raffic Manager 8 ▼ T o Quiesce the I/O 8 ▼ T o Unconfigure the c2 Path 8 ▼ T[...]

  • Page 4

    Contents iv For Internal Use Only Command Line T est Examples 19 qlctest(1M) 19 switchtest(1M) 20 Storage Automated Diagnostic Environment Event Grid 21 ▼ T o Customize an Event Report 21 3. T roubleshooting the Fibre Channel Links 23 A1/B1 Fibre Channel (FC) Link 23 ▼ T o V erify the Data Host 25 FRU T ests A vailable for A1/B1 FC Link Segment[...]

  • Page 5

    Contents v For Internal Use Only ▼ T o V erify Configuration Settings 47 ▼ T o Clear the Lock File 50 5. T roubleshooting Host Devices 53 Host Event Grid 53 ▼ Using the Host Event Grid 53 Replacing the Master , Alternate Master , and Slave Monitoring Host 57 ▼ T o Replace the Master Host 57 ▼ T o Replace the Alternate Master or Slave Moni[...]

  • Page 6

    vi Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 V irtualization Engine LEDs 72 Power LED Codes 73 Interpreting LED Service and Diagnostic Codes 73 Back Panel Features 74 Ethernet Port LEDs 74 Fibre Channel Link Err or Status Report 75 ▼ T o Check Fibre Channel Link Error Status Manually 76 T ranslating Host Device Names [...]

  • Page 7

    Contents vii For Internal Use Only T roubleshooting the T1/T2 Data Path 102 Notes 102 T1/T2 Notification Events 103 Sun StorEdge T3+ Array Storage Service Processor V erif ication 106 T1/T2 FRU T ests A vailable 107 Notes 108 T1/T2 Isolation Procedur es 108 Sun StorEdge T3+ Array Event Grid 109 ▼ Using the Sun StorEdge T3+ Array Event Grid 109 Re[...]

  • Page 8

    viii Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002[...]

  • Page 9

    List of Figures ix List of Figur es FIGURE 2-1 Sun StorEdge 3900 Series Fibre Channel Link Diagram 16 FIGURE 2-2 Sun StorEdge 6900 Series Fibre Channel Link Diagram 17 FIGURE 3-1 Data Host Notification of Intermittent Problems 23 FIGURE 3-2 Data Host Notification of Severe Link Error 24 FIGURE 3-3 Storage Service Processor Notification 24 FIGURE 3-[...]

  • Page 10

    List of Figures x FIGURE 7-6 Path Failure —I/O Routed through Both HBAs 94 FIGURE 7-7 Virtualization Engine Event Grid 95 FIGURE 8-1 Storage Service Processor Event 103 FIGURE 8-2 Virtualization Engine Alert 105 FIGURE 8-3 Manage Configuration Files Menu 106 FIGURE 8-4 Example Link Test Text Output from the Storage Automated Diagnostic Environmen[...]

  • Page 11

    xi Pr eface The Sun StorEdge 3900 and 6900 Series T roubleshooting Guide pr ovides guidelines for isolating problems in supported conf igurations of the Sun StorEdge TM 3900 and 6900 series. For detailed configuration information, refer to the Sun StorEdge 3900 and 6900 Series Reference Manual . The scope of this troubleshooting guide is limited to[...]

  • Page 12

    xii Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Chapter 7 provides detailed information for tr oubleshooting the virtualization engines. Chapter 8 describes how to troubleshoot the Sun StorEdge T3+ array devices. Also included in this chapter is information about the Explorer Data Collection Utility . Chapter 9 discusses [...]

  • Page 13

    Pref ace xiii T ypographic Conventions Shell Pr ompts T ypeface Meaning Examples AaBbCc123 The names of commands, files, and directories; on-scr een computer output Edit your .login file. Use ls -a to list all files. % You have mail . AaBbCc123 What you type, when contrasted with on-screen computer output % su Password: AaBbCc123 Book titles, new w[...]

  • Page 14

    xiv Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 Related Documentation Product Title Part Number Late-breaking News • Sun StorEdge 3900 and 6900 Series Release Notes 816-3247 Sun StorEdge 3900 and 6900 series hardwar e information • Sun StorEdge 3900 and 6900 Series Site Preparation Guide • Sun StorEdge 3900 and 6900[...]

  • Page 15

    Pref ace xv Accessing Sun Documentation Online A broad selection of Sun system documentation is located at: http://www.sun.com/products-n-solutions/hardware/docs A complete set of Solaris documentation and many other titles are located at: http://docs.sun.com Sun W elcomes Y our Comments Sun is interested in impr oving its documentation and welcome[...]

  • Page 16

    xvi Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002[...]

  • Page 17

    1 CHAPTER 1 Intr oduction The Sun StorEdge 3900 and 6900 series storage subsystems are complete preconf igured storage solutions. The configurations for each of the storage subsystems are shown in T ABLE 1- 1 . T ABLE 1-1 Series System Sun StorEdge Fibre Channel Switch Supported Sun StorEdge T3+ Array P ar tner Gr oups Supported Additional Array Pa[...]

  • Page 18

    2 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 Pr edictive Failur e Analysis Capabilities The Storage Automated Diagnostic Environment software provides the health and monitoring functions for the Sun StorEdge 3900 and 6900 series systems. This software pr ovides the following predictive failure analysis (PF A) capabilit[...]

  • Page 19

    3 CHAPTER 2 General T r oubleshooting Pr ocedur es This chapter contains the following sections: ■ “T r oubleshooting Overview T asks” on page 3 ■ “Multipathing Options in the Sun StorEdge 6900 Series” on page 7 ■ “Fibre Channel Links” on page 15 ■ “Storage Automated Diagnostic Environment Event Grid” on page 21 T r oublesho[...]

  • Page 20

    4 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 1. Discover the error by checking one or more of the following messages or f iles: ■ Storage Automated Diagnostic Environment alerts or email messages ■ /var/adm/ messages ■ Sun StorEdge T3+ array syslog file ■ Storage Service Processor messages ■ /var/adm/messages[...]

  • Page 21

    Chapter 2 General T roub leshooting Procedures 5 For Internal Use Only 4. Check the status of the Sun StorEdge FC network switch-8 and switch-16 switches using the following tools: ■ Storage Automated Diagnostic Environment device monitoring r eports ■ Run the SEcfg script, which displays and shows the Sun StorEdge T3+ array configuration ■ L[...]

  • Page 22

    6 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 8. V erify the f ix using the following tools: ■ Storage Automated Diagnostic Environment GUI T opology V iew and Diagnostic T ests ■ /var/adm/messages on the data host 9. Return the path to service by using one of the following methods: ■ Multipathing software ■ Res[...]

  • Page 23

    Chapter 2 General T roub leshooting Procedures 7 For Internal Use Only Multipathing Options in the Sun StorEdge 6900 Series Using the virtualization engines presents several challenges in how multipathing is handled in the Sun StorEdge 6900 series. Unlike Sun StorEdge T3+ array and Sun StorEdge network FC switch-8 and switch- 16 switch installation[...]

  • Page 24

    8 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 Note that in the Class and State fields, the virtualization engines ar e presented as two primary/ONLINE devices. The current Sun StorEdge T raf fic Manager design does not enable you to manually halt the I/O (that is, you cannot perform a failover to the secondary path) whe[...]

  • Page 25

    Chapter 2 General T roub leshooting Procedures 9 For Internal Use Only 2. Using Storage Automated Diagnostic Environment T opology GUI, determine which virtualization engine is in the path you need to disable. 3. Use the world wide name (WWN) of the virtualization engine that is in the unconf igure command, as follows: 4. V erify that I/O has halte[...]

  • Page 26

    10 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 ▼ T o Suspend the I/O Use one of the following methods to suspend the I/O while the failover occurs: 1. Stop all customer applications that are accessing the Sun StorEdge T3+ array . 2. Manually pull the link from the Sun StorEdge T3+ array to the switch and wait for a Su[...]

  • Page 27

    Chapter 2 General T roub leshooting Procedures 11 For Internal Use Only ▼ T o V iew the VxDisk Properties 1. T ype the following: From the VxDisk output, notice that there ar e two physical paths to the LUN: ■ c20t2B000060220041F4d0s2 ■ c23t2B000060220041F9d0s2 Both of these paths are curr ently enabled with VxDMP . # vxdisk list Disk_1 Devic[...]

  • Page 28

    12 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 2. Use the luxadm (1M) command to display further information about the underlying LUN. # luxadm display /dev/rdsk/c20t2B000060220041F4d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/c20t2B000060220041F4d0s2 Status(Port A): O.K. Vendor: SUN Product ID: SESS01 WWN(Node): 2a000060[...]

  • Page 29

    Chapter 2 General T roub leshooting Procedures 13 For Internal Use Only ▼ T o Quiesce the I/O on the A3/B3 Link 1. Determine the path you want to disable. 2. Disable the path by typing the following: 3. V erify that the path is disabled: Steps 1 and 2 halt I/O only up to the A3/B3 link. I/O will continue to move over the T1 & T2 paths, as wel[...]

  • Page 30

    14 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 ▼ T o Return the Path to Pr oduction 1. T ype: 2. V erify that the path has been re-enabled by typing: # vxdmpadm enable ctlr=<c#> # vxdmpadm listctlr all[...]

  • Page 31

    Chapter 2 General T roub leshooting Procedures 15 For Internal Use Only Fibr e Channel Links The following sections provide tr oubleshooting information for the basic components and Fibre Channel links, listed in T ABLE 2-1 . Note – In an actual Sun StorEdge 3900 or 6900 series configuration, there could be more Sun StorEdge T3+ arrays than are s[...]

  • Page 32

    16 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 Fibr e Channel Link Diagrams FIGURE 2-1 shows the basic components and the Fibre Channel links for a Sun StorEdge 3900 series system: ■ A1 to B1— HBA to Sun StorEdge FC network switch-8 and switch-16 switch link ■ A4 to B4 —Sun StorEdge FC network switch-8 and switc[...]

  • Page 33

    Chapter 2 General T roub leshooting Procedures 17 For Internal Use Only FIGURE 2-2 shows the basic components and the Fibre Channel links for a Sun StorEdge 6900 series system: ■ A1 to B1— HBA to Sun StorEdge network FC switch-8 and switch-16 switch link ■ A2 to B2— Sun StorEdge network FC switch-8 and switch-16 switch to virtualization eng[...]

  • Page 34

    18 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 Host Side T r oubleshooting Host-side troubleshooting r efers to the messages and errors the data host detects. Usually , these messages appear in the /var/adm/messages file. Storage Service Pr ocessor Side T roubleshooting Storage Service Processor-side T r oubleshooting r[...]

  • Page 35

    Chapter 2 General T roub leshooting Procedures 19 For Internal Use Only Command Line T est Examples T o run a single Sun StorEdge diagnostic test fr om the command line rather than through the Storage Automated Diagnostic Environment interface, you must log into the appropriate Host or Slave for testing the components. The following two tests, the [...]

  • Page 36

    20 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 switchtest (1M) switchtest (1M) is used to diagnose the Sun StorEdge network FC switch-8 and switch-16 switch devices. The switchtest process also provides command line access to switch diagnostics. switchtest supports testing on local and remote switches. switchtest runs t[...]

  • Page 37

    Chapter 2 General T roub leshooting Procedures 21 For Internal Use Only Storage Automated Diagnostic Envir onment Event Grid The Storage Automated Diagnostic Environment generates component-specific event grids that describe the severity of an Event, whether action is required, a description of the event, and recommended action. Refer to Chapters 5[...]

  • Page 38

    22 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002[...]

  • Page 39

    23 CHAPTER 3 T r oubleshooting the Fibr e Channel Links A1/B1 Fibr e Channel (FC) Link If a problem occurs with the A1/B1 FC link: ■ In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over . ■ In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over , but a severe pr oblem can cause a path to go off[...]

  • Page 40

    24 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 FIGURE 3-2 Data Host Notif ication of Severe Link Err or FIGURE 3-3 Storage Service Pr ocessor Notification Note – An A1/B1 FC link error can cause a port in sw1a or sw1b to change state. Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category[...]

  • Page 41

    Chapter 3 T roubleshooting the Fibre Channel Links 25 For Internal Use Only ▼ T o V erify the Data Host An error in the A1/B1 FC link can cause a path to go of fline in the multipathing software. CODE EXAMPLE 3-1 luxadm (1M) Display # luxadm display /dev/rdsk/c6t29000060220041F96257354230303052d0s2 DEVICE PROPERTIES for disk: /dev/rdsk/ c6t290000[...]

  • Page 42

    26 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 An error in the A1/B1 FC link can also cause a device to enter the “unusable” state in cfgadm . In this case, the output for luxadm -e port will show that a device that was “connected” changed to an “unconnected” state. CODE EXAMPLE 3-2 cfgadm -al Display FRU T [...]

  • Page 43

    Chapter 3 T roubleshooting the Fibre Channel Links 27 For Internal Use Only CODE EXAMPLE 3-3 switchtest (1M) called with options Note – The Storage Automated Diagnostic Environment automatically resets the transfer size if it notes that it is about to test a switch to HBA connection. This is done both in the Storage Automated Diagnostic Environme[...]

  • Page 44

    28 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 ▼ T o Isolate the A1/B1 FC Link 1. Quiesce the I/O on the A1/B1 FC link path. 2. Run switchtest or qlctest to test the entire link. 3. Break the connection by uncabling the link. 4. Insert a loopback connector into the switch port. 5. Rerun switchtest . a. If switchtest f[...]

  • Page 45

    Chapter 3 T roubleshooting the Fibre Channel Links 29 For Internal Use Only A2/B2 Fibr e Channel (FC) Link If a problem occurs with the A2/B2 FC link: ■ In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over . ■ In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over , but a severe pr oblem can ca[...]

  • Page 46

    30 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 FIGURE 3-5 A2/B2 FC Link Storage Service Processor Side Event Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch Key: switch:100000c0dd0061bb EventType: StateChangeEvent.X.port.1 EventTime: 01/08/2002 17:38:32 ’port.1’ in SWITC[...]

  • Page 47

    Chapter 3 T roubleshooting the Fibre Channel Links 31 For Internal Use Only ▼ T o V erify the Host Side An error in the A2/B2 FC link can r esult in a device being listed as in an “unusable” state in cfgadm , but no HBAs are listed as in the “unconnected” state in luxadm output. The multipathing software will note an OFFLINE path.[...]

  • Page 48

    32 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 CODE EXAMPLE 3-4 cfgadm -al # cfgadm -al Ap_Id Type Receptacle Occupant Condition c0 scsi-bus connected configured unknown <snip> # luxadm -e port Found path to 2 HBA ports /devices/pci@6,4000/SUNW,qlc@2/fp@0,0:devctl CONNECTED /devices/pci@6,4000/SUNW,qlc@3/fp@0,0:de[...]

  • Page 49

    Chapter 3 T roubleshooting the Fibre Channel Links 33 For Internal Use Only Note – Y ou can f ind procedures for r estoring virtualization engine settings in the Sun StorEdge 3900 and 6900 Series Reference Manual . ▼ T o V erify the A2/B2 FC Link Y ou can check the A2/B2 FC link using the Storage Automated Diagnostic Environment, Diagnose—T e[...]

  • Page 50

    34 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 5. If the switch or the GBIC show no errors, replace the remaining components in the following order: a. Replace the virtualization engine-side GBIC, recable the link, and monitor the link for errors. b. Replace the cable, recable the link, and monitor the link for errors. [...]

  • Page 51

    Chapter 3 T roubleshooting the Fibre Channel Links 35 For Internal Use Only A3/B3 Fibr e Channel (FC) Link If a problem occurs with the A3/B3 FC link: ■ In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over . ■ In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over , but a severe pr oblem can ca[...]

  • Page 52

    36 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 FIGURE 3-7 A3/B3 FC Link Storage Service Processor -Side Event FIGURE 3-8 A3/B3 FC Link Storage Service Processor -Side Event Site : FSDE LAB Broomfield CO Source : diag.xxxxx.xxx.com Severity : Normal Category : Switch Key: switch:100000c0dd0057bd EventType: StateChangeEve[...]

  • Page 53

    Chapter 3 T roubleshooting the Fibre Channel Links 37 For Internal Use Only ▼ T o V erify the Host Side An error in the A3/B3 FC link r esults in a device being listed as in an “unusable” state in cfgadm , but no HBAs are listed as in the “unconnected” state in luxadm output. The multipathing software will note an “of fline” path. COD[...]

  • Page 54

    38 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 CODE EXAMPLE 3-6 VxDMP Error Message ▼ T o V erify the Storage Service Pr ocessor Y ou can check the A3/B3 FC link using the Storage Automated Diagnostic Environment, Diagnose—T est from T opology functionality . Storage Automated Diagnostic Environment’s implementati[...]

  • Page 55

    Chapter 3 T roubleshooting the Fibre Channel Links 39 For Internal Use Only ▼ T o Isolate the A3/B3 FC Link 1. Quiesce the I/O on the A3/B3 FC link path. 2. Break the connection by uncabling the link. 3. Insert the loopback connector into the switch port. 4. Run switchtest : a. If the test fails, replace the GBIC and rerun switchtest . b. If the [...]

  • Page 56

    40 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 A4/B4 Fibr e Channel (FC) Link If a problem occurs with the A4/B4 FC link: ■ In a Sun StorEdge 3900 series system, the Sun StorEdge T3+ array will fail over . ■ In a Sun StorEdge 6900 series system, no Sun StorEdge T3+ array will fail over , but a severe pr oblem can ca[...]

  • Page 57

    Chapter 3 T roubleshooting the Fibre Channel Links 41 For Internal Use Only FIGURE 3-10 Storage Service Processor Notification Site : FSDE LAB Broomfield CO Source : diag Severity : Warning Category : Switch DeviceId : switch:100000c0dd0061bb EventType: LogEvent.MessageLog EventTime: 01/29/2002 14:25:05 Change in Port Statistics on switch diag-sw1b[...]

  • Page 58

    42 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 ▼ T o V erify the Data Host A problem in the A4/B4 FC Link appears dif ferently on the data host, depending on if the array is a Sun StorEdge 3900 series or a Sun StorEdge 6900 seriesdevice. Sun StorEdge 3900 Series In a Sun StorEdge 3900 series device, the data host mult[...]

  • Page 59

    Chapter 3 T roubleshooting the Fibre Channel Links 43 For Internal Use Only T o verify the failover luxadm display can be used, the failed path will be marked OFFLINE, as shown in CODE EXAMPLE 3-7 . CODE EXAMPLE 3-7 Failed Path marked OFFLINE Note – This type of error may also cause the device to show up "unusable" in cfgadm , as shown [...]

  • Page 60

    44 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 CODE EXAMPLE 3-8 Failed Path marked “unusable” FRU tests available for the A4/B4 FC Link Segment ■ The switchtest can only be run from the Storage Service Pr ocessor ■ The linktest will be able to isolate the switch and the GBIC on the switch. It will not be able to[...]

  • Page 61

    Chapter 3 T roubleshooting the Fibre Channel Links 45 For Internal Use Only 5. Rerun switchtest . a. If switchtest fails, replace the GBIC and rerun switchtest . b. If the test fails again, replace the switch. 6. If switchtest passes, assume that the suspect components are the cable and the Sun StorEdge T3+ array controller . a. Replace the cable. [...]

  • Page 62

    46 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002[...]

  • Page 63

    47 CHAPTER 4 Conf iguration Settings This chapter contains the following sections: ■ “V erifying Configuration Settings” on page 47 ■ “T o Clear the Lock File” on page 50 For a complete listing of SUNWsecfg Error Messages and recommended action, r efer to Appendix B. V erifying Conf iguration Settings During the course of troubleshootin[...]

  • Page 64

    48 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 Note – For cluster configurations and systems that ar e attached to W indows NT , the default configurations may not match the current installed conf iguration. Be aware of this when running the verification scripts. Certain items may be f lagged as F AIL in these special[...]

  • Page 65

    Chapter 4 Configuration Settings 49 For Internal Use Only 10. If anything is marked F AIL, check the /var/adm/log/SEcfglog f ile for the details of the failure. In this example, the mirror setting in the Sun StorEdge T3+ array system settings is “off.” The SA VED CONFIGURA TION setting for this parameter , which is the default setting, should b[...]

  • Page 66

    50 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 1 1. Fix the F AIL condition, and then verify the settings again. If you interrupt any of the SUNWsecfg scripts (by typing a Control-C default font, for example), a lock file might r emain in the /opt/SUNWsecfg/etc directory , causing subsequent commands to fail. Use the fo[...]

  • Page 67

    Chapter 4 Configuration Settings 51 For Internal Use Only CODE EXAMPLE 4-2 savevemap output When savevemap: <ve-pair> EXIT is displayed, the savevemap process has successfully exited. Tue Jan 29 16:12:34 MST 2002 savevemap: v1 ENTER. Tue Jan 29 16:12:34 MST 2002 checkslicd: v1 ENTER. Tue Jan 29 16:12:42 MST 2002 checkslicd: v1 EXIT. Tue Jan 2[...]

  • Page 68

    52 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002[...]

  • Page 69

    53 CHAPTER 5 T r oubleshooting Host Devices This chapter describes how to tr oubleshoot components associated with a Sun StorEdge 3900 or 6900 series Host. This chapter contains the following sections: ■ “Using the Host Event Grid” on page 53 ■ “T o Replace the Master Host” on page 57 ■ “T o Replace the Alternate Master or Slave Mon[...]

  • Page 70

    54 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 FIGURE 5-1 Host Event Grid[...]

  • Page 71

    Chapter 5 T roubleshooting Host De vices 55 For Internal Use Only T ABLE 5-1 lists all the host events in the Storage Automated Diagnostic Environment. T ABLE 5-1 Storage Automated Diagnostic Envir onment Event Grid for the Host Category Component EventT ype Sev Action Description Information host hba Alarm+ Y ellow [ Info ] status of hba /devices/[...]

  • Page 72

    56 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 host lun.VE Alarm- Red Y [ Info ] The state of lun.VE.c14t50020 F2300003EE5d0s2. statusA on diag. xxxxx.xxx .com changed from OK to ERROR (target=ve:diag244- ve0/90.0.0.40) luxadm display reported a change in the port status of one of its paths. The Storage Automated Diagno[...]

  • Page 73

    Chapter 5 T roubleshooting Host De vices 57 For Internal Use Only Replacing the Master , Alternate Master , and Slave Monitoring Host The following procedur es are a high-level overview of the procedur es that are detailed in the Storage Automated Diagnostic Environment User ’ s Guide . Follow these procedur es when replacing a master , alternate[...]

  • Page 74

    58 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 5. Choose Utilities -> System -> Recover Conf ig. Refer to Chapter 7 of the Storage Automated Diagnostic Environment User ’ s Guide for detailed instructions. a. In the Recover Conf ig window , enter the IP address of any alternate master or slave monitoring host (a[...]

  • Page 75

    Chapter 5 T roubleshooting Host De vices 59 For Internal Use Only 7. Choose Maintenance -> General Maintenance -> Maintain Hosts. Refer to Chapter 3, “Maintenance,” of the Storage Automated Diagnostic User ’ s Guide for detailed instructions. 8. In the Maintain Hosts window , select the new host. 9. Conf igure the options as needed. 10.[...]

  • Page 76

    60 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002[...]

  • Page 77

    61 CHAPTER 6 T r oubleshooting Sun StorEdge FC Switch-8 and Switch-16 Devices This chapter describes how to troubleshoot the switch components associated with a Sun StorEdge 3900 or 6900 series system. This chapter contains the following sections: ■ “Sun StorEdge Network FC Switch-8 and Switch-16 Switch Description” on page 61 ■ “Switch E[...]

  • Page 78

    62 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 These switches can be monitored thr ough the SANSurfer GUI, which is available on the Storage Service Processor . Y ou configure and modify the switches using the Configuration Utilities. Do not conf igure or modify the switches using any method other than the SUNWsecfg too[...]

  • Page 79

    Chapter 6 T roubleshooting Sun StorEdge FC Switch-8 and Switch-16 De vices 63 For Internal Use Only FIGURE 6-1 Switch Event Grid[...]

  • Page 80

    64 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 T ABLE 6-1 lists the switch events. T ABLE 6-1 Storage Automated Diagnostic Envir onment Event Grid for Switches Cat Component EventT ype Sev Action Description Information/Action switch port statistics Log Y ellow Y [ Info/Action ] Change in port statistics on switch diag1[...]

  • Page 81

    Chapter 6 T roubleshooting Sun StorEdge FC Switch-8 and Switch-16 De vices 65 For Internal Use Only switch enclosure Audit Auditing a new switch called ras d2-swb1 (ip=xxx.0.0.41) 10002000007a609 switch oob Comm_ Established Communication regained with sw1a (ip= xxx . 20.67.213) switch oob Comm_Lost Down Y es [ Info/ Action ] Lost communication wit[...]

  • Page 82

    66 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 switch enclosure Discovery [ Info ] Discovered a new switch called ras d2-swb1 (ip=xxx.0.0.41) 10002000007a609 Discovery events occur the very first time the agent probes a storage device. It creates a detailed description of the device monitored and sends it using any acti[...]

  • Page 83

    Chapter 6 T roubleshooting Sun StorEdge FC Switch-8 and Switch-16 De vices 67 For Internal Use Only switch port StateChange+ [ Info/Action ] port.1 in SWITCH diag185 ( ip= xxx.20.67.185 )i s now A vailable (status- state changed from OFFLINE to ONLINE) Port on switch is now available. switch port StateChange- Red Y [ Info/Action ] port.1 in SWITCH [...]

  • Page 84

    68 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide • March 2002 Replacing the Master Midplane Follow this procedur e when replacing the master midplane in a Sun StorEdge network FC switch-8 or switch-16 switch or a Brocade Silkworm switch. This procedur e is detailed in the Storage Automated Diagnostic Environment User ’ s Guide . ▼[...]

  • Page 85

    69 CHAPTER 7 T r oubleshooting V irtualization Engine Devices This chapter describes how to troubleshoot the virtualization engine component of a Sun StorEdge 6900 series system. This chapter contains the following sections: ■ “V irtualization Engine Description” on page 69 ■ “T ranslating Host Device Names” on page 78 ■ “Sun StorEd[...]

  • Page 86

    70 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 V irtualization Engine Diagnostics The virtualization engine monitors the following components: ■ V irtualization engine router ■ Sun StorEdge T3+ array ■ Cabling among the router and storage Service Request Numbers The service request numbers ar e used to inform the [...]

  • Page 87

    Chapter 7 T roubleshooting Virtualization Engine Devices 71 For Internal Use Only ▼ T o Display Log Files and Retrieve SRNs Use the /opt/svengine/sduc/sreadlog command to display log files and retrieve the Service Request Numbers (SRN) for errors that need action. Data is returned in the following format: TimeStamp : nnn : T xxxxx.uuuuuuuu SRN= m[...]

  • Page 88

    72 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 ▼ T o Clear the Log ● Use the /opt/svengine/sduc/sclrlog command. V irtualization Engine LEDs T ABLE 7-1 describes the LEDs on the back of the virtualization engine.. Item Description TimeStamp January 3, 2002 10:13 nnn v1 (virtualization engine pair v1) uuuuuuuu 290000[...]

  • Page 89

    Chapter 7 T roubleshooting Virtualization Engine Devices 73 For Internal Use Only Power LED Codes The virtualization engine LEDs are shown in FIGURE 7-1 . FIGURE 7-1 V irtualization Engine Front Panel LEDs Interpr eting LED Service and Diagnostic Codes The Status LED communicates the status of the virtualization engine in decimal numbers. Each deci[...]

  • Page 90

    74 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 Back Panel Featur es The back panel of the virtualization engine contains the Sun StorEdge network FC switch-8 or switch-16 switches and a socket for the AC power input, and various data ports and LEDs. Ethernet Port LEDs The Ethernet port LEDs indicate the speed, activity [...]

  • Page 91

    Chapter 7 T roubleshooting Virtualization Engine Devices 75 For Internal Use Only Fibr e Channel Link Error Status Report The virtualization engine’s host-side and device-side interfaces provide statistical data for the counts listed in T ABLE 7-4 . T ABLE 7-4 V irtualization Engine Statistical Data Count T ype Description Link Failure Count The [...]

  • Page 92

    76 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 ▼ T o Check Fibr e Channel Link Error Status Manually The Storage Automated Diagnostic Environment, which runs on the Storage Service Processor , monitors the Fibre Channel link status of the virtualization engine. The virtualization engine must be power-cycled to reset t[...]

  • Page 93

    Chapter 7 T roubleshooting Virtualization Engine Devices 77 For Internal Use Only CODE EXAMPLE 7-1 Fibre Channel Link Err or Status Example Note – v1 repr esents the first virtualization engine pair Note – The SLIC daemon must be running for the /opt/svengine/sduc/svstat -d v1 command to work. # /opt/svengine/sduc/svstat -d v1 I00001 Host Side [...]

  • Page 94

    78 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 T ranslating Host Device Names Y ou can translate host device names to VLUN, disk pool, and physical Sun StorEdge T3+ array LUNs. The luxadm output for a host device, shown in CODE EXAMPLE 7-2 , does not include the unique VLUN serial number that is needed to identify this [...]

  • Page 95

    Chapter 7 T roubleshooting Virtualization Engine Devices 79 For Internal Use Only ▼ T o Display the VLUN Serial Number Devices That Ar e Not Sun StorEdge T raff ic Manager-Enabled 1. Use the format -e command. 2. T ype the disk on which you are working at the format prompt. 3. T ype inquiry at the scsi prompt. 4. Find the VLUN serial number in th[...]

  • Page 96

    80 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 Sun StorEdge T raf fic Manager -Enabled Devices 1. If the devices support the Sun StorEdge T raff ic Manager software, you can use this shortcut. 2. T ype: The /dev/rdsk/c#t# repr esents the Global Unique Identifier of the device. It is 32 bits long. ■ The first 16 bits c[...]

  • Page 97

    Chapter 7 T roubleshooting Virtualization Engine Devices 81 For Internal Use Only ▼ T o V iew the V irtualization Engine Map The virtualization engine map is stored on the Storage Service Processor . 1. T o view the virtualization engine map, type: Note – This example uses the virtualization engine map file, which could include old information.[...]

  • Page 98

    82 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 2. Y ou can optionally establish a telnet connection to the virtualization engine and run the runsecfg utility to poll a live snapshot of the virtualization engine map. Refer to “T o Replace a Failed V irtualization Engine” on page 84 for telnet instructions. From the v[...]

  • Page 99

    Chapter 7 T roubleshooting Virtualization Engine Devices 83 For Internal Use Only ▼ T o Failback the V irtualization Engine In the event of a Sun StorEdge T3+ array LUN failover , use the following procedur e to fail the LUN back to its original controller . 1. From the Storage Service Processor , type: where: The failback command will always be [...]

  • Page 100

    84 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 For detailed information about the SUNWsecfg scripts, refer to the Sun StorEdge 3900 and 6900 Series Reference Manual . ▼ T o Replace a Failed V irtualization Engine 1. Replace the old (failed) virtualization engine unit with a new unit. 2. Identify the MAC address of the[...]

  • Page 101

    Chapter 7 T roubleshooting Virtualization Engine Devices 85 For Internal Use Only 1 1. Enable the switch port: 12. Reset the virtualization engine: 13. Find the initiator number for the new and old number: The new unit will not have any zones defined. 14. If zones were present before the replacement, type the following: 15. V erify the new unit by [...]

  • Page 102

    86 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 ▼ T o Manually Clear the SAN Database It is occasionally necessary to manually clear the SAN database on the virtualization engine routers. Caution – This procedur e will wipe out the SAN database and will remove the configuration of disk pools, Multipath drives, Zoning[...]

  • Page 103

    Chapter 7 T roubleshooting Virtualization Engine Devices 87 For Internal Use Only Stopping and Restarting the SLIC Daemon Follow this procedur e to restart the SLIC daemon if the SLIC daemon becomes unresponsive, or if messages such as the following are displayed: connect: Connection refused or Socket error encountered.. ▼ T o Restart the SLIC Da[...]

  • Page 104

    88 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 3. Remove the segments by typing the following: Check the ipcrm (1m) man page for details. 4. Restart the SLIC daemon 5. Conf irm that the SLIC daemon is running: The message queues, shared memory , and semaphores have been removed. # ipcrm -m 301 -m 302 -m 303 -s 196608 -s[...]

  • Page 105

    Chapter 7 T roubleshooting Virtualization Engine Devices 89 For Internal Use Only Sun StorEdge 6900 Series Multipathing Example One Sun StorEdge T3+ array partner pair with 1 500GB RAID 5 LUN per brick (2 LUNs total) Currently , there is one 10GB VLUN cr eated from each physical LUN, for a total of two VLUNs. In a Sun StorEdge 6900 series, there ar[...]

  • Page 106

    90 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 In the event of a path failure after the second tier of Sun StorEdge network FC switch-8 and switch-16 switches (or in the event of both T Ports failing between the switches), the virtualization engines force a LUN failover of the affected Sun StorEdge T3+ array and routes [...]

  • Page 107

    Chapter 7 T roubleshooting Virtualization Engine Devices 91 For Internal Use Only FIGURE 7-3 Primary Data Paths to the Alternate Master[...]

  • Page 108

    92 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 FIGURE 7-4 Primary Data Paths to the Master Sun StorEdge T3+ Array[...]

  • Page 109

    Chapter 7 T roubleshooting Virtualization Engine Devices 93 For Internal Use Only FIGURE 7-5 Path Failure—Befor e the Second T ier of Switches[...]

  • Page 110

    94 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 FIGURE 7-6 Path Failure —I/O Routed thr ough Both HBAs[...]

  • Page 111

    Chapter 7 T roubleshooting Virtualization Engine Devices 95 For Internal Use Only V irtualization Engine Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort virtualization engine events by component, category , or event type. The Storage Automated Diagnostic Environment GUI displays an event grid that describes th[...]

  • Page 112

    96 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002 T ABLE 7-5 lists the V irtualization Engine Events. T ABLE 7-5 Storage Automated Diagnostic Envir onment Event Grid for V irtualization Engine Category Component EventT ype Sev Action Description virtualization engine enclosure Alarm Y ellow V olume E00012 on v1a changed ma[...]

  • Page 113

    Chapter 7 T roubleshooting Virtualization Engine Devices 97 For Internal Use Only virtualization engine ve_diag Diagnostic T est- Red ve_diag (diag240) on ve-1 (ip=xxx.20.67.213) failed virtualization engine veluntest Diagnostic T est- Red veluntest (diag240) on ve-1 (ip=xxx.20.67.213) failed virtualization engine enclosure Discovery [ Info ] Disco[...]

  • Page 114

    98 Sun StorEdge 3900 and 6900 Series T roub leshooting Guide — March 2002[...]

  • Page 115

    99 CHAPTER 8 T r oubleshooting the Sun StorEdge T3+ Array Devices This chapter contains the following sections: ■ “Explorer Data Collection Utility” on page 99 ■ “Sun StorEdge T3+ Array Event Grid” on page 109 Explor er Data Collection Utility The Explorer Data Collection Utility script is included on the Storage Service Processor in th[...]

  • Page 116

    100 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 Do not accept automatic emailing of the Explorer Data Collection Utility output, unless the Storage Service Processor is pr operly set up to handle mail correctly . Before running the Explor er Data Collection Utility , make sur e that the switch and Sun StorEdge T3+ array [...]

  • Page 117

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 101 For Internal Use Only CODE EXAMPLE 8-2 Editing Sun StorEdge T3+ array information using vi Note – xxxx repr esents Sun StorEdge T3+ array passwords. ■ Y ou can now run /opt/SUNWexplo/bin/explorer to collect information about the Storage Service Processor operating system, the S[...]

  • Page 118

    102 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 T r oubleshooting the T1/T2 Data Path Notes ■ There ar e two T Port links for redundancy . ■ If one of the two links is lost, no Sun StorEdge T3+ array LUN failover will occur , and no pathing failures will be noted. ■ If both T Port links fail, there will be a Sun St[...]

  • Page 119

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 103 For Internal Use Only T1/T2 Notif ication Events The example below shows a typical port failure event FIGURE 8-1 Storage Service Processor Event Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Error (Actionable) Category : Switch DeviceId : switch:100000c0[...]

  • Page 120

    104 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 If both T Ports go off line, you might see messages like the following. Note the virtualization engine Event alerting the LUN failover . Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Warning (Actionable) Category : Ve DeviceId : ve:6257335A-30303[...]

  • Page 121

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 105 For Internal Use Only FIGURE 8-2 V irtualization Engine Alert ...continued from previous page... ---------------------------------------------------------------------- Site : Lab 3286 - DSQA1 Broomfield Source : diag.xxxxx.xxx.com Severity : Warning Category : Message DeviceId : me[...]

  • Page 122

    106 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 Sun StorEdge T3+ Array Storage Service Pr ocessor V erif ication 1. Run port listmap on the Sun StorEdge T3+ array to see the failover event. 2. Compare the virtualization engine conf iguration to a saved conf iguration by running /opt/SUNWsecfg/runsecfg and choosing V erif[...]

  • Page 123

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 107 For Internal Use Only T1/T2 FRU T ests A vailable ■ Switch - switchtest ■ Link - linktest Running linktest from the Storage Automated Diagnostic Envir onment GUI will guide the Service Engineer to discover the failed FRU. Once the test has completed its run, an email message, s[...]

  • Page 124

    108 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 Notes ■ When inserting a loopback connector into the T Port, there will be NO green light indicating a proper insertion. However , the test will run and be valid. There is currently an RFE to addr ess this issue. ■ If only one of the links has failed and the I/O is trav[...]

  • Page 125

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 109 For Internal Use Only Sun StorEdge T3+ Array Event Grid The Storage Automated Diagnostic Environment Event Grid enables you to sort Sun StorEdge T3+ array events by component, category , or event type. The Storage Automated Diagnostic Environment GUI displays an event grid that des[...]

  • Page 126

    110 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 The following table lists all of the events for the Sun StorEdge T3+ array . Category Component EventT ype Sev Action Description Information t3 power .temp Alarm+ The state of power .u1pcu1.PowT e mp on diag213 (ip=xxx.20.67.213) is Normal t3 disk.port Alarm- Red Y [ Info/[...]

  • Page 127

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 111 For Internal Use Only t3 power . battery Alarm- Red Y [ Info/ Action ] The state of power .u1pcu1.BatStat e on diag213 (ip=xxx.20.67.213) is Fault Possible causes are: 1. V oltage level on power supply and battery have moved out of acceptable thresholds. 2. The internal PCU temp ha[...]

  • Page 128

    112 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 t3 power . output Alarm- Red Y [ Info/ Action ] The state of power .u1pcu1.PowOu tput on diag213 (ip=xxx.20.67.21 3 ) is Fault Information: The state of the power in the Sun StorEdge T3+ array power cooling unit is not optimal. Recommended action: 1. T elnet to affected Sun[...]

  • Page 129

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 113 For Internal Use Only t3 enclosur e Alarm. time Discrepancy Y ello w [ Action ] T ime of T3 diag213 (ip=xxx.20.67.213) is differ ent from host: T3=Fri Oct 26 10:16:17 200, Host=2001-10-26 12:21:04 Recommended action: Fix the date and time on the Sun StorEdge T3+ array using the dat[...]

  • Page 130

    114 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 t3 ib Comm_Lost Down Y [ Info/ Action ] Lost communication (InBandwith diag213 (ip=xxx.20.67.21 3) ( last reboot was 2001-09-27 15:22:00) Information: InBand. This event is established using luxadm . This monitoring may not be activated for a particular Sun StorEdge T3+ arr[...]

  • Page 131

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 115 For Internal Use Only t3 t3ofdg Diagnostic T est- Red t3ofdg (diag240) on diag213 ( ip= xxx. 20.67.213 ) failed t3 t3test Diagnostic T est- Red t3test ( diag240 )o n diag213 (ip= xxx. 20.67.213 ) failed t3 t3volverify Diagnostic T est- Red t3volverify ( diag240 ) on diag213 ( ip =x[...]

  • Page 132

    116 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 t3 power Insert Component [ Info ] ’ power.u1pcu2’(TE CTROL-CAN.300- 1454- 01(50).008275 ) was added to T3 diag213 (ip=xxx.20.67.21 3 ) t3 enclosur e Location Change Location of t3 rasd2-t3b0 (ip=xxx.0.0.40 ) was changed t3 enclosur e QuiesceEnd Quiesce End on t3 d2-t3b[...]

  • Page 133

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 117 For Internal Use Only t3 disk Remove Component Red Y [ Info/ Action ] disk.u2d3(SEAGAT E.ST318203FSUN18 G.LRG07139 ) was removed fr om diag158 (ip=xxx. 20.67.158 ) Information: The Sun StorEdge T3+ array has reported a disk has been removed from the chassis. Recommended action: Rep[...]

  • Page 134

    118 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 t3 disk State Change+ disk.u1d5 in Sun StorEdge T3+ array rasd3-t3b1 (ip=xxx. 0.0.41 )i s now A vailable (status-state changed from fault- disabled to ready- enabled ) t3 interface. loopcard State Change+ [ Info ] loopcard.u1l1 ( SLR -MI.375-0085-01- G-G4.070924 )i n T3 msp[...]

  • Page 135

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 119 For Internal Use Only t3 contr oller State Change- Red Y [ Info/ Action ] controller.u1ctr in T3 diag213 (ip=xxx. 20.67.213 ) is now Not-A vailable (status-state changed from unknown to ready-disabled ) Information: The Sun StorEdge T3+ array controller has been disabled. Recommend[...]

  • Page 136

    120 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 t3 interface. loopcard StateChange- Red Y [ Info/ Action ] Information: The Sun StorEdge T3+ array has indicated that the loopcard is no longer in an optimal state. Recommended action: 1. T elnet to the affected Sun StorEdge T3+ array . 2. V erify loopcard state with fru st[...]

  • Page 137

    Chapter 8 T roubleshooting the Sun StorEdge T3+ Arr ay De vices 121 For Internal Use Only t3 volume StateChange- Red Y [ Info/ Action ] Information: The Sun StorEdge T3+ array has reported that a power cooling unit has been disabled. Recommended action: 1. Check the Sun StorEdge T3+ array syslog for battery hold times. 2. If < 6 minutes, replace[...]

  • Page 138

    122 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 Replacing the Master Midplane Follow this procedur e when replacing the master midplane in a Sun StorEdge T3+ array . This procedure is detailed in the Storage Automated Diagnostic Environment User ’ s Guide . ▼ T o Replace the Master Midplane 1. Choose Maintenance --&g[...]

  • Page 139

    123 CHAPTER 9 T r oubleshooting Ethernet Hubs The Sun StorEdge 3900 and 6900 series uses an Ethernet hub as the backbone for the internal service network. The allocation of Ethernet ports are as follows: ■ 1—Storage Service Processor (per subsystem) ■ 1—for each Fibre Channel Switch ■ 1—for each V irtualization Engine ■ 2—for each S[...]

  • Page 140

    124 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002[...]

  • Page 141

    125 APPENDIX For Internal Use Only A V irtualization Engine Refer ences This Appendix contains the following T ables: ■ T able A-1 “SRN and SNMP Reference” ■ T able A-2 “ SRN/SNMP Single Point of Failure T able ” ■ T able A-3 “Port Communication” ■ T able A-4 “Service Codes” T ABLE A-1 provides an explanation of Service Requ[...]

  • Page 142

    126 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 70005 W rite error is detected by master . If the initiator is master , then it has detected a write error on a member within a mirr or drive. If a spare drive is available, it will be brought in and used to r eplace the failed drive. If no spare is available, r eplace the f[...]

  • Page 143

    Appendix A Vir tualization Engine References 127 7009A Read degrade recorded . A mirr or drive was written to, causing it to enter the degrade state. Reinsert the missing drive, or r eplace it with a drive of equal or greater capacity . 7009B W rite degrade recorded . If a spare drive is available, it will be brought in and used to replace the fail[...]

  • Page 144

    128 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 72005 Failed to check for SAN changes. 72006 Failed to read SAN event log. 72007 SLIC daemon connection is down. W ait for 1-5 minutes for backup daemon to come up. If it doesn’t, check the network connection for virtualization engine halt, or hardwar e failure. T ABLE A-2[...]

  • Page 145

    Appendix A Vir tualization Engine References 129 T ABLE A-4 provides service codes for the virtualization engine. T ABLE A-3 Port Communication P or t Port P or t Number Daemon Management Programs 20000 Daemon Daemon 20001 Daemon virtualization engine 25000 virtualization engine virtualization engine 25001 T ABLE A-4 Service Codes Code Number Cause[...]

  • Page 146

    130 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 54 Unauthorized cabling configuration. • Check cabling. 57 T oo many HBAs attempting to log in. • Check cabling. 60 Node mapping table cleared using SW2. • No action r equired. 62 Improper SW2 setting. • Correct SW2 setting. • Cycle virtualization engine power . 12[...]

  • Page 147

    131 APPENDIX For Internal Use Only B SUNWsecfg Err or Messages The Sun StorEdge 3900 and 6900 Series Reference Manual lists and defines the command utilities that configur e the various components of the Sun StorEdge 3900 and 6900 series storage systems. The information in this appendix expands on that information by providing r ecommendations for [...]

  • Page 148

    132 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 . T ABLE B-1 V irtualization Engine SUNWsecfg Error Messages Message Description and Cause of Error Suggested Action Common to virtualization engines Invalid virtualization engine pair name $vepair , or virtualization engine is unavailable. Confirm that the configuration loc[...]

  • Page 149

    Appendix B SUNWsecfg Error Messages 133 Common to virtualization engine 1. Device-side operating mode is not set properly . 2. Device-side UID reporting scheme is not set properly . 3. Host-side operating mode is not set properly . 4. Host-side LUN mapping mode is not set properly . 5. Host-side Command Queue Depth is not set properly . 6. Host-sid[...]

  • Page 150

    134 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 createvezone Invalid WWN $wwn on $vepair initiator $init , or virtualization engine is unavailable. WWN that has already been specif ied has a SLIC zone and/or an HBA alias assigned. Note that for a WWN to be available for createvezone , the zone name in the map file ( showv[...]

  • Page 151

    Appendix B SUNWsecfg Error Messages 135 T ABLE B-2 Sun StorEdge Network FC Switch-8 and Switch-16 Switch SUNWsecfg Error Messages Message Description and Cause of Error Suggested Action Common Switch Sun StorEdge system type entered, ${cab_type} , does not match system type discovered, ${boxtype }. Either call the command with the -f force option t[...]

  • Page 152

    136 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 setswitchflash Invalid flash f ile $flashfile . Check the number of ports on switch $switch . Y ou might be attempting to download a flash f ile for an 8-port switch to a 16- port switch. Check showswitch -s $switch and look for “number of ports.” Ensure that this matche[...]

  • Page 153

    Appendix B SUNWsecfg Error Messages 137 T ABLE B-3 Sun StorEdge T3+ Array SUNWsecfg Error Messages Message Description and Cause of Error Suggested Action Common to Sun StorEdge T3+ array Present conf iguration does not match Reference conf igurations Check the present Sun StorEdge T3+ array configuration with showt3 -n <t3> command and verif[...]

  • Page 154

    138 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 checkt3config Snapshot configuration f iles are not present. Unable to check conf iguration. Make sure that the snapshot f iles are saved and have read permissions in the /opt/SUNWsecfg/etc/t3name/ directory . If the snapshot f iles are not available, , create them by using [...]

  • Page 155

    Appendix B SUNWsecfg Error Messages 139 restoret3config Error while the block size compar e command is executing. The $BRICK_IP{$IPADD} command is aborted. The Sun StorEdge T3+ array block size parameter is differ ent from the snapshot file. The Sun StorEdge T3+ array may have been reconf igured. Run restoret3config . restoret3config $LUN conf igur[...]

  • Page 156

    140 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002 T ABLE B-4 Other SUNWsecfg Error Messages Message Description and Cause of Error Suggested Action Common to all components If the Sun StorEdge 3900 or 6900 series has multiple (more than two) failur es (for example, both virtualization engines and two switches are down), the[...]

  • Page 157

    Appendix B SUNWsecfg Error Messages 141 setupswitch Exit V alues T ABLE 9-1 lists the setupswitch exit values. The associated messages are logged in the /var/adm/log/SEcfglog log file. T ABLE 9-1 setupswitch Exit V alues Severity Level Message T ype Message Meaning 0 INFO All switch settings are pr operly set. The switch setting matches the default[...]

  • Page 158

    142 Sun StorEdge 3900 and 6900 Series Troubleshooting Guide • March 2002[...]

  • Page 159

    Index 14 3 Index A accessing documentation online, xv C checkswitch used to diagnose and troubleshooting switch, 62 comments sending documentation comments, xv configuration settings, 47 verification of, 47 D data host verification for Sun StorEdge 39x0 series, 42 for Sun StorEdge 69x0 series, 42 diagrams fibre channel link, 15 , 1 6 documentatio[...]

  • Page 160

    Index 144 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002 H health functions for Sun StorEdge 3900 and 6900 series, 2 host device names translating, 78 host devices troubleshooting, 53 host event grid, 53 host side troubleshooting, 18 I IO suspension of, 10, 1 3 isolation procedur es for A2/B2 link, 3 3 L link error example [...]

  • Page 161

    Index 145 For Internal Use Only notification events, 103 T1/T2 data path troubleshooting, 102 test examples command line, 1 9 qlctest(1M), 19 switchtest(1M), 20 thresholds used in PF A, 2 troubleshooting broad steps, 3 check status of Sun StorEdge T3+ array, 4 check status of the Sun StorEdge FC Network Switch-8 and Switch-16, 5 check status of th[...]

  • Page 162

    Index 146 Sun StorEdge 3900 and 6900 Series T roubleshooting Guide • March 2002[...]