Cisco Networkers 2006 - SAN-3608 - Advanced SAN Troubleshooting

Advanced SAN Troubleshooting
SAN-3608
SAN -3608 12527_04_2006_c1
2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
Agenda/First Half
Overview of Data Gathering Capabilities of SAN/OS Network DesignKeep Troubleshooting in Mind Analyzer Uses Trace and Fabric Manager Tool Uses Tie Each Learned Skill to a Lab Exercise
SAN -3608 12527_04_2006_c1
Cisco Public
MDS Switch Troubleshooting Levels

Command line debugs and show commands
Used by network Level 23 engineers
Analyzers and SPAN

Used by Level 3 engineers
Fabric manager and device manager

Used by network operators and Level 1 engineers
General statistics gathering from show commands and GUI managers

Network admins and operators Level 1 engineering
SAN -3608 12527_04_2006_c1
Cisco Public
Command Line Debugging

Available debugs depend on features enabled in SAN/OS Many, many different options to select when turning on debugs Where is it output going?
LogfileData file in switch memory Capture to direct to screen via console, telnet or SSH
Requires admin privileges to run debugs Debugs can only be run from CLI No debugging available from Fabric Manager or Device Manager
N 12527_04_2006_c1
SAN -3608 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
Debug Logging
1. Display captured debug to screen
Tip: use show debug to see name of debug file TOP-9216i# show debug logfile networkers_debug
2. Copy debug file off MDS to a server

TOP-9216i# copy log:networkers_debugs ftp: Enter hostname for the ftp server: 10.91.42.166 Enter username: networkers Password: networkers
To delete the debug logfile

TOP-9216i# clear debug-logfile networkers_debugs Or TOP-9216i# undebug all Or debug logfile will be cleared and over written when next debug logfile is created, only one debug logfile is allowed by system
SAN -3608 12527_04_2006_c1
Cisco Public
Debugs to Direct Telnet Window

Use a telnet/SSH or console application that will capture the expected output to buffer or file Undebug all or no debug of specific debug command is required to turn trace off The debugs are not persistent across reboots Most debugs are very readable and sensible to understand, some not
N 12527_04_2006_c1
SAN -3608
Cisco Public
Debug Outputs
Debug Example
TOP-9216i# debug flogi ? action Configure debugging of flogi actions all Configure all debug flags of flogi demux Configure debugging of flogi message demux error Configure debugging of flogi error event Configure debugging of flogi FSM and Events ha Configure debugging of flogi HA init Configure debugging of flogi adds, deletes and inits timers Configure debugging of flogi message timers trace Configure debugging of flogi trace warning Configure debugging of flogi warning
Some debugs are selfexplanatory, others not Drilling down to most single item you wish to view like the interface or VSAN helps in narrowing output Debugs are usually run as part of a recreate for an issue such as a bad Fibre Channel ID being assigned or a Host login issue Some debugs require expertise in state machine and good general knowledge of FC and SCSI standards
N 12527_04_2006_c1
SAN -3608
Cisco Public
Show Commands
Hundreds of show commands Unlike debug commands, much information from show commands is available via the fabric manager and device manager Many show commands can be several screen pages. Use show terminal command to see information about your telnet terminal properties.
Screen will pause at each screen unless terminal length is temporally configured different for current output
Setting terminal length = 0 helpful in gathering long outputs, i.e. show tech-support detail
N 12527_04_2006_c1
Cisco Public
Design for Troubleshooting

Leverage VSAN design to support troubleshooting mythology Have a SPAN port allocated in port count needs Integrate a Cisco Port Analyzer Adapter (PAA) into SAN design Provision an Analyzer (Finisar, Agilent, other) in the network, keep it operational with FTP access to extract traces Design for Syslog servers and scheduled configuration saves (use CFS to simplify)
SAN -3608 12527_04_2006_c1
Cisco Public
How Do We Troubleshoot the Network

SPAN (Port Monitor) on Cisco L2 Switch Run Ethereal on IP Host
mds-speed.server2 qlogic:networkers1
FC Analyzer Remote
Ethernet
Out-of-Band Management Network
FC Analyzer Local
Server2
microsoft:server1 mds-speed.server3 FC Port Channel FC ISL Ethereal PC
Terminal
BOT-9216i
TOP-9216i
Server3 PAA
Clarion Cx300 Server1 Seagate 92:7a:47-S
Fabric Manager and Device Manager Debugs and Show Commands, Telnet or Console
SAN -3608 12527_04_2006_c1
PC
Seagate 92:7b:94-S
Seagate 92:7b:c5-S
Seagate 67:67:87-S
Span FC Ports, Port Channels, iSCSI or FCIP Port to SD Port on MDS

Cisco Public 10
Gathering Protocol Traces for Analysis

Using built-in FC Analyzer (CLI)
Linux version from Ethereal.com
Using Ethereal on PC (local and remote) Using the MDS port analyzer adapter w/SPAN Using an external FC Analyzer tester in line or with SPAN
Non-Disruptive to Switch Operations and Traffic on the SAN

SAN -3608 12527_04_2006_c1 2006 Cisco Systems, Inc. All rights reserved.
Cisco Public
11
MDS FC Analyzer (SAN/OS Embedded)

Output is displayed to the console in readable sniffer-like format Is only used to monitor Fibre Channel traffic to and from supervisor on the MDS9000
Traffic-like fabric logins, FSPF routing, switch-to-switch control traffic
Output can go direct to your console screen or to a workstation running Ethereal program
Note: SPAN Is Used for FC Port to FC Port Monitoring

Cisco Public
12
FC Analyzer Options
Local or remoteWhere to send the trace. Can be to local devices or remote PAA attached to different MDS switch.
Brief or detailedHeader information vs. full output of frame including hex. Detail is default.
Limit-captured-framesNumber of frames to capture. Default is FC analyzer will trace 100 frames. Specifying zero is unlimited frame capture. Limit-capture-sizeAllows to capture N number of bytes of frame. Useful for not capturing frame data when it is not relevant to troubleshooting.
Cisco Public
13
Display-Filter Options
When not specified, FC analyzer will capture all traffic on VSAN 1
Example: fcanalyzer local brief
To specify a different VSAN, use a display-filter and specify the VSAN to be captured in hex or decimal
Example for 100 (note the two equal signs): fcanalyzer local display-filter mdshdr.vsan==0x64 or mdshdr.vsan==100
To capture for a specific address in VSAN 100:

Example:
fcanalyzer local brief display-filter ((fc.d_id==64.01.00)or(fc.s_id==64.01.00))
Recommend to use remote capture method for ease of filter capabilities on the Ethereal GUI
Cisco Public
14
Using Write Option for fcanalyzer

Using write option sends output of the fcanalyzer to a file on the switch, directory called volatile. This trace file can be then copied off MDS switch and viewed with ethereal app on PC
Example: Capture 250 frames of all traffic on VSAN 200
fcanalyzer local display-filter mdshdr.vsan==0xc8 write volatile:capture.trc limit-captured-frames 250
The file name on volatile: filesystem will have extra characters appended. Issue following command to see contents of filesystem and then copy the file via tftp/ftp
dir volatile: copy volatile:capture_00001_20050321172628.trc tftp://<tftp server ip addr>/capture.trc Added Characters
When File Is Written
Cisco Public
15
Use of SPAN Feature

Used for FC port to FC port analyzing Same type of tool as used on Cisco Catalyst products. Cisco Catalyst uses port monitor. Can be left configured on switch Ingress and egress ports are sent to an FC-port setup as a SPAN Destination (SD-port type) No limits to where the ports are located on the MDS switch network Used to output to third-party test equipment or to Cisco Port Analyzer Adapter
N 12527_04_2006_c1
Cisco Public
16
FCAnalyzer Local Brief

Capture Is Done in Configuration Mode
TOP-9216i# conf t Enter configuration commands, one per line. End with CNTL/Z. TOP-9216i(config)# fcanalyzer local brief display-filter mdshdr.vsan==2 Warning: Couldn't obtain netmask info (eth2: no IPv4 address assigned). Capturing on eth2 2.829871 2.853261 2.853422 2.853592 2.853565 2.859648 2.860885 2.861007 2.861175 2.862053 2.865904 2.865981 2.866153 2.866297 2.866445 2.866496 2.866615 2.868792 2.868857 2.871132 2.872013 2.872021 2.872139 2.872163 2.872234 2.891239 2.891359 2.891469 2.891613 2.900160 2.900394 2.901916 2.902151 2.908296 2.908444 2.919880 00.00.00 -> ff.ff.fe ff.ff.fe -> 7e.01.00 7e.01.00 -> ff.ff.fc ff.ff.fc -> 7e.01.00 7e.01.00 -> ff.ff.fd ff.fc.7e -> 7e.01.00 7e.01.00 -> ff.fc.7e ff.fc.7e -> 7e.01.00 7e.01.00 -> ff.fc.7e 7e.01.00 -> ff.ff.fc ff.fc.7e -> ff.fc.0a ff.fc.7e -> ff.fc.cb ff.fc.0a -> ff.fc.7e ff.fc.cb -> ff.fc.7e ff.fc.0a -> ff.fc.7e ff.fc.7e -> ff.fc.0a ff.ff.fd -> 7e.01.00 ff.fc.cb -> ff.fc.7e ff.fc.7e -> ff.fc.cb ff.fc.7e -> 7e.01.00 7e.01.00 -> ff.fc.7e 7e.01.00 -> ff.fc.7e ff.fc.7e -> 7e.01.00 ff.fc.cb -> ff.fc.7e ff.fc.7e -> ff.fc.cb ff.fc.7e -> ff.fc.cb ff.ff.fc -> 7e.01.00 7e.01.00 -> ff.ff.fc ff.fc.cb -> ff.fc.7e ff.ff.fc -> 7e.01.00 7e.01.00 -> ff.ff.fc ff.ff.fc -> 7e.01.00 7e.01.00 -> ff.ff.fc ff.ff.fc -> 7e.01.00 7e.01.00 -> ff.ff.fc ff.ff.fc -> 7e.01.00 0x2288 0xffff FC ELS FLOGI 0x2288 0xc728 FC ELS ACC (FLOGI) 0x22a0 0xffff FC ELS PLOGI 0x22a0 0xc729 FC ELS ACC (PLOGI) 0x22b8 0xffff FC ELS SCR 0xc72c 0xffff FC ELS PLOGI 0xc72c 0x22d0 FC ELS ACC (PLOGI) 0xc72b 0xffff FC ELS PRLI 0xc72b 0x22e8 FC ELS ACC (PRLI) 0x2300 0xffff dNS RFT_ID 0xc72e 0xffff SW_ILS SW_RSCN 0xc72f 0xffff SW_ILS SW_RSCN 0xc72e 0x77f9 FC Link Ctl, ACK1 0xc72f 0x77fa FC Link Ctl, ACK1 0xc72e 0x77f9 SW_ILS SW_ACC (SW_RSCN) 0xc72e 0x77f9 FC Link Ctl, ACK1 0x22b8 0xc72a FC ELS ACC (SCR) 0xc72f 0x77fa SW_ILS SW_ACC (SW_RSCN) 0xc72f 0x77fa FC Link Ctl, ACK1 0xc730 0xffff FC ELS LOGO 0xc730 0x2318 FC ELS ACC (LOGO) 0x2318 0xffff FC ELS PLOGI 0x2318 0xc731 FC ELS LS_RJT (PLOGI) 0x77fb 0xffff dNS GE_ID 0x77fb 0xc732 FC Link Ctl, ACK1 0x77fb 0xc732 dNS ACC (GE_ID) 0x2300 0xc72d dNS ACC (RFT_ID) 0x2330 0xffff dNS RFF_ID 0x77fb 0xc732 FC Link Ctl, ACK1 0x2330 0xc733 dNS ACC (RFF_ID) 0x2378 0xffff dNS RNN_ID 0x2378 0xc734 dNS ACC (RNN_ID) 0x23a8 0xffff dNS RSNN_NN 0x23a8 0xc735 dNS ACC (RSNN_NN) 0x23c0 0xffff dNS GNN_FT 0x23c0 0xc736 dNS ACC (GNN_FT)
Brief Option Used to get Single Line Caption Along with a Display Filter to Narrow Output to only VSAN 2 Control-C Stops Trace Capture
Display filters are a must for narrowing output on a busy network. See MDS Config Guide for other filter types. Capture is done by default to console screen, so make sure you are able to save output to large capture buffer or log with your telnet application
N 12527_04_2006_c1
Cisco Public
17
FCAnalyzer Local Detail

(PLOGI)
Port Login
VSAN 2
Fabric Domain Controller
Class 3 Device
FCID of HBA Device

SAN -3608
N 12527_04_2006_c1
Cisco Public
18
MDS FC Analyzer Remote

For capture of internal MDS9000 switch traffic, FC traffic to and from supervisor not FC port-to-FC port traffic Make sure MDS can ping workstation
TOP-9216i
At DOS Prompt:
ethereal -i rpcap://10.91.42.183/eth2
Or at Ethereal Interface Drop-Down:

rpcap://10.91.52.183/eth2
Management 10/100 Port on MDS Address 10.91.42.183 SCSI HBA Workstation Running Ethereal IP Address 10.91.42.167
N 12527_04_2006_c1
Cisco Public
19
Ethereal on PC
Filters applied after capture or before. Filters are stored in application cache for later use.
Same Output as FC Analyzer Local but in Nice Format with the Ethereal GUI
SAN -3608 12527_04_2006_c1
Cisco Public
20
MDS 9000 Port Analyzer Adapter

MDS Port Analyzer Adapter is a protocol converter
Converts FC frames into Ethernet frames
FC-specific information is carried in encapsulation header

Enables use of Ethernet protocol analyzers to decode FC protocols
Cisco has released decodes for all FC protocols, the encapsulation header and SCSI decoder to public domain
Part of the popular Ethereal protocol analyzer
SAN -3608 12527_04_2006_c1
Cisco Public
21
Port Analyzer Connection Using Port Analyzer with Ethereal

For Capture of FC Device to FC Device, Sends Fibre Channel Traffic to an IP-Attached Workstation Running Ethereal
Configure Port Analyzer Adapter (dip-switch settings, mostly plug and play) Configure SPAN port on MDS9000 and what device ports you wish to sniff
Set Ethereal on PC to capture on local Ethernet interface
TOP-9216i FC
Cisco Port Analyzer Adapter
FC
SD Port FC 2/16
Raid Array
Workstation L2 Network Running Ethereal

Cisco Public
22
PAA & Ethereal
Capture on Ethernet Interface Connected to PAA
FCPING from Domain Controller in Domain 6f to FCID 6f.02.00 ELS Echo
Boardwalk Is Cisco Code for PAA
SPAN Configuration with Single FC Port Analyzer Tool

TOP-9216i interface fc1/13 switchport description Span Destination TX RX
FC 1/2
switchport mode SD switchport speed 1000 no shutdown
FC 1/2 RX input is sent to FC 1/13 SD port FC 1/2 TX input is sent to FC 1/13 SD port Dropped
span session 1 destination interface fc1/13 source interface fc1/2 rx source interface fc1/2 tx
SAN -3608 12527_04_2006_c1
SPAN Destination Port (SD) FC 1/13
TX RX
TX RX
FC Analyzer
Cisco Public
24
RSPAN Requirements
All switches must be running MDS code 1.2 or later No cable should be attached to the SFP in the ST (SPAN Tunnel) port Make sure FC tunnel is UP before starting frame collection FC Analyzer can be a PAA and PC running Ethereal, or a Finisar-like analyzer
SAN -3608 12527_04_2006_c1
Cisco Public
25
Fabric Manager/Device Manager

The MDS Fabric Manager Server is your friend Used in combination with CLI interface helps tremendously in situations were you cannot remember all the configuration steps Use CLI to verify outputs to the managers but trust the manager displays to be proper
SAN -3608 12527_04_2006_c1
Cisco Public
26
Fabric Manager Views

Show Tech Support output Zoning Checks Domain Manager ID across all VSANs FSPF Health
RSCN Statistics and health
SAN -3608 12527_04_2006_c1
Cisco Public
27
Device Manager Views

For Troubleshooting:
Span configuration Monitor traffic Monitor interface Errors and statistics View persistent logs Manage files in flash
SAN -3608 12527_04_2006_c1
Cisco Public
28
Lab 1
Lab intro Debug output example Ethereal tool usage PAA and SPAN usage FM and DM tool navigation
Seagate 92:7b:c5-S fc1/5 Server2 Seagate 92:7b:94-S
fc1/6 fc1/6 fc1/6
PAA
Server3 fc1/2 iscsi1/1
Seagate 92:7a:47-S
fc1/6
TOP-9216i
fc1/1
fc1/14 Clarion Cx300
iqn.1991-05.com. cisco.mds-speed.server3 iscsi1/1
fc1/6 fc1/6 fc1/6 fc1/6
channel1, channel1 fc1/14
iscsi1/1 fc1/1
Seagate 67:67:87-S fc1/5 Bot-9216i iscsi1/1
iqn.1991-05.com. mds-speed.server2
Server1
iqn.1991-05.com. microsoft:server2
SAN -3608 12527_04_2006_c1
Cisco Public
29
Section Agenda/Second Half

Fibre Channel Layer Review Determining Low-Level Interface Physical Issues Best Practices for Troubleshooting Device Connectivity
Lab
Inter-Switch LinkUnderstanding and Troubleshooting Switch Domains and VSANs Zoning Details Inter-VSAN RoutingIVR
Lab
Cisco Public
30
Fibre Channel Layers

Review
Structure Is Divided into Five Levels of Functionality
FC-0defines the physical interface characteristics
Signaling rates, cables, connectors, distance capabilities, etc.
FC-1defines how characters are encoded/decoded for transmission

Transmission characters are given desirable characters
FC-2defines how information is transported

Frames, sequences, exchanges, login sessions
FC-4s FC-3 FC-2 FC-1 FC-0
SCSI
IP Common Services Signaling Protocol Transmission Code Physical Interface FC-PH
FC-3place holder for future functions FC-4defines how different protocols are mapped to use Fiber Channel
SCSI, IP, Virtual Interface Architecture, others
SAN -3608 12527_04_2006_c1
Cisco Public
31
Fibre Channel Transmission Words: FC-1

Ordered sets are transmitted continuously to indicate that specific conditions within the link are encountered
Ordered sets transmitted while the condition exist

Four primitive sequences which can determine where problem exist
Not Operational Sequence (NOS)
Offline Sequence (OLS) Link Reset Sequence (LR) Link Reset Response Sequence (LRR)
SAN -3608 12527_04_2006_c1
Cisco Public
32
Link Initialization Flow Fibre Channel Layer 1 Protocol (FC-1)

Port A Port B
Link Failure Condition

These Are All Special-Ordered Sets of 8B/10B Coding:
AC
AC
NOS
LF
LF = Link Failure State NOS = Not Operational Sequence
LF
OLS LR
OL
LRR
OLS = Offline Sequence

OL = Offline State LR = Link Reset
LR LR
Idle
LRR = Link Reset Response
AC
Idle
LR = Link Recovery State

AC = Active State
AC
Cisco Public
33
Fibre Channel Port Issues

In Order for an F_Port to Come Up on an MDS Switch:
1. The switch port must first acquire bit and word synchronization with the N_Port
2. N_Port must issue a FLOGI to the MDS

Primitive Sequences Counters Can Determine Layer 01 Problems
Tip:
Clear Counters and Monitor to Verify Active Issues, Use Device Manager Monitor Tool to Monitor Live; Set and Activate Threshold Manager to Alert You; TOP-9216i# clear counters interface fc 1/1
Cisco Public
34
Fabric Services Refresher

FC Standards
FFFFFEWell-known address of login server
Basic responsibility to assign FCID
FFFFFCWell-known address of fabric directory name server

Basic responsibility registration of devices and distribution of database
FFFFFDWell-known address of fabric controller

Basic duties responsible for managing fabric, initialization, FSPF routing, responsible for RSCNs and listens for SCNs
FFFcxxCisco Specific, Representing Name Server in a VSAN and Domain xx

Cisco Public
35
Line Card Basics

Show module will display slot locations and type of card General information in interface and statistics can be gathered from switch main-level prompt More detailed gathering of ASIC counters may be required to troubleshoot difficult issues, attach would then be required
Attach to Module with Attach Command, All Modules Can Be Attached to, Including Standby Supervisor and IPS Blade Exit to Detach
SAN -3608 12527_04_2006_c1
Cisco Public
36
Using FC Test Equipment for Physical Issues

Use MDS show commands and debugs to narrow scope of physical issues External analyzers will be required for detail diagnostics of intermittent physical layer issues and use of triggers to trap problem Testers must go inline, disruptive to network connection. Plan accordingly.
TOP-9216i FC
FC
TX RX
TX RX
FC Analyzer
N 12527_04_2006_c1
SAN -3608
Cisco Public
37
Errors on Application Server Logs, User Having Performance Problems

Use logging capabilities on switch to piece together network issues Put together:
1. What interfaces are on the fabric (up, down, flapping) 2. What was health of each interface (virtual and physical interfaces) 3. Examine interface health (FC protocol errors, physical layer errors) 4. Are effects being seen on the complete network fabric or within the VSAN (RSCNs, Zones, ISLs, Errors on common controllers)
SAN -3608 12527_04_2006_c1
Cisco Public
38
Necessary Network Settings

Switch Fabric clocks synchronized
Use NTP along with CFS to simplify job Use fabric manager to set all clocks and time zones
Setup system logs, syslogd server

Use CFS to simplify
Save off switch configurations or even complete show tech-support details regularly
Use CLI scheduler in SAN/OS to simplify process
SAN -3608 12527_04_2006_c1
Cisco Public
39
Monitor Link Init State

(Flow from Bottom to Top)
Attached to Mod 1 FC Port 1/2
module-1# show hardware internal fc-mac port 2 stateinfo (older Line Cards) module-1# show hardware internal fc-mac2 port 2 state-info-log (newer Line Cards F-Port Point to Point Negotiated
LINK: 052 022180c5 LINK: 051 022180c3 LR_RECEIVE(03) =>ACTIVE(01) OLS_TRANSMIT(07) =>LR_RECEIVE(03)
LINK: 050 022180c1 SENTINAL(00) =>OLS_TRANSMIT(07) LOOP: 049 022180c1 HW_ALPAS(0d) MASTER_LISA_WAIT(1f)=> OLD_PORT(3f) LOOP: 048 022180c1 HW_ALPAS(0d) MASTER_LISA(1e)=> MASTER_LISA_WAIT(1f) LOOP: 047 022180c1 HW_ALPAS(0d) MASTER_LIHA_WAIT(1d)=> MASTER_LISA(1e) LOOP: 046 022180c1 HW_ALPAS(0d) MASTER_LIHA(1c)=> MASTER_LIHA_WAIT(1d) LOOP: 045 022180c1 HW_ALPAS(0d) MASTER_LIPA_WAIT(1b)=> MASTER_LIHA(1c)
Port Tries Loop First When Port Is Set to Auto
Interface Shut/ No Shut via Configuration
LOOP: 044 022180c1 HW_ALPAS(0d) MASTER_LIPA(1a)=> MASTER_LIPA_WAIT(1b) LOOP: 043 022180c1 HW_ALPAS(0d) MASTER_LIFA_WAIT(19)=> MASTER_LIPA(1a) LOOP: 042 022180c1 HW_ALPAS(0d) MASTER_LIFA(18)=> MASTER_LIFA_WAIT(19) LOOP: 041 022180c1 HW_X_ARB(0c) MASTER_START(17)=> MASTER_LIFA(18) LOOP: 040 022180c1 HW_LISM0(0a) OPEN_INIT_SELECT_MASTER(06)=> MASTER_START(17) LOOP: 039 022180bf HW_R_LIP(09) OPEN_INIT_START(05)=> OPEN_INIT_SELECT_MASTER(06) LOOP: 038 022180bf HW_X_LIP(08) NORMAL_INITIALIZE(04)=> OPEN_INIT_START(05) LOOP: 037 022180bf HW_R_LIP(09) LPSM_STARTED(01)=> NORMAL_INITIALIZE(04) LOOP: 036 022180b0 HW_OLDP(07) LPSM_DISABLED(00)=> LPSM_STARTED(01) LINK: 035 022170e8 ACTIVE(01) =>SENTINAL(00)
SAN -3608 12527_04_2006_c1
Cisco Public
40
Show flogi Database

FLOGI stands for fabric login Shows which devices are logged into the fabric Only shows devices that have flogied into this switch not complete multi switch network
TOP-9216i# show flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------fc1/1 2 0x7e00ef 50:06:01:60:30:20:54:f0 50:06:01:60:b0:20:54:f0 fc1/2 2 0x7e0100 21:00:00:e0:8b:17:2f:50 20:00:00:e0:8b:17:2f:50 fc1/6 5 0xa3026e 21:00:00:04:cf:92:7b:c5 20:00:00:04:cf:92:7b:c5 fc1/6 5 0xa30271 21:00:00:04:cf:92:7b:94 20:00:00:04:cf:92:7b:94 fc1/6 5 0xa30272 21:00:00:04:cf:92:7a:47 20:00:00:04:cf:92:7a:47 fc1/6 5 0xa30273 21:00:00:04:cf:67:67:87 20:00:00:04:cf:67:67:87 fc1/14 2 0x7e03ef 50:06:01:69:30:20:54:f0 50:06:01:60:b0:20:54:f0 iscsi1/1 2 0x7e0004 21:07:00:0d:ec:20:97:c2 21:08:00:0d:ec:20:97:c2 iscsi1/1 5 0xa30002 21:01:00:0d:ec:20:97:c2 21:12:00:0d:ec:20:97:c2 iscsi1/1 5 0xa30004 21:07:00:0d:ec:1f:82:42 21:11:00:0d:ec:20:97:c2
N 12527_04_2006_c1
SAN -3608
Cisco Public
41
Show Fibre Channel Name Server Database

Logging into fabric with FLOGI and registering with nameserver are separate events Shows all devices in entire fabric FCID shown is the FCID assigned by the switch that device FLOGIed into Initiator or target is attribute registered with switch from device during device login Vendor field is determined by OUI assigned by IEEE
SAN -3608 12527_04_2006_c1
Cisco Public
42
Show fcns database

Information output of show Command
You will Require the Domain List from the Switches to Know Where the Devices Are Physically Connected on the SAN, Could Be IVRed from Other VSAN
sh fcdomain domain-list vsan 2
Target or Initiator Passed to Switch via PROCESS ACCEPT and ACCEPT Frames During Nameserver Login
OUI is Part of the PWWN and Is IEEE Assigned. 00:e0:8b is Assigned to Qlogic, 00:00:c9 is Assigned to Emulex, and 00:04:cf is Assigned to Seagate, Many Others
Cisco Public
43
Show fcns database detail

Gives Detailed Information About Each Device (Only One Shown)
Device Registered and Thus Indicates It Can Converse with Class of Service 3 ---------------------------------------------------------------------------------VSAN:2 FCID:0x7e0100 ----------------------------------------------------------------------------------port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic) node-wwn :20:00:00:e0:8b:17:2f:50 class :3 Registered to Talk SCSI node-ip-addr :0.0.0.0 over Fibre Channel ipa :ff ff ff ff ff ff ff ff fc4-types:fc4_features :scsi-fcp:init symbolic-port-name : symbolic-node-name :QLA2340 FW:v3.03.13 DVR:v9.0.2.17 (w32) port-type :N FWWN This Device Attaches port-ip-addr :0.0.0.0 to the Fabric via fabric-port-wwn :20:02:00:0d:ec:20:97:c0 hard-addr :0x000000 permanent-port-wwn (vendor) :21:00:00:e0:8b:17:2f:50 (Qlogic)
Cisco Public
44
Debugging Name Server Issues

If a Host Is Not Receiving a Proper Response, or if Not Able to Discover a Target, Enable the Following Debug and Shut/No Shut the Host Interface
1. Check if the host and the disk are in the same VSAN (use fcping or fctrace, show VSAN membership) 2. Debug fcns events (this will give the request and response from the fabric name server) 3. Check if the host and the disk are in the same zone (show zoneset active vsan x); look for the * next to each device FCID; missing * means device is not in the fabric
SAN -3608 12527_04_2006_c1
Cisco Public
45
FCAnalyzer Brief Results During HBA Login

CLI View
Device Logging into Fabric Initial Sends FCID with All 0s Switch Sends ACCept Back and Assigns the HBA a FCID HBA Does Port Login to the Directory Server and Receives ACC Back HBA Registers for State Change Registration (SCR) to be Notified by the Switch if There any Changes in Fabric in Same Zone and Receives ACCept from Switch HBA Registers with Name Server, RFT Switch-to-switch communication noted by SW_ILS. Registered state change notifications going out to zone. MDS Begins to Query the Device with RNN_ID, RSNN_NN, GNN_FT
From FC-GS-4 standard Register FC-4 Type, Register Node Name, Register Symbolic Node Name, Get Node Names-FC4 Type, Get Platform Name
Cisco Public
46
HBA Login
Ethereal Application View
HBA and FC port on switch finish physical level handshake and HBA logs into port server a FF.FF.FE Login Server assigns FCID from Domain 6f (VSAN 200) 6F.02.00 HBA logs into Fabric Name Server HBA registers for State Change Notification, response comes back in frame 7 HBA asks Name Server for FC-4 FC devices registered in the server. Response comes back in frame 24 Name Server from VSAN 200 MDS9216-TOP quires HBA Name Server from VSAN 200 MDS9216-Bottom quires HBA HBA Port logs to FC-4 devices received from accept frame 24 HBA talks to SCSI devices
SAN -3608
N 12527_04_2006_c1
Cisco Public
47
Link Events on a Switch

Per module/per switch command Can be taken from Show tech-support detail Helps drill down as to what devices were leaving fabric at time of problem and general reason Shows both physical and virtual interfaces
SAN -3608 12527_04_2006_c1
Cisco Public
48
Gathering Internal Counter Information for Unknown Issues and Plaguing Connectivity Problems
Determine what ports on which switches you need to examine, this is done by narrowing to physical switch, VSAN, zone Look at path through complete network, initiator and target-side along with ISLs One stop shopping on each switch for all the data:
Attach to Line Card Module that has the interface you need information on
module-1# terminal length 0

Set scroll back buffer or log to file on telnet tool
module-1# show hardware internal debug-info interface fc1/2

Based on Line Card type proper information will be output
Cisco Public
49
show hardware internal debug-info

Script Runs Giving Outputs Of:
show hardware internal errors show port-config internal link-events
show port-config internal port-control

show port-config internal port-cfg-error
We Have Already Looked at Details of Theses 2 Outputs
show hardware internal fc-mac2 port 2 link-status show hardware internal fc-mac2 port 2 port-info show hardware internal fc-mac2 port 2 misc-statistics
show hardware internal fc-mac2 port 2 status-reg

show hardware internal fc-mac2 port 2 state-info-log show hardware internal fc-mac2 port 2 gbic-info show hardware internal fc-mac2 port 2 statistics show hardware internal fc-mac2 port 2 flow-control show hardware internal fc-mac2 port 2 config-reg show hardware internal fc-mac2 port 2 mask-reg show hardware internal packet-flow interface fc1/2
Cisco Public
50
Lab 2
HBA Login Lab Physical Layer Troubleshooting Debugging Fibre Channel services
SAN -3608 12527_04_2006_c1
Cisco Public
51
ISL Connection
ISL connections require proper standards operation
FSPF routing Proper capabilities and parameter exchange
Proper domain ID assignment which could be disruptive or non-disruptive to network

Proper merging of fabric zoning
ISL will not fully initialize unless FC-SW rules are complete
TOP-9216i
BOT-9216i
SAN -3608 12527_04_2006_c1
Cisco Public
52
Switch-to-Switch, the ISL E-Port

Fibre Channel Standard FC-SW addresses ISL operation FC incorporates distributive methods across the SAN Fabric E-Ports seen as SW_ILS in protocol field of traces Switches if determine connecting FC port is an ISL link Much can go wrong as part of the initial ISL handshake ISL needs to be analyzed for situations of domain ID assignment, routing, and zoning distribution issues
N 12527_04_2006_c1
Cisco Public
53
Principal Switch Selection

Review
Only one switch is designated principal switch in a fabric The switch with the lowest WWN becomes the principal switch at new install of SAN after that other rules apply Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric Only one upstream link per switch
Switch 1
Switch 2
Switch 3
Switch 4
Switch 5
Switch 6
N 12527_04_2006_c1
SAN -3608
Cisco Public
54
Fabric Configuration: PS Selection

Review
A principal switch shall be selected when at least one inter-switch link (link between two E_Port) is established
The selection process chooses a principal switch, which is then designated to assign domain identifiers to all the switches in the fabric, existing and new
SAN -3608 12527_04_2006_c1
Cisco Public
55
Fabric Configuration Process

Review
The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses The process is summarized in the following steps
Establish link parameters and switch port operating mode Principal switch selection Domain ID distribution Path selection
N 12527_04_2006_c1
SAN -3608
Cisco Public
56
Fabric Configuration Disruptive/Non-Disruptive

One of the following three conditions can trigger Build Fabric, BF (non-disruptive) or Reconfigure Fabric, RCF (disruptive)
Two disjointed fabric are combined together The principal ISL link fails (upstream or downstream) A switch with Domain_ID request for another Domain_ID
Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; it forwards BF/RCF out of all E_ports except the incoming port (only once) and waits for the timer to expire When the timer expires, BF/RCF propagation state is over and principal switch selection begins
Cisco Public
57
Domain IDs
CLI View
Fabric Manager View
Device Manager View
SAN -3608 12527_04_2006_c1
Cisco Public
58
Cisco TE EISL
TE Port SPAN to PAA
ACK1 Filter Applied Exchange Link Parameters Exchange Switch Capabilities Exchange Peer Parameters (This Is Proprietary to MDS Only to Determine if Connecting Switch Is Another MDS) Exchange Fabric Parameters Build Fabric Domain ID Assign by Existing Principal Switch Request Domain ID from New Switch Enhanced Zoning Merge Request Resource Allocation (New in FCSW3 Standard) Zone Merge Request
Cisco Public
59
Domain ID Problems
Show interface Check fabric manager Check event logs
MDS9216-Bottom# sh int fc 1/9 fc1/9 is down (Isolation due to domain overlap) Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:09:00:0d:ec:0d:0c:c0 Admin port mode is E, trunk mode is off Port vsan is 500 Receive data field Size is 2112 Beacon is turned off 5 minutes input rate 360 bits/sec, 45 bytes/sec, 0 frames/sec 5 minutes output rate 264 bits/sec, 33 bytes/sec, 0 frames/sec 2433 frames input, 198208 bytes 0 discards, 0 errors 0 CRC, 0 unknown class 0 too long, 0 too short 2423 frames output, 128924 bytes 0 discards, 0 errors 19 input OLS, 22 LRR, 1 NOS, 26 loop inits 310 output OLS, 107 LRR, 13 NOS, 10 loop inits 255 receive B2B credit remaining 255 transmit B2B credit remaining
SAN -3608 12527_04_2006_c1
Cisco Public
60
Troubleshooting Zoning
Show zoning information from CLI on all switches in fabric
Show zone status
Show zoneset active
Is zone configuration stored and controlled from one switch or any/all switches? Make sure you are working with active zone when in CLI
zone copy active-zoneset full-zoneset vsan 5
*Key concern when working in mixed vendor interop VSANs
Verify distribution and default values (permit-deny) See event logs for clues about issues Verify device in logged into fabric
Cisco Public
61
Zoneset Tools in FM
Where Is Active ZoneSet in Each VSAN Applied? Backup Zone Configuration to File on Server
Current Zoneset Status
SAN -3608 12527_04_2006_c1
Cisco Public
62
Show Zoneset Active

Shows the zoneset activated in the fabric An asterisk to left of device indicates that device has registered with the nameserver Does not show if a zone has been configured but changes have not been activated Zoneset and zone names are case sensitive Inter-VSAN routing zone added to ZoneSet via IVR wizard if this VSAN has member being zoned to another VSAN
Domain a3(163) Is in VSAN 5
N 12527_04_2006_c1
Cisco Public
63
Activate Zoneset Flow Across ISLs

Zoneset Distribution Would Go to Every Domain Within the VSAN
ACAAcquire Change Auth
Lock the Fabric
ACC
Move the Zone Data to the Switches Trigger Switches to Activate the New Zoneset Unlock the Fabric
SFCStage Fabric Conf

ACC UFCUpdate Fabric Conf ACC RCARelease Change Auth ACC
ACK1 Frames Are Not Shown Here

Cisco Public
64
Activate Zoneset with FC Analyzer and VSAN Filter

Only VSAN 100 FC interface 1/16
SAN -3608 12527_04_2006_c1
Cisco Public
65
IVR Troubleshooting Tips

Start out with valid well thought-out IVR design Pay close attention to zoning Show IVR Tech-Support Look at complete IVR path Use CFS (Cisco Fabric Services)
Domain ID limit rules in Interop VSANs also apply to the IVR Domains (97127 rule)
SAN -3608 12527_04_2006_c1
Cisco Public
66
Which Switch, What Zone, What Feature

CLI Displays Will Show Zones from IVR Feature While Displays on FM Only Display All None IVR Zones
TOP-9216i# sh zoneset brief active zoneset name ZoneSet1 vsan 2 zone Server1 zone Server3 zone iSCSIserver3 zone iSCSIserver2 zone IVRZ_Server1-IVR
TOP-9216i# sh ivr zoneset brief active zoneset name IvrZoneSet1 zone name Server2-IVR zone name Server1-IVR
SAN -3608 12527_04_2006_c1
Cisco Public
67
Lab 3
TE port initialization
Detailed look
Domain ID assignments
Adding a domain (switch) to the SAN
Zone merge
Rules of zoning
SAN -3608 12527_04_2006_c1
Cisco Public
68
MDS Documents and Whitepapers on Net

http://cisco.com/en/US/products/hw/ps4159/ps4358/prod_ white_papers_list.html http://www.cisco.com/univercd/cc/td/doc/product/sn5000/mds9000/ interop/index.htm http://www.cisco.com/en/US/products/hw/ps4159/tsd_products_ support_category_home.html http://www.emc.com/interoperability/index.jsp http://www.redbooks.ibm.com/abstracts/sg246397.html?Open ftp://service.boulder.ibm.com/storage/san/cisco/ http://h18006.www1.hp.com/products/storageworks/cmds9509/ index.html
SAN -3608 12527_04_2006_c1
Cisco Public
69
Q and A
SAN -3608 12527_04_2006_c1
Cisco Public
70
Recommended Reading
Continue your Networkers learning experience with further reading from Cisco Press
Check the Recommended Reading flyer for suggested books
Available Onsite at the Cisco Company Store

SAN -3608 12527_04_2006_c1
Cisco Public
71
Complete Your Online Session Evaluation!

Win fabulous prizes; Give us your feedback Receive ten Passport Points for each session evaluation you complete Go to the Internet stations located throughout the Convention Center to complete your session evaluation Drawings will be held in the World of Solutions
Tuesday, June 20 at 12:15 p.m. Wednesday, June 21 at 12:15 p.m. Thursday, June 22 at 12:15 p.m. and 2:00 p.m.
SAN -3608 12527_04_2006_c1
Cisco Public
72
SAN -3608 12527_04_2006_c1
Cisco Public
73

Cisco Networkers 2006 - SAN-3608 - Advanced SAN Troubleshooting

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Networkers 2006 - SAN-3608 - Advanced SAN Troubleshooting

Uploaded by

Copyright:

Available Formats

Advanced SAN Troubleshooting

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

MDS Switch Troubleshooting Levels

Analyzers and SPAN

Fabric manager and device manager

General statistics gathering from show commands and GUI managers

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

Command Line Debugging

2. Copy debug file off MDS to a server

To delete the debug logfile

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

Debugs to Direct Telnet Window

2006 Cisco Systems, Inc. All rights reserved.

2006 Cisco Systems, Inc. All rights reserved.

Design for Troubleshooting

2006 Cisco Systems, Inc. All rights reserved.

How Do We Troubleshoot the Network

microsoft:server1 mds-speed.server3 FC Port Channel FC ISL Ethereal PC

Clarion Cx300 Server1 Seagate 92:7a:47-S

Span FC Ports, Port Channels, iSCSI or FCIP Port to SD Port on MDS

2006 Cisco Systems, Inc. All rights reserved.

Gathering Protocol Traces for Analysis

Non-Disruptive to Switch Operations and Traffic on the SAN

MDS FC Analyzer (SAN/OS Embedded)

Note: SPAN Is Used for FC Port to FC Port Monitoring

To capture for a specific address in VSAN 100:

Using Write Option for fcanalyzer

fcanalyzer local display-filter mdshdr.vsan==0xc8 write volatile:capture.trc limit-captured-frames 250

Use of SPAN Feature

FCAnalyzer Local Brief

FCAnalyzer Local Detail

Fabric Domain Controller

FCID of HBA Device

2006 Cisco Systems, Inc. All rights reserved.

MDS FC Analyzer Remote

Or at Ethereal Interface Drop-Down:

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

MDS 9000 Port Analyzer Adapter

FC-specific information is carried in encapsulation header

2006 Cisco Systems, Inc. All rights reserved.

Port Analyzer Connection Using Port Analyzer with Ethereal

Set Ethereal on PC to capture on local Ethernet interface

Cisco Port Analyzer Adapter

Workstation L2 Network Running Ethereal

PAA & Ethereal

Capture on Ethernet Interface Connected to PAA

FCPING from Domain Controller in Domain 6f to FCID 6f.02.00 ELS Echo

Boardwalk Is Cisco Code for PAA

SPAN Configuration with Single FC Port Analyzer Tool

switchport mode SD switchport speed 1000 no shutdown

SPAN Destination Port (SD) FC 1/13

2006 Cisco Systems, Inc. All rights reserved.

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.

Fabric Manager/Device Manager

SAN -3608 12527_04_2006_c1

2006 Cisco Systems, Inc. All rights reserved.