SAN Extension

Design and Operation

BRKSAN-2704

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
 Design Section Agenda

ƒ Basic SAN Extension Principles

Dual Path, HA, and Transport Options

ƒ Enhancing SAN Extension

Optical: Extended B2B Credits, B2B Credit Spoofing,
Port Tracking
FCIP: Compression, Encryption, Application Acceleration, QOS

ƒ Advanced FCIP SAN Extension Designs

Multiple Tunnels, Multiple Links

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3

SAN Design
Principles

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2

Presentation_ID.scr
 Design Criteria
Factors to Consider for SAN Extension
ƒ Applications using SAN Extension
Synchronous/asynchronous replication, data backup/restore

ƒ Application latency requirements

Applications that use synchronous replication may be impacted

ƒ Application throughput requirements

Determines bandwidth requirements

ƒ Transport options available

What choices are available for SAN Extension

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

Typical SAN Design

Site A
ƒ Servers with two Fibre
Channel connections to
Access
Fabrics FC
storage arrays for high
availability
Replication
Fabrics Use of multipath software is
required in dual fabric host design

DC ƒ SAN extension fabrics

Interconnect typically separate from
Network
host access fabrics
Site B Replication fabric requirements
vary by replication method
Replication deployed
Fabrics FC

“A” “B”
Fabrics Fabrics
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3

Presentation_ID.scr
 Basic HA SAN Extension Network
Site A
High-Availability
Replication Design:
Access
VSANs FC
ƒ Conventional approach is dual
Replication fabrics (e.g., yellow VSAN and
VSANs
blue VSAN) over distance
ƒ “Client protection”—arrays
DC
Interconnect
provide protection against
Network failures in either fabric
Site B ƒ May be augmented with
additional “network protection”
Replication via port channels and/or
VSANs FC
optical protection schemes

“A” “B”
VSANs VSANs
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

SAN Extension Design: Adding Link HA

Site A
Port Channels Increase
Resilience for High-Availability
Access
VSANs
with FC or FCIP Links
FC

Replication
ƒ Appears as a single
VSANs logical link (up to
sixteen member links)
Protecting the fabric
Port Channels from network failure
ƒ Route port-channel
member links over
Site B
diverse geographic paths
Replication ƒ Load balancing on SRCID/
VSANs FC DESTID or SRCID/DESTID/
OXID basis (unidirectional
per VSAN)
“A” “B” SCSI exchange is smallest atomic
VSANs VSANs unit, so frame order kept intact
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8

© 2006, Cisco Systems, Inc. All rights reserved. 4

Presentation_ID.scr
 Hybrid SAN Extension Implementations
Hub and Spoke
FC
FC
FC FC

FC FC
FC

FC
FC IP FC
Network
FC

Central Site Remote Sites

Multi-Hop
Primary DC Secondary DC Backup Site FC
FC FC

FC FC
DWDM IP FC
Network

Synchronous Replication Asynchronous Replication/Backup

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9

SAN Extension
Solutions

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5

Presentation_ID.scr
 SAN Extension Solutions
Internet SAN Extension Design Options
Fibre Channel over IP (FCIP)
• Asynchronous data replication (T1-multiGigabit
Stateful Firewalls • Higher latency but longer distances
Content Caching SONET/SDH
High Server Load Balancing • Mostly asynchronous data replication
Density • Typically OC-3 to OC-48
Multilayer WAAS
LAN
CWDM
Switch
Intrusion Detection • Synchronous and asynchronous data replication
• Low interface density: 1–8 FC/FICON interfaces
Front-End The Integrated Data
• High bandwidth: 1G-2G lambdas
Application Center Encompassing
Servers Extensive Availability DWDM
with iSCSI LAN/SAN services ƒ Synchronous and asynchronous data replication
ƒ High interface density
Back-End ƒ High bandwidth: 2.5G and 10G lambdas
iSCSI Application
Servers
IP WAN The Remote Data Center
High- Backup
Density
Multilayer
and/or
SAN Fibre Channel over IP Hot-Standby
SONET/SDH
Director Fibre Channel over SONET Servers
Fibre Channel over CWDM
Fibre Channel over DWDM CWDM Point-Point Enterprise-
ESCON over DWDM Class Tape
Arrays
Enterprise-Class DWDM
Network Enterprise-Class
Storage Arrays Storage Arrays
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11

SAN Extension Technology Options

Increasing Distance
Data
Center Campus Metro Regional National Global

Dark Fiber Sync Limited by Optics (Power Budget)

Optical

CWDM Sync (1,2 Gbps) Limited by Optics (Power Budget)

DWDM Sync (1,2,4,10 Gbps per λ) Limited by BB_Credits

SONET/SDH Sync (1,2 Gbps + Subrate) Async

IP

MDS9000 FCIP Sync (Metro Eth) Async (WAN,1 Gbps)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12

© 2006, Cisco Systems, Inc. All rights reserved. 6

Presentation_ID.scr
 Dark Fiber
ƒ Single 1/2/4/10 Gbps
FC link per fiber pair
FC
Port Channel SW (850 nm) over 62.5/
2-16 1/2/4/10 125 µm multimode
Gbps FC SW (850 nm) over 50/125 µm
over Two
multimode
Diverse
Paths LW (1310 nm) over 9/125 µm
Distance Based
single mode
Diverse Paths—
on Fiber Type,
Optic Type,
Multiple ƒ Client protection only; Upper
Fiber Pairs
Link Speed Each Path Layer Protocol (ULP), either
SAN or application,
responsible for
failover protection
FC

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

Coarse Wavelength Division

Multiplexing (CWDM)
1470 nm OADM
Mux/Demux 1490 nm
1510 nm
1530 nm
1550 nm
1570 nm
1590 nm
1610 nm

ƒ 8-channel WDM at 20-nm spacing (cf DWDM at <1-nm spacing)

1470, 1490, 1510, 1530, 1550, 1570, 1590, 1610 nm
ƒ “Colored” CWDM SFPs (or GBICs) used in FC switches
(no transponder required)
ƒ Optical multiplexing done in CWDM OADM (optical
add/drop multiplexer)
Passive (unpowered) device; just mirrors and prisms
ƒ Up to 30-dB power budget (36-dB typical) on SM fiber
~100-km point-to-point or ~40-km ring
ƒ 1/2 gigabit Fibre Channel and 1 Gigabit Ethernet currently
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7

Presentation_ID.scr
 Two-Site CWDM Storage Network
ƒ HA resilience against fiber
cut—“client” protection
FC 4-member port channel—
2 x 2 diverse paths
Diverse
Paths: Port channel appears
One Fiber Port Channel as single logical link
Pair Each 2-16 1/2 Gbps
MUX-4 MUX-4 FC over Two E_Port or TE_Port for
Path
Pass Diverse Paths carriage of VSANs

Network Pass Network Load balance by src/dst

MUX-4
(or src/dst/oxid)
MUX-4
2 Gbps Fiber cut will halve capacity
CWDM from 16 Gbps to 8 Gbps but
SFPs not alter fabric topology—
no FSPF change
FC ƒ MUX-8 would double capacity
or leave spare wavelengths
for GigE channels
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

CWDM Optics Without Multiplexor

ƒ CWDM optics do
not require MUX
FC If dark fiber available, can
4 Fibre
Paths
be used like typical SFPs
Between
Each ƒ Can use different wavelengths
Switch
Port Channel
or the same wavelengths on
4 x 2 Gbps all interfaces
ƒ Use of optical attenuators may
be required for shorter
2 Gbps 2 Gbps
CWDM CWDM distance fiber runs
SFPs— SFPs—Same
Optical power meter used to
Different Wavelength
Wavelengths
measure signal strength

FC

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8

Presentation_ID.scr
 Dense Wavelength Division
Multiplexing (DWDM)

ƒ Higher density than CWDM

32 lambdas or channels in narrow band around 1550 nm
at 100G-Hz spacing (0.8 nm)

ƒ Erbium-Doped Fiber Amplifier (EDFA) amplifiable

allows for longer distances than CWDM
ƒ Carriage of 1, 2, 4, or 10 Gbps FC, FICON, GigE,
10 GigE, ESCON, IBM GDPS
ƒ Data center to data center
ƒ Protection options: client, splitter, or linecard

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

DWDM Protection Alternatives

for Storage
Optical Splitter Protection
ƒ Single transponder required
Optical Working Lambda
Splitter ƒ Protects against fiber breaks
ƒ Failover causes loss of light
(and fabric change if only link)

Protected Lambda

Linecard or Y-Cable Protection

Y-cable Working Lambda ƒ Dual transponders required
More expensive than
splitter-based protection

ƒ Transmits over both circuits,

but only one accepted
Protected Lambda

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18

© 2006, Cisco Systems, Inc. All rights reserved. 9

Presentation_ID.scr
 DWDM HA Storage Network Topology
ƒ Client protection
recommended
FC Fabric and application responsible
for failover recovery
Diverse ƒ Port channel provides
Paths— Port Channel
One-Fiber resilience
2–16 1/2/4/10
Pair Each Gbps FC over Port-channel members
Path Two Diverse follow diverse paths
DWDM Paths
Ring Single fiber cut will not affect fabric
(no RSCNs, etc.)
Use “Src/Dst” hash for load
balancing (rather than
“Src/Dst/Oxid” per exchange)
for each extended VSAN
FC

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

Fibre Channel over SONET/SDH

ƒ FC over SONET/SDH (FCoS)
follows same distance rules as
FC
other optical technologies
Port Channel BB_Credits in Fibre Channel
2–16 1/2 Gbps switch limits distance
FC over Two
Diverse Paths ƒ Outage in SONET/SDH
network will not cause loss
SONET / of light
SDH
Network Recovers in <50 ms

Diverse
May cause some loss BB_Credit
Network loss from in flight traffic
Paths MDS9000 will recover
lost BB_Credits
FC

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10

Presentation_ID.scr
 Fibre Channel over IP (FCIP) Design
ƒ Same port channeling and
VSAN trunking rules apply
FC
as with FC links
ƒ Port channel individual FCIP
links to separate Ethernet
Port Channel
Diverse
2–16 FCIP switches/routers
Network
Interfaces Each WAN link carries
Paths
Over Diverse
two FCIP tunnels
Paths
Load balancing on SRCID/DESTID
or SRCID/DESTID/OXID basis
(unidirectionally per VSAN)
Certain replication protocols
require SRCID/DESTID load
balancing
FC
FICON, IBM PPRC,
HP CA-EVA
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

FCIP Frame Detail

Ethernet IP TCP TCP FCIP EISL opt Ethernet
SOF

FC Frame
Header Header Header Opts Header Hdr Hdr CRC32
14 20 20 12 28 4 8 0-16
4

94 Max 2148 (E_Port) + EISL and Opt Headers

FCIP Overhead for EISL and Optional Headers
Ethernet Frames:
If TE_Port, then 8 Bytes
94 Byte Header + 4 Added to FC Frame (After
Byte CRC = 98 Bytes SOF) for VSAN Routing

ƒ Max Fiber Channel frame is 2148 bytes plus optional extras

ƒ FCIP will segment and reassemble FC frames if MTU too
small (TCP payload on second or subsequent packets)
ƒ Jumbo frames may increase performance
IP MTU of 2300 avoids splitting of TCP frames

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11

Presentation_ID.scr
 Storage Traffic and TCP

ƒ Storage traffic:
Quite bursty
Latency sensitive (sync apps)
Requires high, instantaneous throughput

ƒ Traditional TCP:
Tries to be network sociable
Tries to avoid congestion (overrunning downstream routers)
Backs off when congestion detected
Slow to ramp up over long links (slow start and
congestion avoidance)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

MDS FCIP TCP Behavior

ƒ Reduce probability of drops

Bursts controlled through per flow shaping and congestion
window control Æ less likely to overrun routers

ƒ Increased resilience to drops

Uses SACK, fast retransmit and shaping

ƒ Aggressive slow start q

Initial rate controlled by “min-available-bandwidth”
Max rate controlled by “max-bandwidth”
Differences with Normal TCP:
ƒ When congestion occurs with other conventional TCP traffic, FCIP
is more aggressive during recovery (“bullying” the other traffic)
Aggression is proportional to the min-available-bandwidth configuration
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12

Presentation_ID.scr
 Frame Buffering: FCIP and FC
Traffic Flow

TCP Windowing Flow Control

FC FC

BB_Credit BB_Credit
Flow Control Flow Control

GigE Slower GigE

WAN Link
FC Receive Buffers FCIP Receive Buffers

Backlog Here if Queue Can’t Drain Due To:

ƒ Slow WAN link and long RTT
ƒ Packet loss and retransmissions
ƒ Many sources (only one shown)
ƒ Buffer too big

ƒ FCIP presents a lower bandwidth pipe (if WAN link)

Drain rate (send rate) depends upon bandwidth and congestion
ƒ Slow ramp up of traditional TCP can cause FC frame expiry in
some conditions
Mixture of slow link (e.g., <DS3/E3; retransmissions, many sources, big buffers)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

FCIP TCP Packet Shaping: MDS9000

Shaping Avoids Congestion at This Point
Source Destination
Gigabit Gigabit
Ethernet 45 Mbps Ethernet

Traffic Flow

Source Sends
Packets at Rate
Consumable by
Downstream Path Interpacket Gap to Accommodate Slow
Downstream Link (e.g., 45 mbps)

ƒ Shaper sends at a rate consumable by the downstream path

Immediately sends at “minimum-bandwidth” rate (avoids early stages of
traditional slow start)
Ramps up to “maximum-bandwidth” rate (using usual slow start and
congestion avoidance methods)
ƒ Requirements for shaper to engage:
Min-available-bandwidth > 1/20 max-bandwidth
SACK (Selective Ack) must be enabled
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13

Presentation_ID.scr
 MDS9000 FCIP TCP Behavior
ƒ For example: a dedicated link
Entire link is always available,
so… “min bandwidth” = “max
bandwidth”
FCIP will always send at 95% to
100% of max rate without ramp up
Traffic is shaped at sending rate
(max-bw)

Dedicated FCIP Link

After retransmission (congestion),
Minimum Bandwidth = Maximum Bandwidth Rate send er resumes at min
(= max rate)

ƒ Behavior mimics UDP

“blast” but with benefits of
retransmission capability
and shaping

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

Enhancing
SAN Extension

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14

Presentation_ID.scr
 Extending Optical SAN Extension
BB_Credits and Distance

~2 km per Frame
1 Gbps FC

~1 km per Frame
2 Gbps FC

~½ km per Frame
4 Gbps FC

16 Km

ƒ BB_Credits are used to ensure enough FC frames in flight

ƒ A full (2112 byte) FC frame is approx 2-km long @ 1 Gbps, 1-km long @
2 Gbps and ½-km long at 4 Gbps
ƒ As distance increases, the number of available BB_Credits need
to increase as well
ƒ Insufficient BB_Credits will throttle performance—no data will be
transmitted until R_RDY is returned
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Extending Optical SAN Extension

Fibre Channel Frame Buffering

Traffic Flow

FC FC

BB_Credit BB_Credit BB_Credit

Flow Control Flow Control Flow Control

2-8 BB_Credit 16-255 BB_Credit 2-8 BB_Credit

FC Receive FC Receive
Buffers Buffers

ƒ Buffer to buffer credits (BB_Credit) are negotiated between each device in

a FC fabric; no concept of end to end buffering
ƒ One buffer used per FC frame, irregardless of frame size; small FC frame
uses same buffer as large FC frame
ƒ FC frames buffered and queued in intermediate switches
ƒ Hop-by-hop traffic flow paced by return of Receiver Ready (R_RDY)
frames; can only transmit up to the number of BB_Credits before traffic
is throttled
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15

Presentation_ID.scr
 Extending Optical SAN Extension
SAN Network Solutions for Increasing Distance

Shared Dedicated
FX-Port / Extended Max
FX-Port FX-Port E-Port E-Port Credits Speed Distance (*) Disable Ports for
(Fixed) (Default) (Default) (Min-Max) (Min-Max) (Gbps) (km) Max Credits?
1 510
16 Port 1/2 Gbps N/A 16 255 2-255 N/A No
2 255

1 24
32 Port 1/2 Gbps 12 12 12 12 N/A No
2 12

14/2 Port 1/2 Gbps 1 7000 Disable 3 in a 4-Port

256-
N/A 16 255 2-255 Group, if More than
9216i Fabric Switch 3500 2 3500 2400 per Port
12 Port 1/2/4 Gbps N/A 16 250 1 8190
256-
24 Port 1/2/4 Gbps 16 16 250 2-250 2 4095 No
4095
48 Port 1/2/4 Gbps 16 16 125 4 2047
751-
4 Port 10 Gbps N/A 16 750 2-750 10 800 No
4095

(*) Assuming Max Frame Size

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Extending Optical SAN Extension

Optical Solutions for Increasing Distance

ƒ ONS 15454 SL-Series Card

No
Spoofing Spoofing
ƒ Negotiates up 255 BB_Credit
with FC switch
FC
Negotiation

R_Rdy
Frame
B2B

ƒ Spoofs R_RDYs to FC switch

(release 5.0)
B2B Negotiation

ƒ Has 1200 BB_Credits between

Frame Ack
R_Rdy
Frame

Frame

SONET/ SL cards
SDH
Network ƒ Extends distances to 2300 km
Negotiation

@ 1G FC or 1150 km
R_Rdy
Frame
B2B

@ 2G FC

FC

Configuration Note: When Using ONS 15454 R_RDY Spoofing with MDS 9000, Disable BB State
Change Notification on Connected MDS Port: interface fc 1/1 no switchport fcbbscn
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32

© 2006, Cisco Systems, Inc. All rights reserved. 16

Presentation_ID.scr
 Improving Optical Recovery
Port Track for Resilient SAN Extension Solutions

1) MDS Detects Link Failure

2) MDS Brings
Down Array Port Optical Network

3) Array Retries I/O Optical Network

on Alternate Path

ƒ Arrays recover from a link failure via I/O timeouts; however, this
can take several seconds or longer
ƒ MDS port-track addresses this by monitoring the WAN/MAN link
and if it detects a failure, it will bring down the corresponding link
connected to the array
ƒ The array after detecting a link failure will redirect the I/O to
another link without waiting for the I/O to timeout
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

Improving Optical Recovery

Port Tracking and ONS FLC or Squelching

3) MDS Detects Link Failure 1) ONS Detects Link Failure

4) MDS Brings
Down Array Port Optical Network

2) ONS Brings Down

Client Transponder

5) Array Retries I/O Optical Network

on Alternate Path

ƒ The MDS port-tracking feature can be used with the ONS 15530
Forward Laser Control (FLC) or ONS 15454 squelching feature to
further track failures in the network, improving the ability to detect
failed paths
ƒ Forward laser control, squelching and port-tracking offer
end-to-end path failure detection
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17

Presentation_ID.scr
 FCIP Data Compression
ƒ Cisco uses RFC standard compression algorithms implemented in
both hardware and software
ƒ MDS 9000 8-port IP Storage Services Module
Software-based compression for FCIP
ƒ MDS 9000 14/2-port Multiprotocol Services Module
Hardware and software-based compression and hardware-based encryption
for FCIP
ƒ MDS 9000 18/4-port Multiservice Module
Hardware and software-based compression, hardware-based encryption,
and intelligent fabric-based application services
ƒ Three compression algorithms—modes 1–3 plus auto mode
ƒ Compressibility is data stream dependent
All nulls or ones → high compression (>30:1)
Random data (e.g., encrypted) → low compression (~1:1)
ƒ “Typical” rate is around 2:1, but may vary considerably
ƒ Application throughput is the most important factor
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

FCIP Data Compression and

TCP Windowing
ƒ Compression has the effect
of a variable bandwidth path
ƒ TCP window applies to data
stream before compression
If window size not increased,
throughput will not increase
Need to compensate with larger
TCP max window size

ƒ MDS9000 incorporates
moving average feedback
to dynamically adjust TCP window
according to compression rate
ƒ Feedback mechanism is not
available when using IP network-
based compression solutions—
manual adjustment of TCP
window size required
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18

Presentation_ID.scr
 Compression Performance Envelopes

MSM
(SAN-OS 3.x)
Mode - Auto

MPS
(SANOS 2.x)
Mode 1

IPS
(SANOS 2.x)
Mode 1

ƒ IPS = MDS 9000 IP Storage

IPS/MPS
(SANOS 2.x)
Services Module
Mode 2 ƒ MPS = MDS 9000 14/2-Port
Multiprotocol Services Module
IPS/MPS ƒ MSM = MDS 9000 18/
(SANOS 2.x) 4-Port Multiservice Module
Mode 3

1 10 100 1000

Network Speed (Mbps)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

IPSec Encryption for FCIP

FCIP Link Encryption Provides:
ƒ Data confidentiality—sender can encrypt packets
before transmitting them across a network
ƒ Data integrity—receiver can authenticate packets sent
by the IPSec sender to ensure that the data has not
been altered during transmission
ƒ Data origin authentication—receiver can authenticate
the source of the IPSec packets sent; this service is
dependent upon the data integrity service
ƒ Anti-replay protection—receiver can detect and reject
replayed packets

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19

Presentation_ID.scr
 Hardware-Based IPSec Encryption
Remote Tape Backup

Primary Site

Remote Replication IP Network

Tape Backup and Remote Replication

Secured with IPsec

ƒ Hardware-based GigE wire-rate performance with latency

~ 10 µs per packet
ƒ Standards-based IPSec encryption—implements RFC 2402
to 2410, and 2412
IKE for protocol/algorithm negotiation and key generation
Encryption: AES (128 or 256 bit key), DES (56 bit), 3DES (168 bit)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Write Acceleration
ƒ Enables extended distance capabilities for remote replication technologies
ƒ Better performance using FC/FCIP-WA—up to 2X the performance over
given distance
Reduces effective I/O latency within SAN extension solutions
ƒ Built into services modules (IPS, MPS, MSM, SSM)—transparent to
disk arrays
ƒ Highly resilient solution—no data stored in MDS 9000 switch

FCIP Write Acceleration (WA) Write Acceleration (at Various

Link Speeds and Write Sizes)
FC or IP 2.00
Network 1.90
WA WA 1 Gbps
1.80
622 Mbps
WRITE 1.70
XFER_RDY 1.60 155 Mbps
Ratio

DATA XFER_RDY
1.50
45 Mbps
1.40 32kB 45M
1.30 32kB 155M
1.20 32kB 622M
STATUS
1.10
32kB 1G
1.00
Reduction in I/O Latency Equal
1 2 3 4 5 6 7 8 9 10 11 12 13
to One Round Trip Time (RTT)
RTT (ms)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20

Presentation_ID.scr
 Write Acceleration Design
Considerations
High Availability and
Load Balancing:
FC ƒ Can be used for native FC
replication (SSM) or FCIP
replication (IPS, MPS
and MSM)
Port channel ƒ Port channels may be used
for HA
ƒ Equal cost FSPF load
balancing for FCIP Write
Acceleration not supported
ƒ Works with:
EMC SRDF, Mirrorview,
SANCOPY
FC HDS TrueCopy
HP CA-XP, CA-MVA
IBM FlashCopy, FastT
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Additional FC-WA Information

ƒ Fibre Channel write acceleration requires specific

source and destination device configuration
FC-WA requires the initiator and target be connected to the
SSM ports
Configure scsi-flow feature on specific SSM ports
Configure the specific initiator and target PWWNs and
VSANs to use FC-WA
Configure the number of 2K buffers to be used by FC-WA
(Number of concurrent SCSI writes * size of SCSI writes in
bytes)/FCP data frame size in bytes)

ƒ No restrictions on HA design; FSPF and port

channeling may be used

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21

Presentation_ID.scr
 FCIP Tape Acceleration

Server Tape
CMD Cmd-1
Xfer_Rdy
Data Cmd-2
Status
Cmd-3

Write Filemarks Cmd

Write Filemarks Status

ƒ Tape acceleration is an enhancement of write acceleration

ƒ Local MDS proxies as a tape library
ƒ Remote MDS proxies as a backup server
ƒ Status proxied
ƒ Write Filemarks checkpoints process
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

FCIP Tape Acceleration Flow Control

Server Flow Tape
CMD Cmd-1 Control
Xfer_Rdy
Data Cmd-2
Status
Cmd-3
-1
tus
Cmd-4 Sta

Xfer_Rdy

ƒ Default for tape acceleration flow control buffering is “automatic”

Adjusts to change in WAN latencies
Adjusts to speed of the tape library
ƒ Flow control buffer can be manually set
Is not normally required
Fixes buffer size, network or device performance changes will not be adjusted
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22

Presentation_ID.scr
 FCIP Tape Read Acceleration
Server Tape
Tape Read “N” Cmd
Tape Read “N” Cmd
CMD
Data
Status
Tape Read “N+1” Cmd

Tape Read “N+1” Cmd

Tape Read “N+2” Cmd
Tape Read “N+2” Cmd

ƒ After first Read command received, Tape-end FCIP sends Read

commands to prefetch data
ƒ Host-end FCIP buffers the prefetched data
ƒ For all Tape Read commands from the host
Host-end FCIP sends the cached data for the read and sends a SCSI good status
tape-end FCIP consumes the read command
ƒ When a non-Read command is received, Tape-end FCIP rewinds the tape
A non-sequential read is treated as a nonread by the MDS
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

Tape Acceleration Throughput Results

Performance Remains Linear with Increasing RTT or Distance Even

at 10 ms or 500 KM, Tape Performance Without Tape Acceleration
Is less than 10 MB

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46

© 2006, Cisco Systems, Inc. All rights reserved. 23

Presentation_ID.scr
 FCIP TA Design Requirements

ƒ Tape acceleration is not supported over port channels

or equal cost FSPF paths
Requirement due to state information kept in interface

ƒ HA design requires costing parallel links so one path

preferred over another
HA through redundant links

ƒ Can use separate VSAN and FCIP tunnel to allow

port channels for write acceleration

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

SAN Extension Fabric Stability

ƒ Connecting existing SAN fabrics or extending a
SAN fabric creates SAN design challenges
Minimize the impact of change in fabric services across
geographically distributed sites
Limit fabric control traffic such as RSCNs and build/reconfigure
fabric (BF/RCF) to local VSANs
Connecting SAN fabrics with the same domain IDs
ƒ Inter-VSAN Routing (IVR) can be used to address
these challenges
IVR enables a layer-3 hierarchical layer design to SAN fabrics
to isolate domains
IVR only sends selective RSCNs to edge switches, preventing
disruption of fabric services
IVR with NAT allows two existing SAN fabrics with the same
domain ID to be connected through a third transit VSAN
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24

Presentation_ID.scr
 SAN Extension with IVR
Site A Local
VSAN_5
ƒ Any failure in transit VSAN_20
(network equipment, physical
FC
Replication
VSAN_10
or logical failure) will not
disrupt VSAN_10 or

Inter-VSAN Connection between

VSAN_30 fabric

Completely Isolated Fabrics

Transit
ƒ Works with any transport
VSAN_20 service (FC, SONET/SDH,
(IVR)
DWDM/CWDM, FCIP)

Site B
Host to Local Array Fabric is VSAN_5
Site A Replication Fabric is VSAN_10
FC
Replication Site B Replication Fabric is VSAN_30
VSAN_30
SAN Extension Fabric is VSAN_20

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

QoS for FCIP SAN Extension

Most FCIP Implementations Use Dedicated Links, However:
ƒ Acceptance of FCIP generating interest in converged
IP network for FCIP SAN Extension
ƒ No generally accepted DSCP values for FCIP traffic
(unlike voice with DSCP EF)
QoS—define marking and classification
Mark DSCP according an agreed value
Separate consideration of FCIP data and control packet

ƒ Bandwidth reservation
FCIP has no support for reservation protocol
Simulated using min/max B/W command
(can be considered as a type of reservation)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25

Presentation_ID.scr
 FCIP QoS Mapping Proposal

ƒ Synchronous data replication: bursty, high bandwidth

Can be mapped into mission critical (AF31/DSCP 26)

ƒ Asynchronous data: bursty, low to medium b/w

Can be mapped into transactional data (AF21/DSCP 18)
Can also mapped into bulk data (AF11/DSCP 10)

ƒ Backup data: 150 ~ 500 ms, constant (during backup),

medium b/w
Can be mapped into bulk data (AF11 / DSCP 10)

ƒ Control packets
Both control and data traffic can be assigned the same class
If needed can assign CS6 or DSCP 48
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

Classification and Marking Design

Modified for SAN Traffic

L3 Classification L2
Application
IPP PHB DSCP CoS
Routing / FCIP control 6 CS6 48 6
Voice 5 EF 46 5
Video Conferencing 4 AF41 34 4
Streaming Video 4 CS4 32 4
FCIP SYNC 3 AF31* 26 3
Call Signaling 3 CS3* 24 3

FCIP ASYNC 2 AF21 18 2

Network Management 2 CS2 16 2

FCIP backup /FCIP Async 1 AF11 10 1
Scavenger 1 CS1 8 1
Best Effort 0 0 0 0
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26

Presentation_ID.scr
 Advanced FCIP
Designs

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Advanced FCIP TA Design

Scaling TA Beyond 1 FCIP Tunnel/GE Interface

FCIP1 VSAN 10
FCIP2 VSAN 20
FCIP3 VSAN 30
MDS1
MDS2 FCIP4 VSAN 40 MDS2

VSAN 100
Eight Tape
Libraries
(Three Tapes
Each)
Distributed
over 4 VSANs
FCIP tape acceleration is limited to one parallel path, but …
• Move the target tapes devices into separate VSANs
• Make each FCIP interface only a member of one tape target
VSAN and ensure trunking turned off
• Use IVR to allow all backup servers to access all tape devices
in all target VSANs
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27

Presentation_ID.scr
 Advanced FCIP TA Design
Bidirectional Multiple Path TA

FCIP1 VSAN 10

FCIP2 VSAN 20
MDS2 MDS2

VSAN 100 VSAN 200

IVR Site 1 Backup Servers in IVR Site 2 Backup Servers in

VSAN 100 to Local and Remote VSAN 200 to Local and Remote
Tapes in VSAN 10 and VSAN 20 Tapes in VSAN 10 and VSAN 20

Both Local and Remote Backup Servers Need to Access Tape

in Two Locations
• Move the target tapes devices into separate VSANs
• Make each FCIP interface only a member of one tape target VSAN and
ensure trunking turned off
• Use IVR to allow all backup servers to access all tape devices in all
target VSANs
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

FCIP WA and TA—Multiple Tunnels

Port channel FCIP interfaces 11 & 21 for
replication only
int fcip 11 int fcip 11 VSAN 100
VSAN 100 GigE1/1.100 VSAN 100 GigE1/1.100 Replication
Replication GigE1/1 int fcip 12 int fcip 12 GigE1/1
GigE1/1.200 VSAN 300 GigE1/1.200
VSAN 300
int fcip 21 int fcip 21 Tape Drives
GigE1/2.100 VSAN 100 GigE1/2.100
VSAN 200 GigE1/2 int fcip 22 int fcip 22 GigE1/2
GigE1/2.200 VSAN 400 GigE1/2.200 VSAN 400
Backup Serv
Tape Drives

Do not port channel FCIP interfaces 12 &

22 for tape backup

ƒ Using GE sub-interfaces, multiple FCIP tunnels and port channeling to

enable high b/w FCIP
ƒ Use separate VSANs for data replication (100), backup servers (200)
and tape backup (300, 400)
ƒ Port-channel FCIP tunnels for replication traffic
ƒ Distribute tape drives over multiple VSANs and FCIP tunnels to permit
tape acceleration, do not port channel or use equal cost paths
ƒ Use IVR to allow access from backup servers (VSAN 200) to tape drives
(VSANs 300, 400)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28

Presentation_ID.scr
 FCIP WA and TA—Multiple Tunnels
Port Channel FCIP Interfaces
11 and 21 for Replication Only
int fcip 11 int fcip 11 VSAN 100
VSAN 100 GigE1/1.100 VSAN 100 GigE1/1.100
Replication
Replication GigE1/1 int fcip 12 int fcip 12 GigE1/1
GigE1/1.200 VSAN 300 GigE1/1.200
VSAN 300
int fcip 21 int fcip 21 Tape Drives
GigE1/2.100 VSAN 100 GigE1/2.100
VSAN 200 GigE1/2 int fcip 22 int fcip 22 GigE1/2
Backup Serv GigE1/2.200 VSAN 400 GigE1/2.200 VSAN 400
Tape Drives
Do not Port Channel FCIP Interfaces
12 and 22 for Tape Backup
fcip profile 11 interface fcip 22
ip address 10.10.100.1 switchport trunk mode off
tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 use profile 22
fcip profile 12 peer-info ipaddr 10.11.200.2
ip address 10.10.200.1 write-accelerator tape-accelerator
tcp max-bandwidth-mbps 250 min-available-bw-mbps 200
fcip profile 21 interface GigabitEthernet 1/1
ip address 10.11.100.1 no shut
tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 interface Gigabit Ethernet 1/1.100
fcip profile 22 ip address 10.10.100.1 255.255.255.0
ip address 10.11.200.1 interface Gigabit Ethernet 1/1.100
tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 ip address 10.10.200.1 255.255.255.0

interface fcip 11 interface GigabitEthernet 1/2

switchport trunk mode off no shut
channel-group 100 force interface Gigabit Ethernet 1/2.100
use profile 11 ip address 10.11.100.1 255.255.255.0
peer-info ipaddr 10.10.100.2 interface Gigabit Ethernet 1/2.100
write-accelerator ip address 10.11.200.1 255.255.255.0
interface fcip 12
switchport trunk mode off interface port-channel 100
use profile 12 no shut
peer-info ipaddr 10.10.200.2 switchport trunk mode off
write-accelerator tape-accelerator switchport mode E
interface fcip 21 channel mode active
switchport trunk mode off
channel-group 100 force vsan database
use profile 21 vsan 100 interface fcip 11
peer-info ipaddr 10.11.100.2 vsan 100 interface fcip 21
write-accelerator vsan 100 interface portchannel 100
vsan 300 interface fcip 12
vsan 400 interface fcip 22
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

FCIP WA and TA—Multiple Tunnels

Port Channel FCIP Interfaces
11 and 21 for Replication Only
int fcip 11 int fcip 11 VSAN 100
VSAN 100 GigE1/1.100 VSAN 100 GigE1/1.100
Replication
Replication GigE1/1 int fcip 12 int fcip 12 GigE1/1
GigE1/1.200 VSAN 300 GigE1/1.200
VSAN 300
int fcip 21 int fcip 21 Tape Drives
GigE1/2.100 VSAN 100 GigE1/2.100
VSAN 200 GigE1/2 int fcip 22 int fcip 22 GigE1/2
Backup Serv GigE1/2.200 VSAN 400 GigE1/2.200 VSAN 400
Tape Drives
Do not Port Channel FCIP Interfaces
12 and 22 for Tape Backup
fcip profile 11 interface fcip 22
ip address 10.10.100.2 switchport trunk mode off
tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 use profile 22
fcip profile 12 peer-info ipaddr 10.11.200.1
ip address 10.10.200.2 write-accelerator tape-accelerator
tcp max-bandwidth-mbps 250 min-available-bw-mbps 200
fcip profile 21 interface GigabitEthernet 1/1
ip address 10.11.100.2 no shut
tcp max-bandwidth-mbps 750 min-available-bw-mbps 700 interface Gigabit Ethernet 1/1.100
fcip profile 22 ip address 10.10.100.2 255.255.255.0
ip address 10.11.200.2 interface Gigabit Ethernet 1/1.100
tcp max-bandwidth-mbps 250 min-available-bw-mbps 200 ip address 10.10.200.2 255.255.255.0

interface fcip 11 interface GigabitEthernet 1/2

switchport trunk mode off no shut
channel-group 100 force interface Gigabit Ethernet 1/2.100
use profile 11 ip address 10.11.100.2 255.255.255.0
peer-info ipaddr 10.10.100.1 interface Gigabit Ethernet 1/2.100
write-accelerator ip address 10.11.200.2 255.255.255.0
interface fcip 12
switchport trunk mode off interface port-channel 100
use profile 12 no shut
peer-info ipaddr 10.10.200.1 switchport trunk mode off
write-accelerator tape-accelerator switchport mode E
interface fcip 21 channel mode active
switchport trunk mode off
channel-group 100 force vsan database
use profile 21 vsan 100 interface fcip 11
peer-info ipaddr 10.11.100.1 vsan 100 interface fcip 21
write-accelerator vsan 100 interface portchannel 100
vsan 300 interface fcip 12
vsan 400 interface fcip 22
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29

Presentation_ID.scr
 FCIP—Configuration

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

MDS FCIP Configuration Guidelines

ƒ TCP parameter settings

ƒ Understanding FC buffers and FCIP
ƒ Monitoring for FC Frame expiry and why it is important
ƒ MDS packet shaping methods so not to over
run network
ƒ Understanding MTU and the WAN network

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30

Presentation_ID.scr
 MDS FCIP—TCP Parameters

ƒ Select the proper bandwidth settings

ƒ Use ips measure-rtt or ping command to set initial RTT
fcip profile 1
ip address 200.200.200.1
tcp max-bandwidth-mbps 800 min-available-bandwidth-mbps 500 round-trip-time-us 80

MDS9216-TOP# ips measure-rtt 200.200.200.2 int gig 2/4

Round trip time is 82 micro seconds (0.08 milli seconds)
or
MDS9216-TOP# ping 200.200.200.1
--- 200.200.200.1 ping statistics ---
rtt min/avg/max/mdev = 0.068/0.079/0.091 ms

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

MDS FCIP—TCP Parameters

ƒ Always use Selective ACK (SACK)
ƒ If slow link (<T3/E3):
Keep tcp cwm burstsize low (<20 kB)
Tune FC receive buffers—BB_Credits and performance buffers (fcrxbbcredit)
ƒ If shared link (other traffic):
Determine available bandwidth (link b/w – other traffic b/w) and set max-
bandwidth = available bandwidth and min-available-bandwidth = 80–95%
max-bandwidth
Consider QoS policies; carving out bandwidth
ƒ If shared link (other FCIP tunnel):
Determine available bandwidth (link b/w – other traffic b/w) and set
max-bandwidth = 50% available bandwidth and min-available-bandwidth =
80–95% max-bandwidth
ƒ If a dedicated link:
Set min=95% max and max=path bandwidth (FCIP will send at this rate)
If GigE all the way, set tcp cwm burstsize to 100 kB

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31

Presentation_ID.scr
 FCIP Configuration Example
fcip profile 10
ip address 10.1.4.2
tcp max-bandwidth-mbps 100 min-available-bandwidth-mbps 80 round-
trip-time-ms 30
10
FC interface fcip50 RTT Will Autconfigure and
switchport mode E
Adapt to Network Changes
no shutdown
switchport trunk allowed vsan 100 During Idle Periods
10.1.4.2 use-profile 10
100
peer-info ipaddr 10.4.8.2

Shared interface GigabitEthernet2/5 Jumbo Frame MTU—

ip address 10.1.4.2 255.255.255.0 2300 Bytes Will Handle
155-Mbps switchport mtu 2300
WAN Link Largest FC Frame
no shutdown
(100 Mb for
Storage)
ƒ Three steps for FCIP
30-ms RTT configuration—GigE interface,
FCIP profile, and FCIP interface
10.4.8.2 ƒ Min-bandwidth set to minimum
bandwidth available (through
QoS or other means); sender
FC 100 will start at this rate
ƒ Peer FCIP interface
configured similarly
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

Fibre Channel Frame Buffering

Traffic Flow

FC
FC
BB_Credit BB_Credit BB_Credit
Flow Control Flow Control Flow Control

FC Receive FC Receive
Buffers Buffers

ƒ FC frames buffered and queued in intermediate switches

ƒ Buffer depth controlled by Rx BB_Credit parms
switchport fcrxbbcredit <x>
switchport fcrxbbcredit performance-buffers <y>
Performance buffers added by default unless configured otherwise
ƒ Hop-by-hop traffic flow paced by return of R_RDY frames
ƒ Frames cannot sit in any switch for >500 ms (or they will expire)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32

Presentation_ID.scr
 Frame Buffering: FCIP and FC
Traffic Flow

FC
FC
TCP Windowing Flow Control
BB_Credit BB_Credit
Flow Control Flow Control

TCP Send GigE Slower GigE TCP Send

Buffers WAN Link Buffers

FC Receive FCIP Receive

Buffers Buffers
Increase TCP Send Buffer if : Backlog Here If Queue Can’t Drain Due to:
ƒ Slow WAN link is preventing FC traffic ƒ Slow WAN link and long RTT
destine to other devices not across ƒ Packet loss and retransmissions
FCIP link to have issues
ƒ Many sources (only one shown)
ƒ Buffer too big

ƒ FCIP presents a lower bandwidth pipe (If WAN link)

Drain rate (send rate) depends upon bandwidth and congestion
ƒ Slow ramp up of traditional TCP can cause FC frame expiry in
some conditions
Mixture of slow link (e.g., <DS3/E3; retransmissions, many sources, big buffers)
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

Frame Expiration
ƒ 500-ms timer is fixed (not configurable)
ƒ Behavior possible whenever FC Rx buffers cannot
drain at a sufficient rate (at least 2x #Rx
buffers/second)
Caused by one or more of:
too many sources (speed mismatch)
slow downstream (slow receiver)
FC Rx buffers too deep
ƒ Possible situations are:
1. Long optical links with low BB_Credits (i.e., can’t drain
quickly)— misconfiguration
2. FCIP over long, slow links (e.g., T1 or T3 with 50 + ms RTT)
with occasional packet loss/retransmission
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33

Presentation_ID.scr
 Monitor FCIP Frame Timeouts
MDS9216-TOP# show ips stats dma int gigabitethernet 2/4
Dma-bridge ASIC Statistics for port GigabitEthernet2/4
Hardware Egress Counters
1030008889 Good, 0 bad protocol, 0 bad header cksum, 0 bad FC CRC Monitor for 500 ms
Hardware Ingress Counters
Timeout Issues
1843269599 Good, 0 protocol error, 0 header checksum error
0 FC CRC error, 0 iSCSI CRC error, 0 parity error
Software Egress Counters
1030008581 good frames, 0 bad header cksum, 0 bad FIFO SOP
0 parity error, 0 FC CRC error, 0 timestamp expired error
0 unregistered port index, 0 unknown internal type
0 RDL ok, 0 RDL drop (too big), 0 RDL ttl_1
1278210315 idle poll count, 116279 loopback
0 FCC PQ, 0 FCC EQ, 0 FCC generated
Flow Control: 0 [0], 0 [1], 0 [2], 0 [3]
Software Ingress Counters
1843269472 Good frames, 0 header cksum error, 0 FC CRC error
0 iSCSI CRC error, 0 descriptor SOP error, 0 parity error
141281 frames soft queued, 0 current Q, 732 max Q, 0 low memory
0 out of memory drop, 0 queue full drop
0 RDL ok, 0 RDL drop (too big)
Flow Control: 0 [0], 141281 [1], 0 [2], 0 [3]

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

TCP Window and Buffer with FCIP

ƒ You cannot configure the TCP window size directly.

This value is automatically calculated from the
product of the maximum bandwidth x RTT.
ƒ In SAN-OS 2.0 and later, the RTT will dynamically
adjust up to eight times the configured value in the
MDS9216-TOP# sh interface fcip 2 counters FCIP profile. The TCP sender dynamically changes the
--snip-- maximum window size accordingly.
TCP Parameters
Path MTU 1500 bytes
Current retransmission timeout is 200 ms
Round trip time: Smoothed 2 ms, Variance: 4
Advertized window: Current: 97 KB, Maximum: 97 KB, Scale: 5
Peer receive window: Current: 98 KB, Maximum: 98 KB, Scale: 5
Congestion window: Current: 52 KB, Slow start threshold: 103 KB
Current Send Buffer Size: 97 KB, Requested Send Buffer Size: 0 KB
CWM Burst Size: 50 KB TCP Send Buffer Can Be
--snip-- Increased to Allow Faster
IP compression statistics Removal of Frame off FC
Interface RX Buffer
4059579130004 rxbytes, 273626035160 rxbytes compressed
2287961989360 txbytes 633365852164 txbytes compressed, 0 txbytes non-compressed
3.61 tx compression ratio

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34

Presentation_ID.scr
 TCP Send Buffer Interworkings
FC Interface on
Storage Array MDS9000
Traffic Transmitting
FC Interface FCIP Interface over WAN at Capable
FCIP Virtual Output
Traffic Queues Network Rate

IP
TCP send- Network
Local
Server 28 Buffers buffer
Traffic Total over All
VOQ Default FC

HBA
Attached
Server

TCP Send Buffer Would Only Require Adjustment if Other Local

Exchanges Were Occurring Off the Same FC Port That Is Being
Used by FCIP Application; the Minimum Size is 14k and Is
Dynamically Calculated from Bandwidth x rtt Product
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

FCIP—
Capacity Planning

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70

© 2006, Cisco Systems, Inc. All rights reserved. 35

Presentation_ID.scr
 FCIP—Capacity Planning
SAN Extension Tuner (SET)
SAN Extension Tuner—Lightweight Tool Built into
the IPS and MPS to Assist in FCIP Tuning by
Generating Various SCSI Traffic Flows
N-port: 10:00:00:00:00:00:00:01 N-port: 11:00:00:00:00:00:00:03
Gig3/3 Gig2/2
IP Network

FCIP Tunnel

Gig3/1 Gig2/1

ƒ Creates a virtual N-port on an IPS or MPS port that can act as both
initiator and target to generate traffic
ƒ Measures throughput and response time per I/O over the
FCIP tunnels
ƒ Model effect of storage data in a shared network
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71

Using SAN Extension Tuner

ƒ San Extension-Tuner allows you to configure a unused
iSCSI interface as a FC initiator and target
ƒ Used to create virtual FC devices in the MDS to
generate FC traffic
ƒ FC traffic is sent across FCIP link (ISL) for WAN load
test, help tune TCP parameters and exercise features;
to achieve desired throughput across FCIP a number of
parameters must be tuned to keep WAN pipe full
TCP parameters (Window size, SACK…)
Outstanding SCSI I/Os allowed by application
Transfer size used by application
ƒ Requires 2.0 SAN/OS or greater
Note: MSM 18+4/9222i require SAN-OS 3.3(1) or higher
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72

© 2006, Cisco Systems, Inc. All rights reserved. 36

Presentation_ID.scr
 SAN Extension Tuner

ƒ Only read-and-write SCSI commands are used,

entire SCSI suite in not supported
ƒ Multiply virtual N-ports can be created to overload FCIP
link to stress WAN provisioning, compression results,
troubleshoot issues without using FC real devices
ƒ Use data provided with tool or use your own
ƒ Simple Setup Wizard
ƒ Wizard creates zone for the virtual N-ports

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73

SET Use from Fabric Manager

Click on
FCIP Link

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74

© 2006, Cisco Systems, Inc. All rights reserved. 37

Presentation_ID.scr
 SET—Monitor Results,
Make Adjustments
Monitor ISL Link
Utilization Tool

Auto Created
Zone

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

FCIP—Troubleshooting

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38

Presentation_ID.scr
 Troubleshooting Section Agenda

ƒ Tools available is SAN-OS

ƒ Practical use of SPAN, PAA and Cisco analyzers
in troubleshooting

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

FCIP CLI Commands

Useful Commands
show interface fcip nnn Show fcip interface statistics
show interface gigabitethernet n/n Show Gigabit Ethernet interface statistics
show ips stats all Show all ips stats
show ips stats buffer Show ips buffer stats
show ips stats dma-bridge Show dma-bridge stats
show ips stats hw-comp Show hw compression stats
show ips stats icmp Show icmp stats
show ips stats ip Show ip stats
show ips stats mac Show ethernet mac stats
show ips stats tcp Show tcp stats

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39

Presentation_ID.scr
 Command Line Debugging

ƒ Available debugs depend on features enabled in

SAN-OS
ƒ Many different options to select when turning
on debugs
Interface and protocol level debugging available

ƒ Where is it output going?

Logfile—data file in switch memory
Capture to direct to screen via console, telnet or ssh

ƒ Requires admin privileges to run debugs

ƒ Debugs can only be run from cli, no debug interface
in fabric manager or device manager
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

FCIP Debugs
You Must Attach to the Module
You Are Debugging
module-2# debug ips fcip ?
all Configure FCIP debugging
ctlpkt Configure FCIP Control Pkt debugging
ctlpkt-detail Configure verbose FCIP Control Pkt debugging
datapkt Configure FCIP Data Pkt debugging
datapkt-detail Configure verbose FCIP Data Pkt debugging
err Configure FCIP Error debugging
fsm Configure FCIP FSM debugging
tape-acc Configure FCIP Tape accelerator
tape-acc-det Configure FCIP Tape accelerator debug detail
tape-acc-err Configure FCIP Tape accelerator error
write-acc Configure FCIP Write accelerator
write-acc-err Configure debug FCIP Write accelerator errors
write-acc-event Configure debug FCIP Write accelerator events
write-acc-pkt Configure debug FCIP Write accelerator packet)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40

Presentation_ID.scr
 Gathering Protocol Traces for Analysis

ƒ Using built-in FCanalyzer (CLI)

ƒ Using Cisco Protocol and Traffic Analyzer on PC
(local and remote)
ƒ Using the MDS Port Analyzer Adapter (PAA)
ƒ Using SPAN
ƒ Using an external FCanalyzer

All non-disruptive to switch operations and traffic

on the SAN

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

MDS FCanalyzer
(SAN/OS Imbedded)

ƒ Output is displayed to the console in readable sniffer

like format
ƒ Is only used to monitor Fibre Channel traffic to and from
supervisor on the MDS9000
Traffic-like fabric logins, FSPF routing, switch-to -switch
control traffic

ƒ Output can go direct to your console screen or to a

workstation running a color Ethereal program
Note: SPAN is used not only for FC port-to-FC port monitoring,
but also used to SPAN iSCSI and FCIP ports

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41

Presentation_ID.scr
 FCanalyzer Options

ƒ Local or remote—where to send the trace; can be

to local devices or remote PAA attached to different
MDS switch
ƒ Brief or detailed—header information vs. full output of
frame including hex; detail is default
ƒ Limit-captured-frames—number of frames to capture;
default is fcanalyzer will trace 100 frames; specifying
zero is unlimited frame capture
ƒ Limit-capture-size—allows to capture N number of
bytes of frame; useful for not capturing frame data
when it is not relevant to troubleshooting

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

Use of SPAN Feature

ƒ Used for FC port to FC port analyzing

ƒ Same type of tool as used on Cisco Catalyst® products
(Catalyst uses port monitor)
ƒ Can be left configured on switch
ƒ Ingress and egress ports are sent to an FC-port setup
as a SPAN destination (SD-port type)
ƒ No limits to where the ports are located on the
MDS switch fabric
ƒ Used to output to third-party test equipment or to
Cisco Port Analyzer Adapter

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42

Presentation_ID.scr
 Using the FCanalyzer with FCIP
MDS9216-TOP# conf t
Enter configuration commands, one per line. End with CNTL/Z.

MDS9216-TOP(config)# fcanalyzer local brief display-filter mdshdr.vsan==300

Warning: Couldn't obtain netmask info (eth2: no IPv4 address assigned).

Capturing on eth2

8.986146 ff.ff.fd -> ff.ff.fd 0x46a6 0xffff SW_ILS ELP ƒ Filter on VSAN 300
8.986233 ff.ff.fd -> ff.ff.fd 0x46a6 0x469a FC Link Ctl, ACK1
8.987140 ff.ff.fd -> ff.ff.fd 0x469b 0xffff SW_ILS ELP ƒ No shut of FCIP
8.987539
8.988273
ff.ff.fd -> ff.ff.fd
ff.ff.fd -> ff.ff.fd
0x469b 0x46a7 FC Link Ctl, ACK1
0x469b 0x46a7 SW_ILS SW_RJT (ELP)
interface
8.988790 ff.ff.fd -> ff.ff.fd 0x46a6 0x469a SW_ILS SW_ACC (ELP)
8.989438 ff.ff.fd -> ff.ff.fd 0x469b 0x46a7 FC Link Ctl, ACK1
ƒ Complete view
8.991501 ff.ff.fd -> ff.ff.fd 0x46a6 0x469a FC Link Ctl, ACK1 of standard ISL
8.992965 ff.ff.fd -> ff.ff.fd 0x46a8 0xffff SW_ILS EFP
8.993037 ff.ff.fd -> ff.ff.fd 0x46a8 0x469c FC Link Ctl, ACK1
initialization and
9.001146 ff.ff.fd -> ff.ff.fd 0x469d 0xffff SW_ILS EFP switch fabric merge
9.001603 ff.ff.fd -> ff.ff.fd 0x469d 0x46a9 FC Link Ctl, ACK1
9.001719 ff.ff.fd -> ff.ff.fd 0x46a8 0x469c SW_ILS SW_ACC (EFP) ƒ You do not capture
FCIP protocol; FCIP
protocol is viewed
with debugs and/or
SPAN (monitor) of
WAN network
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85

SPAN of FCIP Interface to PAA and

Cisco Protocol Analyzer

PC w/Cisco
Protocol
Analyzer
(Ethereal)

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 86

© 2006, Cisco Systems, Inc. All rights reserved. 43

Presentation_ID.scr
 PAA Capture Modes
No Truncate Mode (NTM)
Fibre Channel Frames Are
Encapsulated into Ethernet
Frames Without any Ethernet Header SOF-T FC EOF-T Enet
Modification to the Payload). 14 Bytes VSAN FC Frame 0–2164 Bytes CRC VSAN CRC
Ethernet Devices must 2 Bytes 4 Bytes 4 Bytes 4 Bytes
(EDA + SDA + Type)
Support Jumbo Frames
for this Mode to Work.
Ethernet Truncate Mode
(ETM)
Ethernet Header SOF-T FC EOF-T Enet
The PAA Truncates a 14 bytes VSAN FC Frame 0–1490 Bytes CRC VSAN CRC
Fibre Channel Frame to a 2 Bytes 4 Bytes 4 Bytes 4 Bytes
Maximum Payload of 1496 (EDA + SDA + Type)
Bytes. The Fibre Channel
Payload Transmitted Is
1472 Bytes.
Shallow Truncate Mode Ethernet Header SOF-T FC Frame FC EOF-T Enet
(STM) 14 Bytes VSAN CRC VSAN CRC
2 Bytes 0–256 Bytes 4 Bytes 4 Bytes 4 Bytes
In STM, the PAA Truncates (EDA + SDA + Type)
the Fibre Channel Frame to
256 Bytes.
Deep Truncate Mode (DTM)
FC
In DTM, the PAA Truncates Ethernet Header SOF-T Frame FC EOF-T Enet
the Fibre Channel Frame to 14 Bytes VSAN CRC VSAN CRC
0–64
64 Bytes. By Default, the (EDA + SDA + Type) 2 Bytes 4 Bytes 4 Bytes 4 Bytes
Bytes
PAA Is Configured in Ethernet Frame
DTM Mode.
Ethernet Payload
FC Frame
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 87

Display Filters Can

Be Applied Before
Or After Capture

Right Click on
Field to Filter

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88

© 2006, Cisco Systems, Inc. All rights reserved. 44

Presentation_ID.scr
 GiGE Monitoring of FCIP
FCIP GigE Interface Must Be Up and Healthy
ƒ show ips stats ip interface gigabitethernet 2/2 detail
ƒ show ips stats tcp interface gigabitethernet 2/2 detail
ƒ show ips stats mac interface gigabitethernet 2/2

Device Manager Monitor

Views Can Be Actively
Kept Open and Graphed
in Real Time
BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 89

Q and A

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 90

© 2006, Cisco Systems, Inc. All rights reserved. 45

Presentation_ID.scr
 Recommended Reading

ƒ Continue your Cisco Live

learning experience with further
reading from Cisco Press
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 91

Complete Your Online

Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 92

© 2006, Cisco Systems, Inc. All rights reserved. 46

Presentation_ID.scr
 BRKSAN-2704
14571_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 93

© 2006, Cisco Systems, Inc. All rights reserved. 47

Presentation_ID.scr