Professional Documents
Culture Documents
Materials
Book:
Troubleshooting and Maintaining
Cisco IP Networks (TSHOOT)
Foundation Learning Guide:
Foundation learning for the CCNP
TSHOOT 642-832
By Amir Ranjbar
Book
ISBN-10: 1-58705-876-6
ISBN-13: 978-1-58705-876-9
eBook
ISBN-10: 1-58714-170-1
ISBN-13: 978-1-58714-170-6
Chapter 4 Objectives
Troubleshooting VLANs
Switch C:
Forward
The MAC address table never contains an entry for the broadcast
MAC address (FFFF:FFFF:FFFF).
Switch C will flood the frame on all ports:
All access ports in that VLAN (VLAN 10)
All trunks that this VLAN is allowed, active, and not pruned on
Except the port it came in from
Switches D and E repeat this process as they receive the frame
10
Because all switches now have an entry in their MAC address table for the
MAC address of Host A they will:
Learns
Record Host Bs MAC address and corresponding interface and VLAN
in their MAC address table (if they did not already have that entry)
Forwards
Forward the frame containing the ARP reply on the path to Host A only
No flooding
11
12
13
Misconfiguration
Missing or wrong VLANs
Misconfigured VTP settings
Wrong VLAN setting on access
ports
Missing or misconfigured trunks
Native VLAN mismatch
VLANs not allowed on trunk
15
show vlan:
Verifies VLAN existence and port-to-VLAN associations.
Lists all VLANS that were created on the switch (either manually or
through the VLAN Trunking Protocol [VTP]).
Note: Trunks are not listed because they do not belong to any VLAN, in
particular.
19
20
21
traceroute mac:
You specify a source and destination MAC address
Shows a list of switch hops that a frame from that source MAC address to
that destination MAC address.
Discovers the Layer 2.
This command requires that Cisco Discovery Protocol (CDP) is enabled on
all the switches in the network (or at least within the path).
22
Troubleshooting STP
23
24
L2 Loops
Switch (Bridge) loops can occur
any time there is a redundant
path or loop in the bridge
network.
The switches will flip flop the
MAC address table entries
(creating extremely high CPU
utilization).
Unicasts, unknown unicasts and
broadcasts are all problems.
25
26
Link Speed
10 Gbps
1 Gbps
100 Mbps
19
10
10 Mbps
100
100
27
28
29
30
Who wins?
My BID is
32768.0005.5E0D.9315
My BID is
32768.0060.47B0.5850
My BID is
32768.0003.E461.46EC
My BID is
32768.0001.964E.7EBB
I win!
Root Bridge
31
Root Bridge, Access2 sends out BPDUs, containing a Root Path Cost of 0.
Switches receive these BPDUs and adds the Path Cost of the FastEthernet interface to the
Root Path Cost contained in the BPDU.
This value is used internally and used in BPDUs to other switches.
Path Cost
BPDU
BPDU
Cost=0+19=19
Cost=0+19=19
19
19
0
BPDU
Cost=0+19=19
19
0
BPDU
Cost=0
Root Bridge
33
BPDU
BPDU
Cost=0+19=19
Cost=0+19=19
19
19
0
BPDU
Cost=0+19=19
19
0
BPDU
Cost=0
Root Bridge
34
Switches now send BPDUs with their Root Path Cost out other interfaces.
Switches receive BPDU and add their path cost.
Path Cost
BPDU
BPDU
Cost=4+19=23
Cost=4+19=23
19
19
BPDU
BPDU
Cost=19
Cost=19
0
19
Root Bridge
35
BPDU
BPDU
Cost=4+19=23
Cost=4+19=23
19
19
BPDU
BPDU
Cost=19
Cost=19
0
19
Root Bridge
36
Path Cost
BPDU
Cost=4+19=23
BPDU
BPDU
Cost=19+19=38
Cost=19
19
23
23
19
0
19
BPDU
Cost=4+19=23
19
0
0
Root Bridge
37
Final Results
Ports show BPDU Received Root Path Cost + Path Cost = Root Path Cost of Interface,
after the best BPDU is received on that port from the neighboring switch.
This is the cost of reaching the Root Bridge from this interface towards the neighboring
switch.
Now lets see how this is used!
Path Cost
19+4=23
23+4=27
19+19=38
19
19+4=23
23+4=27
19+19=38
19+4=23
19
19+4=23
19+4=23
19+4=23
0
0
19
0
Root Bridge
38
Next:
Elect Root Ports
Elect Root Ports
Every non-Root bridge must select one Root Port.
A bridges Root Port is the port closest to the Root
Elect Designated Ports
Bridge.
Non-Designated Ports: All other ports
Bridges use the cost to determine closeness.
Path Cost
These values
would be the
Root Path Cost
if this interface
was used to
reach the Root
Bridge.
23
27
38
19
23
27
38
23
19
23
23
23
0
19
0
Root Bridge
39
Path Cost
If I go
through
Core it costs
27.
If I go
through D2
it costs 38.
If I go
through A1 it
costs 23.
If I go through
A2 it costs 19.
This is the best
path to the
Root!
40
19
23
23
19
23
23
38
23
38
RP
27
27
23
19
RP
0
0
RP
0
Root Bridge
41
RP
My BID is
32768.0005.5E0D.9315
Lower BID
19
23
23
19
23
23
My BID is
32768.0060.47B0.5850
38
23
38
RP
27
27
23
19
RP
0
0
RP
0
Root Bridge
42
STP Convergence
Step 1 Elect one Root Bridge
Step 2 Elect Root Ports
Step 3 Elect Designated Ports
A Designated Port functions as the single bridge port that both sends and
receives traffic to and from that segment and the Root Bridge.
Each segment in a bridged network has one Designated Port, chosen
based on cumulative Root Path Cost to the Root Bridge.
The switch containing the Designated Port is referred to as the Designated
Bridge for that segment.
To locate Designated Ports, lets take a look at each segment.
Segments perspective: From a device on this segment, Which switch
should I go through to reach the Root Bridge?
43
RP
23
23
?
19
19
19
19
?
19
? ?
RP
19
19
RP
19
19
19
19
RP
?
0
0
0
Root Bridge
44
Because Access 2 has the lower Root Path Cost it becomes the Designated Port for that
segment.
RP 23
23
19
19
19
19
My19designated
What is my
portbest
will be
path
019
via Access
to the2 Root
(Fa0/5).
Bridge,
Its the
19
19
19
best path,
via lowest
AccessRoot
1 or Path,
0 via
Access
Bridge.
2?
RP to the Root
19
19 RP
19 RP
0
DP 0
0
Root Bridge
45
Because Access 2 has the lower Root Path Cost it becomes the Designated Port for those
segments.
RP 23
19
19
19
19
19
RP
19
19
19 RP
23
19
19
19 RP
?
DP
DP 0
0
0 DP
Root Bridge
46
RP 23
23
32768.0005.5E0D.9315
19
19
19
19
RP
?
DP 19
32768.0003.E461.46EC
Lower BID
19
19 RP
19
0 DP
Root Bridge
47
RP 23
23
NDP
DP
19
DP
NDP
19
19
DP
19
19
RP
19
19 RP
19
NDP
DP
X
X
19
19
NDP
19 RP
DP
0 DP
DP 0
0
Root Bridge
48
Path Cost
show spanning-tree
Priority
Address
Hello Time
Aging Time
Interface
---------------Gi0/1
Gi0/2
Role
---Root
Altn
Sts
--FWD
BLK
Cost
--------4
4
Prio.Nbr
-------128.25
128.26
Type
-------------------------------P2p
P2p
49
Path Cost
50
Port Cost/Port ID
0/2
0/1
If the path cost and bridge IDs are equal (as in the case of parallel links), the
switch goes to the port priority as a tiebreaker.
Lowest port priority wins (all ports set to 32).
You can set the priority from 0 63.
If all ports have the same priority, the port with the lowest port number forwards
frames.
51
Port Cost/Port ID
Fa 0/3 has a lower Port ID than Fa 04.
More later (Fast EtherChannel)
RP
19
NDP
19
DP
DP
52
53
Disabled
Blocking
Listening
Learning
Forwarding
BPDUs
Data
Port State
BPDU
MAC-Add Table
Data frames
Duration
Disabled
None sent/received
No update
None sent/received
Until no shutdown
Receive only
No update
None sent/received
Continuous if loop
detected
None sent/received
No update
Building active topology. Thinks port can be selected root or designated port.
Returns to blocking (NDP) if cannot become root or designated port.
Learning
Updating Table
None sent/received
Building bridging table. Switch can now learn source MAC Addresses but is not
formally receiving frames in order to forward them.
Forwarding Receive and send
Updating Table
Continuous if up and no
loop detected
802.1w
vs
RSTP (Reminder)
Root Bridge: Same election process as 802.1D (lowest BID)
Ports
Root Port (802.1D Root Port)
The one switch port on each switch that has the best
root path cost to the root.
Designated Port (802.1D Designated Port)
The switch port on a network segment that has the
best root path cost to the root.
Alternate Port (802.1D Blocking Port)
A port with an alternate path the root.
An alternate port receives more useful BPDUs from
another switch and is a port blocked.
Similar to how Cisco UplinkFast works.
Backup Port (802.1D Blocking Port)
A port that provides a redundant (but less desirable)
connection to a segment where another switch port
already connects.
A backup port receives more useful BPDUs from the
same switch it is on and is a port blocked.
Rick Graziani
graziani@cabrillo.edu
Disabled
Disabled
Discarding
Enabled
Blocking
Discarding
Enabled
Listening
Discarding
Enabled
Learning
Learning
Enabled
Forwarding
Forwarding
RSTP defines port states based on what it does with incoming data frames.
Discarding
Incoming frames are dropped
No MAC Addresses learned
Combination of 802.1D (Disabled), Blocking and Listening
Learning
Incoming frames are dropped
MAC Addresses learned
Forwarding
Incoming frames are forward.
57
STP BPDUs
RSTP BPDUs
Disabled
Not Sent/Received
Discarding
Not Sent/Received
Blocking
Receive only
Discarding
Sent/Received
Listening
Sent/Received
Discarding
Sent/Received
Learning
Sent/Received
Learning
Sent/Received
Forwarding
Sent/Received
Forwarding
Sent/Received
59
Port Cost/Port ID
Distribution1# show spanning-tree
VLAN0001
Spanning tree enabled protocol ieee
Root ID
Priority
32769
Address
0009.7c0b.e7c0
Cost
19
Port
3 (FastEthernet0/3)
Hello Time
2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority
32769 (priority 32768 sys-id-ext 1)
Address
000b.fd13.9080
Hello Time
2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300
Interface
Port ID
Designated
Name
Prio.Nbr
Cost Sts
Cost Bridge ID
---------------- -------- --------- --- --------- -------------------Fa0/1
128.1
19 BLK
19 32769 000b.befa.eec0
Fa0/2
128.2
19 BLK
19 32769 000b.befa.eec0
Fa0/3
128.3
19 FWD
0 32769 0009.7c0b.e7c0
Fa0/4
128.4
19 BLK
0 32769 0009.7c0b.e7c0
Fa0/5
128.5
19 FWD
19 32769 000b.fd13.9080
Gi0/1
128.25
4 FWD
19 32769 000b.fd13.9080
Port ID
Prio.Nbr
-------128.1
128.2
128.1
128.2
128.5
128.25
61
62
Troubleshooting STP
Remove all redundancy
Find the root cause of the problem.
STP disabled on a switch
Cabling problem (unidirectional link)
63
Troubleshooting Etherchannel
64
Etherchannel Bundle
65
67
Forming EtherChannels
EtherChannel
on
on
desirable
auto
active
passive
68
Configuring PAgP
DLS1(config)# port-channel load-balance dst-ip
DLS1(config)# interface range fa 0/11 - 12
DLS1(config-if-range)# switchport trunk encapsulation dot1q
DLS1(config-if-range)# switchport mode trunk
DLS1(config-if-range)# channel-protocol pagp
DLS1(config-if-range)# channel-group 1 mode desirable
DLS2(config)# port-channel load-balance src-dst-ip
DLS2(config)# interface range fa 0/11 - 12
DLS2(config-if-range)# switchport trunk encapsulation dot1q
DLS2(config-if-range)# channel-protocol pagp
DLS2(config-if-range)# channel-group 1 mode auto
Notice:
Load balancing does not have to match but usually it does.
DTP on DLS2 is dyanmic auto (result is trunk with DLS1)
PAgP configured on both ends
69
Verifying
DLS1# show etherchannel protocol
Group: 1
---------Protocol: PAgP
70
71
=
=
=
=
Up Mstr In-Bndl
1
Mode = Desirable-Sl
Po1
GC
= 0x00010001
0
Load = 0x00
Gcchange = 0
Pseudo port-channel = Po1
Protocol =
PAgP
Flags:
Local information:
Port
Fa0/11
Flags State
SC
Timers
Hello
Partner PAgP
Interval Count
Priority
Learning Group
Method Ifindex
Partner's information:
Port
Fa0/11
Partner
Name
DLS2
Partner
Device ID
001b.8fc8.0080
Partner
Port
Age
Partner Group
Flags
Cap.
72
interface FastEthernet0/13
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode active
!
interface FastEthernet0/14
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode active
73
Default
29xx 36xx
Default non-IP
traffic
The header fields that are used as a key into the hash calculation.
The choice of header fields to be hashed does affect the distribution.
Example:
Only the destination MAC address is used
If 90% of all frames are destined for a single MAC address (for instance,
the MAC address of the default gateway)
All of that traffic would end up on the same physical link.
If you see an uneven distribution of traffic over the links in the channel, you
should examine the hashing method and the traffic mix to determine the
cause.
76
Troubleshooting Switched
Virtual Interfaces and InterVLAN Routing
77
Physical Interface
78
79
80
81
SVI Interfaces
- Logical Interfaces
82
83
Creating VLANs
DLS1: Create and name the user VLANs: 10, 11,
20 and 21.
DLS1: Create and name a Management VLAN
(used to telnet into switches)
DLS1: Create and name a NATIVE VLAN other
than VLAN 1 (default)
DLS1: Create and name a Garbage VLAN
(assigned to all unused ports.)
All ports that are not used (trunks and
access) will be assigned as an access port
to this VLAN.
DLS1
vlan 2
name NATIVE
vlan 10
name Engineering
vlan 11
name IT
vlan 20
name Sales
vlan 21
name Administration
vlan 99
name ManagementVLAN
vlan 222
name GarbageVLAN
84
Management VLAN
For each device in the network
we configured it to be a member
of the management VLAN.
On each switch
Switch(config)# inter vlan 99
Switch(config-if)# description Management VLAN
Switch(config-if)# ip address 172.16.99.x 255.255.255.0
Switch(config-if)# no shutdown
85
Default Gateway
Configure DLS1 to be the
default gateway for VLANs 10
and 11.
All hosts on these VLANs will
use these addresses as their
default gateway addresses.
DLS1(config)# inter vlan 10
DLS1(config-if)# description Engineering VLAN
DLS1(config-if)# ip address 172.16.10.1 255.255.255.0
DLS1(config-if)# no shutdown
DLS1(config)# inter vlan 11
DLS1(config-if)# description IT VLAN
DLS1(config-if)# ip address 172.16.11.1 255.255.255.0
DLS1(config-if)# no shutdown
86
Default Gateway
Configure DLS2 to be the
default gateway for VLANs 20
and 21.
All hosts on these VLANs will
use these addresses as their
default gateway addresses.
DLS2(config)# inter vlan 20
DLS2(config-if)# description Sales VLAN
DLS2(config-if)# ip address 172.16.20.1 255.255.255.0
DLS2(config-if)# no shut
DLS2(config)# inter vlan 21
DLS2(config-if)# description Administration VLAN
DLS2(config-if)# ip address 172.16.21.1 255.255.255.0
DLS2(config-if)# no shut
87
Verifying
Verify IP addresses
Protocol
FastEthernet0/1
192.168.4.6
YES manual up
up
GigabitEthernet0/1
192.168.1.1
YES manual up
up
Vlan10
172.16.10.1
YES manual up
up
Vlan11
172.16.11.1
YES manual up
up
Port-channel 1
DLS1#
unassigned
YES manual up
up
88
89
CEF
The information in the CEF data structures should accurately reflect the
information in the control plane data structures.
When troubleshooting IP routing under normal circumstances:
Check the control plane data structures
If the control plane information is correct, but packets are not being
forwarded as expected:
Check the CEF data structures and verify that they are in line with the
control plane information.
91
DLS1#show ip cef
Prefix
0.0.0.0/0
0.0.0.0/32
1.1.1.0/24
172.16.10.0/24
172.16.10.0/32
172.16.10.1/32
172.16.10.255/32
172.16.11.0/24
172.16.11.0/32
172.16.11.1/32
172.16.11.255/32
172.16.20.0/24
172.16.21.0/24
172.16.99.0/24
172.16.99.0/32
172.16.99.1/32
172.16.99.255/32
192.168.1.0/30
192.168.1.0/32
192.168.1.1/32
192.168.1.2/32
192.168.1.3/32
192.168.1.4/30
192.168.1.4/32
192.168.1.5/32
192.168.1.6/32
192.168.1.7/32
192.168.1.8/30
224.0.0.0/4
224.0.0.0/24
255.255.255.255/32
DLS1#
Next Hop
no route
receive
192.168.1.5
attached
receive
receive
receive
attached
receive
receive
receive
192.168.1.2
192.168.1.2
attached
receive
receive
receive
attached
receive
receive
192.168.1.2
receive
attached
receive
192.168.1.5
receive
receive
192.168.1.2
drop
receive
receive
Interface
FastEthernet0/1
Vlan10
Vlan10
Vlan10
Vlan10
Vlan11
Vlan11
Vlan11
Vlan11
GigabitEthernet0/1
GigabitEthernet0/1
Vlan99
Vlan99
Vlan99
Vlan99
GigabitEthernet0/1
GigabitEthernet0/1
GigabitEthernet0/1
GigabitEthernet0/1
GigabitEthernet0/1
FastEthernet0/1
FastEthernet0/1
FastEthernet0/1
FastEthernet0/1
FastEthernet0/1
GigabitEthernet0/1
show ip cef:
This command displays the content of the CEF FIB.
The FIB reflects the content of the routing table with all the recursive
lookups already resolved already, and the output interface determined for
each destination prefix.
The FIB also holds additional entries for:
directly connected hosts
the routers own IP addresses
multicast and broadcast addresses
92
IP
GigabitEthernet0/1
IP
FastEthernet0/1
Address
172.16.99.2
0 packets, 0 bytes
epoch 0
sourced in sev-epoch 3
Encap length 14
0000603E24584400055E6D393C0800
ARP
192.168.1.2
0 packets, 0 bytes
epoch 0
sourced in sev-epoch 3
Encap length 14
0000902B293019000C85B044190800
ARP
192.168.1.5
0 packets, 0 bytes
epoch 0
sourced in sev-epoch 3
Encap length 14
0000024A0A4301000C85B044010800
ARP
show adjacency:
This commandDLS1#
displays the content of the CEF adjacency table.
This table contains the Layer 2 frame information, such as the destination
MAC address.
93
Troubleshooting First-Hop
Redundancy Protocols (FHRP)
94
FHRPs such as HSRP, VRRP, and GLBP all serve the same purpose.
These protocols provide services which are entirely transparent to the hosts:
Redundant default gateway on a subnet
Failover
Load balancing
They do it by:
Electing a single router that controls the virtual IP address
Tracking availability of the active router
Determining if control of the virtual IP address should be handed over to
another router
95
Hello
Hello
Every 3 seconds (default) both the active and the standby router send hello
packets.
If hellos are not received for 10 seconds (the default hold time), the standby
takes on the active role.
This means that for a period of 10 seconds hosts will lose connectivity due to lack
of an active router to forward packets.
If the failure is caused by administrative actions such as a shutdown of an
interface the active HSRP router sends a resign message
Standby router to assume the active role immediately.
96
The10-second hold time does not come into play.
priority value (100 by default) - The router with the higher priority is elected
as the active HSRP router
Tie is broken using the IP address of the contenders.
preempt option Router will take over the active role immediately.
Sends out a coup message, telling the current active router that it will
take over the active role due to its higher priority.
97
Verifying FHRP
98
Useful to know the virtual MAC address used for the standby group
Verifies the correct operation of ARP and the Layer 2 connectivity between
the end host and the active HSRP router.
HSRP is usually not at the root of the problem
Usually the underlying switched network.
Example: A broadcast storm and you notice very frequent HSRP state
changes on the Layer 3 switches that are connected to the affected
VLANs.
99
100
debugstandbyterse
R1 comes up on the segment
Has a higher priority than the current active router
preempt option so it sends out a coup message to take over the active role.
R2 loses its active role, causing it to step back to the role of a non-active, non-standby
HSRP router.
There is no standby router on the segment, R2 moves to the speak state to
announce its eligibility for the standby role.
R2 does not see another (better) candidate for the role of standby router for 10 seconds
and, thus, promotes itself to the standby role.
101
Alternatives to HSRP
103
104
Understanding and
Troubleshooting Common VTP
Issues (EXTRA)
Both switches are VTP Servers and in the same Domain, but different VLAN
information. Lets see what happens when trunking is enabled between the
switches
VTP Domain = West
VTP Mode = Server
Config Rev = 34
VLANs = 1, 10, 11, 12 , 30
When two switches with same Domain Name and same Configuration
Revision Numbers exchange VTP information:
No change
If Switch A adds a new VLAN, VLAN 30, Config Rev is increased by 1.
Switch A will send VTP information to Switch B who will synchronize its
VLAN information with Switch A, losing current local VLANs
106
Note: Because Pod2 2690s and 3560s are incorrectly defaulting to dynamic
desirable they will trunk by default, which we do not want in this example.
This was also done on ALS1 and ALS2 to prevent any trunking.
107
Default VLANs
108
109
Same on DLS2.
:
:
:
:
:
:
2
0
1005
5
Server
110
:
:
:
:
:
:
2
3
1005
8
Server
West
Verified.
112
:
:
:
:
:
:
2
3
1005
8
Server
West
Now on DLS2: Add VTP Domain Name and configure different VLANs
Configuration Revision changed to 3
Still no trunking
113
Verified.
114
Mode
on
on
Encapsulation
802.1q
802.1q
Status
trunking
trunking
Native vlan
1
1
115
:
:
:
:
:
:
2
3
1005
8
Server
West
:
:
:
:
:
:
2
3
1005
8
Server
West
116
117
:
:
:
:
:
:
2
4
1005
9
Server
West
118
Verified.
119
:
:
:
:
:
:
2
4
1005
9
Server
West
120
121
20
name
vlan
name
vlan
name
WestAcct
21
WestMngt
22
WestManuf
:
:
:
:
:
:
2
7
1005
12
Server
West
Verified.
123
:
:
:
:
:
:
2
7
1005
12
Server
West
125
126
127
:
:
:
:
:
:
2
10
1005
12
Server
West
:
:
:
:
:
:
2
10
1005
12
Server
West
128
:
:
:
:
:
:
2
13
1005
9
Client
West
130
131
:
:
:
:
:
:
2
10
1005
12
Server
West
132
:
:
:
:
:
:
2
13
1005
9
Server
West
:
:
:
:
:
:
2
13
1005
9
Client
West
DLS2 (Switch B) is
brought online (no
shutdown on DLS1).
DLS2 (Client) has
higher Configuration
Revision number 13.
DLS1 (Switch A) with
lower revision
number (10) updates
its VLAN information
to be in sync with
DLS2 including its
Configuration
Revision number to
13.
133
Fix it
DLS1(config)# vlan
DLS1(config-vlan)#
DLS1(config-vlan)#
DLS1(config-vlan)#
DLS1(config-vlan)#
DLS1(config-vlan)#
10
name
vlan
name
vlan
name
WestSales
11
WestEng
12
WestAdmin
135
136
:
:
:
:
:
:
2
16
1005
12
Server
West
:
:
:
:
:
:
2
16
1005
12
Client
West
Still in sync!
137
How to make sure switch has Lower Config Rev: VTP Mode
VTP Domain = West
VTP Mode = Server
Config Rev = 10
VLANs = 1, 10, 11, 12, 20, 21, 22, 30
?
device to client mode.
device to server mode.
device to transparent mode.
138
How to make sure switch has Lower Config Rev: VTP Domain
VTP Domain = West
VTP Mode = Server
Config Rev = 16
VLANs = 1, 10, 11, 12, 20, 21, 22, 30
139