You are on page 1of 89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Alcatel-Lucent Advanced Troubleshooting (AT)


Lab & Troubleshooting Guide
Revision 1.2.1
November 27, 2007
Software Version: 7750 SR OS 4.0

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

1/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute


Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

2/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Table of Contents
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide.........1
Troubleshooting Methodology .................................................................................... 1-8
General Layered Approach ......................................................................................... 1-8
Quantitative vs. Qualitative Service Issues ................................................................. 1-8
Services Context.......................................................................................................... 1-8
Fault Determination vs. Fault Resolution vs. Fault Isolation ..................................... 1-9
The OSI Model A Layered Approach to Network Architecture and Troubleshooting1-10
The Physical Layer .................................................................................................... 1-11
General .......................................................................................................................... 1-11
Alcatel-Lucent Services Model Context ....................................................................... 1-11

The Data Link Layer ................................................................................................. 1-12


General .......................................................................................................................... 1-12
Alcatel-Lucent Services Model Context ....................................................................... 1-13

The Network Layer ................................................................................................... 1-15


General .......................................................................................................................... 1-15
Alcatel-Lucent Services Model Context ....................................................................... 1-16

The Transport Layer .................................................................................................. 1-18


General .......................................................................................................................... 1-18
Alcatel-Lucent Services Model Context ....................................................................... 1-18

The Application Layer (TCP/IP = Layers 5, 6, and 7 of the OSI Model) ................. 1-19
General .......................................................................................................................... 1-19

Lab 1

Network discovery and backing up images .............................. 1-21

Section 1.1 - Physical connectivity ........................................................................... 1-21


Objective: ...................................................................................................................... 1-21
Syntax: .......................................................................................................................... 1-22
Exercise: ........................................................................................................................ 1-22
Verification:................................................................................................................... 1-23

Section 1.2 - IP Connectivity and IGP Configuration............................................... 1-24


Objective: ...................................................................................................................... 1-24
Exercise: ........................................................................................................................ 1-24
Verification:................................................................................................................... 1-24

Section 1.3 Save your configuration to the TFTP Server ...................................... 1-25
Objective: ...................................................................................................................... 1-25
Exercise: ........................................................................................................................ 1-25
Verification:................................................................................................................... 1-25
Questions:................................................................................................................. 1-26

Lab 2

Setting event logs and debugs .................................................... 2-27

Section 2.1 Configure Event Logs ......................................................................... 2-27


Objective: ...................................................................................................................... 2-27
Syntax: .......................................................................................................................... 2-27
Exercise: ........................................................................................................................ 2-27
Verification:................................................................................................................... 2-27

Section 2.2 Use Debugs on the router for analysis ................................................ 2-28
Objective: ...................................................................................................................... 2-28
Exercise: ........................................................................................................................ 2-28

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

3/89

Section 2.3 Logging a Packet with a Filter ............................................................ 2-28

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective: ...................................................................................................................... 2-28


Exercise: ........................................................................................................................ 2-28
Verification:................................................................................................................... 2-28

Lab 3

Troubleshooting OSPF ............................................................... 3-30


Objective: ...................................................................................................................... 3-30
Syntax: .......................................................................................................................... 3-30

Section 3.1 OSPF Adjacency Troubleshooting ....................................................... 3-31


Exercise: ........................................................................................................................ 3-31
Exercise: ........................................................................................................................ 3-32
Exercise: ........................................................................................................................ 3-32
Exercise: ........................................................................................................................ 3-32
Exercise: ........................................................................................................................ 3-33

Lab 4

Troubleshooting RIP-OSPF redistribution .............................. 4-34


Objective: ...................................................................................................................... 4-34

Section 4.1 ................................................................................................................. 4-34


Exercise: ........................................................................................................................ 4-34

Section 4.2 ................................................................................................................. 4-35


Exercise: ........................................................................................................................ 4-35

Section 4.3 ................................................................................................................. 4-35


Exercise: ........................................................................................................................ 4-35

Section 4.4 ................................................................................................................. 4-35


Exercise: ........................................................................................................................ 4-35

Lab 5

Troubleshooting ISIS .................................................................. 5-36


Objective: ...................................................................................................................... 5-36

Section 5.1 ISIS Network troubleshooting............................................................. 5-37


Exercise: ........................................................................................................................ 5-37
Exercise: ........................................................................................................................ 5-38

Section 5.3 ISIS Network Troubleshooting ........................................................... 5-38


Exercise: ........................................................................................................................ 5-38

Lab 6

Troubleshooting BGP ................................................................. 6-39


Objective: ...................................................................................................................... 6-39

Section 6.1 ................................................................................................................. 6-39


Exercise: ........................................................................................................................ 6-39

Section 6.2 ................................................................................................................. 6-40


Exercise: ........................................................................................................................ 6-40

Section 6.3 BGP peering problem ............................................................................. 6-41


Objective: ...................................................................................................................... 6-41
Exercise: ........................................................................................................................ 6-41

Section 6.4 BGP Route redistribution problem ........................................................ 6-42


Exercise: ........................................................................................................................ 6-42

Section 6.5 BGP Route Policy problem .................................................................... 6-42


Exercise: ........................................................................................................................ 6-42

Lab 7

MPLS LSPs With RSVP Signaling ........................................... 7-43


Objective: ...................................................................................................................... 7-43

Section 7.1 ................................................................................................................. 7-43


Exercise: ........................................................................................................................ 7-43

Section 7.2 ................................................................................................................. 7-44

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

4/89

Exercise: ........................................................................................................................ 7-44

Section 7.3 ................................................................................................................. 7-44

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Exercise: ........................................................................................................................ 7-44

Section 7.4 ................................................................................................................. 7-44


Exercise: ........................................................................................................................ 7-44

Lab 8

Troubleshooting VPLS Service Connectivity ........................... 8-45


Objective: ...................................................................................................................... 8-45

Section 8.1 Troubleshooting VPLS........................................................................... 8-45


Exercise: ........................................................................................................................ 8-45

Section 8.2 Troubleshooting VPLS........................................................................... 8-46


Exercise: ........................................................................................................................ 8-46

Section 8.3 Troubleshooting H-VPLS ...................................................................... 8-47


Exercise: ........................................................................................................................ 8-47

Section 8.4 Tracking packet flow in a VPLS with layer 3 termination .................... 8-49
Objection: ...................................................................................................................... 8-49
Exercise: ........................................................................................................................ 8-50

Lab 9

VPRN problems .......................................................................... 9-51


Objective: ...................................................................................................................... 9-51

Section 9.1 Basic VPRN problems ........................................................................... 9-51


Exercise: ........................................................................................................................ 9-51

Section 9.2 Basic VPRN problems ........................................................................... 9-52


Exercise: ........................................................................................................................ 9-52

Section 9.3 Basic VPRN problems ........................................................................... 9-52


Exercise: ........................................................................................................................ 9-52

Section 9.4 VPRN spoke-sdp termination problems ................................................ 9-53


Exercise: ........................................................................................................................ 9-53

A. Common troubleshooting scenarios........................................................... 9-54


A.1. Layer 1 & Layer 2 Problems .............................................................................. 9-54
A.1.1.
A.1.2.
A.1.3.
A.1.4.
A.1.5.

How to show Layer 1 & Layer 2 alarms .......................................................... 9-54


Verify cards, MDAs and ports configuration ................................................... 9-54
How to show or clear statistics on a port or a LAG or a SAP .......................... 9-55
How to show or modify the operational status of a port .................................. 9-55
How to loop ports ............................................................................................. 9-55

A.2. OSPF Problems .................................................................................................. 9-57


A.2.1. Commands common to any OSPF troubleshooting ......................................... 9-57
A.2.2. OSPF Adjacency Does Not Come Up .............................................................. 9-59
A.2.3. Isolating Intermittent Reachability In An OSPF Environment ........................ 9-61

A.3. ISIS Problems .................................................................................................... 9-61


A.3.1. Commands common to any ISIS troubleshooting ............................................ 9-61
A.3.2. ISIS Adjacency Does Not Come Up ................................................................ 9-63
A.3.3. Isolating Intermittent Reachability In An ISIS Environment ........................... 9-64

A.4. BGP Problems .................................................................................................... 9-65


A.4.1. Commands common to any BGP troubleshooting ........................................... 9-65
A.4.2. BGP peer session not established ..................................................................... 9-66
A.4.3. BGP load balancing issue ................................................................................. 9-67

A.5. Prefix-list (Access-list) in the Route Policy....................................................... 9-69


A.6. Black holing Problems ....................................................................................... 9-72
A.7. LDP not established ........................................................................................... 9-73
A.8. RSVP Signaled LSP Problems........................................................................... 9-74
Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

5/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.8.1. Commands common to any RSVP troubleshooting ......................................... 9-74


A.8.2. RSVP MPLS Tools Perform Commands ......................................................... 9-77
A.8.3. RSVP Signaled LSP Does Not Come Up......................................................... 9-78

A.9. Service Distribution Path (SDP) Problems ........................................................ 9-80


A.9.1. Commands common to any SDP troubleshooting............................................ 9-80
A.9.2. SDP Does Not Come Up .................................................................................. 9-81

A.10.

Layer 2 Service Connectivity Problem .................................................. 9-81

A.10.1. Commands common to Layer 2 Service troubleshooting ................................ 9-82


A.10.2. Layer 2 Service Troubleshooting Steps............................................................ 9-83
A.10.3. Service SDP Binding Does Not Come Up ....................................................... 9-83
A.10.4. Service SAP Binding Does Not Come Up ....................................................... 9-84

A.11.

Layer 3 VPRN Problem ......................................................................... 9-85

A.11.1. Commands common to VPRN Service troubleshooting .................................. 9-85


A.11.2. Layer 3 VPRN Component Requirements ....................................................... 9-86
A.11.3. Route Selection Process for VPRNS when BGP is being used as the PE-CE
routing protocol ................................................................................................ 9-87
A.11.4. Routes not Getting Populated in to the VRF .................................................... 9-87

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

6/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Table of Figures
Figure 1 Demarcations in the Alcatel-Lucent Services Model ....................................... 1-9
Figure 2 The OSI Model .............................................................................................. 1-10
Figure 3 OSI Layer 1 ................................................................................................... 1-11
Figure 4 OSI Layer 2 ................................................................................................... 1-12
Figure 5 OSI Layer 3 ................................................................................................... 1-15
Figure 6 OSI Layer 4 .................................................................................................. 1-18
Figure 7 Layers 5, 6 and 7 of the OSI Model .............................................................. 1-19
Figure 1-1: Physical Connectivity ................................................................................... 1-21
Figure 1-2: IP Connectivity ............................................................................................ 1-24
Figure 1-3: TFTP Server Location ................................................................................... 1-25
Figure 3-1: OSPF Topology............................................................................................. 3-30
Figure 3-2: OSPF Topology............................................................................................. 3-33
Figure 4-1: OSPF-RIP Topology ..................................................................................... 4-34
Figure 5-1: IS-IS Topology.............................................................................................. 5-36
Figure 5-2: IS-IS Topology.............................................................................................. 5-38
Figure 6-1: BGP Topology .............................................................................................. 6-39
Figure 6-2: BGP Route Reflector Topology .................................................................... 6-40
Figure 6-3: BGP and OSPF topology .............................................................................. 6-41
Figure 7-1: Full Mesh of LSPs......................................................................................... 7-43
Figure 8-1: VPLS 4000 .................................................................................................... 8-45
Figure 8-2: H-VPLS ......................................................................................................... 8-47
Figure 8-3: VPLS with Layer 3 Termination ................................................................... 8-49
Figure 9-1: VPRN Setup .................................................................................................. 9-51
Figure 9-2: VPRN With Spoke Termination ................................................................... 9-53

Table of Tables
Table 1-1: Router remote access addresses ..................................................................... 1-22
Table 1-2: Lab 1 command list ........................................................................................ 1-22
Table 2-1: Lab 2 command list ........................................................................................ 2-27
Table 3-1: Lab 3 configuration commands ...................................................................... 3-31
Table 5-1: IS-IS Commands ............................................................................................ 5-37
Table 8-1: Command List ................................................................................................ 8-50

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

7/89

Troubleshooting Methodology

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

General Layered Approach


An effective troubleshooting methodology will use layered network architecture as leverage towards
finding a problem, or at least isolating the problem. Walking up through the protocol stack in an organized,
methodological approach is key to fault determination and isolation.

Start

Problem
Reported

Details Requested of
Customer as reqd
Ticket
Opened

As the troubleshooter progresses up the protocol stack,


troubleshooting becomes more complex and involves more
specialized knowledge of the network architecture and interrelated
systems. It is the nature of a troubleshooter to use ones own internal
database of past experience and network knowledge to attempt a cutand-try approach to troubleshooting. That is, to try possible solutions
based on the symptoms of a problem without taking the time to look
for the root cause. That is often what is accomplished with this
approach.the superficiality of the problem might be dealt with, but
in not properly dealing with the root cause of a network problem or
outage, the underlying problems might cause unwanted interaction
with other features or services as they are added to the network
architecture. Often the quick fix is required to quickly deal with an
issue in order that the customers traffic might proceed through the
network, but just like a bandage on a wound, eventually it needs to
come off for proper and full healing to take place.

Quantitative vs. Qualitative Service Issues


Fault Isolation

Fault
Determination

Fault
Resolution

Customer Notified / Testing of


Resolution / Signoff
Ticket Closed

Stop

Problems are not necessarily binary in nature. They are not always
on or off. Our experience tells us that sudden network outages,
while hopefully rare, are often caused by some type of hardware
malfunction or physical cabling plant issues. Fibre backbone torn up
by contractors, power outages of equipment in buildings whose
network equipment has not been properly configured with a backup
power supply, EMI sources and sinks, etc., are typically fairly
straightforward to isolate and rectify.

Services Context
Qualitative issues are somewhat more difficult to troubleshoot and
isolate since they often deal with, at least from the customers
perspective, intangiblesthings that are often subject and hard to
quantify. One of the major strengths of the Alcatel-Lucent Services
Routers and Ethernet Services Switches is that they have a rich suite
of tools that can be used to apply metrics to quality of service issues.
Because of the Alcatel-Lucent model of building services, (or service
tunnels) across an IP/MPLS core and the subsequent ability to
segregate customer traffic and manage it in discrete flows, it is
relatively straightforward to troubleshoot QoS issues. SAA, OA&M,
show, and debug commands can be used both in real-time, as well as
for statistical reporting afterwards.

Because of this service-oriented approach to network architecture, Alcatel-Lucent devices


cannot be configured, managed or troubleshooted in the manner of other routers and

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

8/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

switches. Troubleshooting up the protocol stack is the preferred approach but it should be
viewed in the context of not only the underlying switched and IP/MPLS infrastructure but
equally importantly in the context of services.

Fault Determination vs. Fault Resolution vs. Fault Isolation


It is recognized that troubleshooting in a practical sense is designed to resolve a problem, though it would
be wonderful to be able to determine what the problem is to begin with. Customers pay us on a problem
resolution basis, not a problem determination basis. Most SLAs have monetary indemnification built in
to them such that the service provider will pay the customer for outages on a sliding, pro-rated scale of
time-based penalties.
A good troubleshooting approach understands that fault isolation is the single most important first step
towards fault determination then eventually fault resolution. The Alcatel-Lucent services model formalizes
the natural demarcations in a network with network and access ports.

Figure 1 Demarcations in the Alcatel-Lucent Services Model

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

9/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

The OSI Model A Layered Approach to Network


Architecture and Troubleshooting

Figure 2 The OSI Model

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

10/89

The Physical Layer

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 3 OSI Layer 1

General
Troubleshooting Layer 1 can be relatively straightforward. This is the Is it plugged in? layer. In
summary, the physical layer is responsible for bit-encoding and physical cabling topology. It defines how
binary 1s and 0s are represented on the wire and defines the physical interface that devices have with the
cabling topology. For example, Ethernet uses Manchester binary encoding at layer 1 and uses protocols
such as the nWay protocol to negotiate speed and duplex of nodes connecting to the cabling plant.
Common issues as this layer are:
speed, and duplex (for multi-access, non point-to-point links)basic PnP issues.
bit encoding (for example NRZI vs. RZI) for point-to-point serial links
improper category, grade and installation of cabling plant
excessive collisions (multi-access, shared topologies like wireless Ethernet access points, wired
Ethernet hubs)
signal loss due to attenuation, impedance, cable length, cross-talk, EMI and capacitance (cabling
plant again)
mismatched layer 1 setting between nodes (for example laser emitter settings on fibre optic shorthaul vs. long-haul circuits).

Alcatel-Lucent Services Model Context


If the physical layer is down, then all higher layers will be down as well. Services which are built on the
higher layers will not be established and may appear administratively up and operationally down.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

11/89

The Data Link Layer

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 4 OSI Layer 2

General
The Data Link layer frames / organizes the binary 1s and 0s into frames, labeling each frame with an
address field. For multi-access links such as Ethernet and FDDI, the frame labels the data with both a
source and destination address since the underlying physical topology might be shared media. In point-topoint, non-shared data link configurations such as serial links, a single address field is required since the
data link itself is essentially being identified. These point-to-point data link configurations, often called
virtual circuits, are typical of packet-switched technologies such as ATM, Frame Relay and X.25.
Services that create a pseudo layer 2 are called VLL (Virtual Link Layer) services. ePipe, aPipe and fPipe
create pseudo point-to-point data link configurations. VPLS creates a pseudo multi-access data link
configuration. Troubleshooting these services is discussed briefly in their separate contexts below.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

12/89

Alcatel-Lucent Services Model Context

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

QoS solutions will often use information in the layer 2 PDU for classification and marking decisions. For
example source/destination SAPs (IEEE service access points), 801.q VLAN ID, 802.1p priority bits and
source/destination MAC addresses might be used to either segregate traffic into different forwarding
classes or to immediately dispatch high-priority traffic to LLQs. Problems often arise if the QoS solution
doesnt properly classify the customers traffic or else properly respect the layer 2 markings which indicate
priority.
VLL Services
PE B

PE C

PE A

IP / MPLS
ePipe
service

Network
PE D

Some Alcatel-Lucent services mimic or emulate layer 2 of the OSI model. For example, VLLs such as
aPipe, ePipe, and fPipe services create logical vs. physical point-to-point data links between peer devices.
The service appears as a single data link or wire between the peer devices. Customer devices (represented
as buildings in the diagram above) that use the service must be on the same IP subnet as a result. Many
problems with VLL type services can be rectified by visualizing the logical layout of the network.
Knowledge of framing, basic organization of the layer 2 PDU including 802.1p, Q-in-Q, 802.1q, as well as
STP are crucial.
The aPipe service uses ATM for transport. Customers traffic is mapped to ATM VCs, creating a VLL
mapped directly to an ATM VC.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

13/89

VPLS Service

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

PE B
VPLS Service
IP/LSP FullMesh

PE C

PE A

IP / MPLS
Network
PE D

The VPLS service extends the customers switched network across the IP/MPLS core. It provides for a
layer 2 class of VPN solution. The IP/MPLS core appears as a logical switch (pictured as a solid rectangle
in the above picture) to the customers equipment. As with VLL services, a thorough knowledge of
trunking, STP and layer 2 prioritization would be useful. For example, if IEEE 802.2 LLC frames are
switched across the network, the network may experience a higher load that Ethernet II frames since the
former is connection-oriented. As such retransmission and basic sequencing and acknowledgement may
create timing issues for the customers traffic if a proper QoS solution where this traffic is not given higher
priority than other flows is not implemented.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

14/89

The Network Layer

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 5 OSI Layer 3

General
According to RFC 791, The Internet Protocol is responsible for providing addressing of the logical data
link between TCP/IP end systems. While it does not, itself, create or manage the logical data link (this is
the responsibility of the Transport Layer), it is responsible for packaging and fragmenting/reassembling
customer data for delivery by intermediate systems (routers) and providing enough information such that
routers can make informed decisions as to where next to transmit the data on a hop-by-hop basis. QoS
solutions can modify this PHB (per-hop behaviour) by respecting the markings in the ToS byte of the IP
header which indicate the requirement for differentiated services in the IP network. Traffic can be marked,
classified and forwarded based on other information found in the IP header including IP version number,
source/destination address, protocol number, etc. Knowledge of the organization of an IP packet, IP
addressing in general and IP address planning / subnetting in general is crucial. Furthermore, access lists
can be created to filter and block traffic based on just about any piece of information in the IP packet
header.
Dynamic routing protocols, both IGPs and EGPs, can take advantage of good subnetting and route
summarization techniquescareful IP address planning to reduce routing table sizes and the complexity
of the forwarding of IP datagrams throughout an IP network. Basic, common sense principles of 1 subnet =
1 wire (pseudo or physical) must be adhered to. Many troubleshooting exercises begin and end with
careful analysis of the per-hop behaviour of traffic through an IP network. Given the network layers
responsibility to address the logical endpoints for TCP and UDP sessions (as well as stateless, tunnelled
traffic such as GRE and IPSec) and routers responsibility to find the best path between the endpoints, it is
hard to overstate its importance.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

15/89

Alcatel-Lucent Services Model Context

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

VPRN (RFC 2547bis)


MP-BGP
Route
Exchange
For all
Services

VPRN
Service
Red

RI-1
RI-2

PE B

PE C

PE A
RI-1

RI-1

RI-2

RI-2

IP / MPLS
Network
PE D

RI-1

VPRN
Service
Green

RI-2

VPRN is a class of VPN that allows the connection of multiple sites in a routed domain over a provider
managed IP/MPLS network. From the customers perspective it looks like all sites are connected to a
private routed network administered by the service provider for that customer only. The Service provider,
however, can reuse the IP/MPLS infrastructure to offer multiple services.
Each VPRN appears like an additional routing instance, routes for a service between the various PEs are
exchanged using MP-BGP. From a troubleshooting perspective this is critical because each service
contains a separate instance of routed traffic, segregating the customers routed traffic and allowing traffic
conditioning, shaping, modified PHBs and other QoS policies to be managed individually. More
importantly, unless there is some catastrophic, universal problem in the IP/MPLS core, each customers
traffic can be troubleshooted separately too. This is the P for Private in VPRN. Effectively the
customers traffic flows in virtual, private, managed layer 3 domains in the same way the with the VPLS
service the customers traffic flows in virtual, private, managed layer 2 domains.
Since the SDPs tie into the providers IP/MPLS core and are signalled within the context of IP tunnels or
MPLS/RSVP, they are sensitive to bad design and other faults in the underlying layer 3 infrastructure.
Also, another troubleshooting nexus is the use of the customers own IP address structure, often using their
own RFC 1918 addresses when connecting to sites through the service providers service tunnels. This
often means that troubleshooting is done on two planes, the service provider network and the customers
virtual reality network.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

16/89

IES Service

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Internet
Company C

PE C

CE C

PE A

CE A

Company A

Service Provider
Network

PE B
CE B

Company B

From the customers perspective the IES provides a direct connection to the Internet. The Service provider
can apply all billing, ingress/egress shaping and policing to the customer. Unlike the VPRN service, there
is no separation at the network layer for the customer and the service provider. This coupling of customer
traffic to the service providers network may make troubleshooting more problematic in some
circumstances as natural demarcation points may be hard to find at the network layer. However, using
hierarchical routing protocols, good route summarization techniques and redistribution between the access
layer and the edge may aid in fault isolation.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

17/89

The Transport Layer

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 6 OSI Layer 4

General
In an IP network there are two transport layer protocols, TCP and UDP. TCP is connection-oriented and
UDP is connectionless. TCP and UDP share source and destination port numbers that can be used as
discriminators for a variety of filtering and QoS decisions. These port numbers are associated with and
identify application layer protocols on the server side of the connection such as Telnet, FTP, SMTP, etc. In
addition, because TCP is connection oriented, it has additional fields that manage the state of the
connection (the control field) as well as sequencing (sequence number and acknowledgement number) as
well as receiving windows size, etc. As a consequence, TCP has more overhead per unit data than UDP.
From a troubleshooting standpoint, what makes TCP somewhat problematic is that the devices inside the
service providers network tend to be ignorant of the TCP session parameters since this is largely the
responsibility of the endpoints of the TCP connectionthe customers devices. While these TCP
parameters may used for input classification decisions, for example, as part of a QoS solution, congestion
issues in the service providers network can be exacerbated by the retransmission of data segments and
other spurious signalling initiated by the customer end systems, causing a cascading effect in extreme
situations. Furthermore, customers applications may time out if there is not sufficient priority given to
TCP connection setup in the QoS solution offered by the service provider.

Alcatel-Lucent Services Model Context


The transport layer can affect all types of services. Congestion control mechanisms within the context of a
QoS solution can be rendered ineffective without understanding the slow start, congestion control and
congestion avoidance mechanisms built into TCP. QoS solutions which use WRED in a slope policy, for
example, scale well in networks where the customer has a number of TCP sessions. Because TCP is
connection-oriented, it can often hide issues at lower layers through its various mechanisms. What might
appear to be a slow network might, in fact, be a port flapping on an SR because of a speed / duplex issue at
layer 1. If resilience is designed in at layer 3 (routes in hold-down, aggregation, summary routes, etc.) the
problem might be very difficult to find without the right tools.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

18/89

The Application Layer (TCP/IP = Layers 5, 6, and 7 of the OSI Model)

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 7 Layers 5, 6 and 7 of the OSI Model

General
Layers 5, 6, and 7 of the OSI model are often represented as a layer 5 of the TCP/IP stack. Since there is
no RFC that defines an application layer nor for that matter a layered network architecture, this
simplification is debatable and can often lead to errors when troubleshooting a network. For example, layer
5 of the OSI model, the session layer, is a management layer that indicated to the transport layer what level
or type of service is required for the transport of application layer protocols.
The requirement for encryption at the transport layer, connection-oriented vs. connectionless transport, etc.,
is indicated by the session layer. While, some may view this as an unnecessary distinction in
troubleshooting a service-oriented solution which does not classify traffic based on any information
above layer 4.knowledge of timers, synchronization and other application layer behaviours will be useful
in conceptualizing problems. For example, knowing how a VoIP call, is setup, calls progress and then
sessions are torn down might prove useful.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

19/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

While classification and differentiated services for an application layer protocol occur at a lower layer of
the OSI model in the Alcatel-Lucent services model, the symptoms of inadequate network provisioning and
outages will be seen first at the application layer. This, of course, is what prompts our customers to call us
in the first place.
As was indicated earlier, as we progress up through the layered network protocol stack, more specialized
knowledge of the protocols is required.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

20/89

Lab 1 Network discovery and backing up images

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 1.1 - Physical connectivity


Objective:
Verify the operation and physical connectivity of the routers, which are connected according to the
following diagram.

Figure 1-1: Physical Connectivity

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

21/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Pod Number

Router Name

Connect Address

Pod 1 -

P1R1

192.168.161.221

P1R2

192.168.161.241

P1R3

192.168.161.231

P2R1

192.168.161.222

P2R2

192.168.161.242

P2R3

192.168.161.232

P3R1

192.168.161.223

P3R2

192.168.161.243

P3R3

192.168.161.233

P4R1

192.168.161.224

P4R2

192.168.161.244

P4R3

192.168.161.234

Pod 2 -

Pod 3 -

Pod 4 -

Table 1-1: Router remote access addresses

Syntax:
Commands required for this exercise are found in Table 1-2. Detail may be found in Module 1, IGP
Review. Each command may have additional parameters possible. Use the ? character for help and to
explore all command line options. Other commands may also be used, including those from previous
courses.

Lab 1 Configuration Command List


telnet
ping
Show port
Show router interface
Show router route-table
Show router status
Show router ospf neighbors
Show router ospf database
admin save

Table 1-2: Lab 1 command list

Exercise:
1.

2.

Connect to the routers in your Pod using the addresses provided by your instructor. Fill in the
required fields for Table 1. The username and password for all devices is admin. If you are
unable to connect to any of the routers, notify your instructor.
Verify the router has the initial configuration uploaded.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

22/89

Verification:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

1.
2.

Determine if you can connect to your routers


Ensure that a basic configuration is loaded on the router

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

23/89

Section 1.2 - IP Connectivity and IGP Configuration

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Identify the IP addressing and routing protocol for your pods topology.

Figure 1-2: IP Connectivity

Exercise:
Determine the IP addressing and pod configuration for your assigned pod.
1. Identify the naming convention for each router in your pod
2. Document the IP addressing and subnet masking on each interface of each router and fill in the
diagram above. This addressing will not change throughout the entire course. This diagram can
be used as reference for the labs that follow.
3. Ensure network connectivity by use of Ping, and ssh
4. Determine what routing protocol is configured on the router (note: The routing protocol will
change in labs that follow)
5. Ensure routing is operational
6. Document the network topology

Verification:
1.
2.
3.

Verify that all required interfaces are active and operational.


Verify that all adjacencies are up
Verify that all expected routes are populated in the routing table

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

24/89

4.

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Make sure your lab topology is complete and accurate. The following labs will not provide
addressing. It is your responsibility to use your documentation from this lab for the subsequent
troubleshooting labs.

Section 1.3 Save your configuration to the TFTP Server


Objective:
On both the Core and Edge router save your configuration files to the TFTP server

INSERT TFTP LOCATION DIAGRAM


Figure 1-3: TFTP Server Location

Exercise:
Ensure you can access the TFTP server and then save the configuration files to it.
1.
Your instructor will provide you with the IP address of the TFTP server for each pod.
2.
Ping the TFTP server and ensure its operational
3.
Save your Core routers configuration file using the following naming convention:
a. podXcore-student
4.
Save your Edge routers configuration file using the following naming convention
a. podXedge-student
5.
If possible access the TFTP server and ensure the file has been saved

Verification:
Make sure the files have been saved on the TFTP server.
1.
Access your TFTP server and look for the file you saved
2.
If unable to access the TFTP server directly, ask your instructor to verify the file has been
saved to the TFTP server.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

25/89

Questions:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

What command would be used to ensure all ports are active?_____________________________________


What command can be used to determine the naming convention used for the interfaces? ______________
______________________________________________________________________________________
What routing protocol is currently active on your routers? _______________________________________
Are all routers in the Same Area? _________________________________________________
What command can be used to determine if all routers are in the same area?
__________________________________________________________

Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

26/89

Lab 2 Setting event logs and debugs

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 2.1 Configure Event Logs


Objective:
Configure event logs on your R1 router.

Syntax:
Commands required for this exercise are found in Table 2-1. Detail may be found in Module 2,
Troubleshooting Methodology. Each command may have additional parameters possible. Use the ?
character for help and to explore all command line options. Other commands may also be used, including
those from previous courses.

Lab 2 Configuration Command List


log-id log-id
log filter filter-id
from {[main] [security] [change] [debug-trace]}
to {console | session | syslog syslog-id | snmp [max-size] | file file-id | memory [maxsize]}
Show log log-id
Clear log X
No debug
Debug router ip packet <ip address> headers <protocol number>
Ping <ip address>

Table 2-1: Lab 2 command list

Exercise:
For this exercise each student will create a log on one of the routers in their POD.
1. Configure an event log on the R1 and R2 router using log-id 1
2. define the source as all events from the main event stream
3. define the destination to memory with a maximum size of 100
4. Create log filter so only link down and link up events are logged.

Verification:
1.
2.
3.
4.
5.

Using show commands ensure the log is created in your router


Clear log 1 and log 99
On R1 shutdown the port facing R2 within your pod
Examine log 1 and confirm only interface down and interface up events are logged
Compare this to the information displayed in log 99

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

27/89

Section 2.2 Use Debugs on the router for analysis

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Using Debugs examine the output and determine usability.

Exercise:
For this exercise each student will enable debug on one of the routers in their POD.
Use Debug for packet analysis. It is always a good idea to have multiple sessions open when enabling
debug.
1. Open 2 sessions to a router within your POD
2. From one of your sessions create a log file with log-id 2
3. Define the source of information that of debug-trace
4. Forward the information to session
5. Enable debug for IP packet
6. Initiate a ping between the R1 and R2 router in your pod
7. Notice from the debug output there is a lot more information than the ping
8. From the session that is not being used for debug administratively shutdown the log that is being
used for debug (note: this releases any system resources that were being used for debug). Other
methods for turning debug off are disconnecting the session that is being used for debug or using
the command no debug.
9. Disable the debug for IP packet
10. Enable debug for IP packet again but this time only enable it for icmp traffic on the interface
between R1 and R2 in your pod. What command did you use to do this?
____________________________________________________________________________
11. Turn off debug

Section 2.3 Logging a Packet with a Filter


Objective:
Log packets to memory with a filter.

Exercise:
For this exercise each student will create filter to log packets to memory on one of the routers in their POD.
1. Create a filter log to log to memory
2. On your R1 router Configure an IP filter to match the address of your R1 system address and your
R2 system address for ICMP protocol. The goal is to log echo requests and echo replies between
your R1 and R2 router. The default action should be forward and the action for each entry should
be forward. The same exercise can be done on your R3 router for pings between the R3 router and
R2 router in your POD.
3. clear the filter log to endues it is empty
4. Initiate a ping from your R1 router to the system address of your R2 router

Verification:
1.
2.
3.

show the contents of the filter log.


You should only see icmp traffic with the source and destination address as your R1 system
address or R2 system address
You should see icmp traffic egressing and ingressing on the interface the filter was applied to.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

28/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Questions:
1. What must be created prior to activating debug on the router? _______________
_________________________________________________________________
2. Where can the debug output be sent? _________________________________
_________________________________________________________________
3. Debug outputs are turned off by what command? _________________________
_________________________________________________________________
4. To log telnet debug traffic to a file what command would you execute when
activating the debug? _______________________________________________
__________________________________________________________________
Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

29/89

Lab 3 Troubleshooting OSPF

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Troubleshoot network problems in a OSPF routing environment

Figure 3-1: OSPF Topology

Syntax:
In this lab we will troubleshoot some typical OSPF problems. Several troubleshooting tools will be used.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

30/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Lab 3 troubleshooting command list


configure log log-id <log-id> from debug-trace
configure log log-id <log-id> to session
debug router ospf <options>
show router status
show router ospf neighbor <detail>
show router ospf interface <detail>
show router ospf database [type {router|network|summary|asbr-summary|external|nssa|all}]
[area <area-id>] [adv-router <router-id>] [<link-state-id>] <detail>
sow log log-id 99 application ospf
show router route-table

Table 3-1: Lab 3 configuration commands

Section 3.1 OSPF Adjacency Troubleshooting


Exercise:
Load the configuration file pxry_ospf1.cfg into the routers.
In this section the students will work on their own POD in pairs.
After the routers are pointing to this configuration and the routers have been rebooted, routers pxr3 will be
inaccessible for the students. Take a few minutes to look at the configuration set up.
1. Are all adjacencies up? If not, which adjacencies are down?
2. What is the state of the adjacency? Can you find the different adjacency states the router went
through in the log?
3. When the adjacency gets stuck in this state, what is generally the cause of it? Start a debug trace to
prove this. What OSPF message are you looking for? What command can you use to show the
locally configured operational value for this parameter?
4. Provide 2 ways to solve this problem.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

31/89

Section 3.2 OSPF Route Redistribution Problem

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Exercise:
In this section the students will work on their own POD.
Take a look at the OSPF configuration on pxr2.
1. Can you see any routing policy configured? What show command can you use to see if any
policies have been applied?
2. What type of route should you see in the OSPF database if the policy were functioning as
intended?
3. What is the solution to this problem?

Section 3.3 OSPF Router ID Problem


Exercise:
Before you start this section, the teacher will change something in the network.
In this section the students will verify this change in their POD.
Take a look at the OSPF database.
1. Do you see any additional entries (hint: look for new router-id on pxr1)?
2. Where does this entry come from? How did you find out?
3. How can you remove this LSA?

Section 3.4 OSPF route intermittent reachable


Exercise:
Before you start this section, the teacher will change something in the network.
Network 192.168.1.0/24 has been added to the network. The network administrator in area 2 has reported
intermittent connectivity to 192.168.1.0/24 network. They have been using 192.168.1.1 to test
connectivity.
1. Where is 192.168.1.0/24 being advertised from?
2. What commands can be use to isolate this type of problem in a OSPF environment?

Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

32/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 3.5 OSPF route redistribution problem on NSSA


:

Figure 3-2: OSPF Topology


A network administrator is trying to redistribute the 192.168.1.0 and 192.168.3.0 networks into Area 1 and
2 respectively. Users in area 1 however can not reach 192.168.1.0 and users in area 2 can not reach
192.168.3.0.

Exercise:
Load the pxry_ospf2.cfg configuration into all routers.
Isolate the problem.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

33/89

Lab 4 Troubleshooting RIP-OSPF redistribution

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Troubleshoot some typical problems that arise in RIP networks and when redistribution is used for RIP and
OSPF.

Figure 4-1: OSPF-RIP Topology


Before you start with the individual sections, load the pxry_ospf_rip.cfg configuration on each
router. Find the problem in each section but only solve the problem after the teacher tells you to.
There are several problems in this setup. Each problem will be covered in a separate section.

Section 4.1
Exercise:
When you logon to pxr2 or pxr3 you can see that not all routes from the backbone or from the other pods
can be found in the route table.
1. Whats wrong?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

34/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 4.2
Exercise:
The system address of pxr3 does not appear in the route-table of the other routers. Router pxr3 however
does receive RIP routes from the other routers.
1. Whats wrong?

Section 4.3
Exercise:
The link address of pxr1-pxr2 does not appear in the route-table of routers in other PODs.
1. Whats wrong?

Section 4.4
Exercise:
The system address of p2r2 and p2r4 does not appear in the route-table of the other routers.
1. Whats wrong?

Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

35/89

Lab 5 Troubleshooting ISIS

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Trouble shoot ISIS related network problems

Figure 5-1: IS-IS Topology

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

36/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Lab 5 troubleshooting command list


configure log log-id <log-id> from debug-trace
configure log log-id <log-id> to session
debug router isis <options>
show router status
show router isis adjacency <detail>
show router isis interface <detail>
show router isis database
sow log log-id 99 application isis
Show router route-table

Table 5-1: IS-IS Commands

Section 5.1 ISIS Network troubleshooting


Exercise:
In the IS-IS topology all networks and routers should be reachable from all routers. There is currently a
problem within the network preventing this from happening. Load the configuration file pxry_isis-1.cfg
into the routers.

1.

Identify the problem in the network. Do not implement any changes.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

37/89

Section 5.2 ISIS Network Traffic Flow

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 5-2: IS-IS Topology

Exercise:
There have been reports of asymmetrical traffic flow in the above network. Confirm the reports and
identify where this is occurring in the network.
Load configuration file pxry_isis-2.cfg.
1. Identify all link metrics and fill in the information in space provided on the topology
2. What is the cause of the asymmetrical traffic flow?

Section 5.3 ISIS Network Troubleshooting


Exercise:
Load the pxry_isis-3.cfg configuration in each router.
The topology is the same as in the previous sections. All networks and routers should be accessible from all
routers. The client opened a ticket because not all networks are reachable. Since this is a live network, you
are not allowed to make changes to the configuration.
1. What is the problem?
2. What are useful commands or tools to troubleshoot this problem?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

38/89

Lab 6 Troubleshooting BGP

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Troubleshoot BGP network related problems.

Figure 6-1: BGP Topology

Section 6.1
Exercise:
Load the pxry_bgp-1.cfg configuration in each router.
In the above network P4R1 is the preferred exit point for all traffic destined outside of AS 6510. All
networks and routers should be reachable from AS 6510. Notice P3R1 and P4R1 are the only routers
within AS 6510 running BGP. Both P3R1 and P4R1 are injecting default routes into the IGP.
1. Identify which routers and networks are not reachable
2. What is the result of the current configuration
3. Identify 2 possible solutions or recommendations

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

39/89

Section 6.2

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 6-2: BGP Route Reflector Topology

Exercise:
Load config pxry_bgp-2.cfg
In the above network P1R1 and P3R1 are router reflector servers for AS 6500. All networks in all 3
autonomous systems should be reachable from any router in any autonomous system. So far it has been
determined that there is no connectivity between AS 6510 and AS 6520.
Note. There are a number prefixes being advertised by AS 6510 and AS 6520 which will not be reachable
but are used to populate the routing tables with a larger number of entries. All networks that are physically
on each router should be reachable.
1. Identify if there are any other connectivity problems in the above network.
2. Identify the BGP peering sessions
3. Which prefixes are being populated in the routing table?
4. What is the Flag value of the BGP routes that are not being installed? What command did you use
to find this?
5. Identify 2 possible solutions to the problem

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

40/89

Section 6.3 BGP peering problem

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Troubleshoot BGP network related problems.

Figure 6-3: BGP and OSPF topology


Before you start with the individual sections, load the pxry_bgp-3 configuration on each router. Find
the problem in each section but only solve the problem after the teacher tells you to.
There are several problems in this setup. Each problem will be covered in a separate section.

Exercise:
There seems to be a problem with the BGP peering sessions on pxr3.
1. Whats wrong?
2. On top of show commands and checking configurations, what extra tools can you use to
troubleshoot these problems?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

41/89

Section 6.4 BGP Route redistribution problem

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Exercise:
Solve the problem in Section 6.3. All BGP peering should be up when you start section 6.4.
The routers in AS 6500 can not reach the p1r3 or p3r3 router.
1. Whats wrong?
2. How could you solve this problem

Section 6.5 BGP Route Policy problem


Exercise:
Solve the problem in Section 6.4. All routers should be reachable now.
In this section there is a problem with 192.168.[1,2,101,102].0/24 routes inserted on routers P1R3 and
P3R3.
Work in 2 teams, 1 looking at the problem with routes 192.168.1.0/24 and 192.168.2.0/24, the other team
looking at the routes 192.168.101.0/24 and 192.168.102.0/24.
The administrator wants to prepend the AS-path for the 192.168/16 routes. The administrator also wants the
routers in AS 6500 to choose P3R1 as the exit point for 192.168.102.0/24, P4R2 as the exit point for
192.168.101.0/24, P1R1 as the exit point for 192.168.2.0/24, and P2R2 as the exit point for 192.168.1.0/24.
A policy has been implemented on P1R2 and P3R2 but it does not seem to functioning as intended.
1. Whats wrong?
2. How can you solve this problem?
3. What command can you use to see the changes between the RIB in and RIB out?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

42/89

Lab 7 MPLS LSPs With RSVP Signaling

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Objective:
Troubleshoot MPLS RSVP signaled LSPs.

Figure 7-1: Full Mesh of LSPs

Section 7.1
Exercise:
Load pxry_fullmesh_lsps.cfg on all routers.
In the above network a full mesh of LSPs has been setup between all routers. For this exercise each student
will work a on a single router.
1. What command can be used to view RSVP sessions?
2. View only originating RSVP sessions.
3. Enable debug for only one of the originating RSVP sessions. Use debug router rsvp lsp-id
<lspname>::<pathname> packet path detail
4. Disable Debug for the single LSP
5. View only terminating RSVP sessions
6. Enable debug for only one of the terminating RSVP session. Use debug router rsvp lsp-id
<lspname> packet path detail
7. Disable debug for the single LSP
8. View only transit RSVP sessions
9. Enable debug for only of the terminating RSVP sessions. Use debug router rsvp lsp-id <lspname>
packet path detail
10. What other options are available for limiting the debug output for RSVP packets?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

43/89

Section 7.2

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Before you begin this section the instructor must implement a change.

Exercise:
There are several LSPs that are down. Use the tools perform router mpls cspf command to isolate where
the problem is in the network. Refer to the RSVP Signaled LSP Problems section in the appendix of the
lab guide. Try the options that are available in the command. Each node has visibility of the entire TE
network. There is no need to go to each router in the network to see which path cspf will return to an
endpoint. The from option can be used to determine what another router will return as its path to an
endpoint
1. Where in the network is problem?

Section 7.3
Before you begin this section the instructor has to make changes.

Exercise:
There are a number of LSPs that are down in the network.
1. Determine which LSPs are down. What do all the LSPs have in common?
2. What is the failure code of the LSP Path (show router mpls lsp <lspname> path <path name>
detail)?
3. What are the LSPs relying on to get the information required to signal the LSP?

Section 7.4
Exercise:
Before this lab begins the instructor must make changes
There is a mesh of LSPs between all R2 routers with FRR enabled. All expected detours are not coming
up.
1. View the LSPs and path configuration
2. For each of the FRR enabled LSPs determine where the expected detours should be.
3. Isolate where the problem is in the network.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

44/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Lab 8 Troubleshooting VPLS Service Connectivity


Objective:
Troubleshoot VPLS service connectivity.

Figure 8-1: VPLS 4000

Section 8.1 Troubleshooting VPLS


Exercise:
Load pxry-VPLS4000.cfg on all R1 and R2 routers. The R3 routers are used so the SAPs on the R2
routers are operational
All sites within VPLS 4000 are not reachable. From your R2 router follow the steps below to familiarize
yourself with a logical approach to troubleshooting layer 2 services.
1. View the basic service information show service id 4000 base
2. The output of the above command indicates the operational states of the service and the
components the service is bound to.
3. View the details of the component that is in the down state. In this case there are 1 or 2 mesh-sdp
bindings down. show service id 4000 sdp <sdp> detail
4. The output of this command will have further details for the reason the mesh-sdp is down. The
Flags in the output indicate the SDP is operationally down
5. View the details of the SDP that is operationally down. show service sdp <sdp-id> detail
6. From the output of this command we can see that the flags indicate the transport tunnel is down.
The associated LSP is also displayed in the output of the above command.
7. View the detailed output of the LSP that the SDP is bound to. show router mpls lsp <lsp-name>
detail. This command will output the paths that the LSP is using
8. View the detailed output of the LSP path. show router mpls lsp <lsp-name> path <path-name>
detail.
9. From the output of the command above we can see the explicit hops that the LSP is taking.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

45/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

10. Use the tools perform router mpls cspf to <address> command each explicit hop in the list. In
this case the second one fails. The 3rd explicit hop takes a path that is not expected. This is a good
starting point to help isolate the problem
11. Open a connection to the router that is the second in the path
12. Are all expected interfaces operationally up? show router interface
13. Can you ping the adjacent interface on the link? ping <address>
14. Are all routing adjacencies up? show router ospf neighbor
15. Is MPLS enabled on the right interfaces? show router mpls interface
16. What is the problem?

Section 8.2 Troubleshooting VPLS


Before this exercise begins the instructor must make a change.

Exercise:
A fix for the previous sections must be implemented before continuing with this section.
Connectivity between all sites in VPLS 4000 is down. Work in groups of 2 for this exercise.
1.
2.

Determine which sites are not reachable from each pod


Using the same methodology in the previous lab, isolate and determine the cause of the problem

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

46/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 8.3 Troubleshooting H-VPLS

Figure 8-2: H-VPLS

Exercise:
Load pxry_m-vpls.cfg on all routers
In the above diagram VPLS 6000 is fully meshed in Metro 1. VPLS 8000 is fully meshed in Metro 2. The
2 metro networks are connected via redundant Spokes between P1R1 and P3R1 and between P2R1 and
P4R1. Management VPLS 10000 has been created for the redundant Spoke connectivity. There are reports
that some sites are not reachable. It seems there maybe packets getting dropped. The R3 routers are CE
devices in this topology. Each is configured with a router interface on the 192.168.10.x/24.
1. You have full access to all equipment
2. Verify the reports of instability
3. Identify the cause of the problem

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

47/89

Notes

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute


Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

48/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Section 8.4 Tracking packet flow in a VPLS with layer 3 termination


Objection:
Trace the path of a packet through a VPLS with layer 3 termination

Figure 8-3: VPLS with Layer 3 Termination

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

49/89

Lab 8.4 Command list

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Show router interface <interface name> detail


Show router route-table
Show router arp
Show router ldp bindings
Ping source <source address> <destination address>
Traceroute source <source address> <destination address>
Show service id <id> fdb detail
show service fdb-mac
Show service sap-using
Show service service-using
Show service id <id> base

Table 8-1: Command List

Exercise:
Load pxry_trace.cfg on all routers.
Trace the path taken when initiating a ping from 192.168.1.2 to 192.168.3.2. Use 192.168.1.2 as the
source address when initiating a ping from P1R3. Include the return path from 192.168.3.2 to
192.168.1.2.

Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

50/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Lab 9 VPRN problems


Objective:
Troubleshoot VPRN services.

Figure 9-1: VPRN Setup

Section 9.1 Basic VPRN problems


Exercise:
Load pxry_vprnhns.cfg on all routers
There are 4 Hub and Spoke VPRNs configured, 100, 200, 300 and 400. Students will work in pairs to
troubleshoot one of the 4 VPRNs. P1R2 is the Hub for VPRN 100, P2R2 is the Hub for VPRN 200, P2R3
is the Hub for VPRN 300 and P2R4 is the Hub for VPRN 400. All traffic between Spoke sites route via the
Hub Site. All sites should be reachable. The R3 routers are not used for this lab but are required to have the
SAP operational on the R2 routers.
1. Login to each PE router
2. Document the VPRN configuration that you are troubleshooting
3. All VPRN interfaces should be reachable.
4. Determine which site(s) are not reachable
5. Isolate the problem

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

51/89

Section 9.2 Basic VPRN problems

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Exercise:
The instructor will implement a change before you begin this section.
VPRN 100, 200, 300 and 400 are configured in the same way as section 9.1. All VPRN interfaces should
be reachable from all VPRN interfaces within the same VPRN. Refer to the Layer 3 VPRN Problem
section in the appendix.
1. Isolate the problem

Section 9.3 Basic VPRN problems


Exercise:
Load pxry-services1.cfg
The Pxr3 routers are CPE routers that are connected through a VPRN service. They have an interface
192.168.x.100/24 that is connected to the PxR2 router. The CPEs can not ping each other though...
In this lab the students will be working in couples per POD. There are several problems in this lab. The
questions will guide the student through the problems. Note down the commands you used for each
of the problems.
1.
2.
3.
4.
5.

What is the service-id of this VPRN?


Can you see routes in the VRF?
After you see the routes in the VRF, can you ping the other CPEs? What is wrong with the ping?
Troubleshoot on the CPEs.
Can you ping the system address of the CPEs? Fix the problem.
What commands can you use to show the transport label and the service label?

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

52/89

Section 9.4 VPRN spoke-sdp termination problems

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Figure 9-2: VPRN With Spoke Termination

Exercise:
Load pxry-services1.cfg
The Pxr3 routers are CPE routers that are connected through a VPRN service. They have an interface
10.99.x.1/24 that is connected to the PxR2 router. The CPEs can not ping the gateway 10.99.x.100.
1.
2.

What is the problem?


What commands did you use to troubleshoot this?

Notes

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

53/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.

Common troubleshooting scenarios

This section covers some common troubleshooting scenarios that might happen in a 7750 SR network.

A.1.

Layer 1 & Layer 2 Problems

This section describes methods/commands that can be used to troubleshoot a Layer 1 or layer 2 (i.e. IOM,
MDA and port level) problem of 7750 SRs. More details of how to verify hardware operational status are
described in Section 4.

A.1.1. How to show Layer 1 & Layer 2 alarms


7750 SR has two default memory logs (Log-id 99 & 100) containing all the events from the main
application. All severity levels of alarms are recorded in log-id 99, where log-id 100 only contains serious
errors.
There are several ways to view the alarms of a specific subject, such as alarms related to a particular port.
One method is to create a new log that only monitors the specific subject. Refer to
7750_SR_OS_System_Guide_2.0.pdf for more details of how to configure a log.
Another much simpler way is to view the specific subject in the default log-id 99. The following shows
what command(s) to use for Layer 1& 2 alarms.
What To Check
Show alarms of a particular port
(ex. port 1/1/1)

CLI Command
show log log-id 99 subject 1/1/1

Show alarms related to the chassis

show log log-id 99 application


chassis

Show alarms of a particular IOM


(ex. IOM Slot #1)

show log log-id 99 subject Card 1

Show alarms of a SF/CPM


(ex. SF/CPM Slot #A)

show log log-id 99 subject Card A

Show alarms of a particular MDA


(ex. MDA 1/1)

show log log-id 99 subject Mda 1/1

Note: All the commands are Case Sensitive.

A.1.2. Verify cards, MDAs and ports configuration


The following CLI commands are commonly used for checking the detailed configuration of cards, MDAs
or ports. Refer to Section 4 for more information on hardware operational status.
What To Check
Chassis
configuration & status

CLI Command
show chassis
show chassis environment
show chassis power-supply

IOM or SF/CPM
configuration & status

show card
show card <A/B> detail
show card <slot-number> detail

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

54/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

MDA
configuration & status

show mda
show mda detail

port
configuration & status

show port
show port <slot/mda/port>
show
port
<slot/mda/port[.sonet-sdhindex]>
show port <port-id> detail
show port <port-id> ppp [detail]
show lag <lag-id>
show lag <lag-id> detail

Link Aggregation Group (LAG)

Display logical interfaces associated


with a port

show port <slot/mda/port> associations

What To Check
show statistics of a port

CLI Command
show port <slot/mda/port> count

show statistics of a LAG

show lag <lag-id> detail statistics

show counters of a SAP

show service id <service-id>


<port-id[:encap-val]> detail

clear counters of a port

clear
port
statistics

clear counters of a LAG

clear lag <lag-id> statistics

clear counters of a SAP

clear service statistics sap <portid[:encap-val]> counters

<

sap

slot/mda/port

>

A.1.3. How to show or clear statistics on a port or a LAG or a SAP

A.1.4. How to show or modify the operational status of a port


Troubleshooting note: Ports by default are administratively down.
If a port is correctly configured but not up, most likely the port is administratively down.
What To Check
To display the administrative
status of a port
To modify the administrative
status of a port

CLI Command
show port <slot/mda/port>
config port <slot/mda/port>
[no] shutdown

A.1.5. How to loop ports


Ethernet ports:
You can NOT loop Ethernet ports using CLI commands.
SONET/SDH ports:
You can use CLI command to loopback a SONET/SDH port.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

55/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

NOTE:
1) The SONET/SDH port must be in a shut down state to activate any type of loopback.
2) When you loop back a SONET/SDH port, make sure it is not line timing.
3) The loopback setting is never saved to the generated/saved configuration file.
Task
To activate a loopback on the
SONET/SDH port

CLI Command
config port <port-id> <enter>
config>port# sonet-sdh loopback {line|internal}
Description:
line Set the port into line loopback state.
internal Set the port into internal loopback state.

To disable the loopback on the


SONET/SDH port

config>port# sonet-sdh no loopback

TDM ports:
You can use CLI to put a specified TDM port or channel into a loopback mode.
NOTE:
1) The corresponding port or channel must be in a shutdown state in order for the loopback mode to be
enabled. The upper level port or channel or parallel channels should not be affected by the loopback mode.
2) When you loop back a port, make sure it is not line timeing.
3) The loopback setting is never saved to the generated/saved configuration file.
Task
To activate a loopback on a
DS3 port

CLI Command
config port <port-id> <enter>
config>port#tdm
ds3
{line|internal|remote}

To disable this specific


loopback
To activate a loopback on a
E3 port

config>port# tdm ds3 no loopback

To disable this specific


loopback
To activate a loopback on a
DS1 channel

config>port# tdm e3 no loopback

To disable this specific


loopback
To activate a loopback on a
E1 channel

config>port# tdm ds1 no loopback

To disable this specific


loopback

config>port# tdm e1 no loopback

config>port#tdm
{line|internal|remote}

config>port#tdm
{line|internal|remote}

loopback

e3

loopback

ds1

loopback

config>port#
tdm
{line|internal|remote}

e1

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

loopback

56/89

A.2.

OSPF Problems

This section provides information on how to troubleshoot an OSPF related problem.

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.2.1. Commands common to any OSPF troubleshooting


show commands used to check OSPF related configuration
The following commands are commonly used for checking OSPF related configuration:
show
show
show
show
show

router
router
router
router
router

ospf
ospf
ospf
ospf
ospf

area
interface
neighbor
status
database

View the OSPF related alarms/logs


To view the OSPF related alarms or log messages, use the command:
show log log-id 99 application ospf
Using Debug to troubleshoot an OSPF related problem
The debug router ospf command allows the user to troubleshoot an OSPF related issue in many
circumstances. The following are the choices of events that can be logged:
SR12# debug router ospf
- no ospf
- ospf
[no] area
[no] area-range

for an OSPF area


for an OSPF area

[no]
[no]

for an OSPF cspf


for an OSPF

[no]
[no]
[no]
[no]
[no]
[no]
[no]
[no]
[no]

- Enable/disable debugging
- Enable/disable debugging
range
cspf
- Enable/disable debugging
interface
- Enable/disable debugging
interface
leak
- Enable/disable debugging
lsdb
- Enable/disable debugging
statedatabase (LSDB)
misc
- Enable/disable debugging
OSPF events
neighbor
- Enable/disable debugging
neighbor
nssa-range
- Enable/disable debugging
packet
- Enable/disable debugging
rtm
- Enable/disable debugging
spf
- Enable/disable debugging
virtual-neighb* - Enable/disable debugging
virtual neighbor

for OSPF leaks


for an OSPF linkfor miscellaneous
for an OSPF
for
for
for
for
for

an NSSA range
OSPF packets
OSPF rtm
OSPF spf
an OSPF

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results.
Note that if the log destination is session, when the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

57/89

2) To stop the debug, use either of the following commands to stop the debug at different levels:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Command
debug router ospf no
packet
debug router no ospf
no debug

Explanation
Disable debugging for OSPF packets
Disable debugging for all OSPF messages
Disable debugging for all applications

3) The debug will stop if a router is rebooted for some reason.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

58/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.2.2. OSPF Adjacency Does Not Come Up


Symptom: Router OSPF doesnt come up.
The following table outlines the problems that might cause this symptom and describes suggested actions to
resolve the problems.
Possible Problem
1. Link/Interface Status

2. MTU Mismatch

Suggested Action
To verify if the port is up:
show port
To verify that interface has been assigned a port
show router interface <int-name> detail
or
config router interface <int-name>
config>router>if# info [detail]
To bind an interface to a physical port, use the command:
config router interface <int-name>
config>router>if# port-id[:encap-val]
Note: encap-val - 0
for null
- [0..4094] for dot1q
The MTU can be set at the port level or at the IP level. To view the
MTU settings, use the following commands:
show port displays MTU at the port level.
show router ospf interface <int-name> detail
displays the IP MTU.

Use the commands below to modify MTU setting if it is wrong.


To set the MTU at the port level:
config port <port-id> ethernet mtu <value>
To set the MTU at the IP level:
config router ospf area <area-id> interface
<int-name> mtu <value>
Enable debug for OSPF database descriptor packet
debug router ospf packet dbdescr

3. Mismatched Interface
Type

To display the interface type , use the command:


show router ospf interface <int-name> detail
Look at the IF Type under State category.

To modify the interface type, use the command:


config router ospf area <area-id> interface
<int-name> interface-type {broadcast|point-topoint}

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

59/89

4. Mismatched subnet
mask or IP address

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

5. Interface not
configured in OSPF

Check the router and its neighbors interface to see if the subnet mask
or IP address matches each other. Use the command:
show router interface
To verify if the interface has been configured in OSPF, use the
commands:
show router interface to display router interfaces
show router ospf interface to display router interfaces in
OSPF
To configure an interface in OSPF, use the command:
config router ospf area <area-id> interface
<int-name>

6. Router-id not unique


Note: If the router-id
has to be changed the
OSPF process has to be
shutdown/no shutdown
for the new route-id to
take effect

Make sure the router has a unique Router ID.


Normally a router uses its system interface as its Router ID. A router
ID can also be configured specifically. If neither the system interface
or router ID are implicitly specified, then the router ID is inherited
from the last four bytes of the MAC address.
To view the router-id, use the command:
show router ospf status
To view the system(loopback) interfaces, use the command:
show router interface system
To add system interface(loopback) to OSPF, use the command:
config router ospf area <area-id> interface
system

7. Neighbor is
configured for
authentication

If the routers OSPF neighbor is configured for authentication, the


router must be configured to match the authentication. To view the
authentication configuration of an interface, use commands:
config router ospf area <area-id> interface
<int-name>
config>router>ospf>area>if# info detail
To configure the authentication on the interface level, use commands:
config router ospf area <area-id> interface
<int-name> authentication-type
{password|message-digest}
config router ospf area <area-id> interface
<int-name> message-digest-key <key #> md5 <md5key>
The following example displays interface authentication configuration
command usage:
Example:
config>router# ospf
config>router>ospf$ area 0.0.0.40
config>router>ospf>area# interface to-274ferg
config>router>ospf>area>if# authentication-type
password
config>router>ospf>area>if# authentication-key
dilbert
config>router>ospf>area>if# no shutdown
config>router>ospf>area>if# exit

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

60/89

8. Incorrect area

To view the area of the interface, use the command:


show router ospf interface

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

To modify the area setting, and configure OSPF on an interface, use:


config router ospf area <area-id>
config>router>ospf>area# interface <int-name>
9. Mismatched
hello/dead interval
timers

To display the interval timers setting for an interface, use the


command:
show router ospf interface <int-name> detail
To modify the interval timers, use the command:
config router ospf area <area-id> interface
<int-name> {dead-interval|hello-interval}
<value>

A.2.3. Isolating Intermittent Reachability In An OSPF Environment


Symptom: Intermittent connectivity to a network
The following table outlines the problems that might cause this symptom and describes suggested actions to
resolve the problems.
Task
Check for network stability.

Command
show router ospf spf
This command will display spf calculation statistics. The total number of
spf runs should not continually increment when the output is refreshed in
a stable network. This command can be used to determine if network
instability is caused by networks external to OSPF or internal to OSPF.

Check the age of LSAs in


the database

Show router ospf database


Checking the age of an LSA is a good way of isolating the location of the
instability. A router that is experiencing link flapping will have to
generate new LSA for every time the link flaps. Go to the router that is
generating the new LSAs to further isolate the problem.

Note: In multi area environments it will be required to check the source of the summary LSAs. Once the
source of the summary LSAs has been determined run the above commands from the source of the
summary LSAs.

A.3.

ISIS Problems

This section provides information on how to troubleshoot an ISIS related problem.

A.3.1. Commands common to any ISIS troubleshooting


show commands used to check ISIS related configuration
The following commands are commonly used for checking OSPF related configuration:
show
show
show
show
show
show
show
show
show

router
router
router
router
router
router
router
router
router

isis
isis
isis
isis
isis
isis
isis
isis
isis

status
interface
adjacency
routes
database
statistics
spf
spf-log
summary-address

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

61/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

View the ISIS related alarms/logs


To view the ISIS related alarms or log messages, use the command:
show log log-id 99 application isis
Using Debug to troubleshoot an isis related problem
The debug router isis command allows the user to troubleshoot an ISIS related issue in many
circumstances. The following are the choices of events that can be logged:
SR12# debug router isis
- isis
- no isis
[no] adjacency

- Enable/disable debugging
adjacency
[no] cspf
- Enable/disable debugging
[no] graceful-resta* - Enable/disable debugging
graceful-restart
[no] interface
- Enable/disable debugging
interface
[no] leak
- Enable/disable debugging
[no] lsdb
- Enable/disable debugging
[no] misc
- Enable/disable debugging
[no] packet
- Enable/disable debugging
[no] rtm
- Enable/disable debugging
[no] spf
- Enable/disable debugging

for ISIS
for ISIS cspf
for ISIS
for ISIS
for
for
for
for
for
for

ISIS
ISIS
ISIS
ISIS
ISIS
ISIS

leaks
LSDB
misc
packet
RTM
SPF

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command
debug router isis no
packet
debug router no isis
no debug

Explanation
Disable debugging for ISIS packets
Disable debugging for all ISIS messages
Disable debugging for all applications

3) The debug will stop if a router is rebooted for some reason.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

62/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.3.2. ISIS Adjacency Does Not Come Up


Symptom: Router ISIS doesnt come up.
The following table outlines the problems that might cause this symptom and describes suggested actions to
resolve the problems.
Possible Problem
1. Link/Interface Status

Suggested Action
To verify if the port is up:
show port
To verify that interface has been assigned a port
show router interface <int-name> detail
or
config router interface <int-name>
config>router>if# info [detail]
To bind an interface to a physical port, use the command:
config router interface <int-name>
config>router>if# port-id[:encap-val]
Note: encap-val - 0
for null
- [0..4094] for dot1q
Use the commands below to modify MTU setting if it is wrong.
To set the MTU at the port level:
config port <port-id> ethernet mtu <value>
Note: An MTU mismatch will not stop the ISIS adjacency from
coming up but the MTU should match on point-to-point links.

2. Mismatched Interface
Type

To display the interface type , use the command:


show router isis interface <int-name> detail
Look at the IF Type under State category.

To modify the interface type, use the command:


config router isis interface <int-name>
interface-type {broadcast|point-to-point}
3. Mismatched subnet
mask or IP address

4. Interface not
configured in ISIS

Check the router and its neighbors interface to see if the subnet mask
or IP address matches each other. Use the command:
show router interface
debug router isis adjacency
To verify if the interface has been configured in ISIS, use the
commands:
show router interface to display router interfaces
show router isis interface to display router interfaces in
ISIS
To configure an interface in ISIS, use the command:
config router isis interface <int-name>

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

63/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

5. System-id not unique


Note: If the system-id
has to be changed the
ISIS process has to be
shutdown/no shutdown
for the new route-id to
take effect

Make sure the router has a unique System ID.


Normally a router uses its system interface as its System ID. A router
ID can also be configured specifically. If neither the system interface
or router ID are implicitly specified, then the router ID is inherited
from the last four bytes of the MAC address.
To view the system-id, use the command:
show router isis status
To view the system(loopback) interfaces, use the command:
show router interface system
To add system interface(loopback) to isis, use the command:
config router isis interface system

6. Neighbor is
configured for
authentication

If the routers ISIS neighbor is configured for authentication, the router


must be configured to match the authentication. To view the
authentication configuration of an interface, use commands:
config router isis interface <int-name>
config>router>isis>if# info detail
To configure the authentication on the interface level, use commands:
config router isis interface <int-name> helloauthentication-key <authentication key|hashkey> [hash|hash2]
config router ospf interface <int-name>
authentication-type {password|message-digest}

7. Level Capability
mismatch
If the adjacent routers
are in the same area the
interfaces and router
must be level 1 capable.
If the adjacent routers
are in different areas the
interface and router
must be level 2 capable
8. Mismatched
hello/dead interval
timers

note: Authentication can configured at the each level within isis


To view the level-capability of the interface, use the command:
show router isis interface
To modify the level-capabilty setting of the router, use:
config>router>isis# level-capability {1|2}
To modify the level-capability setting of the interface, use
config router isis interface <int-name> levelcapability {level-1|level-2|level-1/2}
To display the interval timers setting for an interface, use the
command:
show router ospf interface <int-name> detail
To modify the interval timers, use the command:
config router ospf area <area-id> interface
<int-name> {dead-interval|hello-interval}
<value>

A.3.3. Isolating Intermittent Reachability In An ISIS Environment


Symptom: Intermittent connectivity to a network
The following table outlines the problems that might cause this symptom and describes suggested actions to
resolve the problems.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

64/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Task
Check for network stability.

Command
show router isis statistics
This command will display ISIS statistics. The total number of spf runs
should not continually increment when the output is refreshed in a stable
network.

Check the age of LSPs in


the database

Show router ISIS database


Checking the age of an LSP is a good way of isolating the location of the
instability. A newly generated LSP will have an age of 1200 and
decrement to 0. A router that is experiencing link flapping will have to
generate new LSP for every time the link flaps. Go to the router that is
generating the new LSP to further isolate the problem.

A.4.

BGP Problems

This section provides information on how to troubleshoot a BGP related problem. Each sub-section
describes a possible problem scenario. Examples of command usage are provided in the sub-sections.

A.4.1. Commands common to any BGP troubleshooting


show commands used to check BGP related configuration
The following commands are commonly used for checking BGP related configuration:
show
show
show
show
show
Show

router bgp summary


router bgp neighbor
router bgp neighbor <ip-address> received-routes
router bgp neighbor <ip-address> advertised-routes
router bgp neighbor <ip-address> detail
route bgp routes <ip-address> hunt

View the BGP related alarms/logs


To view the BGP related alarms or logs, use the command:
show log log-id 99 application bgp
Using Debug to troubleshoot a BGP related problem
The debug router bgp command allows the user to troubleshoot a BGP related issue in many
circumstances. The following are the choices of events can be logged:
SR12# debug router bgp
- bgp
- no bgp
[no] events
[no] keepalive
[no] notification
[no] open

- Enable/disable
events
- Enable/disable
Keepalive
messages
- Enable/disable
Notification
messages
- Enable/disable

debugging for all BGP


debugging for all BGP
debugging for all BGP
debugging for all BGP Open

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

65/89

[no] packets

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

[no] route-refresh
[no] rtm

[no] socket
[no] timers
[no] update

messages
- Enable/disable debugging
packets
- Enable/disable debugging
refresh
- Enable/disable debugging
removal and modification
the system Route Table
Manager
- Enable/disable debugging
sockets
- Enable/disable debugging
timers
- Enable/disable debugging
Update messages

for all BGP


for BGP routefor addition
of BGP routes to
for all BGP
for all BGP
for all BGP

Important Notes:
1) Before enabling the debug, the user must make sure a log is created to view the debug result.
2) To stop the debug, use either of the following commands to stop the debug at different level:
Command
Explanation
debug router bgp no
Disable debugging for all BGP Keepalive messages
keepalive
debug router no bgp
Disable debugging for all BGP messages
no debug
Disable debugging for all applications
3) The debug will stop if a router is rebooted for some reason.

A.4.2. BGP peer session not established


Symptom: Router does not establish a session with its peer.
Possible Problem
1. MTU configuration
mismatch

Suggested Action
To verify if the port MTU size is configured correctly, use command: show
port <port-id>

Use config port <port-id> command to modify the MTU size if it is


improperly configured. For example, to modify an Ethernet port (1/1/1) MTU
size to be 1518 bytes, use command:
config port 1/1/1 ethernet mtu 1518
2. Local or Peer AS
configured improperly
note: If the local-as has
to be changed the BGP
process has to be
shutdown/no shutdown
for the AS to take effect.

To verify if the local or Peer AS is configured correctly, use command:


show router bgp neighbor <neighbor address>
Use config router bgp command to modify AS number if it is the
problem.
For example: to modify the local AS number, use command:
config router bgp local-as <as-number>
To modify the (group level) AS number for the remote peer, use command:
config router bgp group <name> peer-as <as-number>

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

66/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

3. BGP neighbor address


mis-configured

To verify if a BGP neighbor address is configured correctly, use command:


show router bgp neighbor
Use config router bgp group <name> neighbor <ipaddress> command to modify the neighbor address if it is incorrect.

4. EBGP neighbor is not


directly connected

By default the ttl for BGP packets for EBGP sessions is 1. Modify the ttl for
the BGP neighbor with the following command
config router bgp group <name> neighbor <address>
multi-hop <ttl value>

A.4.3. BGP load balancing issue


Route Selection Criteria
When the BGP speaker receives updates from multiple ASs that describe different paths to the same
destination, it must choose the single best path for reaching that destination. Once chosen, BGP propagates
the best path to its neighbors. The process of selecting the best path is as below.
For each prefix in the routing table, the routing protocol selects the best path. Then, the best path is
compared to the next path in list until all paths in the list are exhausted. The following parameters are used
to determine the best path:
1. Routes are not considered if they are unreachable.
2. An RTM's preference is lowered as well as the hierarchy of routes from a different protocol. The lower
the preference is, the higher the chance of the route being the active route.
3. Routes with higher local preference have preference.
4. Routes with the shorter AS path have preference.
5. Routes with the lower origin have preference.
IGP = 0
EGP = 1
INCOMPLETE = 2
6. Routes with the lowest MED metric have preference.
7. Routes learned by an EBGP peer rather than those learned from an IBGP peer are preferred.
8. Routes with the lowest IGP cost to the next-hop path attribute are preferred.
9. Routes with the lowest BGP-ID are preferred.
10. Routes with shortest cluster list are preferred.
11. Routes with lowest IP address are preferred.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

67/89

Commands to adjust BGP attributes for load balancing

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Attributes
Local Preference Attribute

CLI Commands
Local preference can be set at the global level:
config>router>bgp local-preference [0..4294967295]
or group level:
config>router>bgp>group name local-preference
[0..4294967295]
or neighbor level.
config>router>bgp>group name>neighbor ip-addr
local-preference [0..4294967295]
Note: This command enables setting the BGP local-preference attribute in
incoming routes if not specified and configures the default value for the
attribute.
This value is used if the BGP route arrives from a BGP peer without the
local-preference integer set.
The specified value can be overridden by any value set via a route policy.
This configuration parameter can be set at three levels: global level
(applies to all peers), group level (applies to all peers in peer-group) or
neighbor level (only applies to specified peer). The most specific value is
used.

as-path-ignore

MED Attribute

config router bgp as-path-ignore


This command determines whether the AS path is used to determine the
best BGP route.
If this option is present, the AS paths of incoming routes are not used in
the route selection process.
MED value can be set at the global level:
config>router>bgp med-out {number | igp-cost}
or group level:
config>router>bgp>group name med-out {number |
igp-cost}
or neighbor level:
config>router>bgp>group name>neighbor ip-addr medout {number | igp-cost}
number The MED path attribute value expressed as a decimal integer.
Values 0 - 4294967295 (2^32 - 1)
igp-cost The MED is set to the IGP cost of the given IP prefix.
This command enables advertising the Multi-Exit Discriminator (MED)
and assigns the value used for the path attribute for the MED advertised to
BGP peers if the MED is not already set.
The specified value can be overridden by any value set via a route policy.
This configuration parameter can be set at three levels: global level
(applies to all peers), group level (applies to all peers in peer-group) or
neighbor level (only applies to specified peer). The most specific value is
used.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

68/89

always-compare-med

config router bgp always-compare-med {zero |


infinity}

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Note: This command specifies how the Multi-Exit Discriminator (MED)


path attribute is used in the BGP route selection process. The MED
attribute is always used in the route selection process regardless of the
peer AS that advertised the route. This parameter determines what MED
value is inserted in the RIB-IN.
If this parameter is not configured, the router only compares MEDs for
routes from external neighbors that are in the same AS.

A.5.

Prefix-list (Access-list) in the Route Policy

This section describes with an example how prefix lists (aka. access lists) are configured and used in route
policies. Show commands are also provided to troubleshooting a route policy related issue.
Overview of the route policy
Route policies allow you to configure routing according to specifically defined policies. You can create
policies and entries to allow or deny paths based on various parameters such as destination address,
protocol, packet size, and community list.
Policies can be as simple or complex as required. A simple policy can block routes for a specific location or
IP address. More complex policies can be configured using numerous policy statement entries containing
matching conditions to specify whether to accept or reject the route, control how a series of policies are
evaluated, and manipulate the characteristics associated with a route.
There are no default route policies. Each policy must be created explicitly and applied to a policy, a routing
protocol, or to the forwarding table. Policy parameters are modifiable.
Process of provisioning a basic router policy
The following diagram shows the process of how to provision a basic route policy.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

69/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

The following example is focused on how prefix lists are configured and used in a route policy, and how
this route policy applied to BGP. Other parameters such as AS-path, community list and damping
parameters are disregarded.
1) create/edit route policy
SR12>config>router>policy-options#
SR12>config>router>policy-options# begin
2) create/edit prefix lists
SR12>config>router>policy-options# prefix-list
SR12>config>router>policy-options>prefix-list#
. . .
SR12>config>router>policy-options>prefix-list#
SR12>config>router>policy-options# prefix-list
SR12>config>router>policy-options>prefix-list$
SR12>config>router>policy-options>prefix-list$
. . .
SR12>config>router>policy-options>prefix-list$

Deny-routes
prefix 0.0.0.0/8 longer
exit
"permit-routes"
prefix 10.10.1.0/30 exact
prefix 10.10.2.0/24
exit

3) create/edit route policies


SR12>config>router>policy-options# policy-statement "Service ProviderIN"
SR12>config>router>policy-options>policy-statement$ entry 1
SR12>config>router>policy-options>policy-statement>entry$ from prefixlist "D
eny-routes"
SR12>config>router>policy-options>policy-statement>entry# exit

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

70/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

SR12>config>router>policy-options>policy-statement>entry# action reject


SR12>config>router>policy-options>policy-statement>entry# exit
SR12>config>router>policy-options>policy-statement# default-action
accept
SR12>config>router>policy-options>policy-statement>default-action# exit
SR12>config>router>policy-options>policy-statement# exit
SR12>config>router>policy-options# policy-statement "Service ProviderOUT"
SR12>config>router>policy-options>policy-statement$ entry 20
SR12>config>router>policy-options>policy-statement>entry$ from
SR12>config>router>policy-options>policy-statement>entry>from$ prefixlist "p
ermit-routes"
SR12>config>router>policy-options>policy-statement>entry>from$ exit
SR12>config>router>policy-options>policy-statement>entry# action accept
SR12>config>router>policy-options>policy-statement>entry>action# exit
SR12>config>router>policy-options>policy-statement>entry# exit
SR12>config>router>policy-options>policy-statement# default-action
reject
SR12>config>router>policy-options>policy-statement# exit
4) save route policies
SR12>config>router>policy-options# commit
SR12>config>router>policy-options# exit
SR12#
5) Apply route policies created above as the import & export policy for BGP
SR12# config router
SR12>config>router#
SR12>config>router#
SR12>config>router#
SR12>config>router#
SR12>config>router#
SR12#

autonomous-system <as-number>
bgp
import "Service Provider-IN"
export "Service Provider-OUT"
exit

Notes of begin and commit in the policy configuration:


begin
Required in order to enter the mode to create or edit route policies.
The begin command puts the node (not just the session) in a route policy edit mode.
Once begin is entered, until a commit is executed, subsequent users executing the begin
command will be warned that a policy configuration is in progress.
commit

This command is required to save changes made to a route policy.

A commit will save all policy configuration in progress on a node, this include all
session that have entered begin without having exited with a commit regardless of the state of
the route-policy under configuration.

A commit terminates edit mode for all users that are currently in edit mode.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

71/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Troubleshooting Route Policies


To verify how the policy is configured, use command: show router policy
To verify how prefix list is configured in the policy, use command: show router policy prefixlist <name>
The following are example outputs of these commands:
SR12# show router policy
========================================================================
=======
Route Policies
========================================================================
=======
Policy
Description
-----------------------------------------------------------------------------Service Provider-IN
Service Provider-OUT
-----------------------------------------------------------------------------Policies : 2
========================================================================
=======
SR12#
SR12# show router policy prefix-list
==================================
Prefix Lists
==================================
Prefix List Name
---------------------------------Deny-routes
permit-routes
==================================
SR12# show router policy prefix-list Deny-routes
prefix 0.0.0.0/8 longer
. . .
SR12# show router policy prefix-list permit-routes
prefix 10.10.1.0/30 exact
prefix 10.10.2.0/24 exact
. . .
SR12#

A.6.

Black holing Problems

When an AS provides transit service to other ASs and if there are non-BGP transit routers in the AS, transit
traffic might be dropped if the intermediate non-BGP routers havent learned the routes for that traffic via
IGP. In this case, the transit traffic is black-holed.
By default, Alcatel-Lucent 7750SR will not re-advertise learned iBGP routes unless there is an entry in its
routing table learned via an IGP or a static route.
If you believe that you are black holing a route, you can:
1. Check if the route is in the RIB. Use command show router bgp neighbor <ip addr>
{advertised-routes|received-routes} and show route route-table

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

72/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

2. Check if the route is in the FIB. Use command show router fib <slot-number> [<ipprefix/mask]> [longer]]
3. Verify the routing policies for inaccuracies to ensure that packets are not getting filtered.
- To check what policy is applied in IGP (ex. OSPF), use commands:
config router ospf
config>router>ospf# info detail
- To check if the policy is configured correctly, use command:
show router policy <policy-name>

A.7.

LDP not established

This section describes how to troubleshoot problems establishing a LDP.


First make sure the routers OSPF adjacencies are up and running. If there is anything wrong with OSPF,
refer to Section 6.2 for troubleshooting OSPF problems.
If it is not OSPF issue, use the following methods to identify problems in LDP.
View the log messages about LDP
show log log-id 99 subject LDP
Using Debug to troubleshoot a LDP related problem
The debug router ldp command allows the user to troubleshoot a LDP related issue. The following
are the debugging choices.
SR12# debug router ldp
- ldp
- no ldp
[no] interface
[no] peer

+ Enable/disable and configure debugging for


an LDP
interface
+ Enable/disable and configure debugging for
an LDP peer

SR12>debug>router>ldp# interface <interface-name>


[no] event
[no] packet

+ Configure debugging for specific LDP


events
+ Enable/disable debugging for specific LDP
packets

SR12>debug>router>ldp# peer <ip-address>


[no] event
[no] packet

+ Configure debugging for specific LDP


events
+ Enable/disable debugging for specific LDP
packets

Important Notes:
1) Before enabling the debug, the user must make sure a log is created to view the debug result. 2) To
stop the debug, use either of the following commands to stop the debug at different level (more choices
can be found by clicking ? at any level of the CLI syntax):

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

73/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Command
Explanation
debug router ldp interface <int-name>
Disables debugging for specific LDP packets
no packet
debug router ldp no interface <intDisables debugging for LDP interface
name>
no debug
Disables debugging for all applications
3) The debug will stop if a router is rebooted for some reason.
Using show commands to check LDP information
Command
show router ldp bindings
show router ldp bindings active
show
show
show
show
show
show

A.8.

router
router
router
router
router
router

ldp
ldp
ldp
ldp
ldp
ldp

discovery
interface
parameters
peer
session
status

Explanation
To display LDP bindings information
To display LDP active bindings. An active binding must exist for a
prefix in order for an LSP to be active
To display LDP discovery information
To display LDP interface information
To display LDP configured and operation parameters
To display LDP targeted peer information
To display LDP session information
To display LDP operational information

RSVP Signaled LSP Problems

This section provides information on how to troubleshoot an RSVP related problem.

A.8.1. Commands common to any RSVP troubleshooting


show commands used to check RSVP related configuration
The following commands are commonly used for checking RSVP related configuration:
show
show
show
show
show
show
show

router
router
router
router
router
router
router

rsvp
rsvp
rsvp
rsvp
mpls
mpls
mpls

status
interface
session
statistics
lsp
lsp path
lsp active path

View the RSVP related alarms/logs


To view the ISIS related alarms or log messages, use the command:
show log log-id 99 application rsvp
show log log-id 99 application mpls
Using Debug to troubleshoot an isis related problem
The debug router rsvp command allows the user to troubleshoot an rsvp related issue in many
circumstances. The following are the choices of events that can be logged:
SR12# debug router rsvp
- no rsvp
- rsvp [lsp <lsp-name>] [sender <sender-address>] [endpoint
<endpoint-address>] [tunnel-id <tunnel-id>] [lsp-id <lsp-id>]
[interface
<ip-int-name>]

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

74/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

<lsp-name>
<sender-address>
<endpoint-address>
<tunnel-id>
<lsp-id>
<ip-int-name>

:
:
:
:
:
:

[no] event

+ Enable/disable debugging for specific RSVP


events
+ Enable/disable debugging for specific RSVP
packets

[no] packet

[80 chars max]


a.b.c.d
a.b.c.d
[0..4294967295]
[1..65535]
[32 chars max]

SR12# debug router rsvp packet


- no packet
- packet
[no] path

- Enable/disable
packets
[no] patherr
- Enable/disable
packets
[no] pathtear
- Enable/disable
packets
[no] resv
- Enable/disable
packets
[no] resverr
- Enable/disable
packets
[no] resvtear
- Enable/disable
packets
SR12# debug router rsvp event
- event
- no event
[no] bundle
[no] error
[no] hello
[no]
[no]
[no]
[no]

psb
rre
rsb
tcsb

- Enable/disable
messages
- Enable/disable
- Enable/disable
messages
- Enable/disable
- Enable/disable
- Enable/disable
- Enable/disable

debugging for RSVP Path


debugging for RSVP PathErr
debugging for RSVP PathTear
debugging for RSVP Resv
debugging for RSVP ResvErr
debugging for RSVP ResvTear

debugging for RSVP bundle


debugging for RSVP errors
debugging for RSVP Hello
debugging
debugging
debugging
debugging

for
for
for
for

RSVP
RSVP
RSVP
RSVP

psb
rre
rsb
tcsb

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

75/89

2) To stop the debug, use either of the following commands to stop the debug at different levels:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Command
debug router rsvp no
packet
debug router no rsvp
no debug

Explanation
Disable debugging for rsvp packets
Disable debugging for all rsvp messages
Disable debugging for all applications

3) The debug will stop if a router is rebooted for some reason.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

76/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.8.2. RSVP MPLS Tools Perform Commands


Command
tools perform router mpls
cspf

Explanation
Syntax :tools perform router mpls cspf
to <ip-addr> [from <ip-addr>] [bandwidth <bandwidth>]
[include-bitmap<bitmap>] [exclude-bitmap <bitmap>] [hoplimit <limit>] [exclude-address <excl-addr> [<excladdr>...(upto 8 max)]]
<ip-addr>
: a.b.c.d
<bandwidth>
: [1..100000] in Mbps
<bitmap>
: [0..4294967295] - accepted in decimal,
hex(0x) or binary(0b)
<limit>
: [1..255]
<excl-addr>
: a.b.c.d (system or egress ip-address)
Context : tools>perform>
Description: This command does a manual CSPF calculation
based on the constraints provided in the command string. This
tool is very useful for troubleshooting LSPs that are not in the
up state or are not using the optimal path. This command can
only be used if Traffic Engineering is enabled in the IGP as It
relies on the traffic engineering database. The output of the
command is strictly informational and has no impact on any
LSPs. If the CSPF calculation fails it indicates that there is no
path in the TE database to reach the endpoint. The Traffic
Engineering database will not be fully populated if TE is not
enabled on all nodes in the network and if interfaces are not
MPLS or RSVP enabled. This tools command can be used
similar to ping to isolate the source of the problem. For
example, if the CSPF calculation fails to the desired endpoint
the next node in the path can be specified as the endpoint. This
can be continued until the failing node is isolated.
Parameters:
To Address- Endpoint IP address to run CSPF calculation to.
From Address- Starting point IP address for CSPF calculation.
This can be used to determine what path another node in the
network will return based on the constraints in the command
string
Bandwidth- 1..100000 in Mbps. This is used to add bandwidth
as a constraint when performing the CSPF calculation.
Include-bitmap- 0..4294967295 accepted in decimal, hex(0x) or
binary (0b). Only interfaces belonging to the specified
administrative group can be used as part of the CSPF calculation
Exlude-bitmap-0..4294967295 accepted in decimal, hex(0x) or
binary (0b). Only interfaces NOT belonging to the specified
administrative group can be used as part of the CSPF calculation
Hop-Limit-[1..255]. This is used to add a hop limit as a
constraint when performing the CSPF calculation
Exclude-address- IP address. Up to 8 addresses can be
specified. The address specified must be an egress interface
address for the node. By default the CSPF calculation will
follow the IGP to endpoint for the calculation. By excluding
egress addresses alternate paths can be calculated.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

77/89

Tools perform router mpls


resignal

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Syntax: tools perform router mpls resignal lsp <lsp-name> path


<path-name>
<lsp-name>
: [32 chars max]
<path-name>
: [32 chars max]
Context: tools>perform>
Description: This command can be used to perform a manual
make before break resignal of the specified LSP.

A.8.3. RSVP Signaled LSP Does Not Come Up


Symptom: RSVP Signaled LSP doesnt come up.
The following table outlines possible error codes in the output of the show router mpls lsp <lsp-name>
path <path-name> detail command and provides possible causes and suggested actions to resolve the
problems.
Note: When you are reading the Failure Code of a LSP, wait for Retry period because the Failure Code
could change for the next retry period since for every retry period we attempt to compute a path again if the
last attempt to setup a LSP failed.
Error Code
NoError

Suggested Action
Verify that the following are all true.
mpls no shut
rsvp no shut
lsp no shut
lsp path no shut
lsp has a destination address assigned to it.
lsp has primary/secondary path assigned to it
For a non-cspf LSP, RTM is not empty.
In addition, type cli command show router rsvp session status down on
all the LSRs along the LSP path to check if a PATH message reaches all the
LSRs or not.

noRouteToDestination

For a non cspf empty path LSP with Dest Addr not valid/no valid route to
destination.
Verify that MPLS is enabled on all the LSRs in the network.
Verify that the to address is in the RTM. (Reachable by IGP)
For a cspf LSP with invalid strict/loose path.
Verify that all the routes/hops in the strict/loose path are valid and
are in correct sequence according to the network topology.
If a cspf LSP is trying to reserve a BW that none of the end-to-end paths in
the network can fulfill.
Check if the requested BW is too big for the network to handle or
the resources has already being used by other LSPs. Use tools
perform router mpls cspf to verify that there is a cspf path that
can fulfill the BW constraint.
If a cspf LSP cant find an end-to-end path in the network that matches the
color constraint.
Check if link color is assigned correctly for all the routes in the
network. Use tools perform router mpls cspf to verify that there
is a cspf path that can fulfill the color constraint.
For a cspf LSP, downstream routers have ospf/isis te off or opaque LSA
disabled.
Enable te /opaque LSA on all the LSRs that the LSP traverses.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

78/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

noCspfRouteOwner

For a cspf LSP with Dest Addr not valid/ no valid route to destination.
Verify that MPLS is enabled on all the LSRs in the network.
Verify that the Dest Addr is in the IGP database. (Reachable by
IGP)

badNode

For a non cspf LSP with invalid strict/loose path


Verify that all the routes/hops in the strict/loose path are valid and
are in correct sequence according to the network topology. Go to
the node that is being reported by this failure code. Verify
configuration on the node.
For a cspf LSP
The node that is reported will always be itself. This is because
CSPF was unable to return a path to meet the constraints of the
LSP.
Verify TE database. Use the tools perform router mpls cspf to
command to further isolate the cause of the problem
A non cspf frr LSP with empty path.
A LSP with fast reroute enabled needs to have cspf turned on.
However, if the path is a strict hop path all the way to the egress
node, then turning on cspf is not a requirement
Routing Loop is detected when the LSP is setting up. The headend receives
a path message or the LSP it is trying to setup
There is a problem in the strict/loose hop path definition. Double
check it and make sure that a routing loop is not generated by the
strict/loose hop definition itself.
You may be required to add more strict hops to avoid the routing
loop.

looseHopsInFRRLsp

routingLoop

admissionControlError

For a cspf LSP with BW requirement that finds a te path to a destination


from IGP and is trying to send a PATH message to setup a LSP, if there is a
BW change in a downstream router and the opaque LSA hasnt arrived at
the ingress node yet.
In this case, shutdown the LSP and wait for 30s for Routing DB to
converge. Then do a no shut of the LSP, if there is still a path in
the network that can fulfill the BW requirement of the LSP, the
LSP will be setup properly.
For a non cspf LSP with BW request that is trying to setup to the egress
LSR, none of the paths in the network can fulfill it.
In this case, double check if the request BW size is valid according
to your network planning tool. BW resources might have already
been used by other LSPs in the network.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

79/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.9.

Service Distribution Path (SDP) Problems

This section provides information on how to troubleshoot SDP related problems.

A.9.1. Commands common to any SDP troubleshooting


show commands used to check SDP related configuration
The following commands are commonly used for checking SDp related configuration:
show service sdp <sdp-id>
View the SDP related alarms/logs
To view the SDP related alarms or log messages, use the command:
show log log-id 99 application svcmgr
Using Debug to troubleshoot an isis related problem
The debug service sdp command allows the user to troubleshoot an sdp related issue. The following
are the choices of events that can be logged:
SR12# ebug service sdp
- no sdp <sdp-id>
- sdp <sdp-id>
<sdp-id>

: [1..17407]

[no] event-type

- Enable/disable a particular debugging


event type
SR12# debug service sdp 1 event-type
- event-type
{config-change|oper-status-change|keep-alive-tx|keep-aliverx|keep-alive-timeout}
- no event-type
{config-change|oper-status-change|keep-alive-tx|keep-aliverx|keep-alive-timeout}

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command
debug service no sdp
no debug

Explanation
Disable debugging for sdp events
Disable debugging for all applications

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

80/89

3) The debug will stop if a router is rebooted for some reason.

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.9.2. SDP Does Not Come Up


Symptom: SDP is in the down state
The following table outlines possible flag codes in the output of the show service sdp <sdp-id> detail
command and provides possible causes and suggested actions to resolve the problems.
SDPs have dependencies on the following
IGP
Transport Tunnel (RSVP, LDP or GRE)
LDP for Targeted LDP (tLDP) signaling
Flags
SignalingSessDown

Suggested Action
Verify that the following are all true.
LDP is no shut on both nodes
show router ldp status
Endpoint address is in the routing table
show router route-table <prefix>
There is an SDP configured in both directions
show service sdp far-end <ip-address>
show service sdp-using <ip-address>

TransportTunnDown

Verify the following is True


The status of the LSP the SDP is bound to for SDPs with MPLS
delivery
show router mpls lsp <lsp-name>
There is an active LDP binding for the far-end address for SDPs
with LDP Delivery
show router ldp binding active prefix <prefix>
There is a route in the routing table for the far-end address for all
encapsulation types
show router route-table

KAFailure

If message-length is configured for SDP keep Alive verify the SDP MTU
with the following oam command:
oam sdp-mtu
- sdp-mtu <orig-sdp-id> size-inc <start-octets> <end-octets> [step
<step-size>] [timeout <timeout>] [interval <interval>]
<orig-sdp-id>
: [1..17407]
<octets>
: start-octets [40..9198] end-octets [40..9198]
<step-size>
: [1..512]
<timeout>
: [1..10] seconds
<interval>
: [1..10] seconds

A.10. Layer 2 Service Connectivity Problem


This section provides information on how to troubleshoot Layer 2 Service connectivity problems. This
section will be limited to troubleshooting Service and Service Components (SAPs and SDPs) operational
status.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

81/89

A.10.1. Commands common to Layer 2 Service troubleshooting

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

show commands used to check Service related configuration


The following commands are commonly used for checking service related configuration:
show
show
show
show
show
show

service
service
service
service
service
service

id
id
id
id
id
id

<id>
<id>
<id>
<id>
<id>
<id>

base
sdp <sdp-id>
sap <sap-id>
fdb (VPLS only)
stp (VPLS only)
all

View the Service related alarms/logs


To view the service related alarms or log messages, use the command:
show log log-id 99 application svcmgr
Using Debug to troubleshoot an isis related problem
The debug service sdp command allows the user to troubleshoot an sdp related issue. The following
are the choices of events that can be logged:
SR12# ebug service id 1
- id <service-id>
- no id <service-id>
<service-id>

: [1..2147483647]

[no] dhcp

+ Enable/disable and configure DHCP


debugging
- Enable/disable a particular debugging
event type
+ Enable/disable and configure debugging for
a particular SAP
+ Enable/disable and configure debugging for
a particular SDP
+ Enable/disable and configure debugging for
STP

[no] event-type
[no] sap
[no] sdp
[no] stp

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.
For example, log 3 is created to view the debug result:
SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

82/89

2) To stop the debug, use either of the following commands to stop the debug at different levels:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

Command
debug service no service
no debug

Explanation
Disable debugging for service events
Disable debugging for all applications

3) The debug will stop if a router is rebooted for some reason.

A.10.2. Layer 2 Service Troubleshooting Steps


1) View the basic service information with show service id <id> base. From the output of this
command the administrative and operational status of the service and service components (SAPs
and SDPs) can be determined.
2) For any service component that is in the down state view the component specific information with
the following commands.
a. show service id <id> sdp <sdp-id>
b. show service id <id> sap <sap-id>
3) Use the tables in the following sections to determine troubleshooting steps to follow based on the
flag codes in the outputs of the commands used in step 2. For any SDP binding that is down go to
the Service SDP Binding Does Not Come Up Section. For any SAP binding that is down go to
the Service SAP Binding Does Not Come UP section.

A.10.3. Service SDP Binding Does Not Come Up


Symptom: Service SDP binding is in the down state
The following table outlines possible flag codes in the output of the show service <id> sdp <sdp-id>
detail command and provides possible causes and suggested actions to resolve the problems.
SDP bindings have dependencies on the following
IGP
Transport Tunnel (RSVP, LDP or GRE)
LDP for Targeted LDP (tLDP) signaling. An SDP must be configured in both directions for the
SDP to be operational
Service operational status
A SDP binding must be configured in both directions with the same vc-id.
MTU
Flags
ServiceAdminDown

NoIngVCLabel
NoEgrVCLabel
(if both flags are present)

SdpOperDown

Suggested Action
Verify that the following are all true.
The service is administratively enabled
show service <id> base
Verify the following is True
The status of the SDP is operational. If not go to the Service
Distribution Path (SDP) Problems Section
show service sdp <sdp-id>
This indicates the SDP is down.
Go to the Service Distribution Path (SDP) Problems Section

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

83/89

SdpBindAdminDown

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

NoEgrVCLabel
(if this is the only flag
present)

PathMTUTooSmal

ServiceMTUMismatch

Verify the following is True


The status of the sdp binding within the service is administratively
enabled
show service id <id> sdp <sdp-id>
Verify the following is True
A sdp-binding is configured and operational on the far-end
The sdp-binging on the far-end is configured with the same vc-id
on the far-end
The Service is administratively enabled on the far-end
show service id <id> base
This indicates that the vc-mtu is larger than the actual SDP MTU.
Use the show service sdp <sdp-id> command to determine the
SDP MTU
Use the show service <id> base command to determine the
Service MTU.
The service MTU must be equal to or less then the SDP MTU.
This indicates that the services on both end points are configured with
different MTUs.
Use the show router ldp bindings service <serviceid> command. The LMTU and RMTU must match. These MTU values
are the service MTU minus 14 bytes.

A.10.4. Service SAP Binding Does Not Come Up


Symptom: Service SAP binding is in the down state
The following table outlines possible flag codes in the output of the show service <id> sap <sap-id>
detail command and provides possible causes and suggested actions to resolve the problems.
SAP bindings have dependencies on the following
Service operational status
Port operational status
MTU
Flags
PortOperDown

Suggested Action
This indicates that the port for the logical SAP is in the down state.
Verify the port status with the show port <port-id> command

PortMTUTooSmall

This indicates that the port MTU is smaller then the Service MTU. The
SAP MTU must be equal to or greater then the Service MTU
Check the port MTU the show port <port-id> command.
Adjust the port MTU.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

84/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.11. Layer 3 VPRN Problem


This section provides information on how to troubleshoot VPRN Services. This section will be limited to
troubleshooting VPRN Operational status and the importing and exporting of routes into and out of the
VRF.

A.11.1. Commands common to VPRN Service troubleshooting


show commands used to check VPRN related configuration
The following commands are commonly used for checking VPRN service related configuration:
show service id <id> base
show service id <id> sdp <sdp-id>
show service id <id> sap <sap-id>
show service id <id> interface
show service id <id> arp
Show router <service-id> route-table
Note: All router commands can be used to view the routing information specific to
the VPRN. Use show router <service-id> ? for full a list of available commands.
View the Service related alarms/logs
To view the service related alarms or log messages, use the command:
show log log-id 99 application svcmgr
Using Debug to troubleshoot an isis related problem
The debug router <service-id> command allows the user to troubleshoot a VPRN related issue.
The following are the choices of events that can be logged:
SR12# ebug router <service-id>
<service-id>
[no] bgp
[no] igmp
[no] ip
[no] mtrace
[no] ospf
[no] pim
[no] rip
[no] vrrp

: [1..2147483647]

+ Enable/disable and configure debugging for


BGP
+ Enable/disable and configure debugging for
IGMP
+ Enable/disable and configure debugging for
IP
+ Enable/disable and configure debugging for
mtrace
+ Enable/disable and configure debugging for
OSPF
+ Enable/disable and configure debugging for
PIM
+ Enable/disable and configure debugging for
RIP
+ Enable/disable and configure debugging for
VRRP

Important Notes:
1) Before enabling debug, the user must make sure a log is created to view the debug result. The
following is an example log created to view debug results. Note that if the log destination is session, when
the session is closed, the log (log-id) will not be saved.

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

85/89

For example, log 3 is created to view the debug result:

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

SR12>config>log>log-id 3
SR12>config>log>log-id$ from debug-trace
SR12>config>log>log-id$ to session
SR12>config>log>log-id$ no exit
2) To stop the debug, use either of the following commands to stop the debug at different levels:
Command
no debug

Explanation
Disable debugging for all applications

3) The debug will stop if a router is rebooted for some reason.

A.11.2. Layer 3 VPRN Component Requirements


Component
PE Router

Explanation
A PE router is on the Service Provider Edge. Each CE device is
attached, via some sort of attachment circuit, toone or more
Provider Edge (PE) routers.

P Router

P Routers are Routers in the SP's network which do not attach to


CE devices. P routers have no knowledge of the VPN routes
unless the P router is a Route Reflector server for PE routers.

MP-BPG

Multi Protocol BGP must be enabled in order to transport routes


with extended communities between PEs. MP BGP is also used
to transport the inner label for the VPRN service.
A Route Distinguisher (RD) allows for common IP address
prefixes to be present in different routing instances. An RD can
not be shared by different services within a carrier domain.
An RD must be configured within a VPRN service for the
VPRN to be operational. If a RD is not configured the VRF will
not be activated, all layer 3 interfaces within the VPRN will be
in the down state and no routes will be populated in the VRF.

Route-Distinguisher

Route-Target

Routes are distributed using route targets.


A route can only have one Route Distinguisher but can have
multiple Route Targets.
A Route Target Attribute can be thought of as identifying a
collection of VRFs.
Two flavors, Export and Import Targets

Export configures the Route Target to be added to


advertised routes to remote PEs

Import configures the Route Target to be matched


against in received routes from remote PEs

Transport Tunnel

Used to transport the data for the VPRN service between PEs.
Can be delivered by RSVP signaled LSPs, LDP signaled LSPs
or GRE. Auto-bind ldp and Auto-bind GRE are options.
Static
RIP
OSPF
BGP

PE-CE Routing Protocol


Support

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

86/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

A.11.3. Route Selection Process for VPRNS when BGP is being used as the PECE routing protocol
IP-VPNs MSE direct route comparison of BGP and MP-BGP learned routes provides the ability
to compare a route received from a CE peer (inside the VPRN context) to the same route prefix
received as a BGP VPN-IPv4 update from a PE peer. This is required when a CE router is dual
homed and advertises the same customer route prefix to two (or more) PE peers. Each PE router
needs to choose one of the prefixes, which was done previously, based on the Route Table
Preference as opposed to comparing the BGP attributes. The BGP route decision process takes
into account the attribute values of the two routes according following table to decide which is
chosen to be the best route to install in the VRF table:
1. Routes are not considered if they are unreachable.
2. Routes of the protocol with the lowest preference value are selected.
3. BGP routes with higher local preference have preference.
4. BGP routes with the shorter AS path have preference.
(This is checked independent of the as-path-ignore parameter.)
5. BGP routes with the lowest MED metric have preference.
(If MED values are present, they are checked independent of the always-compare-med
parameter.)
6. BGP CE-PE learned routes are preferred over MP-BGP learned routes.

A.11.4. Routes not Getting Populated in to the VRF


Symptom: Some or All routes are not getting populated into the VRF
The following table outlines the problems that might cause this symptom and describes suggested actions to
resolve the problems.

Possible Problem
1. Service Status

Suggested Action
Verify the service is administratively enabled
show service id <id> base
To verify that interface has been assigned a port

2. MP-BGP session
status between PE
routers

Verify the status of the BGP session


Show router bgp neighbor From the output of this command
confirm that the BGP sessions is established and both local and remote
family is configured for VPN-IPv4

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

87/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

3. Route-Distinguisher
is not configured

Verify that a router-distinguisher is configured.


show service id <id> base
Note: A router-distinguisher is required for local VPRNs as well.

4. Route-Targets are not


configured properly

Import targets on the local PE must match the export targets on remote
PEs.
Verify that the expected routes from remote PEs are in the global BGP
routing table (BGP Rib-In). If there are no VPRNs with an import
target on the PE that matches the target in a received route the PE
router will silently drop the advertised route.
show router bgp routes <prefix>
If the expected routes are not present verify that the import target
configured within the VPRN matches the export target on the remote
PE.
Another command to verify the number of VPN routes learned from
remote PE is.
show router bgp summary
Verify the route-targets that are being advertised by the local node.
show router bgp routes <prefix> hunt
Check the community of the output from the above command to
confirm the export target used.
Note: This command will display the RIB-IN and RIB-OUT for the
specified prefix.
Verify the status of the local interface
Show router <service-id> interface
A transport tunnel to the VPN routes next hop must be active before
the route will get installed into the VRF.
A spoke-sdp , auto-bind ldp or auto-bind gre must be configured.
show service id <id> base
From the output confirm that a spoke-sdp is configured or auto-bind is
configured.
If spoke-sdps are used for transport use the following command to
verify the status of the sdp
show service sdp <sdp-id>
If the sdp is not operational go to the Service Distribution Path (SDP)
Problemssection.
Note: Inner Labels are signaled by MP-BGP for VPRNs and signaling
is not required for SDPs being used for a VPRN service
If auto-bind LDP is configured confirm that there is an active binding
for the VPN next-hop address
show router ldp bindings prefix <vpn nexthop/32> active

5. Local interface is not


showing up in the VRF
6. Transport Tunnel is
not operational to next
hop for VPN route

If auto-bind GRE is configured confirm that there is a route in the


routing table for the VPN next-hop address.
Show router route-table <vpn next-hop/32>
7. Is there a import
policy configured?

Confirm the Policy configuration


Show router policy <policy-name>

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

88/89

Alcatel-Lucent Confidential for internal use only -- Do Not Distribute

http:/ /www.alcatel-lucent.com/src

Alcatel-Lucent Advanced Troubleshooting (AT) Lab & Troubleshooting Guide v1.2.1

89/89

You might also like