Deploying Performance Routing

BRKRST-2362
Deploying Performance Routing

Jean-Marc Barozet
BRKRST-2362
2013 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Business and IT are Changing Like Never Before

Drastic Change in Application Type, Delivery, and Consumption
Public/Hybrid Cloud
SaaS/IaaS
Storage
Users/ Machines
Proliferation of Devices
THE NETWORK
Private Cloud
VDI | IaaS
Database
60% of IT professional cites performance as key challenge for cloud

BRKRST-2362
How Application applications Type of applications are areConsumed Delivered

Cisco Public
WAN Challenges
Data Centers
Public SaaS
I dont know, if I am getting my SLA

I dont know, the applications running in my network I dont know, how to isolate performance problems
I dont know, how much non-business traffic is consuming
WAN
Internet
Branch with no direct Internet access
Branch with direct Internet access
I cant do, anything about poor and inconsistent performance
I cant do, anything about my Network outages

I cant do, anything about under utilization of my Expensive WAN links I cant do, anything about unreliability for my WAN links
Network Admin
BRKRST-2362
Cisco Public
Why we need Performance Routing?
Visibility
Control
Report
BRKRST-2362
Cisco Public
Introducing Performance Routing (PfR)

Application aware adaptive routing
Full utilization of expensive WAN bandwidth
Efficient distribution of traffic based upon load, circuit cost and path preference Improved Application Performance Per application best path based on delay, loss, jitter measurements Increased Application Availability Protection from carrier black holes and brownouts
PfR MCs
WAE Cluster
Email VMs
ISR G2
ASR1K ASR1K
Internet DMVPN
Email Path Video Path
Branch
PfR MC/BR
PfR BRs
Headquarter
ASR1K ASR1K
Master Controller (MC) Border Router (BR)
SP A MPLS SP B GETVPN MPLS GETVPN
BRKRST-2362
Cisco Public
Performance Routing 101
BRKRST-2362
Cisco Public
Performance Routing Components
The Decision Maker: Master Controller (MC)

Apply policy, verification, reporting No packet forwarding/ inspection required
MC
The Forwarding Path: Border Router (BR)

Gain network visibility in forwarding path (Learn, measure) Enforce MCs decision (path enforcement)
BR BR
Optimize by:
Reachability, Delay, Loss, Jitter, MOS, Throughput, Load, and/or $Cost
WAN1 WAN2
BRKRST-2362
Cisco Public
Performance Routing Topologies

Internet Edge
ISP1
ISP2
Enterprise WAN
MC/BR
Branch
WAN1
(IP-VPN)
BR BR
MC
BR
MC/BR
WAN2
MC
BR
(IPVPN, DMVPN)
BR
MC/BR
HQ
Optimize by: Reachability, Loss, Delay, Jitter, MOS, Throughput, Load, and/or $Cost
BRKRST-2362
Cisco Public
Basic Configuration
MC (Decision Maker)
key chain pfr key 0 key-string cisco ! pfr master ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external ! learn throughput ! max-range-utilization 15 ! ! Load-Balancing enabled by default !
HQ
BR (Forwarding Path)
MC
! key chain pfr key 0 key-string cisco ! pfr border local Ethernet0/0 master 10.2.3.3 key-chain pfr !
WAN1 (IP-VPN)
BR
BR
1. Provisioning 2. Learning 3. Monitoring and Optimization

MC/BR
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
BR
Learning enabled by default with 15.2(3)T and XE 3.6
BRKRST-2362
Cisco Public
10
What you Need to Enable PfR + WAAS

WCCP establishes tunnels between router and WAAS device PfR needs to be aware of the WCCP Tunnel interfaces
router#show tunnel groups 2 tunnel groups active WCCP : service group 317 in "Default", ver v2, assgnmnt: mask-value set intf: Tunnel0, locally sourced WCCP : service group 318 in "Default", ver v2, assgnmnt: mask-value set intf: Tunnel2, locally sourced
PfR cannot control WAAS Express traffic which is sourced from the router
pfr master border 192.168.254.2 key-chain pfr-keychain interface GigabitEthernet0/2 external max-xmit-utilization percentage 80 link-group secondary interface GigabitEthernet0/1.34 internal interface GigabitEthernet0/1.32 internal interface Tunnel0 internal interface Tunnel2 internal
BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.
Add both interfaces as PfR internal interfaces
Cisco Public
11
Performance Routing The Journey
Learning
Get the Traffic Classes in the MC database
Monitoring (Passive Active)
Get the Traffic Classes Performance Metrics

Check Delay, loss, threshold, Bandwidth and more Use a good performing path per Traffic Class
Choosing Your Policies
Enforcing the Path
BRKRST-2362
Cisco Public
12
Step #1 Learning
Automatic vs Static
Traffic Classes
MC
HQ
Learning
Prefixes ACL DSCP Based Applications Static Automatic
WAN1 (IP-VPN) WAN2 (IPVPN, DMVPN)
BR
BR
Learning Monitoring (Passive Active)

MC/BR MC/BR MC/BR
Choosing Your Policies Enforcing the Path
BR
BRKRST-2362
Cisco Public
13
Learning
PfR Operates on Traffic Classes
PfR determines the traffic classes from the
Traffic Classes
MC
HQ
traffic flowing through the border routers Subsets of the total traffic must be identified, and these traffic subsets are named traffic classes
Dest. IP DSCP Delay 0 Loss Jitter BW
BR
BR
Voice, Video, Critical
The Rest of the Traffic
Prefixes or
10.2.2.0/24
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Dest. IP 10.2.2.0/24
DSCP EF
AppID
Delay 0
Loss
Jitter
BW
Applications
Dest. IP 10.2.2.0/24 DSCP AF31 AppID Delay 0 Loss Jitter BW
MC/BR
MC/BR
MC/BR
BR
10.1.1.0/24 10.1.2.0/24
Cisco Public
10.2.1.0/24 10.2.9.0/24
14
Learning Traffic Classes

On a global basis
MC
HQ
Learn
Global
BR
BR
Global learning Aka Similar to the default class in QoS

MC/BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
15

Or Per Service Class
MC
HQ
Voice - Video
Service Class
Critical Application Rest of the Traffic
BR
BR
Or define Service Classes
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Aka Similar to the class-map concept in QoS
Allows to define:
Specific policies per group
MC/BR MC/BR MC/BR
BR
Specific thresholds per group

Specific monitoring mode per group
Cisco Public
16
Learning Traffic Classes Definition

Automatic learning is enabled by default Default is to learn based on destination prefixes with aggregation mask /24
Type Destination Prefix (Mandatory) ACL Application (Optional) Example 10.0.0.0/8 20.1.1.0/24
10.1.1.0/24 dscp ef
10.1.1.0/24 dst-port 50 10.1.1.0/24 telnet
Well-Known
20.1.0.0/16 ssh
10.1.1.0/24 nbar RTP 20.1.1.0/24 nbar citrix
NBAR
BRKRST-2362
Cisco Public
17
Traffic Class Learning

Configuration Sample
MC pfr master !
HQ
learn throughput
list seq 10 refname <CLASS_NAME_1> traffic-class access-list <ACL1> filter <PREFIX_LIST1> aggregation-type prefix-length <LENGTH1> throughput list seq 20 refname <CLASS_NAME_2> traffic-class access-list <ACL2> filter <PREFIX_LIST2> aggregation-type prefix-length <LENGTH2> throughput [Rest of the traffic]
MC/BR
BR
BR
#1
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
#2
MC/BR
MC/BR
BR
DEFAULT
BRKRST-2362
Cisco Public
18

Disable Global Learning (Optional)
MC pfr master learn throughput traffic-class filter access-list DENY_GLOBAL_LEARN_LIST list seq 10 refname <CLASS_NAME_1> traffic-class access-list <ACL1> filter <PREFIX_LIST1> aggregation-type prefix-length <LENGTH1> throughput ! ! Access-list for disabling global learn. ! ip access-list extended DENY_GLOBAL_LEARN_LIST BR
HQ
BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
deny ip any any

MC/BR MC/BR MC/BR
BR
When you want to use PfR for a subset of the traffic (ie Voice traffic only, rest is controlled by the routing protocol)
Cisco Public
19

Using NetFlow on BRs
Traffic Classes
Destination Prefix DSCP App
HQ
MC
Id
Delay
Loss
Ingress
Egress
BW
BW
BR
Exit
NetFlow Cache
Traffic Classes
WAN1 (IP-VPN)
BR
BR
NetFlow Cache
Automatic Learning is enabled by default once you enable PfR and add the Border Routers MC commands BRs to learn Traffic Classes
MC/BR
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
20

Database Filled with TCs
MC
Destination Prefix 10.1.1.1/32 DSCP EF App Id Delay Loss Ingress BW Egress BW BR BR1 Exit E0/0
HQ
20.2.1.0/24
30.1.1.0/24
AF31
0
BR2
BR1
E0/0
E0/0
NetFlow Cache
BR
BR
NetFlow Cache
Traffic Classes
BRs use their NetFlow cache

Top Talkers based on throughput
BRs aggregate based on the configured destination mask Send the reports to the MC every minute
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
21
Step #2 Measurement
Passive Monitoring
Traffic Classes
MC
HQ
Passive
Reachability Egress BW PfR Netflow Monitoring Flows Need not be symmetrical Delay Loss Ingress BW
Passive Performance Metrics
NetFlow Cache
BR
BR
NetFlow Cache
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Learning Monitoring (Passive Active) Choosing Your Policies Enforcing the Path
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
22
Measuring Passive Mode

For Data, Best Effort Applications
MC
Destination Prefix 10.1.1.1/32 10.1.10.0/24 DSCP 0 AF11 App Id Delay Loss Ingress BW Egress BW BR BR1 BR1 Exit Gi1/1 Gi1/2
HQ
BR
BR
0
BR BR1 BR2 Links Gig1/1 Gig1/2 Ingress
BR2
Egress
Gi1/1
Traffic Classes
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
PfR uses NetFlow to collect and aggregate passive monitoring statistics on a per traffic class basis.
MC Instructs BRs to monitor the performance
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
23
Measuring Passive Mode

For Data, Best Effort Applications
MC
Destination Prefix 10.1.1.1/32 10.1.10.0/24 DSCP 0 AF11 App Id Delay 60 110 Loss 0 0 Ingress BW 20 52 Egress BW 40 60 BR BR1 BR1 Exit Gi1/1 Gi1/2
HQ
89
BR BR1 BR2
1
Links Gig1/1 Gig1/2
34
10
Ingress 200 130
BR2
Egress 40 60
Gi1/1
Traffic Classes
NetFlow Cache
BR
BR
NetFlow Cache
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Border routers collect and report passive monitoring statistics to the master controller approximately once per minute.
BRs gather performance measurements using Netflow BRs report Performance Metrics for Traffic Classes to the Master Controller
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
24
Step #2 Measurement
Active Mode
Active
Reachability Jitter Delay Loss MOS
Traffic Classes Active Performance Metrics
MC
HQ
SLA
IP SLA Responder
BR
BR
PfR enables IP SLA feature

Probes sourced from BR ICMP probes learned or configured TCP, UDP, JITTER need ip sla responder
Probe active path

Learning Monitoring (Passive - Active) Choosing Your Policies Enforcing the Path
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
25
Measuring Active Mode

Hybrid Modes
Fast
Active probes on all path all the time
Passive to measure BW only
Traffic Classes Active Performance Metrics
MC
HQ
SLA
BR
BR
Probe all paths

Active Throughput
Passive to measure BW only Active probing on current exit
MC/BR
MC/BR
MC/BR
BR
Fast Mode is used when fast failover is needed

Cisco Public
26

For Voice, Video and Critical Apps
MC
App Id Ingress BW Egress BW
HQ
SLA
Destination Prefix
DSCP
Delay
Jitter
Loss
BR
Exit
10.1.1.1/32
10.1.10.0/24
EF
AF31 0
BR1
BR1 BR2
Gi1/1
Gi1/2 Gi1/1
BR
BR
Traffic Classes
Active monitoring involves creating a stream of synthetic traffic (IP SLA probes) that replicates a traffic class as closely as possible. MC Instructs BRs to monitor the performance
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
27

For Voice, Video and Critical Apps
MC
App Id Ingress BW Egress BW
HQ
SLA
Destination Prefix
DSCP
Delay
Jitter
Loss
BR
Exit
10.1.1.1/32
10.1.10.0/24
EF
AF31 0
60
110 89
10
15 26
0
0 1
20
52 34
40
60 10
BR1
BR1 BR2
Gi1/1
Gi1/2 Gi1/1
BR
BR
Traffic Classes
BRS gather performance measurements using IP SLA probes The performance metrics of the synthetic traffic are measured The results are applied to the traffic class entry in the Master Controller database BRs report Performance Metrics for Traffic Classes
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
28
PfR and Monitor Mode

Delay Loss
For Your Reference
Unreachable
Jitter
MOS
Bandwidth
Monitor Mode
When to use?
Passive
Internet presence deployment Destination prefixes are all over the internet. These destinations may not respond to ICMP probes. Traffic is TCP traffic Enterprise deployment Traffic is Between Head-quarter/Branch. If there is TCP traffic Traffic is Between Head-quarter/Branch. The only optimization criteria is traffic-class performance (i.e. no optimization based on bandwidth).
Both
Active
Active Throughput
Fast
Traffic is Between Head-quarter/Branch. Traffic is UDP or it is Encrypted

Traffic is time-sensitive like video-stream, VoIP. Traffic is Between Head-quarter/Branch
BRKRST-2362
Cisco Public
29
Step #3 Policy Definitions

Traffic Classes
MC
HQ
Link
Load balancing Max utilization Link grouping $Cost
Application Performance
Reachability
Delay Loss MOS Voice, Video, Critical
BR BR
Jitter
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
30

Resolvers per Group
1. Link-Group
2. Loss
MC
HQ
Voice - Video
3. Jitter 4. Delay 1. Link-Group
BR
BR
Critical Application
2. Loss 4. Delay
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Rest of the Traffic
Load-Balancing
MC/BR MC/BR MC/BR
Multiple resolvers can be assigned Set of Policies per Application Group Resolver Priority
BR
Cisco Public
31
Performance Routing Policies

Option1 - Global - Interface
Global
HQ
MC
pfr master max-range-utilization percent 30 delay relative 200 loss threshold 50000 mode <monitor-mode> resolve delay resolve loss
Per Interface
BR
BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
pfr master border 10.2.4.4 interface Ethernet0/1 external max-xmit-utilization percentage 80 maximum utilization receive percentage 80 link-group WAN1
IOS 15.2(3)T
MC/BR
MC/BR
MC/BR
BR
Cisco Public
32
Performance Routing Policies

Option2 - Per Group (pfr-map)
Application Group #1
HQ
MC
pfr-map MYMAP 10 match pfr learn list <CLASS_NAME_1> <Threshold definitions> <Monitoring mode definition> <Policy definitions> <Preferred Path and Fallback option> <Probe frequency> <Periodic timer definition>
Application Group #2
WAN1 (IP-VPN)
BR
BR
WAN2 (IPVPN, DMVPN)
pfr-map MYMAP 20 match pfr learn list <CLASS_NAME_2> <Threshold definitions> <Monitoring mode definition> <Policy definitions> <Preferred Path and Fallback option> <Probe frequency> <Periodic timer definition>
IOS 15.2(3)T
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
33
Define Your Policy Thresholds?

Absolute vs. Relative
The required characteristic can be represented in two ways Absolute Short Avg exceeds the threshold of N msec.
Used for delay, loss, unreachable, jitter
Relative Short Avg exceeds the Long Avg by X percent.

Used for delay, loss, unreachable
Delay Absolute 70 60 50 40 30 20 10 0 1 3 5 7 9 11 13 15 17 19
OOP Detected Short Avg is higher than the absolute threshold

70 60 50 40 30
Delay Relative
OOP Detected Short Avg is higher than Long Avg by X Percent
Short Term Avg (5 minute)
Short Term Avg (5 minute)
Threshold 35 ms
20 10 0
Long Term Avg (60 minute)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
34
21 23
25 27 29
31 33
BRKRST-2362
Cisco Public
Step #4 Enforce the Path

Automatic or force PBR
MC
HQ
Destination Prefix
BGP
Egress: route injection or Modifying the BGP Local Preference attribute Ingress: BGP AS-PATH Prepend or AS Community
Application
Dynamic PBR NBAR/CCE
BR BR
EIGRP Route Control Static Route Injection PIRO

35
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
Performance Routing The Journey
Learning
pfr master learn <learn-list>
Monitoring (Passive Active)
pfr-map match <list> parameters policy definitions
Enforcing the Path
Automatic, PBR
BRKRST-2362
Cisco Public
36
Performance Routing Domain

Moving PfR to multi-Sites
BRKRST-2362
Cisco Public
Active Mode
Simplifying the Configuration
Active Fast Active Throughput TCP, UDP, JITTER probe need ip sla responder Whats needed: Configure a pfr-map that matches prefixes or applications @ Remote-site1 Define the policies Define the jitter probes And REPEAT for each remote site
MC/BR
HQ pfr-map MYMAP 10 match pfr learn list LEARN_LIST_VIDEO_BRANCH1 set periodic 90 set delay threshold 200 MC set loss threshold 50000 IP SLA Responder SLA set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4BR variance 5 BR no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
38
PfR Multisite System Evolution

Peering & Discovery
Multisite MC Peering Framework
MC to MC Peering Framework can be used to exchange policies, services and feedback Remote Site Discovery
Automatic discovery of branch routers
Simplifies Configuration prefix and target discovery Probing Efficiency sharing of probe data across policies Enhance PfR remote site bandwidth discovery
MC/BR
WAN1 (IP-VPN)
HQ
MC
SLA
IP SLA Responder
BR
BR
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
BR
IP SLA Responder
Cisco Public
39
PfR Domain
HQ
Site HQ
Active Fast Active Throughput

BR
MC
Publish Prefix H1, H2, H3 Responder H
SLA
BR
Each MC announces its inside prefixes, together with probe target address and site names
MC/BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Site 1
Site 2
MC/BR
MC/BR
Site 3
BR
Publish Prefix A Responder 1
Publish Prefix B Responder 2
Publish Prefix C, D, E Responder 3, 4
IP SLA Responder
Cisco Public
40
PfR Domain
Target Discovery
HQ
MC
Prefixes Prefix A Prefix B Prefix C, D, E
Responders Responder1 Responder2 Responder3, 4
BW 1.2 Mbps 2.3 Mbps 10 Mbps
Sites Site 1 Site2 Site3

WAN1 (IP-VPN)
SLA
BR
BR
WAN2 (IPVPN, DMVPN)
Mapping table built on each site Allows automatic jitter probe configuration
Allows automatic probe generation

Remote Bandwidth discovery per remote site
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
41
With Target Discovery Enabled

pfr-map MAP-TEST3 10 match pfr learn list LEARN_LIST_BRANCH1 set periodic 90 set mode route control set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 pfr-map MAP-TEST3 15 pfr master policy-rules MYMAP mc-peer head-end Loopback1 target-discovery [SNIP] ! pfr-map MYMAP 10 match pfr learn list LEARN_LIST_BRANCH
pfr-map MAP-TEST3 15 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 pfr-map MAP-TEST3 15 match pfr-map pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 match pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 set threshold delay threshold 200 pfr-map 15 set loss 50000 match pfr learn list LEARN_LIST_BRANCH2 set periodic 90 set delay threshold 200 pfr-map MAP-TEST3 15 set lossset threshold 50000 match pfr learnMAP-TEST3 list LEARN_LIST_BRANCH2 set periodic 90 delay threshold 200 pfr-map MAP-TEST3 pfr-map 1515 set loss threshold 50000 match pfr learn list LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set delay threshold 200 set loss threshold 50000 match pfr learn list LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set delay threshold 200 set loss threshold 50000 match pfr learn LEARN_LIST_BRANCH2 match pfr learn listlist LEARN_LIST_BRANCH2 set jitter threshold 30 set periodic 90 set mode fast set delay threshold 200 set loss threshold 50000 setmonitor jitter threshold 30 set periodic 90 set mode monitor fast set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set periodic 90 set periodic 90 200 set mode monitor fast set delay threshold set resolve loss priority 2 threshold variance 5 set loss 50000 set jitter threshold 30 set mode monitor fast set delay threshold 200 set resolve loss priority 2 threshold variance 5 set loss 50000 set jitter threshold 30 set mode monitor fast set mode control set delay threshold 200 set resolve loss priority 2 threshold variance 5 set 50000 set resolve jitter priority 3loss variance 5 set jitter threshold 30 route set mode monitor fast set resolve loss priority 2 variance 5 set loss threshold 50000 set resolve jitter priority 3 variance 5 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 threshold variance set delay threshold 200 set loss 50000 set resolve jitter priority 3 variance 5 set jitter threshold 30 5 set resolve delay priority 4 variance 5 variance set mode monitor fast set resolve loss priority 2 set resolve jitter priority 3 threshold variance 5 set jitter 30 5 5 set resolve delay priority 4 variance 5 set mode monitor fast set resolve loss priority 2 variance set resolve jitter priority 3 threshold variance 5 set loss threshold set jitter 3050000 set resolve delay priority 4 variance 5 variance set mode monitor fast no set resolve set resolve loss priority 2 setrange resolve jitter priority 3 variance 5 5 5 set resolve delay priority 4 variance 5 variance set mode monitor fast no set resolve range set resolve loss priority 2 set resolve jitter priority 3 variance set resolve delay priority 4 variance 5 set jitter threshold 305 5 5 set mode monitor fast no set resolve range set resolve loss priority 2 variance no set resolve utilization set resolve jitter priority 3 variance set resolve delay priority 4 variance 5 variance no set resolve range set resolve loss priority 2 no set resolve utilization set resolve jitter priority 3 variance 5 5 set resolve delay priority 4 variance 5 variance no set resolve range set mode monitor fast set resolve loss priority 2 no set resolve utilization set resolve jitter priority 3 variance set probe frequency 4 set resolve delay priority 4 variance 5 5 55 no set resolve range no set resolve utilization set resolve jitter priority 3 variance set probe 4utilization set resolve delay priority 4 variance 5 no set resolve range no frequency set resolve set resolve loss priority 2 variance resolve jitter priority 3 variance set probe 4set set resolve delay priority 4 variance 5 55 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no frequency set resolve utilization set probe frequency 4 set resolve delay priority 4 variance 5 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no set resolve utilization set probe frequency 4utilization set resolve jitter priority variance set resolve delay priority variance 55 set active-probe jitter 20.9.9.9 target-port 2000 43 no set resolve range no set resolve set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 no set resolve range no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4utilization set active-probe jitter 20.9.9.9 target-port 2000 no set resolve set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set resolve nono set resolve utilization set probe frequency 4range set active-probe 20.9.9.9 2000 set probe jitter frequency 4 target-port set active-probe jitter 20.9.9.9 target-port 2000 no set resolve utilization set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 set active-probe jitter 20.9.9.9 target-port 2000 set probe frequency 4 set active-probe jitter 20.9.9.9 target-port 2000 2000 BRKRST-2362 set active-probe jitter 20.9.9.9 target-port 2013
set periodic 90 set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 no set resolve range no set resolve utilization set probe frequency 4
Cisco and/or its affiliates. All rights reserved.
Cisco Public
42
PfR Domain Summary

PfR is now multi-site aware
MC to MC Peering Framework can be used to exchange policies, services and feedback Target Discovery is just the first feature running over this framework Automatic configuration and generation of all jitter probes Reduces IP SLA target configuration per destination and per policy. Improves IP SLA probing efficiency through the sharing of probe data across multiple policies Possible future features: Solving the Asymmetric Routing issue PfR Policies distribution QoS Policies distribution
BRKRST-2362
Cisco Public
43
Internet Edge Load-Balancing
BRKRST-2362
Cisco Public

Internet Edge
Branch
ISP1
ISP2
MC/BR
WAN1
(IP-VPN)
BR BR
MC BR
MC/BR
WAN2
MC
BR
(IPVPN, DMVPN)
BR
MC/BR
HQ
BRKRST-2362
Cisco Public
45
Manual Traffic Engineering

Overview
Problem Statement
Ingress/Egress path are under/over utilized Maximize bandwidth utilization (uplinks with different BW) ISP X ISP Z
ISP Y
Manual Solution
Consider The Traffic Patterns of the Enterprise.
Does the Enterprise Host Content? Does the Enterprise Access Content?
ISP A
ISP B
Not Sure?
Graph Interface Byte Count Use NetFlow Even better, use Flexible NetFlow
Egress
1GE
R1
100M
R2
In General, sites have a 80:20 traffic volume (in bytes) mix. Fix only the 80% Direction.
20% direction doesnt matter unless links are widely varying speeds.
Ingress
Cisco Public
46
Manual Traffic Engineering

BGP Policy Instruments
Tool/Attribute
Longest Match Local Preference AS_Path Communities
Direction of Traffic Flow Affected

Inbound and Outbound Outbound
Implementation
Static or Redistribution / Received
Direction Applied: Inbound Direction Applied: Outbound
Inbound
Inbound
Direction Applied: Outbound
Direction Applied, works together with Route Maps
BRKRST-2362
Cisco Public
47
Automatic Traffic Engineering Load-Balancing with PfR

Solution: PfR used to load balance the traffic New default policies based on load-balancing Cisco ASR1k is typical BR/MC with BR terminating WAN connections BGP routing BRs must be iBGP peers Default routing or Partial routes or Full routes PfR can actively manage up to 20k Prefixes concurrently (with ASR1000) 12.4T/15.0.1M IOS-XE 3.3.0
HQ
R3
iBGP
R4 R5
eBGP
eBGP
ISP1 ISP3 ISP4
ISP2
ISP5
Cisco Public
48
Egress Load Balancing

PfR Solution Used
R3
HQ
Learning
Dest Prefixes (NetFlow)

Passive Global
Egress BW eBGP
R4
iBGP
R5
Monitoring
55% 45%
eBGP
Policies Path Enforcement
Load-Balancing (range) BGP

Inject BGP Route
ISP1 ISP3 ISP4
ISP2
ISP5
BGP Local Pref
BRKRST-2362
Cisco Public
49
Internet Presence
Outbound Load Balancing Only
pfr master max-range-utilization percent 25 logging ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external max-xmit-utilization percentage 90 ! border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external max-xmit-utilization percentage 90 ! learn prefixes 1000 applications 0 expire after time 300 ! ! max prefix total 10000 learn 10000 exporter MYEXPORTER mode monitor passive periodic 600
Link Range Utilization Keep the usage on a set of exit links within a certain percentage range of each other
Max Link Utilization Upper threshold on the amount of traffic a specific link can carry
Max Prefixes Learn 1000 Prefixes Delete Prefix if not relearned in 60 Minutes
Global Policies Load Balancing enabled by default Link OOP if : % Util > Lowest + 10 % Util > 90 Revaluate Exit every 10 Minutes
BRKRST-2362
Cisco Public
50
Ingress Load Balancing

PfR Solution Used
R3
HQ
Learning
Inside Prefixes (BGP)

Passive Global
Ingress BW
eBGP
R4
iBGP
R5
Monitoring
20%
17%
eBGP
Policies Path Enforcement
Load-Balancing (range) BGP

BGP AS-PATH Prepend
ISP1 ISP3 ISP4
ISP2
ISP5
BGP Community
BRKRST-2362
Cisco Public
51
Internet Presence
Inbound Load Balancing
Learning Inside Prefix
pfr master policy-rules MYMAP max-range-utilization percent 25 logging ! [BR PROVISIONNING] ! learn inside bgp prefixes 1000 applications 0 expire after time 300 ! max prefix total 10000 learn 10000 max range receive percent 35 exporter MYEXPORTER mode monitor passive periodic 600 ! pfr-map MYMAP 10 match pfr learn inside !
Configuring Inside Prefix

pfr master policy-rules MYMAP max-range-utilization percent 10 logging ! [BR PROVISIONNING] ! learn prefixes 1000 expire after time 300 ! max prefix total 10000 learn 10000 max range receive percent 5 exporter MYEXPORTER mode monitor passive periodic 600 ! pfr-map MYMAP 10 match ip address prefix-list HQ_PREFIX inside ! ip prefix-list HQ_PREFIX seq 5 permit 10.10.1.0/24 ip prefix-list HQ_PREFIX seq 10 permit 10.10.2.0/24 ip prefix-list HQ_PREFIX seq 15 permit 10.10.3.0/24 ip prefix-list HQ_PREFIX seq 20 permit 10.10.4.0/24
Link Range Utilization - Ingress Keep the usage on a set of exit links within a certain percentage range of each other
BRKRST-2362
Cisco Public
52
Internet Presence
Downgrade Method
AS-PATH PREPEND BGP Community pfr master max-range-utilization percent 10 logging ! border 10.4.5.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external downgrade bgp community 3:100 ! border 10.4.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external downgrade bgp community 4:200 !
Nothing required
Downgrade Community Specific value per Provider
Check the following:

BRKRST-2362
Send-community enabled under bgp neighbor <ip> soft-reconfiguration inbound per neighbor max range receive percent X under pfr master
Cisco Public
53
PfR Master Controller Database

Display Traffic Classes Performance Metrics
R3#sh pfr master traffic-class OER Prefix Statistics: Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms), P - Percentage below threshold, Jit - Jitter (ms), MOS - Mean Opinion Score Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million), E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable U - unknown, * - uncontrolled, + - control more specific, @ - active probe all # - Prefix monitor mode is Special, & - Blackholed Prefix % - Force Next-Hop, ^ - Prefix is denied DstPrefix Appl_ID Dscp Prot SrcPort DstPort SrcPrefix Flags State Time CurrBR CurrI/F Protocol PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos EBw IBw ActSDly ActLDly ActSUn ActLUn ActSJit ActPMOS ActSLos ActLLos -------------------------------------------------------------------------------20.20.0.0/24 N N N N NN INPOLICY 32 10.4.4.4 Et0/1 BGP 53 53 0 0 0 0 12 1 N N N N N N 20.20.8.0/24 53 N 20.20.16.0/24 105 N N N INPOLICY 53 0 N N N 42 0 N 0 N N NN 10.4.4.4 Et0/1 0 12 1 N BGP
Passive Delay (from TCP Syn/Ack)
No Active Delay Passive Mode used
N N N INPOLICY 59 105 0 0 N N N
N NN 10.5.5.5 Et0/1 0 0 27 N N
BGP 1
PfR has calculated the per traffic class (which is per /24 destination in this case) egress bandwidth usage as well as the ingress
BRKRST-2362
Cisco Public
54

Traffic Class Performance
R3#sh pfr master traffic-class performance ip any 20.20.0.0/24 ==============================================================================================
Traffic-class: Destination Prefix : 20.20.0.0/24 Source Prefix : N/A Destination Port : N/A Source Port : N/A DSCP :N Protocol : N/A Application Name: : N/A
General: Control State : Controlled using BGP Traffic-class status : INPOLICY Current Exit : BR 10.4.4.4 interface Et0/1, Tie breaker was Range Time on current exit : 0d 0:4:49 Time remaining in current state : @10 seconds Traffic-class type : Learned Improper config : None Last Out-of-Policy event: No Out-of-Policy Event Average Passive Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Delay : 60% -- Threshold: 50% Loss : 0% -- Threshold: 10% Egress BW : 13 kbps Ingress BW : 1 kbps Time since last update : 0d 0:0:15
TC Performance Performance Details Last Resolver decisions Filter on any type of traffic
Average Active Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Delay : 50% -- Threshold: 50%
============================================================================================== R3#
Cisco Public
55
Load Balancing
Checking Load Balancing Accuracy
R3#sh pfr master exits ============================================================================================== PfR Master Controller Exits:
General Info: ============= E - External I - Internal N/A - Not Applicable Up/ ID Name Border Interface ifIdx IP Address Mask Policy Type Down --- ------------ --------------- ----------- ----- --------------- ---- ----------- ---- ---2 10.4.4.4 Et0/1 2 100.4.81.4 24 Util E UP 1 10.5.5.5 Et0/1 2 100.5.82.5 24 Util E UP Global Exit Policy: =================== Cost: In Policy Exits Performance: ================== Egress Ingress ---------------------------------------------------- -----------------------------------ID Capacity MaxUtil Usage % RSVP POOL OOP Capacity MaxUtil Usage % OOP --- -------- -------- -------- --- -------------- ----- -------- -------- -------- --- ----2 1000 900 232 23 N/A N/A 1000 1000 42 4 N/A 1 1000 900 317 31 N/A N/A 1000 1000 16 1 N/A TC and BW Distribution: ======================= # of TCs BW (kbps) Probe Active Name/ID Current Controlled InPolicy Controlled Total Failed Unreach (count) (fpm) ---- ---------------------------- ---------------------- ------ -------2 23 23 23 509 232 0 0 1 26 26 26 1203 317 0 0 Exit Related TC Stats: ====================== Priority highest nth -----------------Number of TCs with range: 0 Number of TCs with util: 0 Number of TCs with cost: 0
HQ
R3
iBGP
R4 R5
eBGP
eBGP
ISP1 ISP3 ISP4
ISP2
ISP5
0 0 0
Total number of TCs:

R3#
59
BRKRST-2362
Cisco Public
56
Enterprise WAN
Application Based Routing
BRKRST-2362
Cisco Public
ISP1
ISP2
Enterprise WAN
MC/BR
Branch
WAN1
(IP-VPN)
BR BR
MC BR
MC/BR
WAN2
MC
BR
(IPVPN, DMVPN)
BR
MC/BR
HQ
Voice - Video Critical Application Rest of the Traffic
BRKRST-2362
Cisco Public
58
Enterprise WAN Use Case

Blackout and Brownout
Problem Statement:
Recent carrier routing problem cause a network outage (Blackout). Fluctuating performance over the WAN is causing intermittent application problems (Brownout) Secondary/Backup WAN path under utilized
Voice - Video
MC
HQ
BR
BR
Solution: PfR Application based optimization

Protect Voice and Video traffic:
primary path, check delay, loss, jitter fallback secondary
Protect Business Applications:

primary path, check loss, utilization fallback secondary
Best effort Applications Maximize bandwidth utilization:

load balanced across SPs or use the secondary path
MC/BR
MC/BR
MC/BR
BR
Cisco Public
59
Deployment With Target Discovery (Recommended)

ISP-1 ISP-2
MC/BR
WAN1
(IP-VPN)
BR BR
MC
Voice - Video
MC/BR
BR
WAN2
MC
BR
(IPVPN, DMVPN)
BR
HQ
MC/BR
Multisite MC Peering Framework MC to MC Peering Framework can be used to exchange policies, services and feedback Remote Site Discovery Simplifies Configuration prefix and target discovery Probing Efficiency sharing of probe data across policies
Cisco Public
60
Enabling Target Discovery

Ease Configuration for Active Probing
The MC peering system must configure a source loopback interface with an IP address that is reachable (routed) through the network The Head-End MC is configured into unicast-listen mode and each of the Branch MCs will have a static unicast neighbor configuration with the HeadEnd MC peer address
MC
HQ
10.10.0.0/16
R3 LISTEN Shadow Router IP Sla Responder
10.3.3.3
SLA BR
BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
MC/BR
BR
BRANCH SETUP The peering to the head-end

20.9.9.9
30.10.10.10
20.20.0.0/16
Cisco Public
30.30.0.0/16
61
Target Discovery
Configuration Example
! pfr master policy-rules MYMAP mc-peer head-end Loopback0 target-discovery <responder-list HQ_TARGET> <insideprefixes HQ_PREFIX> border 10.4.4.4 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external link-group SP1 ! border 10.5.5.5 key-chain pfr interface Ethernet0/0 internal interface Ethernet0/1 external link-group SP2 ! MC
HQ
10.10.0.0/16 10.3.3.3
SLA BR BR
Shadow Router IP Sla Responder
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
pfr master policy-rules MYMAP mc-peer 10.3.3.3 Loopback0 target-discovery

BRKRST-2362
MC/BR
MC/BR
MC/BR
BR
20.9.9.9
IOS 15.2(3)T
30.10.10.10
20.20.0.0/16
30.30.0.0/16
62
Cisco Public
Target Discovery
Hub Site
MC
HQ
10.10.0.0/16 10.3.3.3
SLA BR BR
R3#sh pfr master target-discovery PfR Target-Discovery Services Mode: Static Domain: 59501 Responder list: HQ_TARGET Inside-prefixes list: HQ_PREFIX SvcRtg: client-handle: 7 sub-handle: 6 pub-seq: 1
PfR Target-Discovery Database (local) Local-ID: 10.3.3.3 Desc: R3 Target-list: 10.4.5.5, 10.4.5.4 Prefix-list: 10.10.4.0/24, 10.10.3.0/24, 10.10.2.0/24, 10.10.1.0/24 PfR Target-Discovery Database (remote) MC-peer: 30.10.10.10 Target-list: 30.30.0.10 Prefix-list: 30.30.0.0/16 Desc: R10
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
MC/BR
BR
MC-peer: 20.9.9.9 Desc: R9 Target-list: 20.20.0.9 Prefix-list: 20.20.0.0/16 R3#
20.9.9.9
30.10.10.10
20.20.0.0/16
30.30.0.0/16
63
Cisco Public
Target Discovery
Branch
MC
HQ
10.10.0.0/16 10.3.3.3
SLA BR BR
R10#sh pfr master target-discovery PfR Target-Discovery Services Mode: Dynamic Domain: 59501 SvcRtg: client-handle: 2 sub-handle: 1 pub-seq: 1
PfR Target-Discovery Database (local)
Local-ID: 30.10.10.10 Desc: R10 Target-list: 30.30.0.10 Prefix-list: 30.30.0.0/16

PfR Target-Discovery Database (remote)
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC-peer: 20.9.9.9 Desc: R9 Target-list: 20.20.0.9 Prefix-list: 20.20.0.0/16

MC-peer: 10.3.3.3 Desc: R3 Target-list: 10.4.5.5, 10.4.5.4 Prefix-list: 10.10.4.0/24, 10.10.3.0/24, 10.10.2.0/24, 10.10.1.0/24 R10#
MC/BR MC/BR MC/BR
BR
20.9.9.9
30.10.10.10
20.20.0.0/16
30.30.0.0/16
64
Cisco Public
Step1 Learning Traffic

Defines your groups
Voice - Video
Traffic Classes
MC
HQ
Traffic Classes Definition

Prefixes Prefixes + DSCP Applications
EXAMPLE Critical Apps DSCP AF31 Mask 24 11 TCs Mask 16 2 TCs
WAN1 (IP-VPN)
SLA
BR BR
Rest of the Traffic
WAN2 (IPVPN, DMVPN)
Check Prefixes and application/DSCP per branch

This will give an idea of the number of Traffic Classes Tune Aggregation Mask as needed
BRKRST-2362
MC/BR
MC/BR
MC/BR
BR
10.10.1.0/24 10.10.2.0/24
Cisco Public
10.20.1.0/24 10.20.9.0/24
65

Traffic Classes
pfr master ! MC
HQ
SLA BR BR
learn throughput ! list seq 10 refname LEARN_VOICE_VIDEO traffic-class access-list VOICE_VIDEO filter BRANCH_PREFIX aggregation-type prefix-length <LENGTH> throughput ! list seq 20 refname LEARN_CRITICAL traffic-class access-list CRITICAL filter BRANCH_PREFIX aggregation-type prefix-length <LENGTH> throughput
! [Rest of the traffic]
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
66

Access-List Traffic Automatically learn based on DSCP values Classes for Voice, Video and Critical Applications Rest of the Traffic falls under global learning (kind of default class)
! ip access-list extended VOICE_VIDEO permit ip any any dscp ef ! ip access-list extended CRITICAL permit ip any any dscp af31 !
MC
HQ
SLA BR BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
ip prefix-list BRANCH_PREFIX seq 5 permit 20.20.0.0/16 ip prefix-list BRANCH_PREFIX seq 10 permit 30.30.0.0/16 !
MC/BR
MC/BR
MC/BR
BR
Branch Prefixes Filters on remote prefixes

Cisco Public
67

Configuration Example with NBAR
MC
HQ
! pfr master learn list seq 10 refname VOICE_VIDEO traffic-class application nbar rtp-audio filter BRANCH thoughput list seq 20 refname CRITICAL traffic-class application nbar citrix filter BRANCH thoughput !
SLA BR BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Use NBAR to identify Application Traffic

NBAR is activated automatically on BR NBAR Support was added in 12.4(20T) release
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
68

Example
MC#sh pfr master learn list Learn-List seq 10 refname LEARN_VOICE_VIDEO Configuration: Traffic-Class Access-list: VOICE_VIDEO Filter: BRANCH_PREFIX Aggregation-type: prefix-length 32 Learn type: throughput Session count: 1000 Max count: 1000 Policies assigned: 10 Status: ACTIVE Stats: Traffic-Class Count: 2 Traffic-Class Learned: Appl Prefix 20.20.1.0/24 ef 256 Appl Prefix 20.20.2.0/24 ef 256 Learn-List seq 20 refname LEARN_CRITICAL Configuration: Traffic-Class Access-list: CRITICAL Filter: BRANCH_PREFIX Aggregation-type: prefix-length 24 Learn type: throughput Session count: 50 Max count: 100 Policies assigned: 20 Status: ACTIVE Stats: Traffic-Class Count: 37 Traffic-Class Learned: Appl Prefix 20.20.14.0/24 af31 256 Appl Prefix 20.20.6.0/24 af31 256 Appl Prefix 30.30.5.0/24 af31 256 Appl Prefix 20.20.8.0/24 af31 256 Appl Prefix 30.30.14.0/24 af31 256 [SNIP]
HQ
MC SLA BR BR
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
MC/BR
BR
Cisco Public
69
Step2 Performance Measurement

PfR Solution Used
MC tells the BR to configure and generate all Jitter probes needed
Traffic Classes
MC
HQ
Monitoring
TD Enabled
Fast Voice/Video
NetFlow
SLA BR BR

NetFlow
Active Critical Apps Passive Rest

Destination DSCP Prefix 10.1.1.0/24 10.1.10.0/24 EF AF31 App Id Delay 60 110 Jitter 10 20 Loss 0 0 Ingress BW 20 52 Egress BW 40 60 BR BR1 BR1 Exit Gi1/1 Gi1/2
Probes
Probes
WAN1 (IP-VPN)
WAN2 (IPVPN, DMVPN)
Destination Prefix 10.1.1.0/24 10.1.10.0/24
EF
DSCP App Id
89
Delay 60 110 89
35
Loss 0 0 1
34
Ingress BW 20 52 34
10
Egress BW 40 60 10
BR2
BR BR1 BR1 BR2
Gi1/1
Exit Gi1/1 Gi1/2 Gi1/1 MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
70
Step3 Policies
Define your policies per group
MC
HQ
Policies
BR BR
SLA
Shadow Router IP Sla Responder The Rest of the Traffic
1. Link-Group
Voice - Video
2. Loss 3. Jitter 4. Delay 1. Link-Group

2. Loss 4. Delay
MC/BR MC/BR MC/BR
BR
Rest of the Traffic
Load-Balancing
BRKRST-2362
Cisco Public
71
Policies Voice and Video
Policies Thresholds Applied to the voice and video traffic Loss, delay and jitter Monitor mode fast Actively probe all exits to get performance metrics Policies Definition List all policies Assign priority Administrative policy: SP1 is the primary path, fallback to SP2 if OOP
pfr-map MYMAP 10 match pfr learn list LEARN_VOICE_VIDEO

set delay threshold 200 set loss threshold 50000 set jitter threshold 30 set mode monitor fast set resolve loss priority 2 variance 5 set resolve jitter priority 3 variance 5 set resolve delay priority 4 variance 5 set link-group SP1 fallback SP2 set probe frequency 4 set periodic 90
IOS 15.2(3)T
Jitter Probe Target Discovery is used No need to manually define the probe target
BRKRST-2362
Cisco Public
72
Policies Critical Applications
Policies Thresholds Applied to the voice and video traffic Loss, delay and jitter
pfr-map MYMAP 20 match pfr learn list LEARN_CRITICAL set delay threshold 120 set loss threshold 200000 set mode monitor active throughput set resolve delay priority 1 variance 20 set resolve loss priority 5 variance 10 set link-group SP1 fallback SP2 set probe frequency 4 set periodic 90
Monitor mode Active Actively probe the current exit to get performance metrics. Mode fast is also an option
Policies Definition List all policies Assign priority Administrative policy: SP1 is the primary path, fallback to SP2 if OOP
IOS 15.2(3)T
Active Probes Automatic configuration and generation of probes
BRKRST-2362
Cisco Public
73
Policies Rest of the Traffic
! pfr master policy-rules MYMAP max-range-utilization percent 22 ! mc-peer head-end Loopback0 target-discovery ! logging ! ! Default Policies ! mode route protocol pbr !
IOS 15.2(3)T
Link Range Utilization Keep the usage on a set of exit links within a certain percentage range of each other
Global Policies Apply for the rest of the traffic Load Balancing enabled by default
BRKRST-2362
Cisco Public
74
Target Discovery
Hub Site Jitter probes Generated
R3#sh pfr master active-probes target-discovery PfR Master Controller active-probes (TD) Border = Border Roter running this probe MC-Peer = Remote MC associated with this target Type = Probe Type Target = Target Address TPort = Target Port N - Not applicable Destination Site Peer Addresses: MC-Peer 30.10.10.10 20.9.9.9 Targets 30.30.0.10 20.20.0.9
WAN1 (IP-VPN)
HQ
10.10.0.0/16 10.3.3.3
SLA BR BR
MC
WAN2 (IPVPN, DMVPN)
The following Probes are running: Border 10.4.4.4 10.4.4.4 10.5.5.5 10.4.4.4 10.4.4.4 10.5.5.5 R3#
Idx 2 2 2 2 2 2
State MC-Peer TD-Actv 30.10.10.10 TD-Actv 30.10.10.10 TD-Actv 30.10.10.10 TD-Actv 20.9.9.9 TD-Actv 20.9.9.9 TD-Actv 20.9.9.9
Type Target TPort jitter 30.30.0.10 5000 jitter 30.30.0.10 5000 jitter 30.30.0.10 5000 jitter 20.20.0.9 5000 jitter 20.20.0.9 5000 jitter 20.20.0.9 5000
MC/BR
MC/BR
MC/BR
BR
20.9.9.9
30.10.10.10
20.20.0.0/16
Cisco Public
30.30.0.0/16
75

Display Traffic Classes Performance Metrics
MC#sh pfr master traffic-class OER Prefix Statistics: Pas - Passive, Act - Active, S - Short term, L - Long term, Dly - Delay (ms), P - Percentage below threshold, Jit - Jitter (ms), MOS - Mean Opinion Score Los - Packet Loss (packets-per-million), Un - Unreachable (flows-per-million), E - Egress, I - Ingress, Bw - Bandwidth (kbps), N - Not applicable U - unknown, * - uncontrolled, + - control more specific, @ - active probe all # - Prefix monitor mode is Special, & - Blackholed Prefix % - Force Next-Hop, ^ - Prefix is denied DstPrefix Appl_ID Dscp Prot SrcPort DstPort SrcPrefix Flags State Time CurrBR CurrI/F Protocol PasSDly PasLDly PasSUn PasLUn PasSLos PasLLos EBw IBw ActSDly ActLDly ActSUn ActLUn ActSJit ActPMOS ActSLos ActLLos -------------------------------------------------------------------------------30.1.8.0/24 N defa 256 N N 0.0.0.0/0 INPOLICY 17 10.4.5.4 Et0/1 PBR 52 52 0 0 0 0 65 7 53 52 0 0 N N N N 30.1.1.0/24 52 54 30.1.9.0/24 52 52 30.1.3.0/24 52 52 30.1.4.0/24 52 54 N ef 256 N N 0.0.0.0/0 INPOLICY 41 10.4.5.5 Et0/1 52 0 0 0 0 66 7 51 0 0 N N N N N defa 256 N N 0.0.0.0/0 INPOLICY @47 10.4.5.4 Et0/1 52 0 0 0 0 64 7 51 0 0 N N N N N af31 256 N N 0.0.0.0/0 INPOLICY 113 10.4.5.4 Et0/1 52 0 0 0 0 65 7 52 0 0 N N N N PBR
Passive Delay (from TCP Syn/Ack)
Active Delay (from probes)
PBR
PBR
N af31 256 N N 0.0.0.0/0 INPOLICY 75 10.4.5.5 Et0/1 PBR 52 0 0 0 0 64 7 51 0 0 N N N N
PfR has calculated the per traffic class (which is per /24 destination in this case) egress bandwidth usage as well as the ingress
BRKRST-2362
Cisco Public
76

Traffic Class Performance
MC#sh pfr master traffic-class performance ip any 20.20.0.0/24 dscp ef ============================================================================================== Traffic-class: Destination Prefix : 20.20.0.0/24 Source Prefix : 0.0.0.0/0 Destination Port : N Source Port :N DSCP : ef Protocol : 256 Application Name: : N/A General: Control State : Controlled using PBR Traffic-class status : INPOLICY Current Exit : BR 10.4.4.4 interface Et0/1, Tie breaker was None Time on current exit : 0d 0:9:32 Time remaining in current state : @34 seconds Last uncontrol reason : Probe frequency changed Time since last uncontrol : 0d 1:12:1 Traffic-class type : Learned Improper config : None Last Out-of-Policy event: No Out-of-Policy Event Average Passive Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Loss : 0 ppm -- Threshold: 50000 ppm Delay : 52 msecs -- Threshold: 200 msecs Egress BW : 7 kbps Ingress BW : 5 kbps Time since last update : 0d 0:0:24 Average Active Performance Current Exit: (Average for last 5 minutes) Unreachable : 0% -- Threshold: 50% Loss : 0 ppm -- Threshold: 50000 ppm Jitter : 0 msec -- Threshold: 3000 msec Delay : 52 msec -- Threshold: 200 msec Last Resolver Decision: BR Interface Status Reason Performance Threshold --------------- ------------ ------------ ------------ ----------- --------10.5.5.5 Et0/1 Eliminated Link Group N/A N/A 10.4.4.4 Et0/1 Best Exit Unreachable N/A N/A ============================================================================================== MC# BRKRST-2362 2013 Cisco and/or its affiliates. All rights reserved.
TC Performance Performance Details Last Resolver decisions Filter on any type of traffic
Cisco Public
77
Enforcing the Path

Dynamic PBR MC instructs BRs to apply Dynamic Route Maps on their inside interfaces Direct connection between BRs is needed
Direct connection GRE/mGRE tunnel
WAN1 (IP-VPN)
HQ
MC
BR
BR
WAN2 (IPVPN, DMVPN)
MC/BR
MC/BR
MC/BR
BR
BRKRST-2362
Cisco Public
78
PBR Example
Troubleshooting Path Enforcement
R4#sh route-map dynamic route-map OER_INTERNAL_RMAP, permit, sequence 0, identifier 402653185 Match clauses: ip address (access-lists): oer#1 Set clauses: ip next-hop 10.4.5.5 interface Ethernet0/0 Policy routing matches: 98809 packets, 67823285 bytes route-map OER_INTERNAL_RMAP, permit, sequence 1, identifier 3439329282 Match clauses: ip address (access-lists): oer#2 Set clauses: ip next-hop 100.4.8.8 interface Ethernet0/1 Policy routing matches: 75247 packets, 51786342 bytes Current active dynamic routemaps = 1 R4#
A dynamic route-map is created by PfR to enforce traffic to the chosen exit point Next hop = R5 A dynamic route-map is created by PfR to enforce traffic to the chosen exit point Next hop = WAN
BRKRST-2362
R4#sh ip access-lists dynamic Extended IP access list oer#1 67108863 permit ip any 30.1.6.0 0.0.0.255 dscp af31 (17981 matches) 134217727 permit ip any 30.1.4.0 0.0.0.255 dscp af31 (17984 matches) 268435455 permit ip any 30.1.5.0 0.0.0.255 dscp af31 (17975 matches) 536870911 permit ip any 30.1.2.0 0.0.0.255 dscp ef (17938 matches) 1073741823 permit ip any 30.1.1.0 0.0.0.255 dscp ef (18002 matches) Extended IP access list oer#2 67108863 permit ip any 30.1.3.0 0.0.0.255 dscp af31 (15000 matches) 134217727 permit ip any 30.1.7.0 0.0.0.255 dscp default (15034 matches) 268435455 permit ip any 30.1.10.0 0.0.0.255 dscp default (15958 matches) 536870911 permit ip any 30.1.9.0 0.0.0.255 dscp default (15937 matches) 1073741823 permit ip any 30.1.8.0 0.0.0.255 dscp default (15966 matches) R4#
Cisco Public
Design Considerations
Scalability on the hub
Number of jitter probes
CPU impact on the BRs
Traffic Classes
MC
HQ
SLA BR BR
Probe Frequency
CPU impact on the BRs
Probes
Probes
Number of Traffic Classes

TCAM space on ASR1k BRs CPU/ Memory usage on MC
How fast is Fast Mode?

Min Probe Frequency: 2 sec
MC/BR MC/BR MC/BR
BR
BRKRST-2362
Cisco Public
80
Network Management
NetFlow v9 Export
SNMP MIB
BRKRST-2362
Cisco Public
PfR NetFlow v9 Export

PfR NetFlow v9 Export
The PfR NetFlow v9 Exporter resides on the PfR Master Controller Exports Passive Metrics, Active Metrics, Events, Configuration
BR
HQ
NetFlow
MC
SNMP Read
BR
PfR SNMP MIB (Traps coming)

TC as a row of cpfrTrafficClassTable TC Status as a row of cpfrTrafficClassStatusTable performance metrics as a row of cpfrTrafficClassMetricsTable
NMS application vendors engaged!
MC/BR
MC/BR
MC/BR
BR
Cisco Prime Assurance engaged!
BRKRST-2362
Cisco Public
82
Traffic Class Identifier

Each TC will have its own Identifier Traffic-Class ID is encoded into application tag as follows:
Classification Engine ID (1 Byte) Classification Engine ID: 17
Type (1 Byte) Type: Type=1: Passive Update, Type=2: Passive Performance, Type=3: Active Update, Type=4: Active Performance, Type=5: Traffic-Class Event
Traffic Class Id (4 Bytes)
BRKRST-2362
Cisco Public
83
Configuration Templates Options

Traffic Class Config Encoded Traffic-Class ID, Source prefix, Source mask, Destination prefix, Destination mask, Protocol, DSCP, Source port min, Source port max, Destination port min, Destination port max, Application Name, Policy id External Interface Config
Internal Interface Config
BR address Internal interface id BR interface name
Policy Config
BR address, External interface id, BR interface name, Link capacity, RSVP reserved bandwidth pool, Maximum ingress bandwidth, Maximum egress bandwidth, BGP Community Link group Name, Cost nickname, Cost type Cost discard rollup count
Policy id pfr-map name State Threshold Priority Variance MOS Link group Name
Reason Config
Reason id Reason text
http://docwiki.cisco.com/wiki/AVC-Export:PfR
Cisco Public
84
Passive/Active Reports
Passive Update Encoded Traffic-Class ID BR Address, External interface ID, Direction State First switched time Last switched time Sum of round-trip-time, Samples, Loss, Unreachable, Bytes, Packets, Flows, Next hop address Passive Performance Encoded Traffic-Class ID BR Address, External interface ID Direction Routing protocol State Reason ID Timestamp Left time Passive short-term unreachable Passive short-term loss Passive short-term round-trip-time Passive long-term unreachable Passive long-term loss Passive long-term round-trip-time Egress bandwidth Ingress bandwidth BGP Prepend BGP Community Active Update Encoded Traffic-Class ID BR Address External Interface ID State Average round-trip-time Minimum round-trip-time Maximum round-trip-time Sum of round-trip-time Unreachable Loss Jitter MOS below counts MOS total counts Initiations Completes Passive Performance Encoded Traffic-Class ID BR Address External interface ID Routing protocol State Reason ID Timestamp Left time Active short-term unreachable Active short-term loss Active short-term round-trip-time Active short-term jitter Active Short-term MOS Active long-term unreachable Active long-term loss Active long-term round-trip-time
Cisco Public
85
Interface Update, Various

External Interface Update BR Address External interface ID State Reason ID Egress bandwidth Ingress bandwidth Cost target bandwidth RSVP bandwidth pool TC total counts Controlled TC counts In Policy TC counts Controlled bandwidth Internal Interface Update BR Address Internal interface ID State Reason ID Traffic Class Performance Encoded Traffic-Class ID BR address External interface ID Direction Routing protocol State Reason ID Timestamp Left time BGP Prepend BGP Community Cost Minimization Performance BR Address External interface ID State Timestamp Counter of left rollup Ingress rollup bandwidth (kbps) Egress rollup bandwidth (kbps) k-th rollup bandwidth (kbps) k-th tier percentage (%) k-th fee
BR Up/Down Alert BR address State
Cisco Public
86
Leverage NetFlow Partners Eco-system

Traffic Analysis
Denial of Service
Billing
More info: http://www.cisco.com/warp/public/732/Tech/nmp/netflow/partners/commercial/

Cisco Public
87
PfR Reporting Tools

NetFlow Partners Plixer, ActionPacked
Cisco Prime Infrastructure 2.x Future (H1/CY13)
LiveAction
Cisco Public
88
Conclusion
Platform Support
Software Packages
URLs
BRKRST-2362
Cisco Public
PfR Platform Support
Cisco ASR 1000 BR in IOS-XE 2.6.1 MC in IOS-XE 3.3.0
Cisco 3900
Cisco 3900 Cisco 3800 12.4, 12.4T 15M/T
Cisco 6500
12.2(33)SXH (Deprecated)
Cisco 2900
Cisco 1900 Cisco 1800 12.4, 12.4T, 15M/T Cisco 2800 12.4, 12.4T 15M/T
Cisco 3800 12.4, 12.4T 15M/T
Cisco 7600
12.2(33)SRB (Limited Support)
Cisco 7200-NPE-G2, 7301 12.4, 12.4T 15M/T
BRKRST-2362
Cisco Public
90
PfR Software Packaging

New Cisco ISR G2 Simplified Feature Sets
New ISR-G2 1900, 2900, 3900 A single IOS Universal Image for all ISR Generation 2
Classic Cisco IOS Software Feature Sets

Existing ISR 1800, 2800, 3800, 7200
ISR Platforms PfR is within the DATA package.

PfR
ASR 1000 Series

ASR1001: Use Universal Image (U or UK9) with Advanced IP Services (AIS) or Advanced Enterprise
Services (AES) technology package license All other ASR1000 (ASR1002-F, ASR1002, ASR1004, ASR1006, ASR1013): Use Advanced IP Services (AIS/AISK9) or Advanced Enterprise Services (AES/AESK9) images
Cisco Public
91
Technical References
Docwiki Performance Routing Home
Technology Overview, Solution Guides, Troubleshooting Guides, FAQ
http://docwiki.cisco.com/wiki/PfR:Home
Performance Routing Technology Overview

http://docwiki.cisco.com/wiki/PfR:Technology_Overview
Performance Routing Solution Guides

http://docwiki.cisco.com/wiki/PfR:Solutions
Performance Routing Troubleshooting Guide

http://docwiki.cisco.com/wiki/PfR:Troubleshooting
Configuration
Understanding Performance Routing
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-understand.html
Basic Configuration
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-basic.html
Advanced Configuration
http://www.cisco.com/en/US/docs/ios/pfr/configuration/guide/pfr-advanced.html
BRKRST-2362
Cisco Public
92
Recommended Reading for BRKRST-2362
BRKRST-2362
Cisco Public
93
Call to Action
Visit the Cisco Campus at the World of Solutions

to experience Cisco innovations in action
Get hands-on experience attending one of the Walk-in Labs Schedule face to face meeting with one of Ciscos engineers at the Meet the Engineer center Discuss your projects challenges at the Technical Solutions Clinics
BRKRST-2362
Cisco Public
94
BRKRST-2362
Cisco Public
95

Deploying Performance Routing

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Deploying Performance Routing

Uploaded by

Copyright:

Available Formats

BRKRST-2362

Deploying Performance Routing

2013 Cisco and/or its affiliates. All rights reserved.

Business and IT are Changing Like Never Before

60% of IT professional cites performance as key challenge for cloud

How Application applications Type of applications are areConsumed Delivered

I dont know, if I am getting my SLA

I dont know, how much non-business traffic is consuming

Branch with no direct Internet access

Branch with direct Internet access

I cant do, anything about poor and inconsistent performance

I cant do, anything about my Network outages

2013 Cisco and/or its affiliates. All rights reserved.

Why we need Performance Routing?

2013 Cisco and/or its affiliates. All rights reserved.

Introducing Performance Routing (PfR)

Email Path Video Path

Master Controller (MC) Border Router (BR)

SP A MPLS SP B GETVPN MPLS GETVPN

2013 Cisco and/or its affiliates. All rights reserved.

Performance Routing 101

2013 Cisco and/or its affiliates. All rights reserved.

Performance Routing Components

The Decision Maker: Master Controller (MC)

The Forwarding Path: Border Router (BR)

2013 Cisco and/or its affiliates. All rights reserved.

Performance Routing Topologies

1. Provisioning 2. Learning 3. Monitoring and Optimization

WAN2 (IPVPN, DMVPN)

Learning enabled by default with 15.2(3)T and XE 3.6

2013 Cisco and/or its affiliates. All rights reserved.

What you Need to Enable PfR + WAAS

Add both interfaces as PfR internal interfaces

Performance Routing The Journey

Get the Traffic Classes in the MC database

Monitoring (Passive Active)

Get the Traffic Classes Performance Metrics

Choosing Your Policies

Enforcing the Path

2013 Cisco and/or its affiliates. All rights reserved.

Learning Monitoring (Passive Active)

Choosing Your Policies Enforcing the Path

2013 Cisco and/or its affiliates. All rights reserved.

Voice, Video, Critical

The Rest of the Traffic

WAN2 (IPVPN, DMVPN)

Learning Traffic Classes

Global learning Aka Similar to the default class in QoS

WAN2 (IPVPN, DMVPN)

2013 Cisco and/or its affiliates. All rights reserved.

Learning Traffic Classes

Critical Application Rest of the Traffic

Voice, Video, Critical

The Rest of the Traffic

Or define Service Classes

WAN2 (IPVPN, DMVPN)

Aka Similar to the class-map concept in QoS

Specific thresholds per group

Learning Traffic Classes Definition

2013 Cisco and/or its affiliates. All rights reserved.

Traffic Class Learning

WAN2 (IPVPN, DMVPN)

2013 Cisco and/or its affiliates. All rights reserved.