Benjamin Franklin was quoted as saying “You may delay but time will not“.  We may also say that “Email may tolerate delay, but VOIP will not“.  Performance Routing (PfR), previously called Optimized Edge Routing (OER), is designed to solve network performance issues, such as delay, that vanilla IP routing can’t.  In this blog, we will use the term OER as that is still the command set used.

Petr and the team have created some awesome new labs on OER, including Volume 1, v5 in the IP Routing section. Our workbook content is extensive, has great explanations for everything and is highly recommended it to anyone preparing for the v4 CCIE RS lab.

One of the big challenges for the OER newcomer is knowing what to expect after configuring OER, and how to see measurable results. My intention in this blog post, is to assist the learner by providing a jump start into OER/PfR, by walking you through a solid configuration, and sharing a few critical show and debug commands to verify that OER is working. For you dynamips/GNS3 fans, the configuration may be replicated using (C3725-ADVENTERPRISEK9-M), Version 12.4(15)T10, RELEASE SOFTWARE (fc3).

So, how does it work?   It is actually very simple.  In the world of OER, there are 2 roles. The brains of the operation, called the master controller, and the border router who owns the exit interfaces. There may be multiple border routers. The key is there needs to be at least 2 or more exit points across the borders being managed by a single controller. Although the controller and border can actually run on the same router, I broke it up into separate devices for this blog post. R1 will be the OER Master Controller, and R2 will be the OER Border. Here is the topology.

Single Border (R2), with 2 exit interfaces Tunnel 1 and Tunnel 2.

First we will create key chain on the controller R1.  The key is used for communications between the controller and the border.

R1(config)# key chain cisco
R1(config-keychain)#  key 1
R1(config-keychain-key)#  key-string cisco

Next, on R1 we will configure the master controller portion. About 98 % of all the configuration for OER is done on the controller. Here we will enable logging, and specify who the border(s) are, along with which interfaces on the border are exit interfaces to the outside world.

R1(config)#oer master
R1(config-oer-mc)#logging
R1(config-oer-mc)#border 136.1.12.2 key-chain cisco
R1(config-oer-mc-br)#interface Tunnel1 external
R1(config-oer-mc-br-if)#interface Tunnel2 external
R1(config-oer-mc-br-if)#interface FastEthernet0/0 internal
R1(config-oer-mc-br)#exit

Next on the controller, we  specify that we will learn prefixes based on throughput and delay, set time periods regarding learning, and tell the controller the network lengths to learn. In this example, due to the small network, we are going to have the controller learn /32 routes. We could also specify a maximum number of prefixes to learn as well. (Note: In a production network, you would not want thousands of static routes for sites visited by users. There is a default maximum, and the default aggregation is /24).

R1(config-oer-mc)#learn
R1(config-oer-mc-learn)#throughput
R1(config-oer-mc-learn)#delay
R1(config-oer-mc-learn)#periodic-interval 1
R1(config-oer-mc-learn)#monitor-period 2
R1(config-oer-mc-learn)#expire after time 30
R1(config-oer-mc-learn)#aggregation-type prefix-length 32
R1(config-oer-mc-learn)#exit

Next, now that we have exited out of “learn mode”, we can configure some of the other properties of the controller, such as how often to make prefix decisions (the backoff period), allow it to put routes in the routing table (mode route control), and choose the best exit interface (mode select-exit best). OER will create /32 static routes that use that use the “best” interface to reach the destination network.

R1(config-oer-mc)#backoff 180 360
R1(config-oer-mc)#mode route control
R1(config-oer-mc)#mode select-exit best

A few additional options include how often to make policy decisions, (periodic command), and which policy items are the most important regarding making those policy decisions (resolve commands).  Highest priority is 1, and lowest is 10.

R1(config-oer-mc)#periodic 90
R1(config-oer-mc)#resolve loss priority 1 variance 1
R1(config-oer-mc)#resolve delay priority 2 variance 1
R1(config-oer-mc)#resolve utilization priority 3 variance 1
R1(config-oer-mc)#resolve range priority 4

That is it for now on the controller. Let’s visit the border router R2 next. We’ll create the key chain, and tell R2 who the controller is. We will also tell R2 which interface to use for sending an active probe, (in the event the controller requests one).  Active probes are one method (from many available within OER) to determine delay.

R2(config)# key chain cisco
R2(config-keychain)#  key 1
R2(config-keychain-key)#  key-string cisco

R2(config)#oer border
R2(config-oer-br)#local FastEthernet0/0
R2(config-oer-br)#master 136.1.12.1 key-chain cisco
R2(config-oer-br)#active-probe address source interface Loopback0

So, how do we verify this? Since we are at R2, let’s start by issuing a show oer border on R2.

R2#show oer border
OER BR 136.1.12.2 ACTIVE, MC 136.1.12.1 UP/DOWN: UP 00:00:03,
Auth Failures: 0
Conn Status: SUCCESS, PORT: 3949
Version: 2.1  MC Version: 2.1
Exits
Fa0/0           INTERNAL
Tu1             EXTERNAL
Tu2             EXTERNAL
R2#

From this we see that we have successfully connected with the controller (R1), and that the controller has told R2 which interfaces are internal and which are external, based on what we configured on R1.

On R1, we can issue a few commands to confirm our configuration and connectivity to our border as well. Any elements shown in the output that we didn’t configure are based on the default settings.   Lets jump over to R1 and take a look.

R1#show oer master
OER state: ENABLED and ACTIVE
Conn Status: SUCCESS, PORT: 3949
Version: 2.1
Number of Border routers: 1
Number of Exits: 2
Number of monitored prefixes: 0 (max 5000)
Max prefixes: total 5000 learn 2500
Prefix count: total 0, learn 0, cfg 0

Border           Status   UP/DOWN             AuthFail  Version
136.1.12.2       ACTIVE   UP       00:00:12          0  2.1
! Note:  Below, items in bold are derived from our configuration.
Global Settings:
max-range-utilization percent 0 recv 0
mode route metric bgp local-pref 5000
mode route metric static tag 5000
trace probe delay 1000
logging

Default Policy Settings:
backoff 180 360 180
delay relative 50
holddown 300
periodic 90
probe frequency 56
mode route control
mode monitor both
mode select-exit best
loss relative 10
jitter threshold 20
mos threshold 3.60 percent 30
unreachable relative 50
resolve loss priority 1 variance 1
resolve delay priority 2 variance 1
resolve utilization priority 3 variance 1
resolve range priority 4 variance 0

Learn Settings:
current state : RETRY
time remaining in current state : 26 seconds
throughput
delay
no inside bgp
no protocol
monitor-period 2
periodic-interval 1
aggregation-type prefix-length 32
prefixes 200
expire after time 30
R1#

As a reference below, on R2, here are the IP addresses and Interfaces in use.   Note, that Serial 0/0 is used only as a transport for the GRE tunnels.

R2#show ip int brief | ex una
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            136.1.12.2      YES NVRAM  up                    up
Serial0/0                  136.1.23.2      YES NVRAM  up                    up
Loopback0                  150.1.2.2       YES NVRAM  up                    up
Tunnel1                    10.0.0.2        YES NVRAM  up                    up
Tunnel2                    20.0.0.2        YES NVRAM  up                    up
R2#

Here is the initial routing table on R2, (before OER has dynamically learned routes). We have only 2 static routes, which originated from the startup configuration.

R2#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 20.0.0.3 to network 0.0.0.0

136.1.0.0/24 is subnetted, 2 subnets
C       136.1.12.0 is directly connected, FastEthernet0/0
C       136.1.23.0 is directly connected, Serial0/0

20.0.0.0/24 is subnetted, 1 subnets
C       20.0.0.0 is directly connected, Tunnel2

10.0.0.0/24 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, Tunnel1

150.1.0.0/24 is subnetted, 1 subnets
C       150.1.2.0 is directly connected, Loopback0

S*   0.0.0.0/0 [1/0] via 20.0.0.3
               [1/0] via 10.0.0.3
R2#

R2#show run | inc route
ip route 0.0.0.0 0.0.0.0 10.0.0.3
ip route 0.0.0.0 0.0.0.0 20.0.0.3

Now for the fun part. (Really!:) Lets generate some traffic in the background, and allow OER to dynamically learn the specific routes, as well as learn the best path to those routes and watch those routes be added to the routing table on R2. We’ll create two SLAs from R1 directed to the loopbacks of R4 and R5.

R1

R1(config)#ip sla 1
R1(config-ip-sla)#tcp-connect 150.1.4.4 4000
R1(config-ip-sla-tcp)#timeout 100
R1(config-ip-sla-tcp)#frequency 1
R1(config-ip-sla-tcp)#exit
R1(config)#ip sla schedule 1 life forever start-time now
R1(config)#
R1(config)#ip sla 2
R1(config-ip-sla)#tcp-connect 150.1.5.5 5000
R1(config-ip-sla-tcp)#timeout 100
R1(config-ip-sla-tcp)#frequency 1
R1(config-ip-sla-tcp)#exit
R1(config)#ip sla schedule 2 life forever start-time now

On R4 and R5, we need to set up the SLA responder, so they will respond correctly to the tcp-connect requests sourced from R1.

R4 and R5

R4(config)#ip sla responder
R5(config)#ip sla responder

On R2, lets see what networks have been learned. (Note: Based on the timers, it may take a ~minute or so for the /32 networks of 150.1.4.4 and 150.1.5.5 to be learned.   I waited at least a couple minutes before issuing the commands below.   You may want to verify that your SLA traffic is reaching R4 and R5 if you need to troubleshoot).

R2#show oer border passive cache prefix

OER Passive Prefix Cache, State: enabled, 278544 bytes
4 active, 4092 inactive, 1222 added
67316 ager polls, 0 flow alloc failures
Active flows timeout in 1 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 25800 bytes
12 active, 1012 inactive, 3666 added, 1222 added to flow
0 alloc failures, 0 force free
1 chunk, 30 chunks added
0 flows not aggregated due to main subcache starvation

Prefix              NextHop         Src If         Dst If          Flows
Pkts   B/Pk  Active   sDly   #Dly  PktLos  #UnRch
------------------------------------------------------------------------
150.1.4.4/32        0.0.0.0         Tu1            Fa0/0             24
34      38    18.5      0      0       0       0
150.1.5.5/32        0.0.0.0         Tu1            Fa0/0             24
34      38    18.1      0      0       0       0
150.1.4.4/32        20.0.0.3        Fa0/0          Tu2               23
35      62    18.5     64      5       0       0
150.1.5.5/32        20.0.0.3        Fa0/0          Tu2               23
35      62    18.1     84      5       0       0
R2#

Now, lets turn on a debug of active probes on the border, and see what we can learn from it.

R2#debug oer border active-probes detail
<snip>
OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 150.1.4.4, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 10.0.0.3
probeIfIndex = 10, SAA index = 339 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 2, Sum of rtt 40, Max rtt 24, Min rtt 16
<snip>
OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 150.1.4.4, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 20.0.0.3
probeIfIndex = 11, SAA index = 340 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 2, Sum of rtt 52, Max rtt 28, Min rtt 24

Based on this, the rtt values from Tunnel 1 (using 10.0.0.3 as the next hop to reach 150.1.4.4) are lower than Tunnel 2 (which is using 20.0.0.3 as the next hop). As a result, OER believes that Tunnel 1 is the best path and will inject a static /32 route for destination 150.1.4.4, using Tunnel 1 as the exit interface and 10.0.0.3 as the next hop. A similar result was discovered for 150.1.5.5 as well. As a result, both /32 routes are dynamically added to the routing table on R2.

R2#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 20.0.0.3 to network 0.0.0.0

136.1.0.0/24 is subnetted, 2 subnets
C       136.1.12.0 is directly connected, FastEthernet0/0
C       136.1.23.0 is directly connected, Serial0/0
20.0.0.0/24 is subnetted, 1 subnets
C       20.0.0.0 is directly connected, Tunnel2
10.0.0.0/24 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, Tunnel1
150.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
C       150.1.2.0/24 is directly connected, Loopback0
! OER dynamically added the 2 static routes below
S       150.1.5.5/32 [1/0] via 10.0.0.3
S       150.1.4.4/32 [1/0] via 10.0.0.3
S*   0.0.0.0/0 [1/0] via 20.0.0.3
               [1/0] via 10.0.0.3
R2#

Also, on R1, the controller, there were a couple messages logged:

%OER_MC-5-NOTICE: Route changed Prefix 150.1.5.5/32, BR 136.1.12.2, i/f Tu1, Reason Delay, OOP Reason Timer Expired
%OER_MC-5-NOTICE: Route changed Prefix 150.1.4.4/32, BR 136.1.12.2, i/f Tu1, Reason Delay, OOP Reason Timer Expired

So now that OER has decided that Tunnel 1, using 10.0.0.3 is the best path for those 2 networks, lets tip the scale in the other direction by using generic  traffic shaping on Tunnel 1, so that the delay there will be worse than Tunnel 2, and watch the results. (Note: in addition to the generic traffic shaping on R2 Tunnel 1, I also ran an extended ping sourced from 150.1.4.4 and 150.1.5.5 to increase the overall delay).

R2(config-if)#traffic-shape rate 8000 1000 0
R2(config-if)#do show traffic-shape

Interface   Tu1
Access Target    Byte   Sustain   Excess    Interval  Increment Adapt
VC     List   Rate      Limit  bits/int  bits/int  (ms)      (bytes)   Active
-             8000      125    1000      0         125       125       -

R2#
! Below is debug output showing Tunnel 2 with next hop of 20.0.0.3
! as having the least delay for destination 150.1.4.4 ...

OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 150.1.4.4, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 10.0.0.3
probeIfIndex = 10, SAA index = 363 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 1, Sum of rtt 88, Max rtt 88, Min rtt 88
OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 150.1.4.4, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 20.0.0.3
probeIfIndex = 11, SAA index = 364 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 1, Sum of rtt 8, Max rtt 8, Min rtt 8
OER BR APE detail: Completed retrieving Probe Statistics.

! Now the debug output for the destination 150.1.5.5 showing
! that Tunnel 2 with next hop of 20.0.0.3 has the least delay ....

probeType = echo, probeTarget = 150.1.5.5, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 10.0.0.3
probeIfIndex = 10, SAA index = 365 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 1, Sum of rtt 72, Max rtt 72, Min rtt 72
OER BR APE detail: Completed retrieving Probe Statistics.
probeType = echo, probeTarget = 150.1.5.5, probeTargetPort = 0
probeSource = 150.1.2.2, probeSourcePort = 0, probeNextHop = 20.0.0.3
probeIfIndex = 11, SAA index = 366 probeToS = 0 policy_seq = 0
OER BR APE detail: Completions 1, Sum of rtt 12, Max rtt 12, Min rtt 12

Back on the master controller (R1), we notice the messages:

%OER_MC-5-NOTICE: Route changed Prefix 150.1.4.4/32, BR 136.1.12.2, i/f Tu2, Reason Delay, OOP Reason Timer Expired
%OER_MC-5-NOTICE: Route changed Prefix 150.1.5.5/32, BR 136.1.12.2, i/f Tu2, Reason Delay, OOP Reason Timer Expired

On R2 (the border), here is the updated routing table:

R2#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 20.0.0.3 to network 0.0.0.0

136.1.0.0/24 is subnetted, 2 subnets
C       136.1.12.0 is directly connected, FastEthernet0/0
C       136.1.23.0 is directly connected, Serial0/0

20.0.0.0/24 is subnetted, 1 subnets
C       20.0.0.0 is directly connected, Tunnel2

10.0.0.0/24 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, Tunnel1

150.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
C       150.1.2.0/24 is directly connected, Loopback0

! Note: the next hop has changed, and will use Tunnel 2 as the exit
S       150.1.5.5/32 [1/0] via 20.0.0.3
S       150.1.4.4/32 [1/0] via 20.0.0.3

S*   0.0.0.0/0 [1/0] via 20.0.0.3
               [1/0] via 10.0.0.3
R2#

For detailed information regarding OER and the hundreds of options available with it, please leverage the content in our updated V5 RS content.

OER is very fun, and very important. Enjoy your studies, and good luck!

Keith.