Just ahead of our brand new CCNA Voice live online bootcamp beginning this Monday, I thought it might be nice to provide an easy-to-follow graphic for those starting out in Voice (or on any other Cisco networking track). This graphic was from last year, but remains quite easy to follow for each and every Cisco track.
Be sure you have a high resolution set if you wish to see the entire thing, otherwise scrolling may be necessary.
Cisco had promised us Christmas 2010 editions of these new exams. Those dates have slipped again.
The new Cisco CCDA 640-864 DESGN and CCDP 642-874 ARCH exams are now scheduled to hit by January 31, 2011.
Both exams promise to test new material regarding virtualization and data centers. Stay tuned to see if these new exams do actually materialize. For those tracking such things - this is the third new date for these exams. :-)
Continuing my review of titles from Petr’s excellent CCDE reading list for his upcoming LIVE and ONLINE CCDE Bootcamps, here are further notes to keep in mind regarding EIGRP.
EIGRP forms "neighbor relationships" as a key part of its operation. Hello packets are used to help maintain the relationship. A hold time dictates the assumption that a neighbor is no longer accessible and causes the removal of topology information learned from that neighbor. This hold timer value is reset when any packet is received from the neighbor, not just a Hello packet.
EIGRP uses the network type in order to dictate default Hello and Hold Time values:
Interestingly, these values are carried in the Hello packets themselves and do not need to match in order for an adjacency to form (unlike OSPF).
By default, EIGRP sends updates and other information to multicast 224.0.0.10 and the associated multicast MAC address of 01-00-5E-00-00-0A.
For multicast packets that need to be reliably delivered, EIGRP waits until a RTO (retransmission timeout) before beginning a recovery action. This RTO value is based off of the SRTT (smooth round-trip time) for the neighbor. These values can be seen in the show ip eigrp neighbor command.
If the router sends out a reliable packet and does not receive an Acknowledgement from a neighbor, the router informs that neighbor to no longer listen to multicast until it is told to once again. The local router then begins unicasting the update information. Once the router begins unicasting, it will try for 16 times or the expiration of the Hold timer, whichever is greater. It will then reset the neighbor and declare a Retransmission Limit Exceeded error.
Note that not all EIGRP packets follow this reliable routine - just Updates and Queries. Hellos and acknowledgements are examples of packets that are not sent reliably.
To start my reading from Petr's excellent CCDE reading list for his upcoming LIVE and ONLINE CCDE Bootcamps, I decided to start with:
EIGRP for IP: Basic Operation and Configuration by Russ White and Alvaro Retana
I was able to grab an Amazon Kindle version for about $9, and EIGRP has always been one of my favorite protocols.
The text dives right in to none other than the composite metric of EIGRP and it brought a smile to my face as I thought about all of the misconceptions I had regarding this topic from early on in my Cisco studies. Let us review some key points regarding this metric and hopefully put some of your own misconceptions to rest.
In the next post on the EIGRP metric, we will examine this at the actual command line, and discuss EIGRP load balancing options. Thanks for reading!
In our current CCDP course - we examined the hot topic of Virtualization in not one, but two lessons.
I thought our students would love to check out this Cisco TechWise TV episode entitled Unlocking the Potential of Virtualization. This episode compliments our course nicely and focuses on:
In order to view, you just need to fill out a quick form. I recommend the use of Internet Explorer and Real Player for the best viewing experience.
I hope you enjoy.
In our CCDP bootcamp, we examined Cisco’s implementation of Virtual Private LAN Services (VPLS) in some detail. One blog that I promised our students was more information about how large enterprises or Internet Service Providers can enhance the scalbility of this solution.
First, let us review the issues that influence its scalability. We covered these in the course, but they are certainly worth repeating here.
Remember that VPLS looks just like an Ethernet switch to the customers. As such, this solution can suffer from the same issues that could hinder a Layer 2 core infrastructure. These are:
The solution for some of these issues is Hierarchical VPLS (H-VPLS). H-VPLS only interconnects the core MPLS network provider edge devices with a full mesh of pseudo-wires, thus reducing the complexity of the full-mesh. The customer provider edge equipment is then connected hierarchically using pseudo-wires to these core devices. The topology now looks like a reduced full-mesh with tree-like connections at the edge of the network. Notice the customer edge equipment no longer connects to each other.
Using H-VPLS, the problem of control-plane scalability can be significantly alleviated. The network is partitioned into as many edge domains as is required. These edge domains are then very efficiently connected using an MPLS core. The MPLS core for the provider allows for the simultaneous usage of L3 MPLS VPNs for those additional customers that desire it. In addition, the MPLS core may limit the extent of the STP domain in non multi-homed scenarios, and therefore, improves performance and limits instabilities.
It is worth mentioning that the main problem with VPLS deployments is not the control-plane but rather data-plane scalability - mainly because of the MAC address table explosion and excessive broadcast/multicast traffic flooding. In addition to improving control-plane scalability by means of H-VPLS pseudo-wire hierarchies, additional techniques can be used to alleviate the issues of the data-plane. The first technique is MAC-in-MAC address stacking, which reduces the number of MAC addresses to be exposed in the core. The second is known as multicast forwarding optimization. This allows for the use of special point-to-multipoint pseudo-wires in the SP core to improve multicast traffic replication.
To summarize, H-VPLS offers the following benefits compared to traditional VPLS solutions:
To complete this blog post, we should mention that there is an alternative to H-VPLS control plane scaling that has been championed by Cisco's rival, Juniper Networks. Juniper's approach utilizes BGP for VPLS pseudo-wire signaling, as opposed to using point-to-point LDP sessions. Scaling BGP by means of route-reflectors is well-known and thus Juniper's approach automatically has a scalable control-plane.
I hope you enjoyed this post. Be sure to watch the blog for more exciting Design-related presentations.
BGP (see [0]) is the de-facto protocol used for Inter-AS connectivity nowadays. Even though it is commonly accepted that BGP protocol design is far from being ideal and there have been attempts to develop a better replacement for BGP, none of them has been successful. To further add to BGP's widespread adoption, MP-BGP extension allows BGP transporting almost any kind of control-plane information, e.g. to providing auto-discovery functions or control-plane interworking for MPLS/BGP VPNs. However, despite BGP's success, the problems with the protocol design did not disappear. One of them is slow convergence, which is a serious limiting factor for many modern applications. In this publication, we are going to discuss some techniques that could be used to improve BGP convergence for Intra-AS deployments.
BGP-Only Convergence Process
Tuning BGP Transport
BGP Fast Peering Session Deactivation
BGP and IGP Interaction
BGP PIC and Multiple-Path Propagation
Practical Scenario: BGP PIC + BGP NHT
Considerations for Implementing BGP PIC
Summary
Further Reading
Appendix: Practical Scenario Baseline Configuration
BGP is a path-vector protocol - in other words, a distance-vector protocol featuring complex metric. In absence of any policies, BGP operates like if routes have metric equal to the length of the AS_PATH attribute. BGP routing polices may override this simple monotonous metric and potentially create divergence conditions in non-trivial BGP topologies (see [7],[8],[9]). While this may be a serious problem at a large scale, we are not going to discuss these pathological cases, but rather talk about convergence in general. Like any distance-vector protocol, BGP routing process accepts multiple incoming routing updates, and advertises only the best routes to its peers. BGP does not utilize periodic updates, and thus route invalidation is not based on expiring any sort of soft state information (e.g prefix-related timers like in RIP). Instead, BGP uses explicit withdrawal section in the triggered UPDATE message to signal neighbors of the loss of the particular path. In addition to the explicit withdrawals, BGP also support implicit signaling, where newer information for the same prefix from the same peer replaces the previously learned information.
Let's have a look at BGP UPDATE message below. As you can see, the UPDATE message may contain both withdrawn prefixes and new routing information. While withdrawn prefixes are listed simply as a collection of NLRIs, new information is grouped around the set of BGP attributes, shared by the group of announced prefixes. In other words, every BGP UPDATE message contains new information pertaining to a set of path attributes, at least prefixes sharing the same AS_PATH attribute. Therefore, every new collection of attributes requires a separate UPDATE message to be sent. This fact is important, as BGP process tries packing as many prefixes per update message as possible, when replicating routing information.
Look at the sample topology below. Let's assume that R1's session to R7 just came up and follow the way that prefix 20.0.0.0/8 takes to propagate through AS 300. In the course of this discussion we skip the complexities associated with BGP policy application and thus ignore the existence of BGP Adj-RIB-In space used for processing the prefixes learned from a peer prior to running the best-path selection process.
The process repeats itself on RR1 and RR2, starting with the incoming UPDATE packet reception, best-path selection and update generation. If for some reason the prefix 20.0.0.0/8 would vanish from AS 100 soon after it has been advertised, it may take as long as "Number_of_Hops x Advertisement_Interval" to reach to R3 and R4, as every hop may delay the fast subsequent update. As we can see, the main limiting factors of BGP convergence are BGP table size, transport-level settings and advertisement delay. The best-path selection time is proportional to the table size as well as time required for update batching.
Let's look at a slightly different scenario to demonstrate how BGP multi-path may potentially improve convergence. Firstly, observing the topology presented on FIG 1, we may notice that AS 300 has two connections to AS 100. Thus, it may be expected to see two paths to every route from AS 100 on every router in AS 300. But this is not always possible in situations where any topology other than BGP full mesh is used inside the AS. In our example, R1 and R2 advertise routing information to the route-reflectors RR1 and RR2. Per the distance-vector behavior, the reflectors will only re-advertise the best-path to AS 100 prefixes, and since both RRs elect paths consistently, they will advertise the same path to R3, R4 and R2. Both R3 and R4 will receive the prefix 10.0.0.0/24 from each of the RRs and use the path via R1. R2 will receive the best path via R1 as well but prefer using its eBGP connection. On contrary, if R1, R2, R3 and R4 were connected in the full mesh, then every router would have seen exits via R1 and R2 and be able to use BGP multi-path if configured. Let's review what happens in the topology on FIG1 when R1 loses connection to AS 100.
If BGP speakers were able to utilize multiple paths at the same time, then it could be possible to alleviate the severity of a network failure. Indeed, if load-balancing is in use, then a failure of an exit point will only affect flows going across this exit point (50% in our case) and only those flows will have to wait for re-convergence time. Even better, it is theoretically possible to do "fast" re-route in the case where multiple equal-cost (equivalent and thus loop--less) paths are available in BGP. Such switchover could be performed in the forwarding engine, as soon as the failure is signaled. However, there are two major problems with the re-route mechanism of this type:
In the following sections we are going to review various technologies developed to accelerate BGP convergence, enabling far better reaction times compared to "pure BGP based" failure detection and repair.
Tuning BGP transport mechanism is a very important factor for improving BGP performance in the cases where purely BGP-based re-convergence process is in use. TCP is the underlying transport used for propagating BGP UPDATE messages, and optimizing TCP performance directly benefits BGP. If you take the full Internet routing table, which is above 300k prefixes (Y2010), then simply transporting the prefixes alone will consume over 10 Megabytes, not to count the path attributes and other information. Tuning TCP transport performance includes the following:
Very detailed information on tuning BGP transport could be found in [10] Chapter 3. We, therefore, skip an in-depth discussion of this topic here.
When using BGP-only convergence mechanic, detecting a link failure is normally based on BGP KEEPALIVE timers, which are 60/180 seconds by default. It could be noted that TCP keepalives could be used for the same purpose, but since BGP already has similar mechanics these are not of any big help. It is possible to tune the BGP keepalive timers to be as low as 1/3 seconds, but the risk of peering session flapping become significant with such settings. Such instability is dangerous since there is no built-in session dampening mechanism in BGP session establishment process. Therefore, some other mechanism should be preferred - either BFD or fast BGP peering session deactivation. The last option is on by default for eBGP sessions, and tracks the outgoing interface associated with the BGP session. As soon as the interface (or the next-hop for multihop eBGP) is reported as down, the BGP session is deactivated. Interface flapping could be effectively dampened using IP Event Dampening in Cisco IOS (see [14]) and hence is less dangerous than BGP peering session flapping. The command to disable fast peering session deactivation is no bgp fast-external-fallover. Notice that this feature is by default off for iBGP sessions, as those are supposed to be routed and restored using the underlying IGP mechanics.
Using BFD is the best option on multipoint interfaces, such as Ethernet, that do not support fast link down detection e.g. by means of Ethernet OAM. BFD is especially attractive in the platforms that implement it in the hardware. The command to activate BFD fallover is neighbor fall-over bfd. In the following sections, we'll discuss the use of IGP for fast reporting of link failures.
BGP prefixes typically rely on recursive next-hop resolution. That is, next-hops associated with BGP prefixes are normally not directly connected, but rather resolved via IGP. The core of BGP and IGP interaction used to be implemented in the BGP Scanner process. This process runs periodically and among other work performs full BGP table walk and validates the BGP next-hop values. The validation consists of resolving the next-hop recursively through the router's RIB and possibly changing the forwarding information in response to IGP events. For example, if R1 crashes on FIG1, it will take 180 seconds for the RRs to notice the failure based on BGP KEEPALIVE message. However, the IGP will probably converge faster and report R1's address as unreachable. This event will be detected during the BGP Scanner process run and all paths via R1 will be invalidated by all BGP speakers in AS 100. The default BGP Scanner run-time is 60 seconds, and could be changed using the command bgp scan-time. Notice that setting this value too low may result in extra burden on router's CPU if you have large BGP tables, since the scanner process has to perform full table walk every time it executes.
The periodic behavior of BGP Scanner is still too slow to effectively respond to IGP events. IGP protocols could be tuned to react to a network change within hundreds of milliseconds (see [6]) and it would be desirable to make BGP aware of such changes as quickly as possible. This could be done with the help of BGP Next-Hop Tracking (NHT) feature. The idea is to make the BGP process register the next-hop values with the RIB "watcher" process and require a "call-back" every time information about the prefix corresponding to the next-hop changes. Typically, the number of registered next-hop values equals the number of exits from the local AS, or the number of PEs in MPLS/BGP VPN environment, so next-hop tracking does not impose heavy memory/CPU requirements. There are normally two types of events: IGP prefix becoming unreachable and IGP prefix metric change. The first event is more important and reported faster than metric change. Overall, IGP delays report of an event for the duration of bgp nextop trigger delay XX interval which is 5 seconds by default. This allows for more consecutive events to be processed and received from IGP and effectively implements event aggregation. This delay is helpful in various "fate sharing" scenarios where a facility failure affects multiple links in the network, and BGP needs to ensure that all IGP nodes have reported this failure and IGP has fully converged. Normally, you should set the NHT delay to be slightly above the time it takes the IGP to fully converge upon a change in the network. In a fast-tuned IGP network, you can set this delay to as low as 0 seconds, so that every IGP event is reported immediately, though this requires careful underlying IGP tuning to avoid oscillations. See [6] for more information on tuning the IGP protocol settings, but in short, you need to tune the SPF delay value in IGP to be conservative enough to capture all changes that could be caused by a failure in the network. Setting SPF delay too low may result is excessive BGP next-hop recalculations and massive best-path process runs.
As a reaction to an IGP next-hop change, the BGP process has to start BGP Router sub-process for re-calculating the best paths. This will affect every prefix that has the next-hop changed as a result of IGP event, and could take significant amount of time, based on number of prefixes associated with this nexthop. For example, if an AS has two connections to the Internet and receives full BGP tables over both connections, then a single exit failure will force full-table walk for over 300k prefixes. After this happens, BGP has to upload the new forwarding information to RIB/FIB, with the overall delay being proportional to the table size. To put it in other words, BGP convergence is non-deterministic in response to an IGP event, e.g. there is no well-defined finite time for the process to complete. However, if the IGP change did not result in any effects to BGP next-hop, e.g. if IGP was able to repair the path upon link failure and the path has the same cost, then BGP is not needed to be informed at all and convergence is handled at IGP level.
The last, less visible contributor to faster convergence is Hierarchical FIB. Look at the figure below - it shows how FIB could be organized as either "flat" or "hierarchical". In the "flat" case, BGP prefixes have their forwarding information directly associated - e.g. the outgoing interface, MAC rewrite, MPLS label information and so on. In such case, any change to a BGP next-hop may require updating a lot of prefixes sharing the same next-hop, which is a time consuming process. If the next-hop value remains the same, and only the output interface changes, the FIB update process still needs walking over all BGP prefixes and reprogramming the forwarding information. In case of "hierarchical" FIB, any IGP change that does not affect BGP prefixes, e.g. output interface change, only requires walking over the IGP prefixes, which are not as numerous as BGP. Therefore, hierarchical FIB organization significantly reduces FIB update latency in the cases where only IGP information needs to be changed. The use of hierarchical FIB is automatic and does not require any special commands. All major networking equipment vendors support this feature.
The last thing to discuss in relation to BGP NHT is IGP route summarization. Summarization hides detailed information and may conceal changes occurring in the network. In such case, BGP process will not be notified of the IGP event and will have to detect failure and re-converge using BGP-only mechanics. Look at the figure below - because of summarization, R1 will not be notified or R2's failure and the BGP process at R1 will have to wait till BGP session times out. Aside from avoiding summarization for the prefixes used for iBGP peering, an alternate solution could be using multi-hop BFD [15]. Additionally, there is some work in progress to allow the separation of routing and reachability information natively in IGP protocols.
You can see now how NHT may allow BGP to react quickly to the events inside its own AS, provided that underlying IGP is properly tuned for fast convergence. This fast convergence process effectively covers core link and node failures, as well as edge link and node failures, provided that all these could be detected by IGP. You may want to look at [1] for detailed convergence breakdowns. Pay special attention that edge link failure requires special handling. If your edge BGP speaker is changing the next-hop value to self for the routes received from another autonomous system, than IGP will only be able to detect failures for paths going to the BGP speaker's own IP address. However, if the edge link fails, the convergence will follow along the BGP path, using BGP withdrawal message propagation through the AS. The best approach in this case is to leave the eBGP next-hop IP address unmodified and advertise the edge link into IGP using the passive interface feature or redistribution. This will allow the IGP to respond to link down condition by quickly propagating the new LSA and synchronously trigger BGP re-convergence on all BGP speakers in the system by informing them of the failed next-hop. In topologies with large BGP tables this takes significantly less time compared to BGP-based convergence process. And lastly, despite all benefits that BGP NHT may provide for recovering from Intra-AS failures, the Inter-AS convergence is still purely BGP driven, based on BGP's distance-vector behavior.
Even though BGP NHT enables fast reaction to IGP events, the convergence time is still not deterministic, because it depends on the number of prefixes BGP needs to be processed for best-path selection. Previously, we discussed how having multiple equal-cost BGP paths could be used for redundancy and fast failover at the forwarding engine level, without involving any BGP best-path selection. What if the paths are unequal - is it possible to use them for backup? In fact, since BGP treats the local AS as a single hop, all BGP speakers select the same path consistently, and changing from one path to another synchronously among all speakers should not create any permanent routing loops. Thus, even in scenarios where equal-cost BGP multi-path is not possible, the secondary paths may still be used for fast failover, provided that a signaling mechanism to detect the primary path failure exists. We already know that BGP NHT could be used to detect a failure and propagate this information quickly to all BGP speakers, triggering local switchover. This switchover does not require any BGP table walks and best-path re-election, but simply is a matter of changing the forwarding information - provided that hierarchical FIB is in use. Therefore, this process does not depend on the number of BGP prefixes, and thus known as Prefix Independent Convergence (PIC) process. You may think of this process as a BGP equivalent to IGP-based Fast Re-Route, though in IGP failure deception is local to the router and in BGP failure detection is local to the AS. BGP PIC could be used any time there are multiple paths to the destination prefix, such on R1 in the example below, where target prefix is reachable via multiple paths:
We have already stated the problem with multiple paths - only one best path is advertised by BGP speakers and the BGP speaker will only accept one path for a given prefix from a given peer. If a BGP speaker receives multiple paths for the same prefix within the same session it simply uses the newest advertisement. A special extension to BGP known as "Add Paths" (see [3] and [16]) allows BGP speaker to propagate and accept multiple paths for the same prefix. The "Add Paths" capability allows peering BGP speakers to negotiate whether they support advertising/receiving multiple paths per prefix and actually advertise such paths. A special 4-byte path-identifier is added to NLRIs to differentiate multiple paths for the same prefix sent across a peering session. Notice that BGP still considers all paths as comparable from the viewpoint of best-path selection process - all paths are stored in the BGP RIB and only one is selected as the best-path. The additional NLRI identifier is only used when prefixes are sent across a peering session to prevent implicit withdrawals by the receiving peer. These identifiers are generated locally and independently for every peering session that supports such capability.
in addition to propagating backup paths, the "Add Paths" capability could be used for other purposes, e.g. overcoming BGP divergence problems described in [9]. Alternatively, if backup paths are required but "Add Path" feature is not implemented, one of your options could be using full-mesh of BGP speakers, such as on the figure below. In this case, multiple exit point information is preserved and allows for implementing BGP PIC functionality.
Pay attention to the fact that BGP PIC is possible even without the "Add Paths" capability in RR scenarios, provided that RRs propagate the alternate paths to the edge nodes. This may require IGP metric manipulation to ensure different exit points are selected by the RRs or using other techniques, such as different RD values for multi-homed site attachment points.
In this hands-on scenario we are going to illustrate the use of IGP tuning, BGP NHT configuration and BGP PIC and demonstrate how they work together. First, look at the topology diagram: R9 is advertising a prefix, and R5, R6 receive this prefix via the RRs. In normal BGP environment, provided that the RRs elect the same path, R5 and R6 would have just one path for R9's prefix. However, we tune the scenario, disabling the connections between R1 and R4 and R2 and R3, so R3 has better cost to exit via R1 and R4 has better cost via R2. This will make the RRs elect different best-paths and propagate them to their clients.
The following is the key piece of configuration for enabling the fast backup path failover to be applied to every router in AS 100. As you can see, the SPF/LSA throttling timers are tuned very aggressively to allow for fastest reaction to IGP events. BGP nexthop trigger delay is set to 0 seconds, thus fully relying on IGP to aggregate underlying events. In any production environment, you should NOT use these values and pick up your own, matching your IGP scale and convergence rate.
router ospf 100
timers throttle spf 1 100 5000
timers throttle lsa all 0 100 5000
timers lsa arrival 50
!
router bgp 100
bgp nexthop trigger delay 0
bgp additional-paths install
no bgp recursion host
The command bgp additional-paths install when executed in non BGP-multipath environment allows for installing backup paths in additional to the best one elected by BGP. This, of course, requires that the additional paths have been advertised by the BGP Route Reflectors. At the moment of writing, Cisco IOS does not support the "Add Paths" capability, so you need to make sure BGP RRs elect different best-paths in order for the edge routers to be able to use additional paths. The command no bgp recursion host requires special explanation on its own. By default, when a BGP prefix loses next-hop, the CEF process will attempt to look-up the next longest-matching prefix for the next-hop to provide fallback. When additional repair paths are present, this functionality is not required and will, in fact, slower the convergence. This is why it's automatically disabled when you type the command bgp additional-paths install and thus typing it with the "no" prefix is not really required.
Now that we have our scenario set up, we are going to demonstrate the fact that at least in current implementation, Cisco IOS BGP process does not exchange/detects the capabilities for "Add Path" feature. Here is a debugging output from a peering session establishment process, which shows that no "Add Path Capability" (code 69, per the RFC draft) is being exchanged during session establishment.
R5#debug ip bgp 10.0.3.3
BGP debugging is on for neighbor 10.0.3.3 for address family: IPv4 Unicast
R5#clear ip bgp 10.0.3.3BGP: 10.0.3.3 active rcv OPEN, version 4, holdtime 180 seconds
BGP: 10.0.3.3 active rcv OPEN w/ OPTION parameter len: 29
BGP: 10.0.3.3 active rcvd OPEN w/ optional parameter type 2 (Capability) len 6
BGP: 10.0.3.3 active OPEN has CAPABILITY code: 1, length 4
BGP: 10.0.3.3 active OPEN has MP_EXT CAP for afi/safi: 1/1
BGP: 10.0.3.3 active rcvd OPEN w/ optional parameter type 2 (Capability) len 2
BGP: 10.0.3.3 active OPEN has CAPABILITY code: 128, length 0
BGP: 10.0.3.3 active OPEN has ROUTE-REFRESH capability(old) for all address-families
BGP: 10.0.3.3 active rcvd OPEN w/ optional parameter type 2 (Capability) len 2
BGP: 10.0.3.3 active OPEN has CAPABILITY code: 2, length 0
BGP: 10.0.3.3 active OPEN has ROUTE-REFRESH capability(new) for all address-families
BGP: 10.0.3.3 active rcvd OPEN w/ optional parameter type 2 (Capability) len 3
BGP: 10.0.3.3 active OPEN has CAPABILITY code: 131, length 1
BGP: 10.0.3.3 active OPEN has MULTISESSION capability, without grouping
BGP: 10.0.3.3 active rcvd OPEN w/ optional parameter type 2 (Capability) len 6
BGP: 10.0.3.3 active OPEN has CAPABILITY code: 65, length 4
BGP: 10.0.3.3 active OPEN has 4-byte ASN CAP for: 100
BGP: nbr global 10.0.3.3 neighbor does not have IPv4 MDT topology activated
BGP: 10.0.3.3 active rcvd OPEN w/ remote AS 100, 4-byte remote AS 100
BGP: 10.0.3.3 active went from OpenSent to OpenConfirm
BGP: 10.0.3.3 active went from OpenConfirm to Established
This means that we need to rely on the BGP RRs to advertise multiple different paths in order for the edge nodes to leverage the backup path capability.
R5#debug ip bgp updates
BGP updates debugging is on for address family: IPv4 Unicast
R5#debug ip bgp addpath
BGP additional-path related events debugging is on
R5#clear ip bgp 10.0.3.3BGP(0): 10.0.3.3 rcvd UPDATE w/ attr: nexthop 20.0.17.7, origin i, localpref 100, metric 0, originator 10.0.1.1, clusterlist 10.0.3.3, merged path 200, AS_PATH
BGP(0): 10.0.3.3 rcvd 20.0.99.0/24
BGP(0): 10.0.3.3 rcvd NEW PATH UPDATE (bp/be - Deny)w/ prefix: 20.0.99.0/24, label 1048577, bp=N, be=N
BGP(0): 10.0.3.3 rcvd UPDATE w/ prefix: 20.0.99.0/24, - DO BESTPATH
BGP(0): Calculating bestpath for 20.0.99.0/24
Here you can see that the RR with IP address 10.0.3.3 sends us an update that has better information than the one we currently know. However, before you enable the bgp additional-paths install there would be just one path installed for the prefix:
R5#show ip route repair-paths 20.0.99.0
Routing entry for 20.0.99.0/24
Known via "bgp 100", distance 200, metric 0
Tag 200, type internal
Last update from 20.0.17.7 00:02:31 ago
Routing Descriptor Blocks:
* 20.0.17.7, from 10.0.3.3, 00:02:31 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 200
MPLS label: none
But as soon as the bgp additional-paths install option has been enabled, the output of the same command looks different:
R5#show ip route repair-paths 20.0.99.0
Routing entry for 20.0.99.0/24
Known via "bgp 100", distance 200, metric 0
Tag 200, type internal
Last update from 20.0.17.7 00:00:03 ago
Routing Descriptor Blocks:
* 20.0.17.7, from 10.0.3.3, 00:00:03 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 200
MPLS label: none
[RPR]20.0.28.8, from 10.0.4.4, 00:00:03 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 200
MPLS label: none
You may also see the second path in the BGP table with the "b" (backup) flag:
R5#show ip bgp 20.0.99.0
BGP routing table entry for 20.0.99.0/24, version 39
Paths: (2 available, best #1, table default)
Additional-path
Not advertised to any peer
200
20.0.17.7 (metric 192) from 10.0.3.3 (10.0.3.3)
Origin IGP, metric 0, localpref 100, valid, internal, best
Originator: 10.0.1.1, Cluster list: 10.0.3.3
200
20.0.28.8 (metric 192) from 10.0.4.4 (10.0.4.4)
Origin IGP, metric 0, localpref 100, valid, internal, backup/repair
Originator: 10.0.2.2, Cluster list: 10.0.4.4
And if you check the CEF entry for this prefix, you will notice there are multiple next-hops and output interfaces that could be used for primary/backup paths:
R5#show ip cef 20.0.99.0 detail
20.0.99.0/24, epoch 0, flags rib only nolabel, rib defined all labels
recursive via 20.0.17.7
recursive via 20.0.17.0/24
nexthop 10.0.35.3 Serial1/0
recursive via 20.0.28.8, repair
recursive via 20.0.28.0/24
nexthop 10.0.35.3 Serial1/0
nexthop 10.0.45.4 Serial1/2
Notice that in oder to use the PIC functionality, BGP multi-path should be turned off - otherwise, equal-cost paths will be used for load-sharing, not for primary/backup behavior. You may opt to using equal-cost multipath if allowed by the network topology, as it offers better resource utilization and CEF switching layer allows for fast path failover in case of equal-cost load-balancing. Now for debugging the fast failover process. We want to shut down R1's connection to R7 and see fast backup path switchover at R5. There are few caveats here, because we have very simplified topology. Firstly, we only have one prefix advertised into BGP on R9. Propagating this prefix through BGP is almost instant, since BGP best-path selection is done quickly and advertisement delay does not apply to a single event. Thus, if we shutdown R1's connection to R7, which is used as primary path, then R1 will detect the link failure and shutdown the session. Immediately after this BGP process will flood an UPDATE with prefix removal and this message would reach R5 and R6 even before OSPF finishes SPF computations. The reason being, of course, single prefix propagated via BGP and no advertisement-interval used to delay to a single event.
It may seems like that disabling BGP fast external fallover on R1 could help us to take BGP out of the equation. However, we still have BGP NHT enabled in R1 - as soon as we shut down the link, the RIB process would report to BGP of the next-hop failure and UPDATE message will be sent right away. Thus, we would also need to disable NTH in R1, using the command no bgp nexthop trigger enable. If we think further, we'll notice that we also need to enable NHT in R3 and R4, just so that they cannot to generate their own UPDATEs to R5 ahead of OSPF notification. Therefore, prior to running experiment we disable BGP NHT in R1, R3, R4 and disable fast external fallover in R1. This will allow the event from R1 propagate via OSPF ahead of BGP UPDATE message and trigger fast switchover on R5. The below is the output of the debugging commands enabled on R5 after we shut down R1's connection to R7.
R5#debug ip ospf spf
OSPF spf events debugging is on
OSPF spf intra events debugging is on
OSPF spf inter events debugging is on
OSPF spf external events debugging is onR5#debug ip bgp addpath
BGP additional-path related events debugging is on
R5 receive the LSA at 26.223 then BGP starts the path switchover at 26.295 - It took 72ms to run SPF, update RIB and inform BGP of the event and then change the paths.
14:00:26.223: OSPF: Detect change in topology Base with MTID-0, in LSA type 1, LSID 10.0.1.1 from 10.0.1.1 area 0
14:00:26.223: OSPF: Schedule SPF in area 0, topology Base with MTID 0
Change in LS ID 10.0.1.1, LSA type R, spf-type Full
….
14:00:26.295: BGP(0): Calculating bestpath for 20.0.99.0/24, New bestpath is 20.0.28.8 :path_count:- 2/0, best-path =20.0.28.8, bestpath runtime :- 4 ms(or 3847 usec) for net 20.0.99.0
14:00:26.299: BGP(0): Calculating backuppath::Backup-Path for 20.0.99.0/24:BUMP-VERSION-BACKUP-DELETE:, backup path runtime :- 0 ms (or 193 usec)14:00:32.439: BGP(0): 10.0.3.3 rcvd UPDATE w/ prefix: 20.0.99.0/24, - DO BESTPATH
14:00:32.443: BGP(0): Calculating bestpath for 20.0.99.0/24, bestpath is 20.0.28.8 :path_count:- 2/0, best-path =20.0.28.8, bestpath runtime :- 0 ms(or 222 usec) for net 20.0.99.0
14:00:32.443: BGP(0): Calculating backuppath::Backup-Path for 20.0.99.0/24, backup path runtime :- 0 ms (or 133 usec)
In the debugging output above, you can see that the BGP process in R5 switched to backup path even before it received the UPDATE message from R3, signaling the change of the best-path in the RR. Notice that the update does not have any path identifiers in the NLRI, as the RR has only a single best-path. Let's see how much time it actually took to run SPF, as compared to overall detection/failover process:
R5#show ip ospf statisticsOSPF Router with ID (10.0.5.5) (Process ID 100)
Area 0: SPF algorithm executed 15 times
Summary OSPF SPF statistic
SPF calculation time
Delta T Intra D-Intra Summ D-Summ Ext D-Ext Total Reason
00:28:00 44 0 0 4 0 4 56 R
…..
As you can see, the total SPF runtime was 56ms. Therefore, the remaining 20ms were spent on updating RIB and triggering the next-hop change event. Of course, all these numbers have only relative meaning, as we are using Dynamips for this simulation, but you may use similar methodology when validating real-world designs.
Even though the Add Paths feature is not yet implemented, it is worth considering the drawbacks of this approach. One drawback is that the amount information needed to be sent and stored is now multiplied by the number of additional paths. Previously, the most stressed routers in BGP AS were route reflectors, that had to carry the largest BGP tables. With the Add-Path functionality, every non-RR speaker now receives all information that RR stores in its BGP table. This puts extra requirement on the edge speakers and should be accounted when planning to use this feature. Furthermore, additional paths will utilize extra memory on the forwarding engines, as now PIC-enabled prefixes have multiple alternate paths. However, since the number of prefixes remains the same, TCAM fast lookup memory resources will not be wasted, and thus only dynamic RAM is being affected the most. You may read more about scalability/performance trade-offs in [17]
Achieving fast BGP convergence is not easy, because BGP is a complicated routing protocol running overlay on top of an IGP process. We found out that tuning purely BGP-based convergence requires the following general steps:
As we noticed previously, pure-BGP based convergence is the only thing available for Inter-AS scenarios. However, for fastest convergence inside a single AS, understanding and tuning BGP and IGP interaction can make BGP converge almost as fast as the underlying IGP. This allows for fast recovery in response to intra-AS link and node failures, as well as to edge link failures. Optimizing BGP and IGP interaction requires the following:
We've also briefly covered some caveats resulting from the future use of "Add-Path" functionality, such as excessive usage of memory resources on router-processor and line-cards and extra toll on BGP best-path process due to the growth of alternate paths. There are few things that were left out of the scope of this paper. We didn't not concentrate on the detailed mechanics of BGP fast peering session deactivation e.g. for multihop sessions and we did not cover the MP-BGP specific features. Some MP-BGP extensions such as the additional import scan interval and edge control plane interworking have their effects on end-to-end convergence, but this is a topic for another discussion.
[0]RFC4271: Border Gateway Protocol
[1]Advanced BGP Convergence Techniques
[2]Graph Overlays on Path Vector: A Possible Next Step in BGP
[3]BGP Add Paths Capability
[4]BGP Convergence in much less than a second
[5]BGP PIC Configuration Guide
[6]OSPF Fast Convergence
[7]An Analysis of BGP Convergence Properties
[8]RFC4451: BGP MULTI_EXIT_DISC (MED) Considerations
[9]RFC3345: Border Gateway Protocol (BGP) Persistent Route Oscillation Condition
[10]BGP Design and Implementation by Randy Zhang
[11]RFC 4274: BGP Protocol Analysis
[12]Day in the Life of a BGP Update in Cisco IOS
[13]RFC 4724: Graceful Restart for BGP
[14]Optimizing IP Event Dampening
[15]RFC 5883: Multihop BFD
[16]BGP Add Path Overview
[17]BGP Add Paths Scaling/Performance Tradeoffs
The below are the initial configurations for the Dynamips topology used to validate BGP PIC behavior.
====R1:====
hostname R1
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
ip address 20.0.17.1 255.255.255.0
no shut
!
interface Serial 1/2
no shut
ip address 10.0.12.1 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.13.1 255.255.255.0
!
interface Serial 1/3
ip address 10.0.14.1 255.255.255.0
!
interface Loopback0
ip address 10.0.1.1 255.255.255.255
!
router ospf 100
router-id 10.0.1.1
network 0.0.0.0 0.0.0.0 area 0
passive-interface Serial 1/0
!
router bgp 100
neighbor 10.0.3.3 remote-as 100
neighbor 10.0.3.3 update-source Loopback0
neighbor 10.0.4.4 remote-as 100
neighbor 10.0.4.4 update-source Loopback0
neighbor 20.0.17.7 remote-as 200====R2:====
hostname R2
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
ip address 20.0.28.2 255.255.255.0
no shut
!
interface Serial 1/2
no shut
ip address 10.0.12.2 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.24.2 255.255.255.0
!
interface Serial 1/3
no shut
ip address 10.0.23.2 255.255.255.0
!
interface Loopback0
ip address 10.0.2.2 255.255.255.255
!
router ospf 100
router-id 10.0.2.2
network 0.0.0.0 0.0.0.0 area 0
passive-interface Serial 1/0
!
router bgp 100
neighbor 10.0.3.3 remote-as 100
neighbor 10.0.3.3 update-source Loopback0
neighbor 10.0.4.4 remote-as 100
neighbor 10.0.4.4 update-source Loopback0
neighbor 20.0.28.8 remote-as 200====R3:====
hostname R3
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 10.0.13.3 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.35.3 255.255.255.0
!
interface Serial 1/2
no shut
ip address 10.0.34.3 255.255.255.0
!
interface Serial 1/3
no shut
ip address 10.0.23.3 255.255.255.0
!
interface Serial 1/4
no shut
ip address 10.0.36.3 255.255.255.0
!
router ospf 100
router-id 10.0.3.3
network 0.0.0.0 0.0.0.0 area 0
!
interface Loopback0
ip address 10.0.3.3 255.255.255.255
!
router bgp 100
neighbor IBGP peer-group
neighbor IBGP remote-as 100
neighbor IBGP update-source Loopback0
neighbor IBGP route-reflector-client
neighbor 10.0.1.1 peer-group IBGP
neighbor 10.0.2.2 peer-group IBGP
neighbor 10.0.5.5 peer-group IBGP
neighbor 10.0.6.6 peer-group IBGP
neighbor 10.0.4.4 remote-as 100
neighbor 10.0.4.4 update-source Loopback0====R4:====
hostname R4
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 10.0.24.4 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.46.4 255.255.255.0
!
interface Serial 1/2
no shut
ip address 10.0.34.4 255.255.255.0
!
interface Serial 1/3
no shut
ip address 10.0.14.4 255.255.255.0
!
interface Serial 1/4
no shut
ip address 10.0.45.4 255.255.255.0
!
router ospf 100
router-id 10.0.4.4
network 0.0.0.0 0.0.0.0 area 0
!
interface Loopback0
ip address 10.0.4.4 255.255.255.255
!
router bgp 100
neighbor IBGP peer-group
neighbor IBGP remote-as 100
neighbor IBGP update-source Loopback0
neighbor IBGP route-reflector-client
neighbor 10.0.1.1 peer-group IBGP
neighbor 10.0.2.2 peer-group IBGP
neighbor 10.0.5.5 peer-group IBGP
neighbor 10.0.6.6 peer-group IBGP
neighbor 10.0.3.3 remote-as 100
neighbor 10.0.3.3 update-source Loopback0====R5:====
hostname R5
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 10.0.35.5 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.56.5 255.255.255.0
!
interface Serial 1/2
no shut
ip address 10.0.45.5 255.255.255.0
!
router ospf 100
router-id 10.0.5.5
network 0.0.0.0 0.0.0.0 area 0
!
interface Loopback0
ip address 10.0.5.5 255.255.255.0
!
router bgp 100
neighbor 10.0.3.3 remote-as 100
neighbor 10.0.3.3 update-source Loopback0
neighbor 10.0.4.4 remote-as 100
neighbor 10.0.4.4 update-source Loopback0====R6:====
hostname R6
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 10.0.46.6 255.255.255.0
!
interface Serial 1/1
no shut
ip address 10.0.56.6 255.255.255.0
!
interface Serial 1/2
no shut
ip address 10.0.36.6 255.255.255.0
!
router ospf 100
router-id 10.0.6.6
network 0.0.0.0 0.0.0.0 area 0
!
interface Loopback0
ip address 10.0.6.6 255.255.255.0
!
router bgp 100
neighbor 10.0.3.3 remote-as 100
neighbor 10.0.3.3 update-source Loopback0
neighbor 10.0.4.4 remote-as 100
neighbor 10.0.4.4 update-source Loopback0====R7:====
hostname R7
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 20.0.17.7 255.255.255.0
!
interface Serial 1/1
no shut
ip address 20.0.78.7 255.255.255.0
!
interface Serial 1/2
no shut
ip address 20.0.79.7 255.255.255.0
!
interface Loopback0
ip address 20.0.7.7 255.255.255.0
!
router ospf 1
router-id 20.0.7.7
network 0.0.0.0 0.0.0.0 area 0
passive-interface Serial 1/0
!
router bgp 200
neighbor 20.0.17.1 remote-as 100
neighbor 20.0.9.9 remote-as 200
neighbor 20.0.9.9 update-source Loopback0
neighbor 20.0.8.8 remote-as 200
neighbor 20.0.8.8 update-source Loopback0====R8:====
hostname R8
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 20.0.28.8 255.255.255.0
!
interface Serial 1/1
no shut
ip address 20.0.78.8 255.255.255.0
!
interface Serial 1/2
no shut
ip address 20.0.89.8 255.255.255.0
!
interface Loopback0
ip address 20.0.8.8 255.255.255.0
!
router ospf 1
router-id 20.0.8.8
network 0.0.0.0 0.0.0.0 area 0
passive-interface Serial 1/0
!
router bgp 200
neighbor 20.0.28.2 remote-as 100
neighbor 20.0.9.9 remote-as 200
neighbor 20.0.9.9 update-source Loopback0
neighbor 20.0.7.7 remote-as 200
neighbor 20.0.7.7 update-source Loopback0====R9:====
hostname R9
!
ip tcp synwait-time 5
no ip domain-lookup
no service timestamps
!
line con 0
logging synch
exec-timeout 0 0
privilege level 15
!
ip routing
!
interface Serial 1/0
no shut
ip address 20.0.79.9 255.255.255.0
!
interface Serial 1/1
no shut
ip address 20.0.89.9 255.255.255.0
!
interface Loopback0
ip address 20.0.9.9 255.255.255.0
!
interface Loopback100
ip address 20.0.99.99 255.255.255.0
!
router ospf 1
router-id 20.0.9.9
network 0.0.0.0 0.0.0.0 area 0
!
router bgp 200
neighbor 20.0.8.8 remote-as 200
neighbor 20.0.8.8 update-source Loopback0
neighbor 20.0.7.7 remote-as 200
neighbor 20.0.7.7 update-source Loopback0
network 20.0.99.0 mask 255.255.255.0
Cisco originally promised us a new CCDP exam (version 2.1) on Nov 8, 2010.
That date is now moved to December 23, 2010. Our Class On Demand was designed to cover you for the old blueprint and the new, so there should be no concern for students. Of course we will be taking the new exam the week following its release and we will be sure to provide any updates to the course that may be required free of charge.
In the meantime, watch blog.ine.com for many posts regarding valuable extra technical information regarding this popular new course. I also want to send out one more thank you to the many students we had that were active participants in the live event. It was an honor to have so many Cisco employees join us, as well as the many highly motivated students from around the world.
Here is the recommended reading list that several asked for from our CCDP Bootcamp. Thanks again to all that attended for the awesome participation and discussions.
Authorized Self-Study Guide Designing Cisco Network Service Architectures (ARCH) Second Edition
Top-Down Network Design, 3rd Edition
IPv6 Theory, Protocol, and Practice
NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures
The INE CCDP Bootcamp covers in depth information about the Nexus line from Cisco Systems. Here is some of the overview information from the course for those that desire a quick introduction to this line of equipment.
With the Nexus gear, Cisco introduces us to yet another operating system called NX-OS. The platforms that run this Operating System include:
What are the key features of this new Operating System? Capabilities include:
Our course ensures you understand all of the management interfaces available with the Nexus line. These include:
If you are interested in learning all about these network components and their differences from the IOS-based devices, be sure to attend the live course - or enjoy the On-Demand version.
