Things sure can get spooky in the Troubleshooting section of the new Version 4 Blueprint! Cisco can present a pretty vague issue; give you a very lame diagram; and then really press you for time to solve the Trouble Ticket. In this blog post, I will walk you through this graveyard and attempt to provide some ideas on an efficient and effective approach. For much more detail and practice, our premier products for this exam section are the Volume 4 workbook (currently being edited and improved upon), and the brand new 5-Day Troubleshooting Bootcamp.
Are you ready to find prefixes that go bump in the night? Here is the sample Trouble Ticket we will attack, and the appropriate portion of the Cisco diagram. You will want to have some scratch paper handy (just like in the actual exam). Diagramming can prove to be more important here then in any exam section. You should practice a diagram now based on the show output that follows.
Trouble Ticket 1
Vampires located in VLAN 666 (behind Amityville) are unable to access any resources located behind Transylvania. Correct this issue without static routing, additional routing protocols, or redistribution of any kind.
The first order of business is to “expand” upon their diagram and determine what protocols are in use and where. A quick show ip protocols on each of the devices in the transit path should do the trick.
Transylvania#show ip protocols Routing Protocol is "eigrp 100" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Default networks flagged in outgoing updates Default networks accepted from incoming updates EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0 EIGRP maximum hopcount 100 EIGRP maximum metric variance 1 Redistributing: eigrp 100 EIGRP NSF-aware route hold timer is 240s Automatic network summarization is not in effect Maximum path: 4 Routing for Networks: 53.234.10.23/32 Routing Information Sources: Gateway Distance Last Update Distance: internal 90 external 170 Routing Protocol is "bgp 65001" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set IGP synchronization is disabled Automatic route summarization is disabled Unicast Aggregate Generation: 10.10.0.0/16 summary-only Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap 53.234.10.44 Maximum path: 1 Routing Information Sources: Gateway Distance Last Update Distance: external 20 internal 200 local 200 HalloweenTown#show ip protocols Routing Protocol is "eigrp 100" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Default networks flagged in outgoing updates Default networks accepted from incoming updates EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0 EIGRP maximum hopcount 100 EIGRP maximum metric variance 1 Redistributing: eigrp 100 EIGRP NSF-aware route hold timer is 240s Automatic network summarization is not in effect Maximum path: 4 Routing for Networks: 53.234.10.44/32 Routing Information Sources: Gateway Distance Last Update Distance: internal 90 external 170 Routing Protocol is "bgp 65001" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set IGP synchronization is disabled Automatic route summarization is disabled Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap 23.10.24.190 53.234.10.23 Maximum path: 1 Routing Information Sources: Gateway Distance Last Update Distance: external 20 internal 200 local 200 Salem#show ip protocols Routing Protocol is "ospf 1" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Router ID 66.150.201.12 Number of areas in this router is 1. 1 normal 0 stub 0 nssa Maximum path: 4 Routing for Networks: Routing on Interfaces Configured Explicitly (Area 0): FastEthernet0/0 Reference bandwidth unit is 100 mbps Routing Information Sources: Gateway Distance Last Update 102.34.102.12 110 00:30:55 Distance: (default is 110) Routing Protocol is "bgp 65000" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set IGP synchronization is disabled Automatic route summarization is disabled Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap 23.10.24.200 Maximum path: 1 Routing Information Sources: Gateway Distance Last Update Distance: external 20 internal 200 local 200 CrystalLake#show ip protocols Routing Protocol is "ospf 1" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Router ID 102.34.102.1 Number of areas in this router is 1. 1 normal 0 stub 0 nssa Maximum path: 4 Routing for Networks: Routing on Interfaces Configured Explicitly (Area 0): FastEthernet0/1 FastEthernet0/0 Reference bandwidth unit is 100 mbps Routing Information Sources: Gateway Distance Last Update Distance: (default is 110) Routing Protocol is "bgp 65000" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set IGP synchronization is disabled Automatic route summarization is disabled Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap 66.150.201.12 Address FiltIn FiltOut DistIn DistOut Weight RouteMap 102.34.102.12 Maximum path: 1 Routing Information Sources: Gateway Distance Last Update Distance: external 20 internal 200 local 200 Amityville#show ip protocols Routing Protocol is "ospf 1" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set Router ID 102.34.102.12 Number of areas in this router is 1. 1 normal 0 stub 0 nssa Maximum path: 4 Routing for Networks: Routing on Interfaces Configured Explicitly (Area 0): FastEthernet0/0 Reference bandwidth unit is 100 mbps Routing Information Sources: Gateway Distance Last Update 102.34.102.1 110 00:33:23 Distance: (default is 110) Routing Protocol is "bgp 65000" Outgoing update filter list for all interfaces is not set Incoming update filter list for all interfaces is not set IGP synchronization is disabled Automatic route summarization is disabled Neighbor(s): Address FiltIn FiltOut DistIn DistOut Weight RouteMap 102.34.102.1 Maximum path: 1 Routing Information Sources: Gateway Distance Last Update Distance: external 20 internal 200 local 200
As I parse the results, I create a more detailed diagram to provide a better picture of what is going on. Looking at the diagram, I immediately contemplate some design issues that could come into play. The iBGP split-horizon rule, next-hop reachability, and synchronization all could come into play here given the protocol configuration.
I decide to start as close the destination as possible and examine the relevant configurations.
Transylvania#show run | section bgp router bgp 65001 no synchronization bgp log-neighbor-changes network 10.0.0.0 aggregate-address 10.10.0.0 255.255.0.0 summary-only neighbor 53.234.10.44 remote-as 65001 no auto-summary
It appears the intent is to advertise prefixes (the Loopback interfaces) beyond Transylvania using BGP prefix aggregation. Let me ensure that configuration is done correctly.
Sure enough, here is our first BGP configuration issue. We need to advertise a component prefix of the aggregate. The little gremlin that attempted this configuration tried to accomplish this with the network statement, but they got it wrong. Subnetting is being done here of the 10.x.x.x space, so you need to reconfigure the network statement as follows:
Transylvania#show ip int brief Interface IP-Address OK? Method Status Protocol FastEthernet0/0 53.234.10.23 YES manual up up FastEthernet0/1 unassigned YES unset administratively down down Loopback1 10.10.10.1 YES manual up up Loopback2 10.10.11.1 YES manual up up Loopback3 10.10.12.1 YES manual up up Transylvania#configure terminal Enter configuration commands, one per line. End with CNTL/Z. Transylvania(config)#router bgp 65001 Transylvania(config-router)#network 10.10.10.0 mask 255.255.255.0
After making this change, I check Salem to see if it is receiving the aggregate from Transylvania and we have reachability:
Salem#show ip bgp BGP table version is 2, local router ID is 66.150.201.12 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path *> 10.10.0.0/16 23.10.24.200 0 65001 i Salem#show ip route bgp 10.0.0.0/16 is subnetted, 1 subnets B 10.10.0.0 [20/0] via 23.10.24.200, 00:06:26 Salem#ping 10.10.10.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.10.10.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 28/33/40 ms
We are on our way to solving this Trouble Ticket…we hope you will follow this blog and continue this (and more) troubleshooting scenarios!
About Anthony Sequeira, #15626:
Anthony Sequeira brings decades of teaching, technical writing, and consulting experience to INE. Anthony began his career as an author and lecturer within the IT community, featuring best-selling titles for Microsoft and Cisco Press. Best known as one of the training voices for the revolutionary e-learning company called KnowledgeNet, Anthony now teaches online and in-classroom exclusively for INE. When not helping his students master Cisco networking, Anthony can be found at the poker tables, or flying the Florida skies in a Cessna.
You can leave a response, or trackback from your own site.
15 Responses to “CCIE R&S Troubleshooting – Sample Trouble Tickets – Part 1”
Leave a Reply
Loading ...
Salem, MA on Halloween… one has to wonder!
On the other hand, it’s a really good thing Anthony wasn’t driving through Pennsylvania at some point in time… Moving through Blue Ball, PA, on to Intercourse, PA and (in theory) finally reaching Paradise, PA, all while avoiding Bird In Hand, PA.
Ahhhh… The trouble tickets that could arise!
Intercourse, PA? Interesting….
hi Anthony,
As we have learnt from V4 victims that there are 30-40 routers and its difficult to find which routers are being effected by the trouble ticket in hand. Is this wise move to draw diagrams for all 10-12 tickets in 120 minutes time frame?
Thanx Anthony
If truble tickets are of this difficulty level then i think i have good chances of acing this section
Haha great stuff Anthony!!!
Loved the horror references!!
Scott are you serious about the Blue Ball PA
and other names, jeez someone must have had a sense of humor naming those locations!
Yes I think the trouble tickets should have good names associated with them, definitely makes them more interesting and enjoyable
Excellent question Nadeem! Just like in the Configuration section – you should draw your own diagram when it is needed. Some Trouble Tickets I would not be able to solve without my own diagram – some will not need one.
Until now, every one get failed. And every one is saying same that even reading troubleshooting twice should be considered a wastage of time.
Same goes with config section, few have drawn their own topologies and at the end they have to regret their decision to do so.
I am not saying your approach is bad, but we should know what is right direction to go with V4.
Are you going to release mock interface for V4
Yeah, Scott is serious about those towns. I live about an hour from most of them. There are all outside of Philadelphia in PA Dutch Country near Lancaster. He left off Smoketown. Cheech and Chong hang out there sometimes
Hi Anthony,
This is good methodology to solve the trouble ticket. Please teach us more for the rest tickets, as well as the 4 switch troubleshoot tickets.
BTW: I failed the lab on 10/27/09 due to limited resource for lab prep by myself. I now joined INE R&S 2.0 program. I think I will be fine for the next lab attempt. Please help me.
Thanks
Dengming
hi Anthony,
Why did you use /24 mask in the bgp network command?
Thanks
Thanks Anthony for another good example the troubleshooting thought/flow process. Sadly, Scott is correct. I live about 20 minutes from all those funny towns in PA. He did forget to mention one of them, Climax PA.
I had to go and see if those were real cities in PA:
http://www.bird-in-hand.com/
http://en.wikipedia.org/wiki/Blue_Ball,_Pennsylvania
http://www.800padutch.com/intercourse.shtml
Even funnier that places exist. You know what would be even funnier? If Scott pulled this out of his Butte, Montana.
Too funny.
Hi Mohammad – we had to correct the network command they tried to use. They used the “classful” network command of 10.0.0.0/8, but the underlying prefixes are of /24, so we need to advertise one of them using the proper mask in order for the aggregate to work.
The new mock labs are indeed done with all materials on screen. At some point – we will create a flash based interface to mimic that of the lab I suppose.
Nice one,
Best Regards,
Khawaja Ahmed
CCNP,
ahmedali497@hotmial.com