Oct
31

Things sure can get spooky in the Troubleshooting section of the new Version 4 Blueprint! Cisco can present a pretty vague issue; give you a very lame diagram; and then really press you for time to solve the Trouble Ticket. In this blog post, I will walk you through this graveyard and attempt to provide some ideas on an efficient and effective approach. For much more detail and practice, our premier products for this exam section are the Volume 4 workbook (currently being edited and improved upon), and the  brand new 5-Day Troubleshooting Bootcamp.

Are you ready to find prefixes that go bump in the night? Here is the sample Trouble Ticket we will attack, and the appropriate portion of the Cisco diagram. You will want to have some scratch paper handy (just like in the actual exam). Diagramming can prove to be more important here then in any exam section. You should practice a diagram now based on the show output that follows.

Screen shot 2009-10-31 at 5.44.38 PM

Trouble Ticket 1
Vampires located in VLAN 666 (behind Amityville)  are unable to access any resources located behind Transylvania. Correct this issue without static routing, additional routing protocols, or redistribution of any kind.

The first order of business is to “expand” upon their diagram and determine what protocols are in use and where. A quick show ip protocols on each of the devices in the transit path should do the trick.

Transylvania#show ip protocols
Routing Protocol is "eigrp 100"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 Default networks flagged in outgoing updates
 Default networks accepted from incoming updates
 EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
 EIGRP maximum hopcount 100
 EIGRP maximum metric variance 1
 Redistributing: eigrp 100
 EIGRP NSF-aware route hold timer is 240s
 Automatic network summarization is not in effect
 Maximum path: 4
 Routing for Networks:
 53.234.10.23/32
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: internal 90 external 170

Routing Protocol is "bgp 65001"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 IGP synchronization is disabled
 Automatic route summarization is disabled
 Unicast Aggregate Generation:
 10.10.0.0/16       summary-only
 Neighbor(s):
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 53.234.10.44                                         
 Maximum path: 1
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: external 20 internal 200 local 200

HalloweenTown#show ip protocols
Routing Protocol is "eigrp 100"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 Default networks flagged in outgoing updates
 Default networks accepted from incoming updates
 EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
 EIGRP maximum hopcount 100
 EIGRP maximum metric variance 1
 Redistributing: eigrp 100
 EIGRP NSF-aware route hold timer is 240s
 Automatic network summarization is not in effect
 Maximum path: 4
 Routing for Networks:
 53.234.10.44/32
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: internal 90 external 170

Routing Protocol is "bgp 65001"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 IGP synchronization is disabled
 Automatic route summarization is disabled
 Neighbor(s):
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 23.10.24.190                                         
 53.234.10.23                                         
 Maximum path: 1
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: external 20 internal 200 local 200

Salem#show ip protocols
Routing Protocol is "ospf 1"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 Router ID 66.150.201.12
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Maximum path: 4
 Routing for Networks:
 Routing on Interfaces Configured Explicitly (Area 0):
 FastEthernet0/0
 Reference bandwidth unit is 100 mbps
 Routing Information Sources:
 Gateway         Distance      Last Update
 102.34.102.12        110      00:30:55
 Distance: (default is 110)

Routing Protocol is "bgp 65000"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 IGP synchronization is disabled
 Automatic route summarization is disabled
 Neighbor(s):
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 23.10.24.200                                         
 Maximum path: 1
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: external 20 internal 200 local 200

CrystalLake#show ip protocols
Routing Protocol is "ospf 1"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 Router ID 102.34.102.1
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Maximum path: 4
 Routing for Networks:
 Routing on Interfaces Configured Explicitly (Area 0):
 FastEthernet0/1
 FastEthernet0/0
 Reference bandwidth unit is 100 mbps
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: (default is 110)

Routing Protocol is "bgp 65000"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 IGP synchronization is disabled
 Automatic route summarization is disabled
 Neighbor(s):
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 66.150.201.12                                        
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 102.34.102.12                                        
 Maximum path: 1
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: external 20 internal 200 local 200

Amityville#show ip protocols
Routing Protocol is "ospf 1"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 Router ID 102.34.102.12
 Number of areas in this router is 1. 1 normal 0 stub 0 nssa
 Maximum path: 4
 Routing for Networks:
 Routing on Interfaces Configured Explicitly (Area 0):
 FastEthernet0/0
 Reference bandwidth unit is 100 mbps
 Routing Information Sources:
 Gateway         Distance      Last Update
 102.34.102.1         110      00:33:23
 Distance: (default is 110)

Routing Protocol is "bgp 65000"
 Outgoing update filter list for all interfaces is not set
 Incoming update filter list for all interfaces is not set
 IGP synchronization is disabled
 Automatic route summarization is disabled
 Neighbor(s):
 Address          FiltIn FiltOut DistIn DistOut Weight RouteMap
 102.34.102.1                                         
 Maximum path: 1
 Routing Information Sources:
 Gateway         Distance      Last Update
 Distance: external 20 internal 200 local 200

As I parse the results, I create a more detailed diagram to provide a better picture of what is going on. Looking at the diagram, I immediately contemplate some design issues that could come into play. The iBGP split-horizon rule, next-hop reachability, and synchronization all could come into play here given the protocol configuration.

I decide to start as close the destination as possible and examine the relevant configurations.

Transylvania#show run | section bgp
router bgp 65001
 no synchronization
 bgp log-neighbor-changes
 network 10.0.0.0
 aggregate-address 10.10.0.0 255.255.0.0 summary-only
 neighbor 53.234.10.44 remote-as 65001
 no auto-summary

It appears the intent is to advertise prefixes (the Loopback interfaces) beyond Transylvania using BGP prefix aggregation. Let me ensure that configuration is done correctly.

Sure enough, here is our first BGP configuration issue. We need to advertise a component prefix of the aggregate. The little gremlin that attempted this configuration tried to accomplish this with the network statement, but they got it wrong. Subnetting is being done here of the 10.x.x.x space, so you need to reconfigure the network statement as follows:

Transylvania#show ip int brief
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            53.234.10.23    YES manual up                    up      
FastEthernet0/1            unassigned      YES unset  administratively down down    
Loopback1                  10.10.10.1      YES manual up                    up      
Loopback2                  10.10.11.1      YES manual up                    up      
Loopback3                  10.10.12.1      YES manual up                    up      
Transylvania#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Transylvania(config)#router bgp 65001
Transylvania(config-router)#network 10.10.10.0 mask 255.255.255.0

After making this change, I check Salem to see if it is receiving the aggregate from Transylvania and we have reachability:

Salem#show ip bgp
BGP table version is 2, local router ID is 66.150.201.12
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
 r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

 Network          Next Hop            Metric LocPrf Weight Path
*> 10.10.0.0/16     23.10.24.200                           0 65001 i
Salem#show ip route bgp
 10.0.0.0/16 is subnetted, 1 subnets
B       10.10.0.0 [20/0] via 23.10.24.200, 00:06:26
Salem#ping 10.10.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/33/40 ms

We are on our way to solving this Trouble Ticket…we hope you will follow this blog and continue this (and more) troubleshooting scenarios!


You can leave a response, or trackback from your own site.

12 Responses to “CCIE R&S Troubleshooting – Sample Trouble Tickets – Part 1”

 
  1. Nadeem Rafi says:

    Hi,
    As we have learnt from V4 victims that there are 30-40 routers and its difficult to find which routers are being effected by the trouble ticket in hand. Is this wise move to draw diagrams for all 10-12 tickets in 120 minutes time frame?

    • Excellent question Nadeem! Just like in the Configuration section – you should draw your own diagram when it is needed. Some Trouble Tickets I would not be able to solve without my own diagram – some will not need one.

  2. kim says:

    Thanx,
    If truble tickets are of this difficulty level then i think i have good chances of acing this section :-)

  3. Ian Finlayson says:

    Haha great stuff!!! :) Loved the horror references!!

    Yes I think the trouble tickets should have good names associated with them, definitely makes them more interesting and enjoyable ;)

  4. Nadeem Rafi says:

    Until now, every one get failed. And every one is saying same that even reading troubleshooting twice should be considered a wastage of time.
    Same goes with config section, few have drawn their own topologies and at the end they have to regret their decision to do so.
    I am not saying your approach is bad, but we should know what is right direction to go with V4.
    Are you going to release mock interface for V4 :)

  5. Dengming Chen says:

    Hi,

    This is good methodology to solve the trouble ticket. Please teach us more for the rest tickets, as well as the 4 switch troubleshoot tickets.

    BTW: I failed the lab on 10/27/09 due to limited resource for lab prep by myself. I now joined INE R&S 2.0 program. I think I will be fine for the next lab attempt. Please help me.

    Thanks
    Dengming

  6. Mohammad Itani says:

    Hi,
    Why did you use /24 mask in the bgp network command?
    Thanks

    • Hi Mohammad – we had to correct the network command they tried to use. They used the “classful” network command of 10.0.0.0/8, but the underlying prefixes are of /24, so we need to advertise one of them using the proper mask in order for the aggregate to work.

  7. shaun says:

    Thanks for another good example the troubleshooting thought/flow process. I live about 20 minutes from all those funny towns in PA. He did forget to mention one of them, Climax PA.

  8. Net_OG says:

    I had to go and see if those were real cities in PA:

    http://www.bird-in-hand.com/

    http://en.wikipedia.org/wiki/Blue_Ball,_Pennsylvania

    http://www.800padutch.com/intercourse.shtml

    Even funnier that places exist. You know what would be even funnier? If Scott pulled this out of his Butte, Montana.

    Too funny.

  9. Khawaja Ahmed Ali says:

    Nice one,

    Best Regards,
    Khawaja Ahmed
    CCNP,
    ahmedali497@hotmial.com

 

Leave a Reply

Categories

CCIE Bloggers