Feb
16

There are two types of troubleshooting that you’ll run into on the CCIE lab:

1. The “Proctor is Evil” Troubleshooting
2. The “Self-Induced” Troubleshooting

The latter type is by far the more time-consuming but also the most important. Basically you messed something up, therefore you have to fix it! (At least if you want the points) The reason it is the most time-consuming is because it could be ANY silly mistake or combination of silly mistakes along the way, and there is no predicting what kinds of things can be done to mess with your own head!

The most important rule with this kind of troubleshooting is time management. Set a time limit of 15 minutes. If you can’t figure something out in 15 minutes (no, I don’t care how “close” you think you are!) go do something else. Whether this involves a bathroom break, a soda/snack break, standing on your head on the high-quality lab chairs or simply moving on to another “service” or “security” task of your lab makes no difference. The idea is to separate your brain from staring at the same thing over and over.

The longer you stare at something, the more you see what you want and not what’s really there. Most self-induced errors are really small, and fairly inane. You know. Those “DUH!” moments once we figure it out. But time management is the consequence we suffer due to silly mistakes. Avoid it!

Anyway… on to the more exciting things. The unpredictable nature of the “Proctor is Evil” Troubleshooting. Having started my training career specializing in the old CIT (Cisco Internetwork Troubleshooting) class, I can greatly appreciate some of the humorous things that MAY get thrown into lab exams. The question becomes, if I don’t know what they are and there are many different things that could go wrong… What the heck do I do about it?!?!

Excellent question! Process, my dear. It’s all about process.

We’ve had many different posts and blogs about the things you should be doing in your lab exam. You know… The stuff about reading your exam, re-drawing your diagrams, L2 diagramming, etc. Been there, done that. But then there’s troubleshooting. Some things are easy to spot. Others are not! Some things may obvious, others are just plain obnoxious.

The one thing to remember is that during the lab exam, you are there for your proctor’s entertainment. As long as this goes through your head, you’ll understand why some things are done. It’s all in good fun! And once you pass the CCIE Lab exam, it’s much easier to laugh about it all!

Start with the basics. IP addressing. You are working on your diagram anyway before your lab. Seriously. Troubleshooting and Diagramming all in one step! This is a good time to quickly verify things; A few different commands to think about, and some differences between them.

1. show ip interface brief | exclude unassigned
2. show interface | include Internet
3. show run | include interface|ip address

Each is slightly different.

Rack1R3(config)#do sh ip int br | exc una
Interface                  IP-Address      OK? Method Status                Protocol
FastEthernet0/0            145.1.36.3      YES manual up                    up
FastEthernet0/1            145.1.3.3       YES manual up                    up
Serial1/2.13               145.1.13.3      YES manual up                    up
Serial1/3.23               145.1.23.3      YES manual up                    up
Loopback0                  150.1.3.3       YES manual up                    up
Loopback111                145.1.133.3     YES manual up                    up
Rack1R3(config)#

This gives me simple IP addresses to check against.

Rack1R3(config)#do sh int | in Internet
Internet address is 145.1.36.3/24
Internet address is 145.1.3.3/24
Internet address is 145.1.13.3/24
Internet address is 145.1.23.3/24
Internet address is 150.1.3.3/24
Internet address is 145.1.133.3/24
Rack1R3(config)#

This gives me IP addresses with VLSM masks that may well be present in my diagram. More information to check. Context can be provided by:

Rack1R3(config)#do sh int | in Internet|is (up|down)
FastEthernet0/0 is up, line protocol is up
Internet address is 145.1.36.3/24
FastEthernet0/1 is up, line protocol is up
Internet address is 145.1.3.3/24
Serial0/0/0 is administratively down, line protocol is down
Serial0/1/0 is administratively down, line protocol is down
Serial0/2/0 is administratively down, line protocol is down
Serial0/2/1 is administratively down, line protocol is down
Serial1/0 is administratively down, line protocol is down
Serial1/1 is administratively down, line protocol is down
Serial1/2 is up, line protocol is up
Serial1/2.13 is up, line protocol is up
Internet address is 145.1.13.3/24
Serial1/3 is up, line protocol is up
Serial1/3.23 is up, line protocol is up
Internet address is 145.1.23.3/24
NVI0 is up, line protocol is up
Loopback0 is up, line protocol is up
Internet address is 150.1.3.3/24
Loopback111 is up, line protocol is up
Internet address is 145.1.133.3/24
Rack1R3(config)#

(I think this is the best option, by the way!)

Rack1R3(config)#do sh run | in interface|ip address
ip telnet source-interface Loopback111
ip ftp source-interface Loopback0
interface Loopback0
ip address 150.1.3.3 255.255.255.0
interface Loopback111
ip address 145.1.133.3 255.255.255.0
interface FastEthernet0/0
ip address 145.1.36.3 255.255.255.0
interface FastEthernet0/1
ip address 145.1.3.3 255.255.255.0
interface Serial0/0/0
no ip address
interface Serial0/1/0
no ip address
interface Serial0/2/0
no ip address
interface Serial0/2/1
no ip address
interface Serial1/0
no ip address
interface Serial1/1
no ip address
interface Serial1/2
no ip address
frame-relay route 112 interface Serial1/3 121
interface Serial1/2.13 point-to-point
ip address 145.1.13.3 255.255.255.0
frame-relay interface-dlci 131
interface Serial1/3
no ip address
frame-relay route 121 interface Serial1/2 112
interface Serial1/3.23 point-to-point
ip address 145.1.23.3 255.255.255.0
frame-relay interface-dlci 132
passive-interface default
no passive-interface FastEthernet0/0
ip nat inside source list 111 interface Loopback0 overload
match ip address 101
match ip address prefix-list NAT-Route
Rack1R3(config)#

This one allows you to see the dotted decimal portion of IPs.

You may do whatever you like the best, but this way you can quickly look at your diagrams and see what’s happening. Check to be sure that your routers match your diagram.

Checking IP address only is nice. Checking the IP and Mask is better as you may not “discover” the error until your routing protocol configurations. At that point, you aren’t thinking about troubleshooting or IP addresses any longer, so it takes more time to figure things out.

Keep it simple!

The next important stage is the L2 diagramming. Your lab may or may not include any sort of physical connection diagram. While your nice L3 diagram may show a link between SW3 and R2, unless they are physically connected to each other, there may be important steps missing in your functionality! The easiest thing to do is go to your switches and use the “show cdp neighbor” command.

A switch’s interfaces are up by default (unlike a routers’). Although, you may have discovered any “down” interfaces by your IP address checks, it’s good to see anyway.

Rack1SW1#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Rack1SW4         Fas 0/19          160          R S I     WS-C3550- Fas 0/13
Rack1SW3         Fas 0/16          174           S I      WS-C3560- Fas 0/13
emanon-3750-Dev  Gig 0/2           170           S I      WS-C3750- Gig 1/0/2
Rack1R1          Fas 0/1           176          R S I     2811      Fas 0/0
Rack1R3          Fas 0/3           119          R S I     2811      Fas 0/0
Rack1R5          Fas 0/5           129          R S I     2811      Fas 0/0
Rack1SW1#

Rack1SW2#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
RS.1.1.BB2       Fas 0/24          156          R S I     2811      Fas 0/0
Rack1SW4         Fas 0/19          144          R S I     WS-C3550- Fas 0/16
Rack1SW3         Fas 0/16          124           S I      WS-C3560- Fas 0/16
Rack1R2          Fas 0/2           157          R S I     2811      Fas 0/0
Rack1R4          Fas 0/4           172          R S I     2811      Fas 0/0
Rack1SW2#

Rack1SW3#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
RS.1.1.BB3       Fas 0/24          130          R S I     2811      Fas 0/0
Rack1SW4         Fas 0/20          177          R S I     WS-C3550- Fas 0/20
Rack1SW4         Fas 0/19          177          R S I     WS-C3550- Fas 0/19
Rack1SW2         Fas 0/16          159          R S I     WS-C3560- Fas 0/16
Rack1SW1         Fas 0/13          174          R S I     WS-C3560- Fas 0/16
Rack1R3          Fas 0/4           163          R S I     2811      Fas 0/1
Rack1R5          Fas 0/6           132          R S I     2811      Fas 0/1
Rack1SW3#

Rack1SW4#sh cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Rack1SW2         Fas 0/16          145          R S I     WS-C3560- Fas 0/19
Rack1SW3         Fas 0/20          170           S I      WS-C3560- Fas 0/20
Rack1SW3         Fas 0/19          170           S I      WS-C3560- Fas 0/19
Rack1SW1         Fas 0/13          161          R S I     WS-C3560- Fas 0/19
Rack1R4          Fas 0/4           159          R S I     2811      Fas 0/1
Rack1R6          Fas 0/6           165          R S I     3825      Gig 0/1
Rack1SW4#

Draw yourself a quick diagram and center the drawing around your switches. Use colors as needed for trunk vs. access ports. Use colors for ISL vs. 802.1Q trunks. Do whatever makes sense for you. But this too will assist in troubleshooting connections (or lack thereof) later in the lab.

Make sure the interfaces match what you are asked for in your lab. Whether on the diagram for Layer3, or in the lab task list for Layer2 connections. Are ports missing? Some quick “show run interface (intf)” commands may help out there, or at least enough to make a quick note to check later while configuring the trunking section!

Those two discovery items may well handle some of your Troubleshooting tasks, and will certainly save time in later verification or Self-Induced Troubleshooting!

That’s enough to whet the appetite a little bit. More on the notorious and/or obnoxious troubleshooting next time. Same Bat time…. Same Bat channel…. ;)


You can leave a response, or trackback from your own site.

13 Responses to “CCIE Troubleshooting: Part 1 (Is it really my fault?)”

 
  1. Jun Prieto says:

    Thanks! This is very helpful and very informative. L2 diagramming, you hit it right on the spot. I always admire how you present your topics.

  2. Nick says:

    Wow i was just going to post a question about proposed initial troubleshooting checklist…

  3. ccie_boy says:

    thanx,
    i had a strange experience with my 3550 a week ago , after reload the running config did not vanished as supposed . Is this any way related to config registry ?
    my question may be silly but wil certainly help if i found myself in similar situation in lab . someome plz explain
    cheers

  4. work4ccie says:

    Thanks.
    This is very helpful!

  5. bam says:

    Hi — any chance you can format your router output in “Courier New”, “Lucida Console” or some other monospaced font? It’s not easy to read router output in proportional fonts.

  6. Bam, Excellent question and I have no earthly idea. But I’ll find out and will edit ASAP!

    ;) (I try to learn something new every day!)

  7. Mido says:

    Thanks,

    IS this method or approach also applicable for the other tracks? (Security and voice)

    I mean if I am doing the security lab, should I start the lab by doing L2 and L3 troubleshooting?

  8. Naveen says:

    Thanks a lot for these helping tips.

  9. Mido,

    I would find it unlikely to work into the Security labs or Voice labs to the same degree it does in R&S. Should you be vaguely aware of things? Sure. But the blueprint and thought process for those labs is a bit different.

  10. Ian says:

    Hi,

    Great articles as always!
    One other thing I was thinking that is probably a good thing to do (To save some time) while you are displaying the Interface commands to check IP Address and Masks etc, is to do the ALT copy with CRT and place each IP Address into a text file so it can help you build your TCL scripts (for ping tests) later etc..

    Keep up the good posts!

    Rgds,
    Ian.

  11. Carlos says:

    Very good tips to save time. Another command worth using is “alias exec” You can save a lot of typing with something like this:
    alias exec sib show ip interface brief
    Cheers!

  12. Jaeson V says:

    For a technical fellow, you’re indeed hilarious. I was about to fall asleep reading through docs and searching in the Internet then I come across your blog and then you put a smile to my face. :P

 

Leave a Reply

Categories

CCIE Bloggers