And we’re back for an exciting conclusion to the CCIE Troubleshooting series. Just what fate is in store for our heroes? Well, if we’re anything like the Friday the 13th movies, you may need to wait about 20+ years to find out! :)

You’re probably asking yourself, “With the amount of stuff we’ve gone through already, you’re really telling me there’s more to be concerned about?”

Yup, that’s exactly what I’m talking about. So what next then? How about little unexpected zingers that may just confuse, confound and otherwise astound you?

Have you ever rebooted your router/switch and had it reply to you “Would you like to enter basic management setup?” Of course you have. All the time in practice labs! But what if it happened in the middle of your actual lab exam? You DID save your configuration, right? You didn’t see any error messages pop up did you? Well, how about that?

How about a bit of preventive medicine? When you start the day, in SecureCRT that you’ll be using in the lab, I would click on the Session Options and then on the Emulation (under Terminal in the left-hand pane). See the part about Scrollback buffer. Set that to AT LEAST 5000 lines. Gives you the ability to review things you’ve typed or what the router has said, or recreate your steps in case of an emergency.

But back to this dilemma. What do you do? Run in circles, scream and shout? While that may be entertaining, it would scare everyone else in the room and not be overly helpful to your cause! Whatever you do, DO NOT enter the setup mode. Before completely panicking, just do a “show start” and see whether your config was really saved or not!

If it did not save, go check your scrollback and see what error you missed. Then hope you have enough scrollback to reconstruct your configuration. THEN run in circles, scream and shout!

You likely were victimized by the configuration-register. Well, ok, you were victimized by the Proctor’s warped sense of humor, but the weapon of choice was the configuration-register. Deal with it. You should have checked ahead of time! “show version | include Config”

Rack1R1(config)#do sh ver | in Config
Configuration register is 0x2102

That would be the sign of a good router.

0×2142 is the sign of a bad router – your startup configuration will be skipped
0×2101 is the sign of a really bad router – you will go into bootstrap or ROMMON mode

Those are the primary ones to think about. There are others that involve the changing of console speed. While particularly entertaining, being that you do not have access to the terminal server configuration or physical access to devices any longer, that’s not something you’ll be worrying about. Back in my day, that was one method of torture potentially bestowed upon us.

So what do you do if you end up in ROMMON? Partly that will depend on what routing platform you happen to be using. But let’s go with the standard ISR routers (28xx and 38xx which I believe are the currently listed devices on the blueprint for Lab Hardware).

rommon 1 >
rommon 1 > ?
alias               set and display aliases command
boot                boot up an external process
break               set/show/clear the breakpoint
confreg             configuration register utility
cont                continue executing a downloaded image
context             display the context of a loaded image
cookie              display contents of motherboard cookie PROM in hex
dev                 list the device table
dir                 list files in file system
dis                 disassemble instruction stream
dnld                serial download a program module
frame               print out a selected stack frame
help                monitor builtin command help
history             monitor command history
iomemset            set IO memory percent
meminfo             main memory information
repeat              repeat a monitor command
reset               system reset
rommon-pref         Select ROMMON
set                 display the monitor variables
showmon             display currently selected ROM monitor
stack               produce a stack trace
sync                write monitor environment to NVRAM
sysret              print out info from last system return
tftpdnld            tftp image download
unalias             unset an alias
unset               unset a monitor variable
xmodem              x/ymodem image download
rommon 2 >    

You’ll have a very specific set of commands there having to do with the boot process and/or reloading IOS through the serial port (not much fun, and not possible over telnet as configured!).

You can use the “dir flash:” command to see what files are available (or “dev” if you need to know the names of current devices like DISK0: or DISK1: for PCMCIA cards) and then “boot flash:(filename)” if there’s any doubt.

Knowing how to get out of ROMMON is a great skill. Murphy’s Law says whatever can go wrong will go wrong, and generally at the most inopportune moment! Know how to reboot!

So what if you get into a router at the beginning of your lab and you see this:

rommon 1>
rommon 1>?
Exec commands:
  access-enable    Create a temporary Access-List entry
  access-profile   Apply user-profile to interface
  call             Voice call
  clear            Reset functions
  connect          Open a terminal connection
  crypto           Encryption related commands.
  disable          Turn off privileged commands
  disconnect       Disconnect an existing network connection
  enable           Turn on privileged commands
  exit             Exit from the EXEC
  help             Description of the interactive help system
  lat              Open a lat connection
  lock             Lock the terminal
  login            Log in as a particular user
  logout           Exit from the EXEC
  modemui          Start a modem-like user interface
  mrinfo           Request neighbor and version information from a multicast
  mstat            Show statistics after multiple multicast traceroutes
  mtrace           Trace reverse multicast path from destination to source
  name-connection  Name an existing network connection
  pad              Open a X.29 PAD connection
  ping             Send echo messages
  ppp              Start IETF Point-to-Point Protocol (PPP)
  release          Release a resource
  renew            Renew a resource
  resume           Resume an active network connection
  rlogin           Open an rlogin connection
  set              Set system parameter (not config)
  show             Show running system information
  slip             Start Serial-line IP (SLIP)
  ssh              Open a secure shell client connection
  systat           Display information about terminal lines
  tclquit          Quit Tool Command Language shell
  telnet           Open a telnet connection
  terminal         Set terminal line parameters
  tn3270           Open a tn3270 connection
  traceroute       Trace route to destination
  tunnel           Open a tunnel connection
  udptn            Open an udptn connection
  where            List active connections
  x28              Become an X.28 PAD
  x3               Set X.3 parameters on PAD

rommon 1>  

That’s entirely different and you have no capability of setting the boot parameters there. Miraculously though, you do have the “enable” command.

I’ll give you a hint, there’s no enable command in ROMMON mode! You aren’t really in ROMMON, you are just being punk’d by the router.

rommon 1>enable
rommon 1>sh run | in rommon
prompt "rommon 1>"
rommon 1> 

No matter what mode you are in, that’s the prompt. Let’s exit back out though, and assume that we’re doing things our “typical” configuration fashion.

rommon 1>
rommon 1>en



The "help" PAD command signal consists of the following elements:

  is the identifier for the type of
                explanatory information requested


What the heck is that??? Ahhhh… More entertainment. That would be the X28 Diagnostic Mode (helps if you are running an X.25 PAD, but since it’s highly unlikely that normal people today even know what that is, chances are you don’t want to run it! And yet here we are. Punk’d again.

The “exit” command will get you out. Only to be placed back to your fake ROMMON prompt! Try typing “enable” fully. (By the way, the prompt won’t change, so don’t believe everything you see!)

rommon 1>enable
rommon 1>sh run | in en
Current configuration : 3980 bytes
no service password-encryption
 enrollment selfsigned
ip http authentication local
Please change these publicly known initial credentials using SDM or the IOS CLI.
alias exec en x28
rommon 1>    

Ahhhh… Aliases. Aren’t they exciting. If your proctor REALLY doesn’t like you, they’ll alias “en” and “enable” to something equally inane. But that’s a start. So let’s get rid of these things.
Don’t forget that “configure terminal” may be necessary to fully type out in case they aliased “conf” to “exit” or something fun like that!

rommon 1>
rommon 1>conf t
Enter configuration commands, one per line.  End with CNTL/Z.

rommon 1>
*Feb 17 04:35:23.451: %SYS-5-CONFIG_I: Configured from console by console
rommon 1>   

Ummmm… Did someone eat the configuration mode?

rommon 1>conf t
Enter configuration commands, one per line.  End with CNTL/Z.

interface fa0/0

ip address ?
  A.B.C.D  IP address
  dhcp     IP Address negotiated via DHCP
  pool     IP Address autoconfigured from a local DHCP pool

ip address
% Incomplete command.

rommon 1>
*Feb 17 04:36:24.755: %SYS-5-CONFIG_I: Configured from console by console
rommon 1>      

Commands appear to work, but we can’t see anything. That would be another command!

rommon 1>conf t
Enter configuration commands, one per line.  End with CNTL/Z.

do sh run | in config
Building configuration...
Current configuration : 4005 bytes
no service prompt config

Fix me!
% Invalid input detected at '^' marker.
service prompt config

Just put the service back on and we’re good. Interesting enough, the hostname shows up while in config mode. Once back in user mode, the prompt comes back.

rommon 1>
rommon 1>
rommon 1>
*Feb 17 04:38:52.911: %SYS-5-CONFIG_I: Configured from console by consoleconf t
Enter configuration commands, one per line.  End with CNTL/Z.
TestRouter(config)#default prompt
*Feb 17 04:39:04.775: %SYS-5-CONFIG_I: Configured from console by console

Other things you may have occur to your routers… Modification of the Exec-Timeout timers. Every 30 seconds may work without detection. Or if someone is really being amusing set “exec-timeout 0 1″ on the console port. Type one character every second or get kicked out.

This is a place for cut/paste if I’ve ever seen one!

There may be any number of other odd things appearing throughout your configurations. With a decent glance these (hopefully) will stand out like a sore thumb. Other than the show commands from the last two days to verify IP addressing and looking for basic “no” things in your config, you should do a “show run” on every device.

When you get into the lab exam, you really have no idea just how much configuration will be in place already. Just like any consulting engagement, you could have a completely blank greenfield deployment. Or you could walk into a semi-dysfunctional existing network to improve/fix/enhance throughout the day. Check out what you have to begin with. Make notes.

Things that are especially important as they may lead to future difficulties when you configure the tasks given to you:

1. Backup-interface configurations — These leave interfaces in a “standby” state which is most definitely not up!
2. Span or Remote-Span configurations — This may involve the copying of information from one port to another. While it’s one way, so OSPF peers won’t show up, RIP advertisements could!
3. “ip classless” command — This may have effects on your routing processes, or at least what is showing up versus expected!
4. kron jobs — I outlined this before regarding time-based redistribution, but anything pre-existing should be noted!
5. EEM (Embedded Event Manager) — Shouldn’t see these anyplace (or rarely!) – See below

Every once and a while, we hear ramblings from people insisting that the proctor got into their racks and changed configuration. Even if using Notepad to track your commands, or the scrollback buffer, there’s insistence that interfaces were configured one minute and had no configuration the next minute. There was no reboot, therefore it must have been the proctor.

While they do have a devlish glint in their eyes most of the time, and look like rather unsavory individuals, the Proctor’s job is not to interfere with anyone’s routers or switches. They have enough to do rather than resort to that level of torture! The Geneva Convention actually prohibits this type of behavior!

So ask yourself… If the proctor doesn’t get into my equipment… And I KNOW that I’ve configured things and they are working…. What could possibly be the cause of it? How about the last two things I mentioned above? Kron or EEM have the ability to execute commands, configure device changes and/or copy files from TFTP devices into the running config. You should be aware of what’s happening on a network at any point.

If you job is to evaluate and improve, it would be silly to rush off to do a list of tasks without understanding the impact along the way, wouldn’t it? Or what forces were working against you?

A simple check of the running configuration before hand can show these anomalies to you. Anything that looks strange needs to be investigated! Nothing worse than working through things in the middle of the day, then seeing:

*Feb 18 16:47:59.973: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or
*Feb 18 16:47:59.997: %PIM-5-NBRCHG: neighbor DOWN on interface FastEthernet0/0 DR
*Feb 18 16:48:00.009: %SYS-5-CONFIG_I: Configured from console by vty0
*Feb 18 16:48:05.473: %OSPF-5-ADJCHG: Process 1, Nbr on OSPF_VL4 from FULL to DOWN, Neighbor Down: Interface down or detached
*Feb 18 16:48:30.041: %SYS-5-CONFIG_I: Configured from console by vty0

And man, I’d agree. That evil proctor just jacked my lab!

Rack1R1(config)#do sh run int fa0/0
Building configuration...

Current configuration : 73 bytes
interface FastEthernet0/0
 no ip address
 duplex auto
 speed auto


Well, that’s way not cool. My scrollback even tells me what I’ve done.

Rack1R1(config)#do sh run int f0/0
Building configuration...

Current configuration : 115 bytes
interface FastEthernet0/0
 ip address
 ip pim sparse-mode
 duplex auto
 speed auto


Well, let’s put it back…

Rack1R1(config)#interface FastEthernet0/0
Rack1R1(config-if)# ip address
Rack1R1(config-if)# ip pim sparse-mode
Rack1R1(config-if)# duplex auto
Rack1R1(config-if)# speed auto
*Feb 18 16:51:22.753: %PIM-5-NBRCHG: neighbor UP on interface FastEthernet0/0
*Feb 18 16:51:22.757: %PIM-5-DRCHG: DR change from neighbor to on interface FastEthernet0/0
*Feb 18 16:51:27.501: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from LOADING to FULL, Loading Done
Rack1R1#do sh run int f0/0
*Feb 18 16:51:29.921: %OSPF-5-ADJCHG: Process 1, Nbr on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or
*Feb 18 16:51:29.929: %PIM-5-NBRCHG: neighbor DOWN on interface FastEthernet0/0 DR
*Feb 18 16:51:30.005: %SYS-5-CONFIG_I: Configured from console by vty0
Rack1R1(config)#do sh run int f0/0
Building configuration...

Current configuration : 73 bytes
interface FastEthernet0/0
 no ip address
 duplex auto
 speed auto


Our neighbors come back, things are good again… But then it’s back down right away. Or it could be much later. Either way, the same frustration ensues!

Rack1R1(config)#do sh run | s event
event manager applet NeverTrustTheseThings
 event timer watchdog time 300
 action 1.0 cli command "enable"
 action 2.0 cli command "configure terminal"
 action 3.0 cli command "default interface FastEthernet0/0"

Well, that would certainly do it. And it may have been hidden before by a startup “logging console warnings” or something like that.

EEM is a POWERFUL tool. Check out the Network Management section of your Documentation. http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html#wp1157622

Rack1R1(config)#no event manager applet NeverTrustTheseThings
Rack1R1(config)#do sh run | s event

So all during the first 30-45 minutes of your lab exam (after the Core Technology Q&A), you should be:

1. Reading through the whole exam
2. Taking notes to remind yourself of things later
3. Redraw the diagram quickly so you can write on it
4. Verify IP addresses quite simply
5. Identify major things in the configs beginning with “no” or altering the configuration register
6. Quickly do “show run” on all devices and scan through for anything that looks strange or out of place
7. Get ready to kick butt and take your number home!

We get stuck in ruts when going through practice labs. We have a process generally dictated by what level of preparation that we’ve done. While there are certainly a good number of tools and labs and documents out there to help you study, there is nothing that compares to the “personalized approach”. By that, I mean, make changes yourself. Or have a buddy make some additional tweaks, tasks, changes to labs for you.

Keep in mind that folks on the CCIE team probably have EVERYONE’s study materials. So while we are indeed pretty cool, they’ll go out of their way to find something we didn’t think of. So outsmart them! Process, process, process.

Good luck in your lab prep, and most certainly in the troubleshooting portion! Don’t forget, this addresses nothing about the self-induced troubleshooting. While going through your lab, you should be verifying things every step of the way with show or debug (or whatever) commands. If properly verified, you will have few, if any, surprises during your lab.

You can leave a response, or trackback from your own site.

10 Responses to “CCIE Troubleshooting: Part 3 (Ummmm…. Houston, We Have a Problem)”

  1. Ioan Branet says:

    Excellent material.
    Thank you for it, I found it very interesting …and your style is unique :)

  2. Mihai says:


    Did I mentioned last time your great style? Sure I did…

  3. Masood says:

    another useful and intersting blog..It would be great if you can write some stuff on routing protocol adjacencies and advertisement Gotcha’s. Like in case of BGP, sometimes you clear a neighbor but i did not work (get routes) clearing same neighbor on the other side would work (mpls vpn).. hope u got it..

  4. Vlad says:

    How often should one expect to run into some of these stuff in the lab?

    It was really cool to see how to play with a friend…but should we really expect this in the lab?

  5. steven says:

    hey, don’t show this stuff here ,-) proctors may read this and get new evil ideas ,-))

    great work! funny to see what kind of ‘evilness’ is possible on those devices ,-)

  6. bam says:

    One other thing that ‘might could use’ a writeup of its own is ‘config confirm’. The symptom — you save the config and then it doesn’t take because the config revert timer kicked off before the unsuspecting engineer could type ‘config confirm’. A barrel of laughs, that one. It would be great if you could include an example on how to create a remote archive too.
    BTW — the formatting on your router output is spectacular!

  7. See, I DO learn new things now and then. :) (Thank BM for helping me on the formatting part!)

    I’ll note a few things for future blogs here and will move forward.

    Vlad – As for expectations in the lab, anything is fair game. These really aren’t new things. These aren’t even new concepts in terms of the CCIE lab. So be prepared. Or as GI Joe says, “Knowing is half the battle.”

  8. Bhuvanesh says:

    These troubleshooting steps are awesome and really wonderful not only lab perspective but also in production network.

    EEM and Kron are undoubtedly outstanding………..

    Thanks a lot for sharing……..


    ……..Simply Superb……….

  9. Swap says:

    awesome stuff..:)

    Hi – that’s the first technical post i saw from you tht doesnt have a diplomatic angle to it..pure tech stuff unlike ur other posts :)

  10. Does the OP know any good dragon gaming websites.


Leave a Reply


CCIE Bloggers