Jan
11

UPDATE: I have received numerous submissions and currently in the process of reviewing them. I'm going to extend the deadline until Wednesday (2012-01-18). At that time all people who submitted working solutions will be awarded 100 tokens!

Recently I have been working with a large enterprise customer that is looking to implement a new change control policy. The main goal of the policy is to be able to track who is making changes to devices in the network, and specifically what those changes are. As opposed to using a full blown network management suite to do this for them, I suggested a simple solution of using TACACS for exec and command accounting (all devices are Cisco), and EEM scripting along with a TFTP server for tracking the actual configuration changes in case they need to roll back to a well-known good working config. The final result worked out very well, and I thought it would make a good CCIE level challenge as well.

So here is the challenge - write an EEM script to manage change control in the network as follows. The first person to submit a working script will win 100 rack rental tokens valid for any rack rental or mock lab session.

Every time a user makes a change to the configuration, the router should automatically TFTP its running configuration to the TFTP server 10.0.0.1 using the following naming convention:

HOSTNAME.YYYY-MM-DD.HHhMMmSSs.ADMIN_NAME.working.cfg

This ensures that if a change is made to the network but not actually saved to NVRAM, and there is a device crash, you can recover the last working running config of the device. Also this naming format tells you when exactly the change was made and by who. Remember that the router always generates a %SYS-5-CONFIG log message when a change is made. So for example suppose the following change was made:

EDGE-ROUTER-1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
EDGE-ROUTER-1(config)#int lo1234
EDGE-ROUTER-1(config-if)#shutdown
EDGE-ROUTER-1(config-if)#
*Jan 11 19:05:49.694: %LINK-5-CHANGED: Interface Loopback1234, changed state to administratively down
*Jan 11 19:05:50.694: %LINEPROTO-5-UPDOWN: Line protocol on Interface Loopback1234, changed state to down
EDGE-ROUTER-1(config-if)#end
EDGE-ROUTER-1#
*Jan 11 19:05:59.054: %SYS-5-CONFIG_I: Configured from console by bmcgahan on console

The router would then TFTP its running config to 10.0.0.1 using the filename EDGE-ROUTER-1.2011-01-11.19h05m59s.bmcgahan.working.cfg

Secondly, the script should also make backups of configs that are actually saved to NVRAM. Similar to the previous requirement, files should be backed up to TFTP using the naming convention HOSTNAME.YYYY-MM-DD.HHhMMmSSs.ADMIN_NAME.startup.cfg. However in this case you need to account for the fact that different admins use different syntax when saving configs. Some of them use "write memory" or shorter variations like "wr m" or just "wr", while others use the "copy run start" variations. However regardless which variation is used, the router spits out the same output afterwards as follows:

EDGE-ROUTER-1#wr
Building configuration...

[OK]
EDGE-ROUTER-1#copy run start
Destination filename [startup-config]?
Building configuration...

[OK]

Lastly make sure that the script doesn't mistake a "show run" output for the same as a "write memory", as the outputs are similar:

EDGE-ROUTER-1#sh run
Building configuration...

Current configuration : 3438 bytes
!
! Last configuration change at 19:05:59 UTC Wed Jan 11 2012 by bmcgahan
version 15.1

Submit your script as a comment and the first one with fully functional requirements wins 100 tokens!

Jan
18

A discussion / introduction to EEM, and basic configurations.

Why EEM?
Embedded Event Manager (EEM) allows you to have event tracking and management functionality directly on the Cisco IOS device, rather than on an external device. By having the configuration locally, actions can still be taken, even if the connection to an external monitoring station is unavailable. Plus, it is a great topic that can be used to challenge (or torment) CCIE candidates.

 
EEM Components:
Let's start by taking a look at the components of EEM: EEM server, event publishers, and event subscribers.

EEM server - an internal process that handles the interaction between the publishers and subscribers.

EEM publisher (detector) - software that screens events, publishes if there is a policy match. Some of the different detectors are CLI, NetFlow, SNMP, syslog, timers and counters. Newer releases have more options available.

EEM subscriber (policy) - defines an event and actions to be taken. There are two policy types, applets and scripts. Applets are defined within CLI, scripts are written using TCL.

For this section, we are just going to look at some of the basic applet configurations, which are defined using the CLI.

Policy Creation:
Basic steps for creating a policy:

1. Select event

2. Define detector options for the event

3. Define variables (if needed)

4. Define actions (optional)

Variables will be covered more in part two. This section is focusing on events and actions.

Start off by creating the applet with the CLI command event manager applet YYY, where YYY is the applet name that you want to use.

For a basic applet configuration, we will have an event and one or more actions.

Defining the event:
For the command syntax, we have the option of just specifying an event, or specifying an event with a tag value. Newer versions allow multiple events to be specified, and the tag values are used to correlate to attribute tags. We will come back to attribute tags in a later example, for now, just be aware that if you only have a single event, the tag is optional. Here are some sample event config lines. As mentioned earlier, without tags and additional configuration, you would only have one of these lines within your applet.

event syslog pattern ".*UPDOWN.*FastEthernet1/0.*"

event none

event track 99 state any

event timer cron cron-entry "15 13 * * 1-5"

There are a variety of different event detectors. For some of these we will go into greater detail, but here is a quick list of what some of the various options watch:

cli - screening CLI input for regex

counter - watch a named counter

interface - generic interface counters / thresholds (absolute or incremental)

ioswdsysmon - watchdog - CPU / memory values/ thresholds

ipsla - watch for SLA events

nf - watch for NetFlow events

none - allows an event to be run manually

oir - hardware - online insertion removal events

resource - event from embedded resource manager

rf - dual RP - redundancy framework

routing - changes to RIB

rpc - invoke from off-box using SSH/SOAP encoding to exchange XML messages

snmp - monitor a MIB object for values / thresholds

snmp-notification - match info in trap/inform messages

syslog - screen for regex match

timer - absolute time of day, countdown, watchdog, CRON

track - Tracking object event

Note: If you do not configure an event, your applet will not show as registered, and you will receive an error message when exiting applet configuration mode.

Detectors:
Let's take a look at some of the detectors in more detail.

None
This detector will let you manually trigger, using the event manager applet YYY run command, where YYY is the name of your applet.

CLI Detector
This detector allows for regex matching of CLI input.

Some of the various options that we have within the CLI detector:

Synchronous vs. Asynchronous
With Synchronous ( sync yes), the CLI command in question is not executed until the policy exits. Whether or not the command runs depends on the value for the variable _exit_status. If _exit_status is 1, the command runs, if it is 0, the command is skipped. (More on configuring variables later in part two.)

With Asynchronous (sync no), the event is published, and is the command can be suppressed (skip yes) or executed (skip no)

Timer Detector
Some of the various options include:

absolute

countdown - counts down to 0

watchdog - counts down to 0 and then resets to initial value

CRON - uses CRON specification

For those of you not familiar with CRON, here is a basic field listing and examples:

minute - range 0 to 59
hour - range 0 to 23
day of month 1 to 31
month of year 1 to 12
day of week - 0 to 6 (Sunday is 0)

For individual fields, an asterisk represents all valid values, a comma can specify a list of values, and a hyphen can specify a range.

"1 1 1 1 *" 1:01 AM, 1 January

"1 9 * * 1-5" 9:01 AM, Monday-Friday

"5 8,16 * * 0,6" 8:05 AM or 4:05 PM, Saturday or Sunday

SNMP trap detector
watch for a trap with a specified OID.

Track detector
Watch for change in a defined track object. Can watch for going down to up (up), up to down (down), or either of those two changes (any)

Actions:
Compared to the event types, the IOS descriptions for actions are a lot more straightforward.

With multiple actions, the tags will be sorted in ascending alphanumeric (lexicographical) order, using the label. Most examples use labels like 1.0, 2.0, 3.0, but using numbers and decimal values is not required.

Here is a sample list of action commands with various tag values, as sorted by the device. Notice that numbers come before letters, and that upper case letters come before lower case. Also, notice that 11 comes before 2 in the sort order, since it is sorted by the order of the individual characters.

action ... puts "where"
action 1.0 puts "Reno"
action 11.0 puts "Today"
action 2.0 puts "EEM"
action RRR puts "ABC"
action abc puts "12345"
action nnn puts nonewline "test123"
action rrr puts "hellothere"

Action prerequisites
This may be self-explanatory, but there are times when you need to have additional configuration outside of the applet. Here are some examples, as well as items that may not already be configured.

Action:   Requirement:
Track      -  Define objects

SNMP     - Trap Basic snmp config, enable traps event-manager

Syslog     - Logging enabled

Individual Actions
OK, so let's look at some of the actions, grouped by functions:

cli - execute a CLI command

gets/puts - used to send to or pull from tty.

mail/syslog/snmp-trap/cns-event - used to send messages

increment/decrement/append - changing variables

if/else/elseif/while/end/break/continue/foreach - conditional operators

wait - pause for a period of time

track - read or set a tracking object

regexp - match

reload/force-switchover - system actions

add/subtract/multiply/divide - hopefully self-explanatory, if you have no idea what these might be used for, then perhaps you have much bigger issues than EEM to worry about.

Applet Examples:
OK, now lets take a look at some examples. Since we are focusing on the available actions, we will use the event value of none, so that we can manually run these, and don't have to wait for an event to be triggered.

Basic example to "no shut" for an interface, and save the config afterwards.

event manager applet NOSHUT

event none

action 1.0 cli command "enable"

action 2.0 cli command "config term"

action 3.0 cli command "interface Ser0/0/0"

action 4.0 cli command "no shut"

action 5.0 cli command "end"

action 6.0 cli command "write mem"

Basic track and put example. Define the Track object first, configure the applet to read the object state. Object state will be stored by default with the variable _track_state. The puts command can output the value of the variable.

track 1 stub-object
default-state down

event manager applet EOT
event none
action 110 track read 1
action 111 puts "Object 1 is $_track_state."

! Run the applet

Router#event manager run EOT

down

!

Next example, still using track and puts. Here, we want to set the value for the object as well. Even though the applet sets the state of the object, that state is not entered into the variable _track_state unless you read the value. Additional discussion on variables will be covered in part two of this blog entry.

event manager applet EOT2

event none

action 110 track set 1 state up

action 110a track read 1

action 111 puts "$_track_state"

action 112 track set 1 state down

action 112a track read 1

action 113 puts "$_track_state"

Router#event manager run EOT2

up

down

If you look at the output of show track, it will show that EEM made the state changes.

Router#show track 1

Track 1

Stub-object

State is Down

3 changes, last change 00:00:10, by EEM

Router#

Basic example using gets, puts, and wait:

event manager applet G

event none maxrun 300

action 1 gets LINE1

action 2 puts "$LINE1"

action 3 puts "start"

action 4 wait 50

action 5 puts "end wait"

action 6 gets LINE2

action 7 puts "$LINE2"

Note:  If you don't configure the "maxrun" parameter when configuring "wait", you may think the applet is broken, and you may not see the items after the wait. The default for "maxrun" is 20 seconds.

Once you have an applet defined, you may want to use it in other locations.  Here, we have the applet titled RUNTEST2 configured to call the applet TEST2 after sending a message to the tty line.

event manager applet TEST2

event none

action 1.0 cli command "enable"

action 2.0 cli command "config term"

action 3.0 cli command "interface Ser0/0/0"

action 4.0 cli command "no shut"

action 5.0 cli command "end"

action 6.0 cli command "write mem"

event manager applet RUNTEST2

event none

action 1 puts "Enabling Ser0/0/0 Interface"

action 2 policy TEST2

Router#event manager run RUNTEST2

Enabling Ser0/0/0 Interface
Router#

Jan 18 12:15:57.063: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:TEST2)

Router#

Jan 18 12:15:59.071: %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to up

Router#

Jan 18 12:16:00.131: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0/0, changed state to up

Router#

As mentioned earlier, the variable "_exit_status" can be used with synchronous CLI events. Here, we are setting the exit status to 0, so that the command isn't run. In addition, a syslog message with priority 0 is configured.

event manager applet NOSAVE
event cli pattern "write mem.*" sync yes
action 1.0 syslog priority emergencies msg "Not Allowed"
action 2.0 set _exit_status "0"

For asynchronous events, you can determing whether the command runs (skip no), or doesn't run (skip yes) keyword options within the event command. Here, we have an applet that is setting a syslog message, and is not allowing the command to execute.

event manager applet NOSAVE2
event cli pattern "write mem.*" sync no skip yes
action 1.0 syslog priority emergencies msg "NO!"

As mentioned earlier, it is possible to have an event without an action. You may receive a warning message that you do not have an action configured, but the event will still trigger. In the case below, the CLI interception will prevent someone from using the write mem command.

event manager applet NOSAVE2
event cli pattern "write me.*" sync no skip yes

More examples, in addition to some more complex configurations, will be covered in part two.

Feb
18

And we're back for an exciting conclusion to the CCIE Troubleshooting series. Just what fate is in store for our heroes? Well, if we're anything like the Friday the 13th movies, you may need to wait about 20+ years to find out! :)

You're probably asking yourself, "With the amount of stuff we've gone through already, you're really telling me there's more to be concerned about?"

Yup, that's exactly what I'm talking about. So what next then? How about little unexpected zingers that may just confuse, confound and otherwise astound you?

Have you ever rebooted your router/switch and had it reply to you "Would you like to enter basic management setup?" Of course you have. All the time in practice labs! But what if it happened in the middle of your actual lab exam? You DID save your configuration, right? You didn't see any error messages pop up did you? Well, how about that?

How about a bit of preventive medicine? When you start the day, in SecureCRT that you'll be using in the lab, I would click on the Session Options and then on the Emulation (under Terminal in the left-hand pane). See the part about Scrollback buffer. Set that to AT LEAST 5000 lines. Gives you the ability to review things you've typed or what the router has said, or recreate your steps in case of an emergency.

But back to this dilemma. What do you do? Run in circles, scream and shout? While that may be entertaining, it would scare everyone else in the room and not be overly helpful to your cause! Whatever you do, DO NOT enter the setup mode. Before completely panicking, just do a "show start" and see whether your config was really saved or not!

If it did not save, go check your scrollback and see what error you missed. Then hope you have enough scrollback to reconstruct your configuration. THEN run in circles, scream and shout!

You likely were victimized by the configuration-register. Well, ok, you were victimized by the Proctor's warped sense of humor, but the weapon of choice was the configuration-register. Deal with it. You should have checked ahead of time! "show version | include Config"

Rack1R1(config)#do sh ver | in Config
Configuration register is 0x2102
Rack1R1(config)#

That would be the sign of a good router.

0x2142 is the sign of a bad router - your startup configuration will be skipped
0x2101 is the sign of a really bad router - you will go into bootstrap or ROMMON mode

Those are the primary ones to think about. There are others that involve the changing of console speed. While particularly entertaining, being that you do not have access to the terminal server configuration or physical access to devices any longer, that's not something you'll be worrying about. Back in my day, that was one method of torture potentially bestowed upon us.

So what do you do if you end up in ROMMON? Partly that will depend on what routing platform you happen to be using. But let's go with the standard ISR routers (28xx and 38xx which I believe are the currently listed devices on the blueprint for Lab Hardware).

rommon 1 >
rommon 1 > ?
alias set and display aliases command
boot boot up an external process
break set/show/clear the breakpoint
confreg configuration register utility
cont continue executing a downloaded image
context display the context of a loaded image
cookie display contents of motherboard cookie PROM in hex
dev list the device table
dir list files in file system
dis disassemble instruction stream
dnld serial download a program module
frame print out a selected stack frame
help monitor builtin command help
history monitor command history
iomemset set IO memory percent
meminfo main memory information
repeat repeat a monitor command
reset system reset
rommon-pref Select ROMMON
set display the monitor variables
showmon display currently selected ROM monitor
stack produce a stack trace
sync write monitor environment to NVRAM
sysret print out info from last system return
tftpdnld tftp image download
unalias unset an alias
unset unset a monitor variable
xmodem x/ymodem image download
rommon 2 >

You'll have a very specific set of commands there having to do with the boot process and/or reloading IOS through the serial port (not much fun, and not possible over telnet as configured!).

You can use the "dir flash:" command to see what files are available (or "dev" if you need to know the names of current devices like DISK0: or DISK1: for PCMCIA cards) and then "boot flash:(filename)" if there's any doubt.

Knowing how to get out of ROMMON is a great skill. Murphy's Law says whatever can go wrong will go wrong, and generally at the most inopportune moment! Know how to reboot!

So what if you get into a router at the beginning of your lab and you see this:

rommon 1>
rommon 1>?
Exec commands:
access-enable Create a temporary Access-List entry
access-profile Apply user-profile to interface
call Voice call
clear Reset functions
connect Open a terminal connection
crypto Encryption related commands.
disable Turn off privileged commands
disconnect Disconnect an existing network connection
enable Turn on privileged commands
exit Exit from the EXEC
help Description of the interactive help system
lat Open a lat connection
lock Lock the terminal
login Log in as a particular user
logout Exit from the EXEC
modemui Start a modem-like user interface
mrinfo Request neighbor and version information from a multicast
router
mstat Show statistics after multiple multicast traceroutes
mtrace Trace reverse multicast path from destination to source
name-connection Name an existing network connection
pad Open a X.29 PAD connection
ping Send echo messages
ppp Start IETF Point-to-Point Protocol (PPP)
release Release a resource
renew Renew a resource
resume Resume an active network connection
rlogin Open an rlogin connection
set Set system parameter (not config)
show Show running system information
slip Start Serial-line IP (SLIP)
ssh Open a secure shell client connection
systat Display information about terminal lines
tclquit Quit Tool Command Language shell
telnet Open a telnet connection
terminal Set terminal line parameters
tn3270 Open a tn3270 connection
traceroute Trace route to destination
tunnel Open a tunnel connection
udptn Open an udptn connection
where List active connections
x28 Become an X.28 PAD
x3 Set X.3 parameters on PAD

rommon 1>

That's entirely different and you have no capability of setting the boot parameters there. Miraculously though, you do have the "enable" command.

I'll give you a hint, there's no enable command in ROMMON mode! You aren't really in ROMMON, you are just being punk'd by the router.

rommon 1>enable
rommon 1>sh run | in rommon
prompt "rommon 1>"
rommon 1>

No matter what mode you are in, that's the prompt. Let's exit back out though, and assume that we're doing things our "typical" configuration fashion.

rommon 1>
rommon 1>en

*
*
*
*?

ERR

The "help" PAD command signal consists of the following elements:

where
is the identifier for the type of
explanatory information requested

*

What the heck is that??? Ahhhh... More entertainment. That would be the X28 Diagnostic Mode (helps if you are running an X.25 PAD, but since it's highly unlikely that normal people today even know what that is, chances are you don't want to run it! And yet here we are. Punk'd again.

The "exit" command will get you out. Only to be placed back to your fake ROMMON prompt! Try typing "enable" fully. (By the way, the prompt won't change, so don't believe everything you see!)

rommon 1>enable
rommon 1>sh run | in en
Current configuration : 3980 bytes
no service password-encryption
boot-end-marker
enrollment selfsigned
ip http authentication local
Please change these publicly known initial credentials using SDM or the IOS CLI.
alias exec en x28
end
rommon 1>

Ahhhh... Aliases. Aren't they exciting. If your proctor REALLY doesn't like you, they'll alias "en" and "enable" to something equally inane. But that's a start. So let's get rid of these things.
Don't forget that "configure terminal" may be necessary to fully type out in case they aliased "conf" to "exit" or something fun like that!

rommon 1>
rommon 1>conf t
Enter configuration commands, one per line. End with CNTL/Z.

exit
rommon 1>
*Feb 17 04:35:23.451: %SYS-5-CONFIG_I: Configured from console by console
rommon 1>

Ummmm... Did someone eat the configuration mode?

rommon 1>conf t
Enter configuration commands, one per line. End with CNTL/Z.

interface fa0/0

ip address ?
A.B.C.D IP address
dhcp IP Address negotiated via DHCP
pool IP Address autoconfigured from a local DHCP pool

ip address
% Incomplete command.

^Z
rommon 1>
*Feb 17 04:36:24.755: %SYS-5-CONFIG_I: Configured from console by console
rommon 1>

Commands appear to work, but we can't see anything. That would be another command!

rommon 1>conf t
Enter configuration commands, one per line. End with CNTL/Z.

do sh run | in config
Building configuration...
Current configuration : 4005 bytes
no service prompt config

Fix me!
^
% Invalid input detected at '^' marker.
service prompt config
TestRouter(config)#

Just put the service back on and we're good. Interesting enough, the hostname shows up while in config mode. Once back in user mode, the prompt comes back.

TestRouter(config)#end
rommon 1>
rommon 1>
rommon 1>
*Feb 17 04:38:52.911: %SYS-5-CONFIG_I: Configured from console by consoleconf t
Enter configuration commands, one per line. End with CNTL/Z.
TestRouter(config)#
TestRouter(config)#
TestRouter(config)#default prompt
TestRouter(config)#exit
TestRouter#
*Feb 17 04:39:04.775: %SYS-5-CONFIG_I: Configured from console by console
TestRouter#
TestRouter#

Other things you may have occur to your routers... Modification of the Exec-Timeout timers. Every 30 seconds may work without detection. Or if someone is really being amusing set "exec-timeout 0 1" on the console port. Type one character every second or get kicked out.

This is a place for cut/paste if I've ever seen one!

There may be any number of other odd things appearing throughout your configurations. With a decent glance these (hopefully) will stand out like a sore thumb. Other than the show commands from the last two days to verify IP addressing and looking for basic "no" things in your config, you should do a "show run" on every device.

When you get into the lab exam, you really have no idea just how much configuration will be in place already. Just like any consulting engagement, you could have a completely blank greenfield deployment. Or you could walk into a semi-dysfunctional existing network to improve/fix/enhance throughout the day. Check out what you have to begin with. Make notes.

Things that are especially important as they may lead to future difficulties when you configure the tasks given to you:

1. Backup-interface configurations -- These leave interfaces in a "standby" state which is most definitely not up!
2. Span or Remote-Span configurations -- This may involve the copying of information from one port to another. While it's one way, so OSPF peers won't show up, RIP advertisements could!
3. "ip classless" command -- This may have effects on your routing processes, or at least what is showing up versus expected!
4. kron jobs -- I outlined this before regarding time-based redistribution, but anything pre-existing should be noted!
5. EEM (Embedded Event Manager) -- Shouldn't see these anyplace (or rarely!) - See below

Every once and a while, we hear ramblings from people insisting that the proctor got into their racks and changed configuration. Even if using Notepad to track your commands, or the scrollback buffer, there's insistence that interfaces were configured one minute and had no configuration the next minute. There was no reboot, therefore it must have been the proctor.

While they do have a devlish glint in their eyes most of the time, and look like rather unsavory individuals, the Proctor's job is not to interfere with anyone's routers or switches. They have enough to do rather than resort to that level of torture! The Geneva Convention actually prohibits this type of behavior!

So ask yourself... If the proctor doesn't get into my equipment... And I KNOW that I've configured things and they are working.... What could possibly be the cause of it? How about the last two things I mentioned above? Kron or EEM have the ability to execute commands, configure device changes and/or copy files from TFTP devices into the running config. You should be aware of what's happening on a network at any point.

If you job is to evaluate and improve, it would be silly to rush off to do a list of tasks without understanding the impact along the way, wouldn't it? Or what forces were working against you?

A simple check of the running configuration before hand can show these anomalies to you. Anything that looks strange needs to be investigated! Nothing worse than working through things in the middle of the day, then seeing:

Rack1R1(config)#
*Feb 18 16:47:59.973: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or
detached
*Feb 18 16:47:59.997: %PIM-5-NBRCHG: neighbor 145.1.17.7 DOWN on interface FastEthernet0/0 DR
*Feb 18 16:48:00.009: %SYS-5-CONFIG_I: Configured from console by vty0
Rack1R1(config)#
*Feb 18 16:48:05.473: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on OSPF_VL4 from FULL to DOWN, Neighbor Down: Interface down or detached
Rack1R1(config)#
*Feb 18 16:48:30.041: %SYS-5-CONFIG_I: Configured from console by vty0
Rack1R1(config)#
Rack1R1(config)#
Rack1R1(config)#

And man, I'd agree. That evil proctor just jacked my lab!

Rack1R1(config)#do sh run int fa0/0
Building configuration...

Current configuration : 73 bytes
!
interface FastEthernet0/0
no ip address
duplex auto
speed auto
end

Rack1R1(config)#

Well, that's way not cool. My scrollback even tells me what I've done.

Rack1R1(config)#do sh run int f0/0
Building configuration...

Current configuration : 115 bytes
!
interface FastEthernet0/0
ip address 145.1.17.1 255.255.255.0
ip pim sparse-mode
duplex auto
speed auto
end

Rack1R1(config)#

Well, let's put it back...

Rack1R1(config)#
Rack1R1(config)#interface FastEthernet0/0
Rack1R1(config-if)#
Rack1R1(config-if)# ip address 145.1.17.1 255.255.255.0
Rack1R1(config-if)#
Rack1R1(config-if)# ip pim sparse-mode
Rack1R1(config-if)#
Rack1R1(config-if)# duplex auto
Rack1R1(config-if)#
Rack1R1(config-if)# speed auto
Rack1R1(config-if)#
Rack1R1(config-if)#
Rack1R1(config-if)#
*Feb 18 16:51:22.753: %PIM-5-NBRCHG: neighbor 145.1.17.7 UP on interface FastEthernet0/0
*Feb 18 16:51:22.757: %PIM-5-DRCHG: DR change from neighbor 0.0.0.0 to 145.1.17.7 on interface FastEthernet0/0
*Feb 18 16:51:27.501: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on FastEthernet0/0 from LOADING to FULL, Loading Done
Rack1R1#do sh run int f0/0
*Feb 18 16:51:29.921: %OSPF-5-ADJCHG: Process 1, Nbr 150.1.7.7 on FastEthernet0/0 from FULL to DOWN, Neighbor Down: Interface down or
detached
*Feb 18 16:51:29.929: %PIM-5-NBRCHG: neighbor 145.1.17.7 DOWN on interface FastEthernet0/0 DR
*Feb 18 16:51:30.005: %SYS-5-CONFIG_I: Configured from console by vty0
Rack1R1(config)#do sh run int f0/0
Building configuration...

Current configuration : 73 bytes
!
interface FastEthernet0/0
no ip address
duplex auto
speed auto
end

Rack1R1(config)#

Our neighbors come back, things are good again... But then it's back down right away. Or it could be much later. Either way, the same frustration ensues!

Rack1R1(config)#do sh run | s event
event manager applet NeverTrustTheseThings
event timer watchdog time 300
action 1.0 cli command "enable"
action 2.0 cli command "configure terminal"
action 3.0 cli command "default interface FastEthernet0/0"
Rack1R1(config)#

Well, that would certainly do it. And it may have been hidden before by a startup "logging console warnings" or something like that.

EEM is a POWERFUL tool. Check out the Network Management section of your Documentation. http://www.cisco.com/en/US/docs/ios/netmgmt/command/reference/nm_06.html#wp1157622

Rack1R1(config)#no event manager applet NeverTrustTheseThings
Rack1R1(config)#do sh run | s event
Rack1R1(config)#

So all during the first 30-45 minutes of your lab exam (after the Core Technology Q&A), you should be:

1. Reading through the whole exam
2. Taking notes to remind yourself of things later
3. Redraw the diagram quickly so you can write on it
4. Verify IP addresses quite simply
5. Identify major things in the configs beginning with "no" or altering the configuration register
6. Quickly do "show run" on all devices and scan through for anything that looks strange or out of place
7. Get ready to kick butt and take your number home!

We get stuck in ruts when going through practice labs. We have a process generally dictated by what level of preparation that we've done. While there are certainly a good number of tools and labs and documents out there to help you study, there is nothing that compares to the "personalized approach". By that, I mean, make changes yourself. Or have a buddy make some additional tweaks, tasks, changes to labs for you.

Keep in mind that folks on the CCIE team probably have EVERYONE's study materials. So while we are indeed pretty cool, they'll go out of their way to find something we didn't think of. So outsmart them! Process, process, process.

Good luck in your lab prep, and most certainly in the troubleshooting portion! Don't forget, this addresses nothing about the self-induced troubleshooting. While going through your lab, you should be verifying things every step of the way with show or debug (or whatever) commands. If properly verified, you will have few, if any, surprises during your lab.

Subscribe to INE Blog Updates