Full Scale Lab 1 has been added to the CCIE Routing & Switching v5 Workbook. More Foundation, Troubleshooting, and Full Scale Labs will be coming soon, including additional updates before the end of the weekend. I will post more information about additional content and its release schedule shortly.

This lab uses a 20 router topology which will be available through our rack rental system shortly.  In the meantime if you have your own lab built on CSR1000v, IOU/IOL, etc. the initial configs are available to download on the lab 1 tasks page.  For technical discussion of this lab, please visit the Full Scale Labs section of our Online Community here.


Having passed the CCIE Voice 10 years ago, and having taught on the technologies surrounding both Voice and Collaboration ever since, one might think that the exam would be easy to pass. I can assure you that no matter how much you know, no CCIE exam is easy to pass. Cisco doesn't allow them to be. Every CCIE track requires hard work and preparation, even if it may, at first glance, seem somewhat of a repeat of things you already know. You may ask since I had the CCIE Voice already, why I didn't simply take the Collaboration Written exam and convert my cert to a CCIE Collaboration? The answer I think is pretty straightforward - it's the challenge!! Seeing if you still have it 10 years later. Seeing if what you've been teaching your students for 10 years is still up to par and still relevant. To take you back to when I passed CCIE Voice ten years ago, the track was literally brand new that year, and Cisco was testing on CallManager version 3.3, SIP wasn't anywhere to be found, and creating a hunt group meant tweaking Attendant Console to make it do things it shouldn't ever be expected to do (like work). I'm quite happy to find that I may still have 'it' and that my content is right on par and not only relevant on all accounts, but as always goes well above and beyond the minimum of what you need to know to pass the exam, and takes you into the deep inner-workings of the technologies and answers the all of the "why" questions. Bear in mind that we never create content with the singular goal of simply getting you "past" the lab exam (the people that can only barely pass the lab can't make it past a technical interview in the real world), but rather our focus is making you a true expert whereby, as a byproduct, you do pass the lab exam and quite handily at that. Over the past 10 years I've had the pleasure of helping over 1,500 people do just this, and it's been so enriching in my life to see their professional and personal lives bettered for it. So what took me so long to getting around to sitting for this new exam? Simply put - my schedule. As some of you may know, I've been teaching a lot of 2-week CCIE Data Center courses and 2-week CCIE Collaboration courses, as well as working on building all of the Collaboration racks and self-paced learning content, and quite frankly just hadn't found time in my schedule to get around to preparing to sit for and take the actual new lab exam until just a few weeks ago. So onto more of what you need to know and what it takes to be ready.

It's quite possible that I may be one of the only people besides Frog that possesses 3 or more CCIE's, where one of them is not Routing and Switching.

Firstly, what it's not. As I mentioned in a previous post, there isn't a whole lot of Cisco's "Collaboration" portfolio in the CCIE Collaboration written or lab exam blueprint. No TelePresence or DX/EX/MX/TX/SX or Codec endpoints (which differ vastly from simple 9971 phones), no MCUs, no WebEx, VCS-C / VCS-E Expressway (now Collab Edge), TMS or TPS in the exam. (Note: VCS/TMS are present in the backbone only – and all the hard work is on the VCS & TMS and out of the control of the student - you simply need to provide interop dialing with it.) This is much more of a Voice exam with a heavy video ephasis and a little bit of Jabber (8% of total score).

What do you need to know to be ready to sit for the exam? Since the new policy is now in effect that if you fail the exam twice that you have to wait 3 months before you're allowed to sit for it again, it is more important than ever to be 100% ready before you go sit for your first attempt, and that you pass on your first or second attempt before your momentum is severely interrupted by that 3 month stint. This is only one of the reasons that Brian, Brian, Petr and I have always recommended that you be able to do all of the CLI portion of your lab (whichever track) ... in Notepad. With no internet connection, no router tab-completion or ? context-sensitive help. And while you may misspell one or two things or occasionally forget an argument, that when you go to paste what you did in notepad, into your Cisco device, that 95% of it is syntactically correct, and that your logic is flawless. And as for your UCM web page configuration - that will obviously take up most of your time. While it's impossible to know what you will need to accomplish before you arrive, you need to be able to digest what they give you for tasks and visualize the entire call flow and any features, and go execute the configuration in UCM with no hesitation. As much as you may not like it - the CCIE exam remains an exam where not only accuracy, but also speed, are key. There is a lot to accomplish in 8 hours. An awful lot.

As I've been advocating heavily for over the past 5 years, you must be absolutely proficient with Globalized dial plans. With the likelihood of more than one cluster, configuring them must be second nature and not even something you think much about - rather something that you simply execute quickly using muscle-memory with absolute knowledge that what you are configuring will work cold. This may take a good deal of practice for some of you that still implement more traditional dial plans on a regular basis in your day jobs. Thankfully we have loads of content to prepare you for this critical key component of the exam. Not only have I just re-recorded the complete dial plan section (videos 83-105 including globalized dial plans as well as dynamic dial plans related to ILS/GDPR and CCD/SAF and Session Management Edition), but we also have loads of labs with heavy emphasis on globalized dialing in our CCIE Voice v3.5 workbooks, with new ones specifically aimed at the Collaboration track coming out very soon. The recent SRNDs as well as a number of Cisco Live can provide a lot of guidance as well. With that very core topic covered, it's on to SIP signaling and video, the other two topics that you will need to know cold. The good news about video is that you don't have to memorize how every endpoint treats video and what CUBE needs to do to pass it, but just a few endpoints - namely the Cisco 9971 phone and the Jabber for Windows client. It's no secret that the industry has heavily gravitated toward SIP trunks over the past 5 years, and that in any production environment today, you are working heavily with SIP and therefore also with CUBE or some flavor of Session Border Controller, and both the CCIE Collaboration written and lab exam reflect those very well. You should be able to read and completely deconstruct every SIP message that you come across in very quick fashion. We prepare you well for this. Video calling and video conferencing is the other bit that you will need to know cold. This guide contains complete samples for video conferencing configurations as well as good info specifically about 9971 phones and their RTP payload type and how it differs from platform to platform (CUCM vs CME). CUBE, Cisco's SBC is something that is heavily used in real-world deployments, is on the blueprint, and should be taken seriously. There is a ton you can do with CUBE, and you should know it well. Read and know this guide inside and out. I will be hosting a live class on CUBE the week of Sept 1-5, and that content will get added to the CCIE Collaboration ATC playlist. Beyond that there are of course the usual topics: Codec Preference and Region control, CME, Unity Connection, Unity Express and Contact Center Express - all of which are important but shouldn't take you very long to think about and configure at all, with the exception of CME - that can be a bit tedious in taking a while to key in all of the CLI configuration with both SIP and SCCP phones to consider, as well as dial plan with Voice Translation Rules.

Speaking earlier of building the new Collaboration racks, I'd like to provide some guidance on a few different options for either building your own racks that will contain everything you will need to completely prepare you for your exam or else building differing stages of a partial rack coupled with supplementing your own rack practice with rental sessions from INE to give you access to the more expensive bits of the rack that you needn't bother with purchasing. Of course there is always the option to rent all of your rack time from us, however this option doesn't obviate the need for roughly $1,000 USD in hardware, as you simply cannot adequately prepare for this exam without having 3x 9971 and 3x 7962 Cisco IP phones physically in front of you (to dial/hear-audio/see-video/hear-audible-results-from-dtmf-key-presses/etc), connected by Layer 2 back to our racks (This is still far less expensive than the $20,000 price tag that it costs to build a full rack). Remotely controlling phones was something you -to some degree- could get away with on the previous version of the CCIE Voice exam, but it is simply not an option with this new version of Collaboration*.

Here is the list of what Cisco has in the actual lab exam and we have mirrored our racks around this build list. Here is a complete list of our hardware and server builds, and throughout that same guide you can find everything you need to know to connect to and use our Collaboration racks.

Option 1 - Complete Rack Rental
This option will provide you with the easiest option in terms of time to get up and running. With this option you should plan to rent roughly 700-1000 hours of rack time.

What you will need:

  • 1x Cisco router for EzVPN and L2TPv3
  • 1x Cisco switch for QinQ tunneling and L2VPN
  • 3x 9971 IP phones with CP-CAM USB backpack
  • 3x 7962 IP phones

Full details for this option in terms of hardware, software and configurations can be found beginning on this page of our Collaborations Rack Rental Guide.

Option 2 - Fully Virtualized Solution Augmented with Lots of Rack Time
This option will provide you with a very inexpensive way to get started in your studies and be able to practice maybe around 25% of the necessary tasks - including globalized dial plans but with all SIP trunking. You will definitely need to rent plenty of rack time to augment your studies with this solution, but this will get you started. I would estimate that you would still need to rent roughly 500-700 hours of rack time with this option.

What you will need:

Cisco uses a UCS C-Series server for their hardware, but this is not necessary as you have no access to the UCSM in the lab exam, so any server will do. A server like this can typically be found online used for around $300-$500 USD.
Access to the Cisco NFR bundle is something that only Partners have access to and only costs around $300 for everything you need, but if you are not a partner, you will not be able to purchase this software. If you have a proper service contract, you may be able to download the software from Cisco.com and register for the 6-month demo license, but I don't believe you will be able to get another 6-month license after the first has expired. You will then need to revert to having to rebuild all of the servers every 60 days. Without either, this will make it impossible to build your own servers without purchasing full licenses - which is an incredibly expensive option.

Option 3 - Entry Level Full Rack (no ISR-G2s) Augmented with Rack Time
This option will provide you with a semi-inexpensive way to get started in your studies and be able to practice approximately 65-70% of the necessary tasks - including all phone features except for Video conferencing. You will still need to rent a decent amount of rack time to augment your studies for video conferencing and full-lab practice sessions with this solution, but this is a great, (comparatively) inexpensive option. I would estimate that you would still need to rent roughly 200-250 hours of rack time with this option.

What you will need:

  • All of Option 2 (Server/software)
  • 4x 2811 ISR (Gen1) routers with PVDM2s and VWIC2-1MFT-T1/E1s for site and PSTN PRI gateways and audio-only transcoding and conferencing
  • All of Option 1 hardware for augmented rack rental sessions

This rack should cost you somewhere close to $5,000 USD to build.

Option 4 - Near Complete Full Rack (1 ISR-G2) Augmented with Rack Time
This option will provide you with a way to practice approximately 95-99% of the necessary tasks - including Video conferencing and the latest CUBE features. You may still wish to rent some rack time simply to have a few full-lab practice sessions with all routers running 15.2(4)M code and the possibility of having multiple video conference bridges and/or video transcoding devices, but you also may find this unnecessary as you can just move around your tasks to accommodate everything on your single ISR-G2.

What you will need:

  • All of Option 2 (Server/software)
  • 1x 2911 ISR-G2 router with 1x PVDM3-32 (or 2x PVDM3-16) and 1x VWIC3-1MFT-T1/E1s for HQ
  • 3x 2811 ISR (Gen1) routers with PVDM2s and VWIC2-1MFT-T1/E1s for Site B/C and PSTN PRI gateways and audio-only transcoding and conferencing
  • SRE-710 for Unity Express

This rack may cost you somewhere close to $12,000 USD to build. There is an embedded demo license with Cisco Unity Express on the SRE module that simply needs to be activated, however it should be noted that it will only last you 60 days, at which time you will need to re-install the software completely to get any sort of extension to this demo period.

Of course you can also always build an entire rack with everything we have listed (all 2911s) in our Rack Rental Guide, but this will cost you probably around $20,000 in hardware alone, before any licensing costs.

*As a side note from above, we do provide a single 7961 phone at each site physically connected to our racks that renters may remotely control, but we do this as a mere courtesy for those that cannot afford to purchase their own right away and simply need to test a few basic dial plan and softkey functions, and we do not intend for this to be a complete replacement to having your own phones connected to us via L2VPN. If you chooe this option initially, just know that at some point during your studies you will in fact need to have them in front of you connected to us with our L2VPN option. Also, we don’t provide 9971 because the remote control is next to impossible (it is impossible to predict a reliable response, and many times the phone simply won’t respond at all). Also, while on our racks, you can practice 2-way and 3-way video with the Jabber for Windows clients we provide at the HQ, Site B and PSTN/Backbone sites. They all have cameras attached and will allow you to practice point-to-point video as well as video conferencing with the PVDM3 video conference bridge that you can build at any of the sites.

Also, for connecting back to our racks via L2VPN, I recommend the Cisco 1841 router and the Catalyst WS-C3560-8PC wwitch since it's an 8-port PoE that is fanless (read quiet), however if you're on a super-tight budget, you can get away with 2611XM (must be XM) router and a Catalyst 3550 switch with PWR-CUBE-4 (note the 3550 Inline Power won't adequately power the 9971 phones). Again guidance for all of this can be found in our Rack Rental Guide.

I hope this has provided some good insight and help for those that are working toward accomplishing the CCIE Collaboration, and please comment below on anything you can think that might be useful to add to this article for yourself or others studying, and I will be happy to update it.


It's rumored that the announcement for the R&S CCIE v5 update should be coming soon (November timeframe) and the switch over for the lab sometime around March 2014. Cisco Live Europe has a R&S v5 Technical Breakout scheduled for anyone attending. The update to version 5 is rumored to be a 100% virtual lab environment similar to how the troubleshooting section of the lab is done now. The major benefit of the lab going virtual is that the topics covered will be platform independent. You will not need to buy 2911's or 3750x's to prepare and can use any relatively newer router or switch to prepare or use a virtualized environment (IOL/IOU/VIRL, GNS3, CSR). The goal of the v5 appears to be to focus on the technologies themselves and less on the hardware and a specific topology. This is the best move Cisco has made for the R&S CCIE program in years as candidates will need to focus more on the technologies themselves and not worry about IOS versions, hardware platforms, physical topologies, etc.

Allegedly the R&S CCIE v5 blueprint will see legacy topics like Frame-Relay removed. Additionally it's possible some of the more lesser used features of the IOS like Zone-Based Firewall, WCCP, IPv6 multicast, and PfR could be removed from the lab. A few of the topics we could see added are IPSec, DMVPN, and Embedded Packet Capture. We may see ISIS added to the written at least if not the lab. This could be the last version of the R&S lab that isn't IOS XE based so we could see it added to the written.

Currently the lab has a 2 hour troubleshooting section and a 6 hour configuration section. The new lab may contain, in addition to the troubleshooting section, a new diagnostic section. This means the lab could have a troubleshooting, diagnostic and configuration section. I would assume the points for this new section would come from the configuration section and the troubleshooting section would remain the same or possibly even slightly higher in points.

So what does this mean for someone currently preparing for the R&S v4 blueprint? If you feel you are close to taking the lab but do not have to scheduled, you should schedule a date ASAP. Once the official v5 announcement comes out from Cisco, it will be hard to schedule a lab date. If you have a date scheduled before March 2014 then you should be fine. If your date is after February 2014 then I would recommend you move it up ASAP. The longer we go into November the more likely the new blueprint date has been pushed back by Cisco.

Additional v4 bootcamps will be added to the schedule before the March 2014 changeover. We will start transitioning the current bootcamps over to version 5 around the first of the year. For the self-paced products we will start releasing labs and videos covering the new blueprint in November. 90% of the material from the version 4 blueprint will carry over to the version 5 lab blueprint. Topics you can skip for the routers will be Frame-Relay, PfR, WCCP, Zone-Based Firewall along with technologies that are not supported in IOU L2/IOL L2. Here is a list of features we may not see for the layer 2 section since the switches will be virtual.

1) QinQ Tunneling
2) ISL trunks
3) DHCP Snooping
4) Layer 3 Port Channel
5) Private VLANs

Post any questions you have about the new blueprint changes and I'll start creating a FAQ below:

Q) I purchased the version 4 self-paced material so will I be covered for the version 5 products?
A) Yes.

Q) Will I have to pay anything for get the new version 5 material?
A) No.

Q) I attended an R&S CCIE v4 bootcamp so can I resit a v5 bootcamp for free?
A) Yes.

Q) Will INE offer racks for the new blueprint?
A) Yes. Although the lab is virtual it is still good to spend part of your preparation using real hardware as that is what you use in your day-to-day job. Towards the end of your preparation you can hone your skills using virtualized environment. We will be using the CSR along with real switches for the virtualized environments.

Q) What about my tokens?
A) Your tokens will carry over.

Q) I'm currently schedule for the lab after the v5 update. Can I still take the v4 lab?
A) No.

Q) Do you feel strongly that the announcement will come out in November?
A) I do feel confident that the announcement will be in November but it could slip since they are trying to align the Cisco 360 update to the lab release.


I hope everyone enjoyed the IPv4 Route Redistribution session on Friday. The turnout was amazing to say the least. We got off to a late start due to needing to split the streams between servers as we had over 2000 people logged in accessing the session. Once we got rolling you can really see how excited I get working on routers ;-) In fact I'll be doing a few of these R&S CCIE sessions a month going forward since the turnout is so good and I really enjoying do this. The flow for these new online sessions will be that I run every session twice to get enough video for a complete product. After that it'll take about one or two weeks to get the final product through editing and into your members site depending on the length of the session. These videos will be free for any AAP member or R&S CCIE Bootcamp customer.

I'm going to be running the IPv4 Route Redistribution session again sometime after next week and span it over two days (6 hours each). I'll publish labs for this next session so after you watch the videos you can go back and reinforce the concepts by doing the labs. Additionally I'll publish the initial and final configurations for the video sessions along with the configurations I capture during each of the breaks. After that I'll go through the videos and get the complete product released.

The next sessions will be MPLS L3 VPNs and IPv4 Multicast. Both of these will be ran twice with the first session being one day in length and the second two days. Also I'll be fitting in a few smaller sessions in between covering other topics.

As a side note the R&S CCIE Version 5 update is just around the corner. The word is that we are looking at a April/May timeframe so I'll try and get all of the R&S topics covered in these sessions by the end of January or mid-February. Most of the sessions will carry over to the R&S CCIE Version 5 blueprint if you aren't planning on taking the R&S CCIE Version 4 lab.


The R&S CCIE Volume 2 workbook has been ported to our new web format (see below). This format allows you to add bookmarks, add notes (both private and public notes) and ensures you always have the latest version. Additionally you can submit feedback directly to the Development and Editing teams from within workbook. We are also working on integrating our IEOC forum directly into the workbook and plan on having it available late October.

For the notes, we are currently implementing a rating system that will allow you to rate a public note posted by someone else. This will allow you to filter public notes that are not above a certain rating if desired. This new public notes feature will be really popular based on the feedback we have received from the beta testers. The R&S Volume 2 in the new web format will be released next week.

R&S Volume 3 workbook has been updated. This updated version will be available later this week in your members site account. This is the last major update to Volume 3 before it is retired and integrated into the new single R&Sv5 blueprint based workbook we are already developing.

Q - Will I automatically get the new workbook when its released next week if I have the current workbook?
A - Yes. The workbook will automatically show up in your members site account next week.

Q - Will I be able to view it offline?
A - Yes. A PDF version will automatically be generated when a change is made to the workbook.

Q - Can I view it on my tablet?
A - Yes the new format is pure HTML5 which is support by all modern tablet web browsers. This is the first step before we release it as an iPad/iPhone app and Android app. The tablet app will allow you to take the workbooks "offline" and still make notes, bookmarks, etc that will sync up when you get back online.

Q - Does this include new material?
A - We had to freeze any development while the workbook was ported to the web. Now that the workbook is up we'll start adding new content. We have 3 additional labs to add to the workbook now.

Lastly Volume I is currently being ported to the web and I'll release it in chunks as each section is finished. Currently the IP Routing section has been ported. I'm reviewing it before release and adding more PfR/OER labs.

R&S Online Workbook


This past Monday I passed the CCIE Data Center Lab Exam in San Jose CA, making me four time Cisco Certified Internetwork Expert (CCIE) #8593 in Routing & Switching, Service Provider, Security, and Data Center, as well as Cisco Certified Design Expert (CCDE) #20130013.  This was my first - and thankfully last - attempt at the DC lab exam, and also my first experience in the San Jose CCIE Lab location.  In this post I’m going to outline my preparation process for CCIE Data Center, as well as to talk about my experience with the actual day of the lab.


The Initial Commitment

When the new CCIE Data Center track was first announced last year, it was a no-brainer that I was going to pursue it.  As I already had 15+ years of experience in Enterprise networking, with a large focus on campus LAN switching, IGP and BGP routing, plus some minor exposure to the Nexus platforms, I thought it would be a cinch.  After all, Nexus is just a fancy Catalyst 6500, right? The major hurdle for the track however was not the technologies, but procuring the equipment.  After debating back and forth for quite a while, Brian Dennis and I decided that INE would hold off on the company Ferraris, and instead invest in the equipment for CCIE Data Center.  One of our deciding factors to invest in the track was the sheer volume of customers at our CCIE Candidate Party at Cisco Live 2012 that kept asking us all night long, “when are you guys going to do Nexus training!”  As they say, ask and you shall receive… or was it if you build it, they will come?

Coincidentally, our initial build plans for DC started in early July 2012, which makes it almost one year to the day from when we committed to the track until when I finally had a chance to take the lab exam.

Originally I had planned to try to get the very first available slot for the DC lab exam, but as always life happened and a few things got in the way, such as the birth of my daughter, as well as a short pit stop along the way to pick up the Cisco Certified Design Expert (CCDE). Anyways, onto my preparation…

Once our equipment build was finalized, which by the way was the most grueling and complicated build of my 15+ year career, Mark Snow and I decided to implement a divide and conquer approach to the blueprint, where we would split the Nexus topics, I would take Storage, he would take Unified Computing System (UCS), and then we’d come back and meet in the middle.  Nexus I assumed would be simple, since I had some experience using it as a basic 10GigE aggregation switch, but none of the advanced DC specific topics (e.g. vPC, FabricPath, OTV, FCoE, etc.)  In hindsight, yes Nexus is just a glorified Cat6k, however there are caveats, caveats, and more caveats galore.  Did I mention Nexus has a lot of caveats?

Recommended Reading or: How I Learned to Stop Worrying and Love the Documentation

Since a lot of the DC specific technologies are so new, there’s not many traditional books that are out there that can help you, unlike something like OSPF that is over 20 years old.  With Nexus the topics are so cutting edge, the NX-OS software team is literally pushing out hotfixes and new features as we speak.  Therefore the only main resource that is available for reading about a lot of these technologies is the Cisco Documentation.  I can already hear the collective groan from the audience about reading the documentation, but I can’t stress this enough, you must read the Nexus documentation if you are serious about these technologies and about passing the CCIE DC Lab Exam.

To give you an idea, this is what my Chrome bookmarks toolbar still looks like today.


Personally the way I did this was to download every single configuration guide for Nexus 7K, 5K, and 2K in PDF format, and then load them on my iPad.  Starting with Nexus 7K I worked from basic system administration up to interfaces, layer 2 switching, layer 3 routing, SAN switching, etc.  Don’t count on having access to the PDF versions of the documentation in the actual lab exam, but for preparation purposes these are much more convenient than clicking through every section of the documentation in HTML format.

Each configuration guide can be downloaded as a single complete PDF.


Note that for MDS you don’t need to read through as much, since the SAN switching syntax is essentially the same between the Nexus 7K, 5K, and MDS, as they all run NX-OS.  The sections of MDS documentation that I did read end-to-end however are the Cisco MDS 9000 Family NX-OS Interfaces Configuration Guide and the Cisco MDS 9000 Family NX-OS IP Services Configuration Guide.  Both of these sections are key, as some topics such as FC Trunking and Port Channeling work differently in MDS than they do in Nexus, and then the IP Storage features such as FCIP and iSCSI are unique to MDS and are not supported in Nexus.

Another key point about the documentation for Data Center, just like for other CCIE tracks and other technologies in general, is that once you know how to use the documentation and where things are located you don’t need to worry about the default values for features, or other inane details about syntax.  For example there was a discussion recently on the CCIE DC Facebook group about how to create a mnemonic device (e.g. All People Seem To Need Data Processing / Please Do Not Throw Sausage Pizza Away) in order to remember in which features higher values are better and in which features lower values are better, e.g. LACP system-priority, vPC role priority, FabricPath root tree priority, etc.  I responded, who cares?  Why waste time remembering default values that likely will change between versions anyways?  Instead, your time would be better spent making sure that you know the manual navigation path for all features that you will be tested on in the exam.

Lower is higher and higher is lower… makes perfect sense, right?


Another point to consider is that in the actual Lab Exam, access to the documentation web pages is not very fast.  I’m assuming this is due to the strict content filtering that all the pages have to go through before they show up on your desktop.  Regardless as to the reason, if you need to use the documentation in the exam and you don’t already know exactly where the page you want is located, you’re gonna have a bad time.

Additionally, don’t limit your reading of the documentation to just the configuration guides.  There are a number of other very useful portions of the documentation that you should read – again, end-to-end, there are no shortcuts here – such as the white papers, design guides, and troubleshooting guides.

The Nexus 7000 White Papers are an essential read.


This is especially true since some of the verification and troubleshooting syntax for Nexus is just out of this world.  I swear whoever works on the actual syntax parser for the NX-OS software team must get paid based on the number of characters that the commands contain.  Did you say that your Fibre Channel over Ethernet isn’t working to your Nexus 2232PP Fabric Extenders that have multiple parent Nexus 5548UP switches paired in Enhanced Virtual Port Channel?  I hope you remember how to troubleshoot them with the command show system internal dcbx info interface ethernet 101/1/1!  Err… how about we just know where to find it in the FCoE troubleshooting guide instead then.

The troubleshooting guides are an often overlooked section of the documentation.


The real point of using the documentation is as follows: you must understand, in detail, the design caveats and hardware caveats that the Nexus, MDS, and UCS platforms have as they relate to the DC technologies.

Pictured above, some light reading on the Design Considerations for Classical Ethernet Integration of the Cisco Nexus 7000 M1 and F1 Modules


Recommended Books

Beyond the documentation, there are a select few regular books that I used during my studies.  The vast majority of them are either available on the Safari Online site, or in the case of the IBM Redbooks, free in PDF form direct from IBM’s website.  These books, in no particular order are:

Cisco Live 365

For those of you that have never heard of Cisco Live 365 before, you’re welcome. ;)  This is where all the video recordings and PDFs of slide decks are from the different Cisco Live (i.e. Cisco Networkers) conventions that have occurred in the past few years, from multiple locations.  A lot of these sessions are used to talk about the introduction of new products, e.g. the new Nexus 7700 that was just announced at Cisco Live 2013 Orlando, while others are technical deep-dives into topics.  In the case of CCIE Data Center there are a lot of really good presentations that I would recommend looking at during your preparation.  You don’t need to have physically attended Cisco Live in the past to get access, just sign up for an account for free and you can search all the content.  The Data Center sessions generally start with “BRKDCT” (Breakout Data Center Technologies), so that’s a good place to start your search.  Notable ones that I personally thought are worth looking at are in no particular order as follows:

  • BRKDCT-2204 - Nexus 7000/5000/2000/1000v Deployment Case Studies
  • BRKDCT-2237 - Versatile architecture of using Nexus 7000 with F1 and M-series I/O modules to deliver FEX, FabricPath edge and Multihop FCoE all at the same time
  • BRKCRS-3145 - Troubleshooting Cisco Nexus 5000 / 2000 Series Switches
  • BRKDCT-2048 - Deploying Virtual Port Channel in NXOS
  • BRKCRS-3146 - Advanced VPC operation and troubleshooting
  • BRKDCT-2081 - Cisco FabricPath Technology and Design
  • BRKDCT-2202 - FabricPath Migration Use Case
  • BRKDCT-1044 - FCoE for the IP Network Engineer
  • BRKSAN-2047 - FCoE - Design, Operations and Management Best Practices
  • BRKCOM-2002 - UCS Supported Storage Architectures and Best Practices with Storage
  • BRKVIR-3013 - Deploying and Troubleshooting the Nexus 1000v virtual switch
  • BRKRST-2930 - Implementing QoS with Nexus and NX-OS
  • BRKCOM-2005 - UCS and Nexus Fabric and VM's - Extending FEX direct to VM's in UCS and Nexus 5500
  • BRKCOM-2003 - UCS Networking - Deep Dive

INE’s Videos, Workbooks, & Classes

Now in my personal case, when I am learning a new technology, I know that I have truly absorbed and understood the topics when I can explain it to someone else in a clear and concise manner, hence my day job, author and instructor at INE.  From the culmination of reading these books, reading the documentation, and testing essentially every feature that the platforms have to offer, Mark Snow and I developed INE’s Nexus, Storage, and UCS classes, as well as the associated workbook labs and the live Bootcamp classes for these technologies.

As I’ve done many write-ups before on these offerings, and without getting too much into a sales pitch, you can find more information here about INE’s CCIE Data Center Video Series, here about INE’s CCIE Data Center Workbook, here about our CCIE Data Center 10-Day Bootcamp, and here about our CCIE Data Center Rack Rentals.  Note that we are currently adding more capacity to rack rentals and adding more Bootcamp classes to the schedule, both of which I’ll be posting separate announcements about shortly.

Read, Test, Rinse, and Repeat

While learning and developing the content for Data Center I followed the same methodology that Brian Dennis and I have been personally using and have been teaching for the past 10 years (yes, I can’t believe it’s been that long).  This methodology is essentially a four step process of learning and testing incrementally.  This is also the same methodology that has helped Brian Dennis obtain five CCIEs, and for me to obtain four CCIEs and the CCDE, so trust me when I say that it works.

The methodology is a basic four step process as follows:

  • Gain a basic understanding of the technologies
  • Gain basic hands-on experience to reinforce and expand your understanding
  • Gain an expert level of understanding
  • Gain an expert level of hands-on experience

It might seem self-explanatory that you need to start at the bottom and work your way up, i.e. A then B then C then D, however over the years we’ve seen so many CCIE candidates try to shortcut this process and try to go from A directly to D.  Traditionally these are the candidates that end up taking the lab exam 5, 6, 7 times before passing, essentially trying to brute force their way through the lab.  Believe it or not, we have had customers in the past that have attempted the CCIE Lab Exam in the same track 10 or more times before passing.  Coincidentally, these are also usually the customers that don’t want to hear that they don’t know something or that their methodology is wrong. Go figure.

Pictured above, how to make a hobby out of visiting building C at Cisco’s San Jose campus.


At least for me personally, obtaining a CCIE is more about the journey than it is the destination.  I feel that I would have cheated myself coming out of the process without truly being an expert at the technologies covered, so I made sure to really take the time and be meticulous about going through everything.

Pictured above, how to astound the engineers at the technical interview for your new job after getting your CCIE.


The CCIE Data Center Written Exam

Before scheduling the Lab Exam, I of course had to tackle the necessary evil that is the CCIE Data Center Written Exam. In my opinion this exam should be renamed the “how many stupid facts can I memorize about the Nexus and UCS platforms exam.”  I didn’t pass the DC written exam on my first attempt, or on my second attempt.  I’m not going to say exactly how many times I took the the DC written exam, but let’s just say that it’s somewhere more than two and somewhere less than infinity, and that I likely have seen every question in the test pool multiple times.

For those of you have passed this exam on your first try, more power to you.  With me on the other hand I try not to memorize any facts that I can quickly look up instead. While whoever wrote the CCIE DC Written Exam may think it's important that the UCS B420 M3 Blade Server has a Matrox G200e video controller with integrated 2D graphics core with hardware acceleration and supports all display resolutions up to 1920 x 1200 x 16 bpp resolution at 60 Hz with a 24-bit color depth for all resolutions less than 1600x1200 and up to 256 MB video memory, I do not, but I digress.

Scheduling the CCIE Data Center Lab Exam

One of the biggest hurdles in obtaining the CCIE DC that I had not initially planned for was the availability, or lack thereof, of lab dates open for scheduling.  I’m not normally one to complain about this, because when I took the CCIE R&S Lab Exam back in January 2002 I believe that I scheduled the date somewhere around July of 2001. Back then it was the norm to have a 6 month wait for dates, so when you went to the lab you had better be really prepared for it, otherwise you had a long 6 months ahead of you afterwards trying to think of what you did wrong.  With Data Center though, this was a completely different ballgame.

By the time I got around to being ready to schedule a date, there was literally not a single date open on the schedule for any location.  Mark Snow had even scheduled a lab date in Brussels Belgium, and was going to fly from Los Angeles in order to take the lab because that was literally his only option.  Luckily right around that time the CCIE Program added  new dates on the schedule, and he was able to move his lab attempt to San Jose, where he ended up passing.

Anyways once these new dates were added to the schedule I knew that I had to act fast, or risk having to wait until 2015 (not really, but that’s what it felt like).  Unfortunately the date that I took was only a week after Cisco Live 2013 Orlando, so I couldn’t help but feel while we were partying it up at the conference I should have been at home studying instead.  Also I would have much preferred to go to RTP over San Jose, since RTP is much closer to Chicago and I’m much more familiar with the area.  In hindsight SJC was probably a better choice anyways, since I have lots of friends in RTP which means there would have been more distractions.

Traveling To San Jose

I scheduled my exam purposely on a Monday, which meant that I could get to San Jose either Friday or Saturday and then leisurely spend the rest of the weekend doing some last minute review and relaxing in the hotel without any distractions.  This is the first time I’ve done it this way, and if you have the option to this is the approach that I would recommend.

Having had all my attempts in RTP in the past I was never worried about travel time, since it’s only about two hours from Chicago.  Normally I would fly in the day before the lab in the afternoon, and then immediately go to the airport after the exam to fly home.  Worst case scenario I could drive from to Chicago to RTP, which I actually have done in the past.  I remember one time when teaching a class at Cisco’s RTP office I left the campus at about 5:30 on a Friday, drove to RDU and parked my car, bought a ticket at the desk, and still had time to make a 6:15 Southwest flight back to Chicago.  I could only dream that Chicago O’Hare or Midway was as delay-free as RDU.

SJC on the other hand doesn’t have as many flights to Chicago, so I wanted to play it safe and arrive more than one day early.  Luckily I did plan it this way, otherwise with the Asiana Flight 214 incident at SFO this past weekend I might not be writing this post at all right now; the moral or the story being that if you have the option to travel an extra day early before the exam, take that option.

For the hotel I stayed at the Hyatt Regency Santa Clara, which was nice.  They have a nice outdoor pool area that I spent some time relaxing with my laptop at.  It’s fairly close to the Cisco campus, being about a 5 – 10 minute cab ride to the office in the morning, and then after the lab I walked back to the hotel which took about a half an hour or so.  If you’re familiar with the area it’s directly next to the Techmart and the Santa Clara Convention Center.

The Day of the Lab

San Jose’s lab starts at 8:15am, so I scheduled a cab from the hotel at 7:20am.  I figured this way even if the cab didn’t show up I’d still have time to walk over to the office.  Admittedly I did arrive much too early to the office, but it’s always better to be early than late.  If you’ve ever had a class with Brian Dennis you’ve probably heard the same joke he’s been telling for the last 10 years: “I’ve been both early for the CCIE Lab Exam and I’ve been late for the CCIE Lab Exam.  The preferred method is to be early.”

Since it was only about 7:30am when I got there I walked around the campus for a while just to try to calm my nerves.  Ultimately I checked in with the receptionist, and made some small talk with some of the other candidates.  I was hoping to go incognito for the day, but immediately the first guy I said hi to said “aren’t you Brian McGahan from INE?”  Oh well… that's the price of being nerd famous.

The proctor Tong Ma came out to the lobby around 8:15am or so to collect us all and check IDs, and then did his spiel about the logistics of the lab location (e.g. where the bathroom was, the breakroom, lunch time, etc.).  8:30am was our official start time, so I sharpened my colored pencils, sat down at my terminal, logged in, and prepared for the fun.

Immediately all around me I heard the other candidates furiously pounding away on their keyboards.  This is what I like to call the “panic approach”. I on the other hand started with a different approach that had already worked for me three times in the past.  I took my first sheet of scratch paper, and started a quick drawing of the diagram I was presented with.  This was my first lab attempt using the new lab delivery system where everything is electronic, but regardless in past attempts you couldn’t draw on their diagrams anyways.

One point of drawing out the diagram for myself was to help me learn the topology, but more importantly so that I could take quick notes as to which technologies would need to be configured in which portions of the network, e.g. which devices were running vPC, FabricPath, OTV, FCoE, etc.

The next step was to read through the exam, to see what technologies were actually being tested on, and to plan my order of attack on how I was going to build the network.  One thing that I have found with my past CCIE tracks is that the order that they give you the questions in isn’t necessarily always the best order that you actually want to configure things.  After all they’re only grading the result at the end of the day, not the actual steps that you used to get there.

Once I had a basic understanding of what was covered, and had taken some notes on my diagram as to which features went where, I took my two other pieces of scratch paper (there were 3 total but you can always ask for more if you need), and drew out my two tables that I use to track my work.  For those of you that have attended a live class with me in the past or watched any videos I’ve done on lab strategy you may be familiar with this, but for those of you that haven’t seen this there’s basically two tables that I use to track my work during the day.  The first of which I use to track which sections that I have configured, how comfortable I am with the answer I gave, and which sections I skipped.  Throughout the day this helps me to know what sections I need to go back to at a later time.  Also at the end of the day this is the sheet I use to go back and check everything with a fine tooth comb.  The end result looks something similar to the picture below, but this one is just something I made up now it’s not from any real lab.

The way I read this at the end of the day is that all the tasks with a Check mark I’m 100% confident that the solution is correct.  Tasks with a ? mean that I configured something, but I’m not 100% if it’s correct or that I answered the question the way that they wanted.  Anything that is blank, like section 2.3 that means that I completely skipped that task, and that I’ll come back to it at a later time.  Once I’m done with all the tasks, I then circle back around to the tasks that I completely skipped to see if I can answer them, then revisit the ones with a ? that I wasn’t 100% sure about.  Finally the “2nd” column is for my double checking, where I start all the way at the beginning of the lab and re-read each question, re-interpret it, verify my answer, and if satisfied check it off again and continue.  In the case of the DC Lab Exam I ended up with two tasks with a question mark and one task with a blank at the end of the day.  In other words by my count there was one task I definitely was getting wrong, two tasks I had completed but wasn’t sure if I interpreted properly exactly what they wanted, and all other tasks I was 100% confident were correct.

The second of these scratch paper tables was to track my timing.  After all if they gave you a week to configure the lab, most candidates would probably pass.  With the 8 hour time limit ticking down though, not so much.  This is why it’s not only important to track your progress throughout the day to see which sections you’re confident about your answer, but also how long it’s taking you to configure them.  The end result of this table looks something like this:

The “Time” column represents the hour of the day.  The lab starts around 8 and ends around 5.  Between 8am and 9am, I got zero points.  The rest of the values in the table are made up, I don’t remember what the point values of the sections in my attempt were, but the first row is actually true.  From 8:30am to 9am I did not configure a single section, and I did not gain a single point.  Why?  Because I spent that half hour drawing the topology, reading through the tasks, and planning my attack.  While most people take the “panic approach” and immediately begin configuring the lab blind, I knew that even though it would cost me time up front to draw and read, it would save me time in the long run.  This did actually save time in the long run, because I finished about 95% of the exam by 2:30pm, which gave me a very relaxed two and half hours at the end of the day to double, triple, and quadruple check my work.

Getting back to the table above, between 9am and 10am, I completed – and was confident with the answers of – sections that were worth 2, 2, 3, 2, 4, 2, and 2 points.  Basically each time I completed a task and it had a check mark in the other table, I wrote the point value down here.  Each time I completed a section I also checked the clock on the wall to make sure I was writing the point value in the correct row.  The logic of using this table is simple:  the exam is broken down into sections totaling 100 points.  Excluding your lunch break, you get 8 hours to configure the lab.  This means at an absolute minimum you need to be averaging 10 points per hour in order to hit your goal of 80 points.  Now ultimately the totals on the right should be consistently be reading above 10 points for the early portion of the day, because you don’t want to configure exactly 80 points worth of sections with zero time left over at the end of the day to check your work.  In this situation it’s very likely that you’re failing the exam.  Instead you want to be consistently be hitting 14 points, 16 points, etc. especially early in the day, because then it makes you more relaxed that you’re not as rushed for time.  Remember that in the CCIE Lab Exam your biggest enemy is stress – well other than simply not knowing the technologies, that’s kind of an issue too – so whatever you can do to help calm yourself down during the day, do it.

For me personally constantly tracking my timing is one of those methods that helps to relax me.  When I hit about 1pm/2pm that day, I looked at my sheets that were tracking my work, sat back and said to myself “there’s no possible way you’re not passing this exam.” Now of course I didn’t really know for a fact that I was passing, ultimately only the score report can tell you that, but based on my point counts and how much time I had left to go back and double check I knew that I was golden.  This brings us to my next point, which is that “golden moment”.

The Golden Moment

Every CCIE Track and its associated CCIE Lab Exam has what has been commonly referred to as the “golden moment”.  This is basically the point in the exam that if you can reach, and you have everything working, your chances of passing are very high, i.e. you’re “golden”.  In the case of CCIE R&S it’s having full IP reachability everywhere.  In Security it’s when all your LAN to LAN and Remote Access VPNs work; in Service Provider it’s when you can ping between all your L2VPN and L3VPN sites; in Voice it’s when you can make all your phones ring, etc.  In the case of CCIE Data Center, the golden moment is undoubtedly marked by one point: can you get your UCS blades to boot from the SAN.

Pictured above, the Zen of Buddha passing over me when I knew that I had reached the golden moment.


The CCIE Data Center Lab Exam is very unique in my opinion based on the fact that essentially all tasks are cumulative and somehow interrelated in the exam.  For example in the case of CCIE R&S, you could theoretically skip complete sections, such as IPv6, Multicast, QoS, etc. and still pass the exam, as long as you can gain enough points from all the other sections.  For Data Center though, this is not the case.  All tasks in the exam essentially are getting you to work towards the golden moment, which is to actually get your servers to boot their OS.  All the technologies are so highly interrelated that the DC lab exam is a delicate house of cards. If one minor task is wrong, you’ve essentially bought yourself a $100 rack rental and a $1400 lunch for the day.

For example let’s take a theoretical CCIE DC lab scenario, and look at how the most minor of mistakes can snowball, and cause you to have a very bad day. Suppose we have two Data Center sites, “DC-A” and “DC-B”.  In DC-A we have our UCS B series blade servers, while in DC-B we have our Fibre Channel SAN.  Our ultimate goal is to get the blades in DC-A to boot from the SAN in DC-B. DC-A and DC-B have a Data Center Interconnect (DCI) provider between them that runs both IPv4 Unicast and IPv4 Multicast routing, and it’s up to us to configure everything so that it functions properly.  On one our our edge Nexus 7Ks though we forgot to enable jumbo MTU on the link to the DCI.  Minor problem, right?  Wrong, we just failed the exam! But why?

The UCS blade server was trying to boot to the Fibre Channel SAN.  Fibre Channel doesn’t natively run over the DCI though because it’s a routed IP network.  To fix this we first sent the FC traffic to our MDS 9200 switches.  The MDS switches in DC-A and DC-B then encapsulated the Fibre Channel into a Fibre Channel over IP (FCIP) tunnel between each other.  Additionally the MDSes were in the same IP subnet and VLAN in both DC-A and DC-B, so their FCIP traffic had to go over an Overlay Transport Virtualization (OTV) tunnel across the DCI.  The OTV tunnel was up and working.  The FCIP tunnel was up and working.  Both the UCS blade and the FC SAN successfully FLOGI’d into the Fibre Channel Fabric.  All of our Fibre Channel Zoning was configured properly.  The UCS blade server’s Service Profile associated properly.  We clicked the “Boot Server” button, connected to the KVM console of the blade, crossed our fingers, and got this:

Pictured above, someone having a very bad day in the CCIE Data Center Lab Exam.


No!  It didn’t boot the VMware ESXi instance!  This means that our Nexus 1000v didn’t come up either! I just lost 22 points and failed the exam!  Why did the CCIE Lab Exam Gods hate me!

Pictured above, 2274 bytes > 1500 bytes


Our OTV tunnel is actually Ethernet over MPLS over GRE over IP, or what is sometimes referred to as a Fancy GRE Tunnel. Our SAN traffic is SCSI over Fibre Channel over TCP over IP.  FCIP frames can go up to about 2300 bytes in size, but our default Ethernet MTU was 1500 bytes.  OTV doesn’t support fragmentation, and sets the DF bit in its header.  This means that since we forgot to type the single command mtu 9216 on the edge switch connected to the DCI, our SCSI over Fibre Channel over TCP over IP over Ethernet over MPLS over GRE over IP was dropped, and we had to do the walk of shame out of building C in San Jose as we knew the CCIE Lab Exam had defeated us that day.

This of course is just one of a thousand different possible examples of how the house of cards that is the Data Center Lab Exam fits together, but you get the idea.  Luckily for me however, when I clicked the “Boot Server” button in UCSM this week, the results were closer to the output below.

Pictured above, someone doing the happy dance in the CCIE DC Lab Exam.


In Conclusion

For those of you still reading I hope that you found this post both helpful and informative.  If you’re still on the fence about pursuing the CCIE Data Center Track I would definitely say that it’s worth it.  If you told me 12 years ago when I got out of server support that I’d be back in the server support market today I’d never have believed it, but without throwing around too many buzzwords like public/private cloud etc. this Data Center space is clearly here to stay, and will only continue to grow.  Especially with how rapidly the market has been changing within the past few years with virtualization and an entire plethora of new technologies and design philosophies, it’s more important than ever to try to differentiate your skill set in the market.

Thanks for all the well wishes, and good luck in your studies!


This evening it's topping 100 degrees in Reno, NV where our rack rental equipment is located. I'm looking at the temperature in our new data center to see how it's holding up in regards to the high exterior temperature. We have temperature sensors for each cage in our existing data center but so far only have a few installed in our new data center. I'm looking to see what the temperature is in a couple new cages in the new data center that don't have a temperature senor installed yet. So how can we get the temperature without a dedicated senor and only Cisco devices installed?

Relatively newer Cisco hardware has the ability to display the numerical temperature values by using the show environment command along with polling via SNMP. For the ISR G1's (1800, 2800, 3800) the internal-ambient, CPU, intake and backplane temperature (3845) and for the ISR G2's (1900, 2900, 3900) the intake left(bezel), intake left, exhaust right(bezel), exhaust right, CPU and power supply unit temperature can be displayed/polled. I wanted to see the temperature of the management access server (3825's) located at the top of each cage. I assumed I would just login and issue the show environment command to see the values. Good idea but I don't want to check it manually every few hours.

I could just login to SolarWinds and see the temperature but as we network engineers know, network management applications aren't that fun to use. You buy them, install them, swear they are the best thing since sliced bread, drool over the fancy graphs and then two months later you never log back into them unless something is wrong. Plus my normal thought process is to always try and do whatever possible via the IOS as it's the most fun.

To start I'll display the values via the show environment command and then poll the values using the SNMP via TCLSH and finally wrap it up with an EEM applet to log the values.

Row8Cage1AS#show environment
Redundant Power System is not present OR in standby mode.

SYS PS1 is present.
Type: AC

AUX(-48V) PS1 is absent.

Fan 1 OK
Fan 2 OK
Fan 3 OK

Fan Speed Setting: Normal

Alert settings:
Intake temperature warning: Enabled, Threshold: 55
Core temperature warning: Enabled, Threshold: 70 (CPU: 95)

Board Temperature: Normal
Internal-ambient temperature = 33, Normal
CPU temperature = 50, Normal
Intake temperature = 32, Normal

Voltage 1(3300) is Normal, Current voltage = 3300 mV
Voltage 2(5150) is Normal, Current voltage = 5153 mV
Voltage 3(2500) is Normal, Current voltage = 2525 mV
Voltage 4(1200) is Normal, Current voltage = 1215 mV

Nominal frequency


Now I need to find the SNMP OID related to temperature for the platform. Note that SNMP has previous been setup on this router.

Row8Cage1AS#show snmp mib | in Temperature

I know it's one of the ciscoEnvMonTemperatureStatusEntry values but which one? One should be the name, another should be the actual values and another should be the thresholds. Seems like Google should be used here but we know the values via the show environment command so we could poll them and see which one matches. We'll learn more this way over using Google. We'll start with the first one and walk down 99.

usage: snmp_getbulk community_string non_repeaters max_repetitions oid [oid2 oid3 ...]
Row8Cage1AS(tcl)#snmp_getbulk XXXXXX 0 99 ciscoEnvMonTemperatureStatusEntry.2
{<obj oid='ciscoEnvMonTemperatureStatusEntry.2.1' val='Internal-ambient'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.2.2' val='CPU'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.2.3' val='Intake'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.1' val='33'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.2' val='50'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.3' val='32'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.4.1' val='70'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.4.2' val='95'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.4.3' val='55'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.5.1' val='0'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.5.2' val='0'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.5.3' val='0'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.6.1' val='1'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.6.2' val='1'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.6.3' val='1'/>}

Row8Cage1AS(tcl)#snmp_getbulk XXXXXX 0 3 ciscoEnvMonTemperatureStatusEntry.3
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.1' val='33'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.2' val='50'/>}
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.3' val='32'/>}

That was easy. The ciscoEnvMonTemperatureStatusEntry.3 is what we are looking for and ciscoEnvMonTemperatureStatusEntry.3.1 is a good value to use as it's giving us the "internal ambient" temperature.

usage: snmp_getone community_string oid [oid2 oid3 ...]
Row8Cage1AS(tcl)#snmp_getone XXXXXX ciscoEnvMonTemperatureStatusEntry.3.1
{<obj oid='ciscoEnvMonTemperatureStatusEntry.3.1' val='33'/>}

Now how about having this value logged every 5 minutes so we can check back in the morning to see any changes overnight. An easy way to do this is to poll the SNMP OID using EEM and log the value returned using syslog if it's above 1 degree which we know it will always be. This way EEM will always log the value to syslog.

Row8Cage1AS#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Row8Cage1AS(config)#event manager applet EEM_TEMP_MON
Row8Cage1AS(config-applet)#event snmp oid ciscoEnvMonTemperatureStatusEntry.3.1 get-type exact entry-op gt entry-val 1 poll-interval 300
Row8Cage1AS(config-applet)#action 1.0 syslog msg "Row8Cage1AS Temperature $_snmp_oid_val"
Jun 7 06:53:42.011: %SYS-5-CONFIG_I: Configured from console by bdennis on vty0
Jun 7 06:54:12.012: %HA_EM-6-LOG: EEM_TEMP_MON: Row8Cage1AS Temperature: 33

We could convert the value to fahrenheit if we wanted by using this TCL expression: set temp [expr {9.0*$_snmp_oid_val/5.0+32.0}].

Of course using a network management application or script on a server would be easier but doing it via the IOS was fun. There are a few other ways to do this via the IOS and one being SNMP MIB BULKSTAT.


Here is our updated and simplified CCIE bootcamp reseat policy:

Bootcamp Retake Policy
INE invites all students to retake a Bootcamp course for only a $500-per-week rack rental fee. Routing & Switching Bootcamps are excluded from the fee and are free of charge. How do you know whether seats are available for a Bootcamp retake? It's simple! As long as the Bootcamp is not labeled “Sold Out” on our website, you can retake it. Unlike other training companies, we do not limit the number of seats for students retaking a course.

It really doesn't get any better and simpler than this. For the non-R&S CCIE bootcamps if you calculate the reseat fee out the $500 is cheaper than you renting the rack yourself for a week especially for our Service Provider bootcamps and new Data Center Bootcamps.


Below are the steps needed to get the CSR1000v working in VMware Fusion on a Mac. Before we look at the steps here is the hardware and software I used:



Basically you need a quad core machine with a hypervisor (VMware, Virtual Box, KVM, etc). I just used VMware Fusion as it's a commonly used hypervisor. Additionally I used two USB NICs to map to the GigabitEthernet1 and GigabitEthernet2 interfaces inside the CSR1000v. You could use any NICs or even subinterfaces if you are using Linux (Mac OS does support VLANs but I personally have not tried it with VMware Fusion). The GigabitEthernet0 is the management interface that is associated with the "Mgmt-intf" VRF so I didn't want to use it for the internal and internet facing interfaces.

Download the CSR1000v OVA (Open Virtualization Archive) file here if you haven't already done so. Note that you will need to use your Cisco CCO login to download.

Once the file has been downloaded, start the installation by finding the .ova file and double-clicking it. It will then open the VMWare Fusion window below:

VMware is now importing the machine:

Now that the machine has been imported you can start it.

Hit enter on the virtual console from the GRUB menu:

The CSR1000v will generate a lot of log messages as it goes through the first bootup process.

The CSR1000v will reboot after this next screen:

Now the CSR1000v is up and running.

We need to map the GigabitEthernet interfaces from the CSR1000v to the USB NICs.

We can verify the CSR1000v's interfaces with VMware. A good way is to check the MAC addresses as VMware will show you the MAC address of each NIC in the settings.

It's now ready to apply your configuration. Last night I used it as my home router (50Mbps down and 20Mbps up) and here was the performance numbers after I activated the 50Mbps throughput demo license/span>

CSR1000v(config)#platform hardware throughput level ?
10000 throughput in kbps
25000 throughput in kbps
50000 throughput in kbps

CSR1000v(config)#platform hardware throughput level

Here is the very basic configuration that I used in the CSR1000v to provide internet access:

CSR1000v#sho run
Building configuration...

Current configuration : 1816 bytes
! Last configuration change at 04:06:59 UTC Fri Apr 19 2013
version 15.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console virtual
platform hardware throughput level 50000
hostname CSR1000v
vrf definition Mgmt-intf
address-family ipv4
address-family ipv6
enable secret 4 wnIsLyS.p9pNIRVWPyb98mg0322nrnyQVqPabl7clC1
no aaa new-model

no ip domain lookup
ip dhcp excluded-address
ip dhcp excluded-address
ip dhcp pool INSIDE
multilink bundle-name authenticated
license accept end user agreement
spanning-tree extend system-id
mode none
ip tftp source-interface GigabitEthernet0
interface Loopback0
ip address
interface GigabitEthernet1
ip address
ip nat inside
negotiation auto
interface GigabitEthernet2
ip address dhcp
ip nat outside
negotiation auto
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address dhcp
negotiation auto
ip nat inside source list ACL_NAT interface GigabitEthernet2 overload
ip forward-protocol nd
no ip http server
no ip http secure-server
ip access-list standard ACL_NAT
ip access-list standard ACL_TELNET
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
access-class ACL_TELNET in
privilege level 15
password 7 XXXXXXXX


Note that you may find it easier to configure the G0 interface within VMware as "host only" so that it'll provide an IP address to the CSR1000v's management interface via DHCP. This way all you need to do is enable telnet access under the VTYs and not have to work on the VMware virtual console as there are limitations with cut/paste along with scroll back.


The Cloud Services Router, CSR1000v, which was first announced at Cisco Live in San Diego last year is now available for download. For those who have never heard of the CSR1000v, in a nutshell it's an IOS XE image running in a virtualized environment (VMWare support now, Citrix XEN, Amazon, Windows Hypervisor and OpenStack in the near future). The CSR1000v is designed as a virtual router (think Quagga but IOS from Cisco) that resides on the hypervisor server as a client instance and provides any services a normal ASR1000 would provide between the other hypervisor's client instances (Linux servers, Windows servers, etc) and the network infrastructure. This could be something as simple as basic routing or NAT all the way up to advanced technologies like MPLS VPNs or LISP. Basically anything that a real ASR1000 could provide you can do in the CSR1000v with a small exception. This early release only supports 50Mbps throughput due to licensing restrictions. This being the case the specifications from Cisco shows the CSR1000v being capable of pushing 1.17Gpbs max and 438Mbps throughput with an IMIX traffic flow. With the 50Mbps limit it means the CSR1000v at this stage would be deployed more for proof-of-concept as opposed to real production. At this time the throughput is limited by the licensing but I'm sure there are plans to allow for higher throughput licensing in the near future.

If you're interested in a presentation from Cisco covering the CSR1000v you can view the Cisco Live session from London: Cisco’s Cloud Services Router (CSR 1000V): Extending the Enterprise Network to the Cloud (2013 London) Note that you will need to create an account if you don't already have an account to view the session.

Personally I've been testing out the CSR1000v over this past week and to say the least I'm really impressed at not only the product itself but the possibilites for the CSR1000v. The version available for download is IOS XE 3.9.0aS aka IOS Version 15.3(2)S which means basically any feature you want or need is in this release.

IOS-XE1#show version
Cisco IOS Software, IOS-XE Software (X86_64_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.3(2)S0a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Sat 30-Mar-13 19:08 by mcpre

IOS XE Version: 03.09.00a.S

Cisco IOS-XE software, Copyright (c) 2005-2013 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE


IOS-XE1 uptime is 4 days, 24 minutes
Uptime for this control processor is 4 days, 25 minutes
System returned to ROM by reload
System image file is "bootflash:csr1000v-packages-adventerprisek9.03.09.00a.S.153-2.S"
Last reload reason:

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

If you require further assistance please contact us by sending email to

cisco CSR1000V (VXE) processor with 1141428K/6147K bytes of memory.
Processor board ID 926O40RARZR
2 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

IOS-XE1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
IOS-XE1(config)#router ?
bgp Border Gateway Protocol (BGP)
eigrp Enhanced Interior Gateway Routing Protocol (EIGRP)
isis ISO IS-IS
iso-igrp IGRP for OSI networks
lisp Locator/ID Separation Protocol
mobile Mobile routes
odr On Demand stub Routes
ospf Open Shortest Path First (OSPF)
ospfv3 OSPFv3
rip Routing Information Protocol (RIP)

IOS-XE1(config)#router bgp 1
IOS-XE1(config-router)#address-family ?
ipv4 Address family
ipv6 Address family
l2vpn Address family
nsap Address family
rtfilter Address family
vpnv4 Address family
vpnv6 Address family

IOS-XE1(config)#ipv6 unicast-routing
IOS-XE1(config)#router ospfv3 1
IOS-XE1(config-router)#address-family ?
ipv4 Address family
ipv6 Address family

IOS-XE1(config-router)#address-family ipv4
Router Address Family configuration commands:
area OSPF area parameters
auto-cost Calculate OSPF interface cost according to bandwidth
bfd BFD configuration commands
compatible Compatibility list
default Set a command to its defaults
default-information Control distribution of default information
default-metric Set metric of redistributed routes
discard-route Enable or disable discard-route installation
distance Define an administrative distance
distribute-list Filter networks in routing updates
event-log Event Logging
exit-address-family Exit from Address Family configuration mode
graceful-restart Graceful-restart options
help Description of the interactive help system
interface-id Source of the interface ID
limit Limit a specific OSPF feature
log-adjacency-changes Log changes in adjacency state
max-lsa Maximum number of non self-generated LSAs to accept
max-metric Set maximum metric
maximum-paths Forward packets over multiple paths
no Negate a command or set its defaults
nsr Enable non-stop routing
passive-interface Suppress routing updates on an interface
prefix-suppression Enable prefix suppression
queue-depth Hello/Router process queue depth
redistribute Redistribute information from another routing protocol
router-id router-id for this OSPF process
shutdown Shutdown the router process
snmp Modify snmp parameters
summary-prefix Configure IP address summaries
timers Adjust routing timers


I've tested a lot of features (LISP, BGP, NAT, etc) over this past week and haven't stumbled across any issues related to the CSR1000v itself. The only issues I've ran into related to not giving the VM the required 4GB of RAM and it would complain when I started enabling features. The CSR1000v doesn't use much CPU but it does require 4GB of RAM per instance. I've had over 30 CSR1000v instances running on my ESXi 5.1 development server without a problem.

So what does this mean for people preparing for a Cisco certification or just looking to learn networking? I hate to use buzzwords but this really is a game changer. The CSR1000v is like "crack" for a networking addict. You want to learn a technology or a feature then the CSR1000v is really the only way to go if you don't have or want to setup real hardware. I have access to tons of real hardware and I would never consider giving it up for GNS3 or IOU but I would for the CSR1000v. The image is very stable and fast plus has everything you could need. Want to learn LISP? Want to learn OTV? Want to learn MPLS VPNs? What to learn VPLS? Want to learn OSPFv3 for IPv4? The CSR1000v is the way to go.

I'll post a detailed instruction tomorrow on how to install and get it up and running based upon my experience running it in VMWare ESXi and on VMWare Workstation but here are a couple quick notes for those who can't wait. VMWare ESXi Enterprise version has a built in console server which will allow to you via the virtual serial console over the VMWare console.

Brians-MacBook-Pro-3:~ ccie2210$ telnet 2004
Connected to
Escape character is '^]'.

IOS-XE4#show version | in LINUX
Cisco IOS Software, IOS-XE Software (X86_64_LINUX_IOSD-ADVENTERPRISEK9-M), Version 15.3(2)S0a, RELEASE SOFTWARE (fc1)

This is really nice to have. If you don't want to fork out the crazy money for the Enterprise licensing from VMWare then you could run the demo for 60 days and then reinstall or you could create a "management" interface in each instance and use it for telnet access. The CSR1000v allows you to create as many interfaces on the router as the host machine will support so creating one dedicated to management isn't a problem.

IOS-XE4#show platform software vnic-if interface-mapping
Interface Name Short Name vNIC Name Mac Addr
GigabitEthernet0 Gi0 eth0 (vmxnet3) 000c.2965.dc56
GigabitEthernet2 Gi2 eth2 (vmxnet3) 000c.2965.dc6a
GigabitEthernet1 Gi1 eth1 (vmxnet3) 000c.2965.dc60

IOS-XE4#show run int g0
Building configuration...

Current configuration : 113 bytes
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address
negotiation auto


Lastly when you install the CSR1000v it comes with a 60 day license for 50Mbps throughput. After that expires it drops to 2.5Mbps which is still fine if you are using it in a test environment or for learning.

IOS-XE4#show platform hardware throughput level
The current throughput level is 50000 kb/s

As I said earlier, tomorrow I'll post a detailed instructions on how to setup a lab using the CSR1000v along with how I'm using the CSR1000v as my home router ;-)

Subscribe to INE Blog Updates