Brian Dennis and I attended Cisco Live! - Networkers this week, and both enjoyed the privilege of sitting down to talk privately with Yusuf Bhaiji (Program Manager over the entire CCIE program) and Ben Ng (Program Manager over the CCIE Voice track) for roughly 45 minutes. It was quite an enjoyable and spirited talk, and I believe it benefited both sides - our side to gain a better understanding of why some of the choices have been made, and theirs possibly to see things a bit more 'through the eyes of the typical hard-studying student'. I would like to take a moment to jot down some of the highlights from our conversation, and then unpack them in a bit more detail, so that you may benefit from the open conversation.
I'll jot down some very simple, high-level topics that were discussed during our conversation, and then unpack them in more detail in the following section.
- Upcoming changes to every CCIE Lab Exam
- Protecting the integrity of the CCIE certification
- Robust, matured results-based grading engine
- Heuristic logic embedded into task wording
- Accuracy and detail of lab score reports
- Cisco's CCIE Lab Delivery System and virtualization for mobile labs
- No re-reads
- CCIE Voice
- Next blueprint version expectation
- Topics for current and next blueprint versions
- CCIE Data Center
- CCIE Storage grows up
- Reason behind Cisco.com CCIE Statistics web page being removed
Let's Unpack This a Bit More
Firstly, if I could sum up our entire conversation into one, clear theme, it would be "Protecting the Integrity of the CCIE Program". Indeed, both Yusuf and Ben clearly stated that central theme was the primary guiding principle behind every meeting they have regarding any aspect of the program, every planning session for the future of the program, and every decision made to add, remove and/or change anything in any of the tracks.
The main focus is to ensure that candidates who are attempting the lab exams are what Cisco is certifying them as: Experts. To ensure that no one is able to memorize any aspect of the exam, and to make sure that those of us that are teaching these potential candidates about the technologies involved, are in fact teaching our students everything they need to know about a given group of technologies necessary to become a true expert - something INE has always worked incredibly hard to do. This focus (and the changes it will affect) implicitly deals with the problems that have arisen in the past with a small minority of people who dishonestly try just to memorize parts of the exam, as well as companies who blatantly cater to that small minority by attempting to gain access to a given exam and publish entire exams, or even parts of them.
Now, this has always been a driving focus for the program, however they realized that while some of their efforts in the past to this end have been successful, that many also may not have succeeded in the manor in which they had intended for them to. Case in point: Core Knowledge/Open-Ended Questions (and their complete removal from all tracks). Another attempt at this from long ago -and still in place but soon to possibly change- was the 'detailed score report' (or shall I say 'lack' of detail in the score report). More on that last bit in a moment.
Upcoming Changes to Every Lab
So, onto the specifics of how they plan to accomplish this. With no definites given as to exactly how or when the implementation of any of these initiatives would be implemented, some things were quite frank and very clearly stated. They talked about how their lab grading engine has been in use for some time now, and has reached quite a mature level with it's results-based scoring (both terms of it's accuracy as well as it's modularity). Now would be remiss if I didn't take a quick aside to say when I mention that it has reached a high level of accuracy, some might immediately jump to the conclusion that it wasn't accurate before, and that might have been the reason for their failed attempt. Just to quell those fears, a (human) proctor has always, and will continue to look over and verify a candidate's lab and scripted grading results before issuing the final grade. Now with the fact that the engine grades based on results-based testing (and not on any specific commands input into the configuration), as well as the key point that it is extremely modular (due to it's structured XML nature), any given task can be tested in the same way, however worded in 10-15 different ways on different iterations of any given lab exam. This leads naturally to the ability for them to have upwards of 25 or 50 labs in rotation at any given time for any given CCIE track and blueprint, which makes it nearly imposible for anyone to actually memorize any/all of the questions or versions of lab exams. Thus making those that pass, truly validated as experts. They spoke of a sort-of heuristic logic woven into all of the task wording to accomplish this. Also talked a lot about was how troubleshooting was one of the best things that they added (added 'back' I should say, since it used to be there in the 2-day exam). So look for that to not only continue, but permeate its way through more exams in more ways. No mention was specifically made as to whether any other track outside of R/S would go to the same sort of segregated TS section that it utilizes - so it may happen, it may not. Just have to wait and see.
They have been developing towards this goal for some time now, they know exactly how they will carry it out (however to disseminate that specific knowledge would be completely counter-intuitive to the very idea of integrity protection - so anyone telling you they 'know' how it's being done clearly doesn't know what they are talking about). They mentioned that it will be occurring soon, though when exactly won't ever be known or disclosed, and it won't be necessary to 'announce' it per-se, since it won't require the upgrade/change of any hardware/software or technology-specific topics.
All of what we've talked about up till this point, this obviously doesn't negatively impact any of our readers here, as you all are true learners, true knowledge-seekers - those who wish to not only know, but to truly understand (to paraphrase Einstein, if I may). However, it may have some positive implications that you have not yet considered. Aside from the obvious fact that it preserves the integrity of the prestigious certification that you have (and continue to) worked so diligently towards achieving, another possible benefit from this is that since the CCIE program has/will-have this new ability to have so many exams in rotation (and thus exponentially-if-not-entirely reducing the possibility of memorization of tasks), they can relax a bit on their very ambiguous failed score reports. For those of you who are not yet aware, while some candidates pass on their first attempt, most do not. In fact the average (very loose average) is roughly 2-3 attempts at any given CCIE lab before a 'Pass' is awarded, and you only receive a broken down score report if you fail - if you pass you simply receive a 'PASS' (and frankly, you don't care that you didn't receive a report). Now they didn't state for certain that this would in fact occur (more detailed reports), but they did indicate that it was being discussed internally as a strong possibility - since the grading engine obviously reports back in extreme detail - although he (Yusuf) said it would never include specifically which tasks you got right vs wrong. However, he did also mention that the purpose of the CCIE Program is not to train candidates or provide feedback - it is to test them. It is the responsibility of us, the CCIE training providers, to not only teach, but pre-test and provide accurate feedback to students before they make the actual journey to the Lab and truly become candidates for admission into the prestigious certification that is the CCIE. That by the way, is something that INE is committed to, and indeed already provides more than adequate resources for CCIE R/S, and will soon be adding for the CCIE Voice as well (more on that later).
Another one of the initiatives of the program is to make it much more widely accesible to everyone, everywhere. This was the impetus behind the original idea of having the lab able to be taken from any Prometric/Vue testing center. But Vue wasn't really ready to handle the stringent requirements, and thus the attempt didn't ultimately succeed. Then came the CCIE Mobile Testing Labs. Those worked. Well. Really well in fact. And now the push is to make every single track able to be tested in that same mobile fashion. However, there are a few challenges to overcome first, before that can become a reality. Take CCIE Voice for instance - in order for the Voice lab to become truly mobile, everything has to be ported to Cisco's CCIE Lab Delivery System (where everything: the tasks, the desktop for CLI/GUI, etc. are all completely virtualized). This means the phones themselves as well. Softphones won't do. They don't behave at all like hardware phones. So they are working on creating a completely virtualized hardware-like phone, that behaves exactly like a hardware phone (but doesn't remotely control one). When they get that completed, then you will begin to see the Voice lab become available in the mobile testing centers (along with every other lab once their similar challenges are met).
One last thing I asked Yusuf about regarding specifically the CCIE Voice lab exam, was why they hadn't yet allowed for re-reads (the ability that, if you fail an exam but think you should have passed, you can pay $300 USD to have your lab re-graded manually). He mentioned that since the overall percentage since they began offering that option was so extremely low of those who request a re-read actually results in a grade being overturned (we're talking like 1 out of every 5,000), not to mention that that number has dropped even more with results-based scripted grading, that the focus was not to add more of that unnecessary burden on the already taxed proctors, but to reduce it. So while he didn't provide any guidance on when (if ever) they will completely eliminate that practice from the other tracks (R/S, Sec, SP), he did say that the focus is on removing that option from those tracks vs. adding that option to other tracks (Voice, WiFi, Storage).
There was a very interesting breakout session this year entitled "CCIE Voice: Cryptography in Cisco Unified Communications", which inevitably led to the question by participants: "If this class is prefaced by the title 'CCIE Voice', does this mean that Cryptography/Security is going to begin being tested in the lab exam?". The answer to this when asked by the attendee and then later in a bit more detail by me was answered with the basic answer of (and I'll paraphrase): "This has been tested in the written exam for some time now, however there is absolutely nothing stopping us from testing this in the lab today with version 7 of the various UC platform servers, and obviously no problem testing it moving forward to UC version 8 (or 9, etc), as even more security has been added to the new version of UC platforms".
Of course, in the main CCIE Voice 8-hour Technical Seminar, the question was asked by some participants (as well as by me in more detail later in private) when we might anticipate the Voice lab being updated to UC platform version 8.x (or beyond, if FCS'd for better than 6 months by time of announcement), to which Ben gave no real guidance publicly, yet in private alluded to (reverting back to our previous mention in this blog post) the desire to virtualize the hardware phones and deliver the exam with the new virtual Lab Delivery System, so that they could support the mobile labs.
Now of course 8.x has been in production for well over a year now in many networks around the globe, but 9.x doesn't FCS (First Customer Ship) until April '12, meaning that they could possibly update the lab to UC platform version 8 anytime (with a standard 6-month pre-announcement of course), but to update to version 9 would mean that they couldn't even announce a new lab based on that version until Oct '12, with it going live around Apr '13 at the earliest. I have no idea at all which they will end up doing, and from talking with Ben, he made it seem like they hadn't reached an internal decision yet either. In fact it probably will largely depend on how quickly they get that virtualized hardware-like phone put into production in all reality.
Either way, it doesn't really matter. INE has you covered. The major new things in CUCM ver 8.x that can be tested are the following:
- Call Control Discovery over the new Services Advertisement Framework (CCD / SAF) with RSVP SIP Preconditions
- Extension Mobility Cross Cluster (EMCC)
- SIP Normalization using the Lua scripting language
They can't really test the Intercompany Media Engine (IME) - it's just not technically possible. EMCC is easy, and within the day of them announcing this being tested I'll have a minimum 2-hour video ready. I already have over 4 hours recorded on every aspect of CCD over SAF and another half hour of the RSVP with SIP Preconditions in our new CCNP Voice product (which should be posted to our CDN in about a week). SIP Normalization with Lua -- eewww -- let's hope they don't test you on it - but either way, we'll have a video for you the week they announce it.
There are way too many small new features in UC 9.x to list here, -- I'll do a post covering those in the not-too-distant future, but needless to say, it won't be hard to add content to what we have to cover it. The best news is that everything you are studying today, is 100% completely relevant to any new version of the lab that could be announced, whenever they decide to do so -- so don't loose an moment of sleep over a possible upgrade -- you'll be 98% ready anyhow.
As for the possible testing of Cryptography/Security, I will be adding labs to our Volume II Workbook here in the upcoming month, with at least one of which will specifically address this topic. Our racks will allow you to test those features related to security to coincide with the release of that lab.
CCIE Data Center
Now, onto Data Center. Data Center was easily, hands-down, the single largest topic discussed at this year's Cisco Live! event. Not to mention that VMware timed the announcement of version 5 of their suite of virtualization products (including vSphere, vCenter Site Recovery Manager, vSphere Storage Appliance, and vCenter Heartbeat Center) to be exactly 1 hour prior to (and thus ending with the beginning of) John Chamber's Keynote address at Cisco Live!.
But here's how everything relates to the CCIE program. It was very clear that the CCIE Storage is going to become the CCIE Data Center. In fact, aside from that being very clearly stated, the breakout this year was entitled: "Cisco Data Center/Storage Certification". It was stated that the following would be what comprised the new CCIE DC.
Written exam will include (this is their wording copied verbatim):
- Revised Smaller version of the existing SAN Track blueprint
- MDS device operation, Advanced FC Features, SAN extension & switch Interop
- SAN Management will be integrated in the new overall DC management
- In addition to new topics to include:
- Basic Data Center L3 topology
- Data Center Access Layer deployment
- L2, vPC, Fabric Multipathing, QoS
- Unified I/O, FCoE , DCBX
- Unified Computing System (UCS)
- Load Balancing techniques and algorithms
- Branch WAN Acceleration
- Data Center Management
Lab exam will look like this (this is their wording copied verbatim):
- MDS will remain in the lab as well as 3rd party FC switches
- We will consider adding DC solutions and technologies that can be deployed on the following Cisco Products:
- MDS SAN Switches
- Nexus 7000, 5000 and 2000
- Cisco Unified Computing Systems (UCS)
- Application Control Engine (ACE)
- Global Site Selector (GSS) in case of DR scenario
- Wide Area Application Services (WAAS)
- Data Center Management for both LAN & SAN
- Virtualization with Nexus 1000v
Of all those things that can be tested, the most obvious ones that would almost have to be included are the Nexus line of DC switches and the UCS blade servers, complete with Fabric Interconnects and Virtual Interface Cards. So those are the things that INE will begin immediately to record video-based lessons on, adding them to our All Access Pass. Guidance wasn't provided on exactly when this track might go into live testing, though sources tell us that we may be less than 12 months away from it going live. Watch this blog for announcements soon on when you might expect the first of those Nexus and UCS training videos. Storage wasn't big. Data Center is already huge. The CCIE Data Center is going to be as well.
I mentioned above two things that I forgot to include, so I will add them in here.
First off I mentioned that we will be adding graded mock labs to the Voice track. Please email me directly if you would be interested in participating in a graded mock lab. It would involve using a dedicated rack for 8.5 hours (8 hours for config and .5 for lunch, just like the real lab) with basic minimal access to the proctor (myself) for basic question clarification, but no real assistance (again, just like the real lab).
Second thing was the reasoning behind the removal of the CCIE stats page from cisco.com. This might sound like a strange reason - I thought so, but after listening to Yusuf talk a bit more about it, it did make good sense in the end. The reason was completely centered around the fact that when one updates his/her cisco.com testing profile with the proper home mailing address, and most importantly home country, and then takes and passes any CCIE exam, his/her CCIE number is forever associated with that country. The problem was, people didn't always stay in that country. They sometimes moved, as is a reasonable assumption. However, the CCIE stats page wasn't designed as a synchronous page that would do a real-time DB lookup each time it was loaded, and so it would always report X number of CCIEs in X country. Yusuf used himself as an example. He's from Pakistan, and so his CCIE was basically 'registered' there (at least so far as that stats page went). Problem is, he moved to Australia for a number of years, but according to that stats page, his CCIE was still in Pakistan. Then he moved to Dubai in the UAE. CCIE? Still in Pakistan. So why should any of that matter? Well, to you and me - it might not. However, when a Cisco Partner in Pakistan (just continuing our example of Yusuf) is told by their Channels team: "You must have 4 CCIEs to become a Gold Partner" (or CCDE's, they count now too for that metric), and maybe the parter reports back: "But there aren't enough CCIEs in Pakistan to accomplish that!", the Cisco Channels team would just pull up that stats page, point to it and show the Partner and say: "Yes there are, see here?" (Again, Pakistan is just an example country, I have no idea how many CCIE/CCDEs there are there, so please don't think I'm being partial for/against that or any other country in any way - nothing is implied). It actually became a very, very big issue with Channels and Partner certifications. And Cisco is a large organization. If any of you have worked for one, you know that a team like the CCIE program has nothing to do with web page programming - that's a completely different part of the business. And to get something changed there requires a requisition to be submitted, go through various levels of approval, and finally implementation. And believe it or not, it took about 4 months for that process to occur, and by the time the implementation of it was being carried out (the removal of that page off of cisco.com), it truly just happened to coincide with a period just following a CCIE R/S change that was resulting in less people passing the lab at that specific point in time. Yusuf stated that it couldn't have been worse timing, however it truly was purely coincidental. He also mentioned that - yes, for a period of time after any type of a change to any lab track, there is always a fall-off in the number of CCIEs awarded for that track, but then it always picks back up. This is perfectly natural for any type of change for a number of reasons. 1) People stop booking a given lab en-masse right after a new version is announced - their basically afraid of what they don't know (aren't we all to some degree?). 2) If you sit a brand new lab version, and have no idea what to expect, you might be thrown for a bit of a loop, and therefore loose a bit of time you would expect to be productive during that lab attempt. By the way, this doesn't have to be. Take a bootcamp course from veterans of the lab such as Brian & Brian (and possibly counting myself as a veteran at this point, I guess I'm getting up there in years :-), my first lab attempt was in '02, so I'm going on 10 years next year .... wow -- although Brian Dennis has his 15 year anniversary coming up in just a few months!), and anyway, you'll be prepared no matter what they throw at you, and there will be no need for you to be counted in with the stats of people that 'don't know what to expect after a new version change'. Anyway, he finally mentioned that while Cisco doesn't (and won't) publish the exact statistics of CCIE Pass/Fail, if you look at the overall average number of passing scores over the life of any blueprint version, those numbers have always been, and will continue to keep trending upward.
Well, that's about all I can think of at the moment. I just finished a long flight from Las Vegas to Minneapolis sitting next to Louie Anderson, and to be honest, I'm not sure how I got any writing done. That guy's funny. I need to see him next time I hit the strip.