My CCIE SP Lab Experience

After about seven months of study time, I sat my CCIE Service Provider lab this week. I passed! Heres my story.

After finishing up my R/S I knew that I wanted to do either the JNCIE SP, or the CCIE SP. Why? Because I really dig routing… like a lot (also tacos), I just think its super interesting, and obviously I am a glutton for punishment. I had a bit of a delay in even deciding what the hell I was going to do while I moved to Seattle and some other life shenanigans, but at some point I decided on the CCIE SP. I picked the CCIE over the JNCIE because I rarely touch Juniper gear in my job (which does make me a bit sad, but is the reality), and the lack of quality study/lab guides was a very big detractor.

After acquiring all the gear required to do the INE labs, I hit the books/rack. My routine would be to wake up at 5 and do as much lab work as I could every morning before heading to work. Sometimes this would be a lot of time, other times I wouldn’t be able to do much. All in all though I was able to average probably about 18ish hours a week of dedicated lab time.

The INE material was (is) awesome. Their SP workbook has two sections, the first is the “technologies labs,” and the second is basically mock labs. The technology labs are deep dives into each of the main topics that are on the blueprint — for example the VPN section (largest by and far) contains basically mini labs for CSC, Inter-AS VPNs, 6PE, 6VPE, etc. I went through all of these several times in order to build up some confidence and understanding. Having not worked for an SP a lot of this was very new to me. I’d poked ISIS and OSPF plenty, I’d spent a lot of time working on basic MPLS VPNs during my R/S studies, but the rest was very new and very overwhelming.

After a month and a half or so of that I was finally able to jump into the mock labs. I started the first one and, pretty much as expected, got my ass handed to me. I did lab one and two multiple times over trying to get to a point where I was fully understanding every issue I ran into — using these times to find topics in the Cisco documentation. I kept at this for a while and eventually realized I needed to go back to revisit some topics — particularly CSC.

CSC (or any other topic) isn’t particularly hard in a vacuum… but it is very easy to get ‘in the weeds’ and miss easy things when you are working through  mock labs, and certainly the real deal. As always, slowing down and focusing on one thing at a time makes all the difference. The lab in general is a huge undertaking and is insanely intimidating, but when you slice it up into little chunks it becomes much more manageable.

I kept churning through the INE labs and jumping back to the Cisco documentation, making little spin-off labs of my own to further test/validate my understanding of topics, and eventually I started feeling pretty good. Good enough to schedule my lab. I had about two months out and that meant zero hour.

The last two weeks before the lab I was feeling even better. I felt like I had finally broken every INE lab in every possible way and knew I knew how to get myself out of all of those situations. I felt like I could hear the labels talking to me (possibly too much caffeine?) and I just kept hammering out the labs. At some point you obviously just kind of remember what the labs are asking you to do, but that wasn’t all bad. I felt good about thoroughly understanding the WHY of what i was doing so now it was just about building insane muscle memory. Don’t stop for question marks, don’t stop to read the doc-cd, just hammer it out and verify as quickly and accurately as possible.

Finally, feeling as good as I could, I left for San Jose! I had an enjoyable day visiting with family in San Francisco, I had a good dinner, I had some beer, and I got to sleep nice and early.

Getting into the lab the next morning was awesome. I was so anxious and ready to roll, but I was able to chill out and take at least a quick read through the whole lab. This is of course important in any track, for SP its all about getting the overall picture of what you are being asked to do — where are the different AS, what is the IGP in each of them, how do they interconnect, who’s your ‘core’ carrier, do you have CSC, do you have TE to worry about, where are the IOS-XR devices (since they do things a touch differently than IOS its very important to be cognizant of where they are), etc.

After the quick read I jumped right in. By lunch I was about half way through the core parts of the lab. I skipped over all the TE and multicast — my logic being that if I run out of time, I’d rather have my core stuff squared away, and that multicast and TE wouldn’t ‘break’ any of that core functionality (other than those tasks of course). After lunch I was basically in clean up mode. I wrapped up a few of the core tasks that I couldn’t finish earlier, and set to work using TCL to ping from every router to make sure everything was still looking okay. Once that things looked solid I jumped into the ‘secondary’ type tasks — things like TE/L2VPN/Multicast/etc.

I had a few questions that I am pretty confident the proctor saved my butt on… The proctor is totally your friend! Just make sure you approach and ask questions that clearly show that you understand the technology and that you are not asking for configuration guidance, but instead clearly state why you think the question leads you to solution 1, but it could also mean solution 2, and what the implications of those solutions are.

I wrapped up by of course going back over everything with a  fine tooth comb. I caught several things that I could have sworn I had taken care of the first time through, but clearly didn’t… TCL is of course your friend! I must have spent as much time in the TCL shell as I did in config mode. Ping everything…. from everywhere…. and then do it again… and a second and third time for good measure.

All in all, I left the lab feeling reasonably good, but since it’s the CCIE there is of course that overwhelming feeling of doubt!! I promptly acquired more beer as soon as I made it to the airport… that helped everything be far more awesome 🙂

By 630 or so I got an email from Cisco… oh man…. I could not freaking open that email fast enough! ‘PASS’! I was anticipating having to wait until the following morning, or at least late that night like I did for R/S, but what a great relief! All in all, I am so glad that I decided to go for the CCIE SP. I learned SO MUCH, and even if I never work for a service provider, having the understanding of whats going on in the background is going to be invaluable.



Pick a track that you are genuinely interested in — don’t let other people’s opinions drive your track selection.

As I’ve said before — INVEST in yourself and your lab journey by getting your own hardware (virtualize people! — I imagine DC would be hard to do this with though…) so that you can lab on your own time, all the time.

Each individual task is totally doable, but they may seem insane/overcomplicated when looking at the big picture — isolate and work on one thing at a time (seems obvious, but it’s really hard to do in the lab sometimes)

Proctors — they are cool. Ask them stuff. Worst case they tell you to re-read the question, best case they help save you some points you may otherwise have lost!

TCL — this should be incredibly obvious as well, but TCL is totally the shit. PING ALL THE THINGS!!!

Tidbit: IOS-XR BGP Allocate Label + Some Inter-AS VPNv4

I ran into a problem while doing an INE mock lab this morning…. it basically kicked my ass, so I figured I’d post about it!

The overall scenario is that there are two BGP domains, AS 1000 and AS 2000. Within each AS, there is some standard IGP routing, and IPv4 BGP — including eBGP between the two domains. There is also some route reflection and some other fun stuff, but that’s mostly irrelevant for the purposes of this post. Below is the INE topology drawing.


After the tasks that setup the basics, the lab rolls into some inter-AS VPN. Essentially, the routers in AS 1000 and 2000 also have loopbacks that are in the same VRF that AS 3000 lives in. The initial VPNv4 task basically is asking to configure the domains so that loopbacks in this VRF are reachable from both of the domains.

So the first thing to consider is that BGP labels will need to be sent between the domains. Thats pretty simple, just send-label in IOS or labeled-unicast in IOS-XR. In addition to that, IOS will require the “mpls bgp forwarding” command on the interfaces between the domains in order to send the labels. For IOS-XR, since the neighbor is on a physical interface, and it’s not a /32 (obviously), a static host route to the neighbor pointing out the connected interface is required. This is because IOS-XR will not install any labels into the forwarding table that have a next hop of something other than a /32.

After this, we need to ensure that each domain has reachability to the ‘PE’ routers loopbacks. This is to ensure that we have a label switched path the whole way through to the end. We also need to make sure that however we learn about the ‘PE’ (which is basically every router since they all have a loopback in the ‘customer’ VRF) loopbacks, and that we get some labels for those. There is an important piece here that basically says that however we learn about that prefix (/32 for the PE), we must also get a label from the same mechanism. IF we were to learn about those /32s via BGP, we would need a BGP label. If we learn the PE loopbacks via IGP, we need to have a label for that via IGP/LDP.

This leads us to the point of the post! In the course of the lab, I was advertising the loopback of each of the PE devices into BGP on each router individually — i.e. on R5 I advertised (loopback0) into BGP locally, and advertised R2s loopback locally, etc. This totally worked — R5  and XR1 both had these prefixes in BGP and while things were configured for normal IPv4 unicast (not labeled) they were advertised across to AS 2000.

Things got a little dicey for me though when I moved the eBGP to labeled unicast. R5 was sending prefixes and labels across to AS 2000, but when shutting down that peering session to test that the inter-AS VPNv4 setup was working across XR1/R3 as well, I was met with crushing defeat!!

Thankfully somebody on the IEOC forums (INEs forum) had this same problem, and Mr. Brian McGahan was there to save the day… here’s what he said:

Only the originator of the BGP route can allocate the label.  This means that whoever you have the network statement or the redistribute statement on you need to do the allocation there.  In your case if you don’t originate the network on XR1 you’d have to go to R5 and then send-label to XR1, and on XR1 send label back to R5.  That’s why in most designs you just have your edge routers originate the BGP networks on behalf of the IGP network, because then you have a single point of control for them.  You can do it either way but it’s good to know that the problem exists in the first place.

So basically IOS-XR, which was configured for ‘allocate-label all’ in order to send BGP labels across to AS 2000, was NOT actually sending any labels!! This was due to the way I was getting the loopback prefixes piped into BGP. Killing the advertisements on the other routers, and then advertising them into BGP on XR1 instead allowed XR1 to send the labels across.

So lesson learned! I’m pretty glad that I ‘messed’ up and was able to come across this because I could totally see Cisco doing something like this on the lab — guess I’ll find out in a few weeks when I sit my first attempt!! 😀