My CCIE Service Provider Lab

Well, since I got on my soapbox and ranted about how you really need to buy your own hardware for certification study, I guess its fitting that I outline my current lab setup.

Right now I’m working toward my CCIE Service Provider. I’ve passed the written, and am shooting for the “recon” attempt end of February/early March. I call it that because, having done the CCIE R/S, I know that I will need to circle back and re-study at least a few sections (going out on a limb and guessing multicast…. again….).

So, how am I preparing? Well I’ve decided to roll with the INE service provider study guide. Seems to me that in terms of prepping for service provider there are not exactly a whole lot of options. INE and IPExpert are the two that I think of, but IPExpert doesn’t have any service provider material…. so i had an easy choice in who to choose (that’s definitely not a knock on INE FYI). Having a study guide makes it REALLY easy to know what kind of hardware you need which helps to give a little focus to the initial setup. To run the INE lab you need:

1x ME3400
1x 3550
6x 7200
2x 2600
2x CRS 12k

The CRS 12ks each need:
1x PRP-2
1x 4GE-SFP-LC
1x POS/OC blade (I’m using OC48X/POS-SR-SC)

So, low hanging fruit — ALL of the routers in the INE topology can be virtualized (yay GNS3)!! Thats a HUGE help. So in GNS, we have 6x 7204VXRs, and two 2611XMs (but I’m using 3725 because I had that image handy and don’t have to worry about memory — more on that in a bit).
Next, switches. We need one ME3400 and one 3550. I have 3550s coming out of my ears, so that was no big deal, but the 3400 is a bit of a strange animal for somebody that lives mostly in enterprise-land (like myself). Thankfully, eBay is our friend, and the 3400 is a bit long in the tooth — I was able to snag a ME3400 for 399$. I suspect that I could have put in bids or waited around to pick it up for a better price, but I was impatient and greedy and wanted it ASAP.
All thats left is the big boy… the 12k… This took a bit more work than a quick search on eBay and a simple PayPal transaction. The first thing that is critical to know is that although you need “two” CRSs, you can actually get away with a single chassis. We can do that by configure an SDR — Secure Domain Router — instance for each “router.” If you’ve lived with Nexus 7ks, this is basically the same thing as a VDC — with the notable difference that entire blades must be allocated to each SDR, you cannot allocate ports or port-groups. What you can do is get a single chassis, buy all the required cards, then allocate cards between the two SDRs. (I’ll post up configs for how to do this next week when I’m back home and can fire up the chassis)

Acquiring a 12k is a bit more of an adventure. I totally lucked out and found somebody on eBay who had just finished their SP lab and was selling the chassis with power and fabric cards, 2x PRP-2, and 2x 4GE-SFP-LC. This tidy package was listed at 3500$ or best offer. I lead with 2700$ (thinking there couldn’t be too many crazy people like me out there looking for a second-hand CRS), and was countered w/ 3100$ splitting the difference. I figured that was fair, so that was that!

Next up I needed the OC blades. This was as easy as poking around on eBay for a bit. 175$ per blade later, those were in the mail — not too bad.

Putting all of this together turned out to be a pretty fun little adventure too. First thing was spinning up the GNS routers. I have a desktop I built relatively recently with a core i7 and 32gb memory that I’m running ESXi on, so I just went ahead and used this setup. I spun up two different Ubuntu VMs just to split the GNS3 load out a bit (having lived the GNS3 dream while studying for my R/S). Each Ubuntu host is running 3x 7204VXRs, and a single 3725. One of those fancy GNS3 switches connects to all the necessary copper ports of the routers, dumping each one into a separate VLAN. The last piece to this is connecting the switch to a “physical interface” in the host. This connection is a trunk in GNS3 and in ESXi, I left them trunking all VLANs just to make it easy.

The special sauce here is that my ESXi host is connecting to a 3750… this is actually super important because the 3750 apparently has some magical powers in regards to this quasi QinQ stuff happening between GNS and the “real” lab. (For details on the “breakout” switch setup check out this great IPExpert post: http://blog.ipexpert.com/2011/02/28/gns3-and-physical-switches-breakout-switch/) The 3750 basically just “pops” the tags that the layer 2 GNS3 switch was stuffing the virtual router traffic into, and then dumps that traffic out to the rest of the lab devices.

The rest of the lab setup is nothing fancy — just cabled up to support the INE topology, then logically configured per the lab guide. This lab will certainly allow for some helpful testing for real life use though — being able to lab two IOS-XR routers connected to the rest of the virtual routers, plus the ME switch will be pretty powerful. In fact I have a customer that this lab will come in handy for early next year I hope!

One final and interesting note about the CRS/blades for those considering building their own lab…. Memory… its a super important piece in CRS land. Each blade runs the IOS-XR software version, and is subject to particular requirements for memory. The OC blades that I ordered came with 512mb of “main memory” for the blade. This would be great… if 3.9.1 didn’t require 1gb of memory. Well memory is cheap and easy to come by right??? NOPE!! Holy crap… we all know that Cisco memory is overly ridiculously hilariously expensive, but for basically every router ever i’ve been able to order the right “type” of memory (pin, speed, etc.) and just rock and roll. Not so for this bad boy. Boot up the chassis (takes FOREVER), and no dice — memory is not “recognized.” Well shit… back to the internets. I was able to find some memory on our good friend eBay for 150$ per pair of 512 sticks (1 set of 2x512mb per blade). Not too bad, but certainly added to the total.

For those adding this up, or wondering how doable this is, here are where my totals stand (I think):

CRS chassis w/ PRP and GE blades = 3100
OC48X blades = 350
Memory for OC48X blades = 300
ME3400 = 400
3550 = “free” (probably as low 50 on eBay)
3750 = “free” (can do this as low as 300 I think on eBay)
Desktop = “free” (because i built this months ago preemptively for this adventure; I spent around 1100 on this)
INE SP Guide = 200 (normally 300, waited it out for holiday sale!!)

Totals:
Including estimated costs for “free” things: 5485 (or tack on an extra 100$ for non holiday sale INE material)
Without “free” things: 4350
I’ll try and remember to post up the important 3750 (“breakout switch”) configs and SDR configs when I get back home to Bellevue.

Here is the relevant 3750 config:

vlan 100
 name R1_F0/0
 !
 vlan 101
 name R1_F0/1
 !
 <snip>
 !
 vlan 800
 name R8_F0/0
 !
 vlan 801
 name R8_F0/1
 !
 interface FastEthernet1/0/1
 description From R1 F0/0
 switchport access vlan 100
 switchport mode dot1q-tunnel
 l2protocol-tunnel cdp
 l2protocol-tunnel stp
 l2protocol-tunnel vtp
 no cdp enable
 no cdp tlv server-location
 no cdp tlv app
 spanning-tree portfast
 !
!
 interface FastEthernet1/0/23
 description Trunk to ESXi Host
 switchport trunk encapsulation dot1q
 switchport mode trunk
 no keepalive
 l2protocol-tunnel cdp
 l2protocol-tunnel stp
 l2protocol-tunnel vtp
 no cdp enable
 no cdp tlv server-location
 no cdp tlv app
Advertisements

DMVPN/MPLS/PfR Part 2: Finally… Some Routing!

Picking up right where we left off with Part 1 of the “DMVPN/MPLS/PfR” series we are going to drop some dynamic routing on top of our setup and see where this takes us.

Frank&Co is using OSPF as their IGP of choice, but their MPLS provider required them (rightfully so!) to peer via BGP to them. So, internal to each of the Frank&Co locations we have some basic OSPF for internal route propagation, but also a bit of BGP to send and learn prefixes over the MPLS. As I’m sure will become painfully evident over the course of future blog posts, I absolutely despise redistribution, and thankfully Frank&Co shares my opinion on that matter. Referring to the diagram, on the “HQ” sites, there is of course BGP running on the “MPLS” routers to deal with the provider, in addition to that, the “MPLS” routers also peer to the “Core” device via iBGP.

Fran&Co Routing Overview

This is of course a pretty common deployment, but it really is quite elegant. Ignoring for the moment the added requirement for the DMVPN terminating on the “WWW” router, we can see that prefixes learnt via the MPLS provider never need to be redistributed back into the IGP. Any routers “south” of the HQ “Core” device that do not have an exact match for a route that exists in the MPLS cloud would “fallback” to a default route (being originated via the “WWW” device)…. which in turn would pipe traffic through the “Core,” where there is a longer match via iBGP! Viola! No redistribution necessary (the other clean fix is MPLS, but that tends to become a licensing/feature support issue).

So why is this at all relevant for our DMVPN discussion? Well it totally matters, that’s why! DMVPN is a little less dynamic without a dynamic routing protocol running over the top of it. So what is the logical routing protocol to use? Well Cisco may tell you to use EIGRP (even though it sucks), OSPF would work too (better than EIGRP). EIGRP (besides sucking) seems like a poor choice because then we would end up with OSPF, BGP, AND EIGRP to deal with. OSPF would be okay, but then we have some interesting issues of path selection — the “Core” (without redistribution) would ALWAYS prefer the OSPF routes via the DMVPN over the MPLS routes (because the prefixes in BGP will be “iBGP” at the “Core” and as such have an AD of 200 vs OSFP’s AD of 120). Alright, so other options: RIP – LOL, NOPE!, Static – again, not very dynamic, ISIS – cool, but not exactly its primary use case, BGP it is!

Next up – eBGP or iBGP?? Well we could do either really, but there are some things to seriously consider.

  •  Full mesh requirement if using iBGP; could configure the hub(s) as route-reflectors to ease this requirement
  • Administrative distance; AD 20 vs AD 200 / eBGP vs iBGP
  • Next hop reachability; are we updating this or not, and if not, how are we learning the next hop

My money (and consequently Frank&Co’s) is on eBGP. I think that it provides the simplest and most scalable solution, eliminates the need for route reflection, and gives you the least amount of headache with respect to path selection. So, enough talk, lets set it up.

This is going to be a super simple BGP setup, for now all we care about is getting our basic peering up. On the HQ “DMVPN” router, we configure the BGP process, set a static router-id, and send communities for kicks, and tack on soft-reconfig.

router bgp 100
 bgp router-id 1.1.1.3
 bgp log-neighbor-changes
 neighbor 1.1.1.1 remote-as 100
 neighbor 1.1.1.1 update-source Loopback0
 neighbor 1.1.1.1 send-community both
 neighbor 1.1.1.1 soft-reconfiguration inbound
 neighbor 100.100.100.180 remote-as 180
 neighbor 100.100.100.180 send-community both
 neighbor 100.100.100.180 soft-reconfiguration inbound
 neighbor 100.100.100.190 remote-as 190
 neighbor 100.100.100.190 send-community both
 neighbor 100.100.100.190 soft-reconfiguration inbound

Excuse my silly lab IP allocations 🙂 Frank&Co is using 1.1.1.x for loopbacks at the “HQ” site, and as seen in the previous post 100.100.100.x for tunnel IPs for connecting to the primary DMVPN hub. Also please excuse the public AS numbers. As you can see, nothing special here; Hub “1” is peering to both of our branch routers, and to the “Core” at the “HQ” site.  As we already have established that we can ping across the tunnels, we have everything we need to establish our eBGP sessions. The spoke side is configured similarly, but since it is compressed into a DMVPN and MPLS terminating router, that’s all happening in one place.

router bgp 180
 bgp router-id 1.1.80.1
 bgp log-neighbor-changes
neighbor 100.100.100.100 remote-as 100
 neighbor 100.100.100.100 send-community both
 neighbor 100.100.100.100 soft-reconfiguration inbound
 neighbor 172.16.80.2 remote-as 1
 neighbor 172.16.80.2 send-community both
 neighbor 172.16.80.2 soft-reconfiguration inbound

Nothing exciting here — just landing the two eBGP sessions here for the MPLS and the DMVPN peers. We can of course very simply see that we have established our BGP peers with the “show ip bgp summary” command. Here is the output of this command on the Branch router:

Neighbor            V   AS    MsgRcvd MsgSent TblVer InQ OutQ Up/Down    State/PfxRcd
100.100.100.100     4   100    13      14        5     0  0   00:09:09   0
172.16.80.2         4   1      12      14        5     0 0    00:09:23   0

Here we can see the two neighbors, 100.100.100.100 being the DMVPN hub router, and the 172.16.80.2 peer being our itty bitty MPLS “cloud” (hate that word…). We aren’t receiving any prefixes yet (or sending), so lets fix that and see how this works.

router bgp 100
 network 10.10.10.10 mask 255.255.255.255
 aggregate-address 10.10.0.0 255.255.0.0 summary-only

On the HQ Core router, we are going to advertise one of our loopbacks, 10.10.10.10/32,  into BGP. In real life we probably would not be just advertising single /32 prefixes, so we will go ahead and advertise the whole 10.10.0.0/16 into BGP. Furthermore, since summarization is… ya know… a thing… and an important thing at that, we will tack on the “summary-only” keyword. The aggregate-address command will advertise the indicated prefix (assuming at least a subset of that prefix is in the BGP RIB), including the “summary-only” syntax will suppress any of the subsets of the aggregate.

Back out on the Branch router, we can see that we are receiving this new prefix, and that we are receiving it via two possible paths:

Network           Next Hop         Metric LocPrf Weight Path
 *> 10.10.0.0/16  100.100.100.100                0      100 i
 *                172.16.80.2                    0      1 100 i

As we can see, we are receiving the prefix via MPLS and over the DMVPN. We can also see that we prefer the path via the DMVPN due to a shorter AS-Path — “100” vs “1 100” — the added hop of the MPLS provider makes it less desirable. Since its likely that the DMVPN connection is a “fallback” or a secondary path, we probably want to make sure that the MPLS is the preferred path. We have several ways to attack this, probably the simplest option since we are using Cisco devices is to set the weight on the Branch device. We can use this to set the weight on a neighbor basis or on specific prefixes by matching with a route-map and access-list/prefix-list. This is a good option, but I would like to have the possibility of multi-path later, so we will require basically all metrics for the prefix to be equal up to IGP cost.

So the simplest way to make things equal is to prepend an extra hop onto prefixes advertised via the DMVPN router at HQ. That way when the prefix is received at the Branch, the AS-Path count is equal. Heres how we make that happen:

ip prefix-list PL_Any seq 5 permit 0.0.0.0/0 le 32
!
route-map RM_DMVPN_Prepend permit 10
 match ip address prefix-list PL_Any
 set as-path prepend 100
!
router bgp 100
 neighbor 100.100.100.180 route-map RM_DMVPN_Prepend out

Above you can see the prefix-list “PL_Any” matching any prefix, that prefix-list is being used in the route-map, the route-map is then adding a single hop to the AS-Path of “100” which is our local AS. Lastly, we attach this route-map to the Branch neighbor outbound.

Now we have equal AS-Paths, so we will need to move further up the BGP path selection process to determine which of the paths is the best. In our case, we will still end up preferring the path via the DMVPN due to the lower router-id. At this point, we can simply use a route-map and a prefix-list to match all traffic and set the local preference of all prefixes received via MPLS to a higher than default value to force the Branch to prefer the MPLS connection.

ip prefix-list PL_Any seq 5 permit 0.0.0.0/0 le 32
!
route-map RM_MPLS_In permit 10
 match ip address prefix-list PL_Any
 set local-preference 9999
!
router bgp 180
 neighbor 172.16.80.2 route-map RM_MPLS_In in

Now we will prefer the MPLS provider since AS-Paths are equal, and the local-prefernce of that peer is higher than the DMVPN peer.

Network           Next Hop         Metric LocPrf Weight Path
 *  10.10.0.0/16  100.100.100.100                0      100 100 i
 *>               172.16.80.2             9999   0      1 100 i

Yay! So finally we are utilizing the MPLS to reach back to HQ. I’ve also advertised the loopback of the Branch into BGP, so now we should have full reachability to HQ from the Branch.

traceroute 10.10.10.10 source lo10
Type escape sequence to abort.
Tracing the route to 10.10.10.10
VRF info: (vrf in name/id, vrf out name/id)
 1 172.16.80.2 0 msec 0 msec 0 msec
 2 172.16.1.1 1 msec 1 msec 0 msec
 3 10.10.1.1 [AS 100] 1 msec 1 msec *

Above we can see that we reach HQ via the MPLS when sourced from the Branch routers loopback10 (using l10 to simulate the Branch “LAN”). If we shutdown the interface that we peer to the MPLS provider on, we automagically re-route to our next best path which is of course via the DMVPN.

interface e0/1
shutdown
!
traceroute 10.10.10.10 source lo10
Type escape sequence to abort.
Tracing the route to 10.10.10.10
VRF info: (vrf in name/id, vrf out name/id)
 1 100.100.100.100 5 msec 5 msec 5 msec
 2 10.10.1.5 [AS 100] 5 msec 5 msec *

At this point we have a pretty solid setup — we can use local-pref/weight and other BGP attributes to control which path we prefer to use, and we can automagically failover to a secondary path. In this post we didn’t outline the configuration of the second hub, however once we start playing with PfR we will be working from the dual hub DMVPN setup, and the magic of PfR will be able to take advantage of having three different paths between the HQ sites and the Branch. 

Last note: At the HQ core device, we are recursing for reachability to the DMVPN network. We could use next-hop-self on the DMVPN router’s peering session to the core to eliminate that kind of wonky recursion — and utilize a next-hop for the Branch (via the DMVPN) that is in the routing table. Either way, in this case things work.