Picking up right where we left off with Part 1 of the “DMVPN/MPLS/PfR” series we are going to drop some dynamic routing on top of our setup and see where this takes us.
Frank&Co is using OSPF as their IGP of choice, but their MPLS provider required them (rightfully so!) to peer via BGP to them. So, internal to each of the Frank&Co locations we have some basic OSPF for internal route propagation, but also a bit of BGP to send and learn prefixes over the MPLS. As I’m sure will become painfully evident over the course of future blog posts, I absolutely despise redistribution, and thankfully Frank&Co shares my opinion on that matter. Referring to the diagram, on the “HQ” sites, there is of course BGP running on the “MPLS” routers to deal with the provider, in addition to that, the “MPLS” routers also peer to the “Core” device via iBGP.
This is of course a pretty common deployment, but it really is quite elegant. Ignoring for the moment the added requirement for the DMVPN terminating on the “WWW” router, we can see that prefixes learnt via the MPLS provider never need to be redistributed back into the IGP. Any routers “south” of the HQ “Core” device that do not have an exact match for a route that exists in the MPLS cloud would “fallback” to a default route (being originated via the “WWW” device)…. which in turn would pipe traffic through the “Core,” where there is a longer match via iBGP! Viola! No redistribution necessary (the other clean fix is MPLS, but that tends to become a licensing/feature support issue).
So why is this at all relevant for our DMVPN discussion? Well it totally matters, that’s why! DMVPN is a little less dynamic without a dynamic routing protocol running over the top of it. So what is the logical routing protocol to use? Well Cisco may tell you to use EIGRP (even though it sucks), OSPF would work too (better than EIGRP). EIGRP (besides sucking) seems like a poor choice because then we would end up with OSPF, BGP, AND EIGRP to deal with. OSPF would be okay, but then we have some interesting issues of path selection — the “Core” (without redistribution) would ALWAYS prefer the OSPF routes via the DMVPN over the MPLS routes (because the prefixes in BGP will be “iBGP” at the “Core” and as such have an AD of 200 vs OSFP’s AD of 120). Alright, so other options: RIP – LOL, NOPE!, Static – again, not very dynamic, ISIS – cool, but not exactly its primary use case, BGP it is!
Next up – eBGP or iBGP?? Well we could do either really, but there are some things to seriously consider.
- Full mesh requirement if using iBGP; could configure the hub(s) as route-reflectors to ease this requirement
- Administrative distance; AD 20 vs AD 200 / eBGP vs iBGP
- Next hop reachability; are we updating this or not, and if not, how are we learning the next hop
My money (and consequently Frank&Co’s) is on eBGP. I think that it provides the simplest and most scalable solution, eliminates the need for route reflection, and gives you the least amount of headache with respect to path selection. So, enough talk, lets set it up.
This is going to be a super simple BGP setup, for now all we care about is getting our basic peering up. On the HQ “DMVPN” router, we configure the BGP process, set a static router-id, and send communities for kicks, and tack on soft-reconfig.
router bgp 100 bgp router-id 22.214.171.124 bgp log-neighbor-changes neighbor 126.96.36.199 remote-as 100 neighbor 188.8.131.52 update-source Loopback0 neighbor 184.108.40.206 send-community both neighbor 220.127.116.11 soft-reconfiguration inbound neighbor 100.100.100.180 remote-as 180 neighbor 100.100.100.180 send-community both neighbor 100.100.100.180 soft-reconfiguration inbound neighbor 100.100.100.190 remote-as 190 neighbor 100.100.100.190 send-community both neighbor 100.100.100.190 soft-reconfiguration inbound
Excuse my silly lab IP allocations 🙂 Frank&Co is using 1.1.1.x for loopbacks at the “HQ” site, and as seen in the previous post 100.100.100.x for tunnel IPs for connecting to the primary DMVPN hub. Also please excuse the public AS numbers. As you can see, nothing special here; Hub “1” is peering to both of our branch routers, and to the “Core” at the “HQ” site. As we already have established that we can ping across the tunnels, we have everything we need to establish our eBGP sessions. The spoke side is configured similarly, but since it is compressed into a DMVPN and MPLS terminating router, that’s all happening in one place.
router bgp 180 bgp router-id 18.104.22.168 bgp log-neighbor-changes neighbor 100.100.100.100 remote-as 100 neighbor 100.100.100.100 send-community both neighbor 100.100.100.100 soft-reconfiguration inbound neighbor 172.16.80.2 remote-as 1 neighbor 172.16.80.2 send-community both neighbor 172.16.80.2 soft-reconfiguration inbound
Nothing exciting here — just landing the two eBGP sessions here for the MPLS and the DMVPN peers. We can of course very simply see that we have established our BGP peers with the “show ip bgp summary” command. Here is the output of this command on the Branch router:
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 100.100.100.100 4 100 13 14 5 0 0 00:09:09 0 172.16.80.2 4 1 12 14 5 0 0 00:09:23 0
Here we can see the two neighbors, 100.100.100.100 being the DMVPN hub router, and the 172.16.80.2 peer being our itty bitty MPLS “cloud” (hate that word…). We aren’t receiving any prefixes yet (or sending), so lets fix that and see how this works.
router bgp 100 network 10.10.10.10 mask 255.255.255.255 aggregate-address 10.10.0.0 255.255.0.0 summary-only
On the HQ Core router, we are going to advertise one of our loopbacks, 10.10.10.10/32, into BGP. In real life we probably would not be just advertising single /32 prefixes, so we will go ahead and advertise the whole 10.10.0.0/16 into BGP. Furthermore, since summarization is… ya know… a thing… and an important thing at that, we will tack on the “summary-only” keyword. The aggregate-address command will advertise the indicated prefix (assuming at least a subset of that prefix is in the BGP RIB), including the “summary-only” syntax will suppress any of the subsets of the aggregate.
Back out on the Branch router, we can see that we are receiving this new prefix, and that we are receiving it via two possible paths:
Network Next Hop Metric LocPrf Weight Path *> 10.10.0.0/16 100.100.100.100 0 100 i * 172.16.80.2 0 1 100 i
As we can see, we are receiving the prefix via MPLS and over the DMVPN. We can also see that we prefer the path via the DMVPN due to a shorter AS-Path — “100” vs “1 100” — the added hop of the MPLS provider makes it less desirable. Since its likely that the DMVPN connection is a “fallback” or a secondary path, we probably want to make sure that the MPLS is the preferred path. We have several ways to attack this, probably the simplest option since we are using Cisco devices is to set the weight on the Branch device. We can use this to set the weight on a neighbor basis or on specific prefixes by matching with a route-map and access-list/prefix-list. This is a good option, but I would like to have the possibility of multi-path later, so we will require basically all metrics for the prefix to be equal up to IGP cost.
So the simplest way to make things equal is to prepend an extra hop onto prefixes advertised via the DMVPN router at HQ. That way when the prefix is received at the Branch, the AS-Path count is equal. Heres how we make that happen:
ip prefix-list PL_Any seq 5 permit 0.0.0.0/0 le 32 ! route-map RM_DMVPN_Prepend permit 10 match ip address prefix-list PL_Any set as-path prepend 100 ! router bgp 100 neighbor 100.100.100.180 route-map RM_DMVPN_Prepend out
Above you can see the prefix-list “PL_Any” matching any prefix, that prefix-list is being used in the route-map, the route-map is then adding a single hop to the AS-Path of “100” which is our local AS. Lastly, we attach this route-map to the Branch neighbor outbound.
Now we have equal AS-Paths, so we will need to move further up the BGP path selection process to determine which of the paths is the best. In our case, we will still end up preferring the path via the DMVPN due to the lower router-id. At this point, we can simply use a route-map and a prefix-list to match all traffic and set the local preference of all prefixes received via MPLS to a higher than default value to force the Branch to prefer the MPLS connection.
ip prefix-list PL_Any seq 5 permit 0.0.0.0/0 le 32 ! route-map RM_MPLS_In permit 10 match ip address prefix-list PL_Any set local-preference 9999 ! router bgp 180 neighbor 172.16.80.2 route-map RM_MPLS_In in
Now we will prefer the MPLS provider since AS-Paths are equal, and the local-prefernce of that peer is higher than the DMVPN peer.
Network Next Hop Metric LocPrf Weight Path * 10.10.0.0/16 100.100.100.100 0 100 100 i *> 172.16.80.2 9999 0 1 100 i
Yay! So finally we are utilizing the MPLS to reach back to HQ. I’ve also advertised the loopback of the Branch into BGP, so now we should have full reachability to HQ from the Branch.
traceroute 10.10.10.10 source lo10 Type escape sequence to abort. Tracing the route to 10.10.10.10 VRF info: (vrf in name/id, vrf out name/id) 1 172.16.80.2 0 msec 0 msec 0 msec 2 172.16.1.1 1 msec 1 msec 0 msec 3 10.10.1.1 [AS 100] 1 msec 1 msec *
Above we can see that we reach HQ via the MPLS when sourced from the Branch routers loopback10 (using l10 to simulate the Branch “LAN”). If we shutdown the interface that we peer to the MPLS provider on, we automagically re-route to our next best path which is of course via the DMVPN.
interface e0/1 shutdown ! traceroute 10.10.10.10 source lo10 Type escape sequence to abort. Tracing the route to 10.10.10.10 VRF info: (vrf in name/id, vrf out name/id) 1 100.100.100.100 5 msec 5 msec 5 msec 2 10.10.1.5 [AS 100] 5 msec 5 msec *
At this point we have a pretty solid setup — we can use local-pref/weight and other BGP attributes to control which path we prefer to use, and we can automagically failover to a secondary path. In this post we didn’t outline the configuration of the second hub, however once we start playing with PfR we will be working from the dual hub DMVPN setup, and the magic of PfR will be able to take advantage of having three different paths between the HQ sites and the Branch.
Last note: At the HQ core device, we are recursing for reachability to the DMVPN network. We could use next-hop-self on the DMVPN router’s peering session to the core to eliminate that kind of wonky recursion — and utilize a next-hop for the Branch (via the DMVPN) that is in the routing table. Either way, in this case things work.