DMVPN/MPLS/PfR Part 1: Basic DMVPN/NHRP

This series will tackle the basics of a current pet project/side lab I’ve got going on at the moment. The scenario is essentially your “standard” type enterprise network with one or two “main” sites, and any number of “branches” connected via MPLS. The idea here is that our customer “Frank&Co.” has this MPLS provider, but wants some additional redundancy. The easy answer is some affordable commercial broadband at the remotes, but how do you securely tie that into the rest of the enterprise network? Well you’ve got some options of course, site to site VPNs, VTI tunnels, DMVPN, you could have brought in a second MPLS provider too if you want to be spendy, or you could get really nutty and look at possibly introducing LISP and GET-VPN (hopefully I’ll be lab-ing that up later, looks like Cisco has a few good docs on this).

All of these options could provide you with some redundancy options, and hopefully we will take a look at all of them in time, but for now “Frank&Co” is interested in DMVPN. On top of the DMVPN and MPLS combo, they are interested in leveraging PfR for some more intelligent routing control (SDN lite perhaps?).

Lets take a look at the basic lab layout:

FrankCo Overview

As you can see, nothing fancy here! Two “HQ” type location with a unique router for terminating the MPLS and for the Internet connection, and a single core device (lets hope its VSS and not literally just one box). For the remote locations, a single router terminates the MPLS and the Internet connections. We will pile on ZBFW for some security on the remote branches in a later post in this series, for now though Frank&Co is just not that concerned with security.

So, now that all that is out-of-the-way, let’s get down to the fun part. DMVPN is often described with the three different phases that I’m sure you have read about before. Simply put; Phase 1 = hub to spoke connectivity, and Phase 2 = Phase 1 + dynamic spoke to spoke tunnels. Phase 3 is a whole new animal that we won’t worry about for now! In addition to the different phases, DMVPN is almost always associated with IPSEC in order to encrypt traffic, but strictly speaking IPSEC is not a mandatory component of DMPVN. So how do these tunnels magically form over the Internet (or any NBMA network)?

Next Hop Routing Protocol of course! NHRP is basically a protocol that functions sort of like ARP, just over the Internet. Your spoke routers, or Clients, are configured to map to the Hub(s), or Server(s). The Clients basically register with the Server so that the Server does not have to be pre-configured for a potentially dynamic Client address. Lets take a look at how this is configured:

Hub/Server:
 interface Tunnel0
 ip address 100.100.100.100 255.255.255.0
 tunnel source Ethernet0/0
 tunnel mode gre multipoint
 ip nhrp network-id 1
Spoke/Client:
 interface Tunnel0
 ip address 100.100.100.180 255.255.255.0
 ip nhrp map 100.100.100.100 192.168.10.1
 ip nhrp network-id 1
 ip nhrp nhs 100.100.100.100
 tunnel source Ethernet0/0
 tunnel mode gre multipoint

Thats really it. On the Hub/Server side, you can see that we’ve completed the basics to spin up a tunnel – source/ip/mode, but we did not need to configure a destination. The “ip nhrp network-id [number]” command enables NHRP on the interface, and eliminates the need for a statically defined destination. The actual number is locally significant to the router, you could use “1” on the hub and “9999” on the spoke; the mapping and next hop server commands on the spoke are what tie the tunnels together.

The Spoke/Client clearly has a little bit more going on, but still, not a whole lot. Again, we have got the basic tunnel stuff, but still no destination. In addition to the network-id, we have two very important lines:

ip nhrp nhs 100.100.100.100
ip nhrp map 100.100.100.100 192.168.10.1

Line one above defines the (in our case) private IP address of the endpoint that is the Next Hop Server (NHS). The second line essentially just tells the Spoke how to get to the server. Here we are mapping the NHS address to a “public” (just pretend with me, it just a lab after all) IP address that is reachable from the Spoke.

So at this point, assuming that you have reachability to the address that NHRP is mapping the NHS to, you should have basic DMVPN connectivity! Well how do you know its working!? The easy peasy verification for this is simply, and intuitively, “show dmvpn.”

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
 N - NATed, L - Local, X - No Socket
 # Ent --> Number of NHRP entries with same NBMA peer
 NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
 UpDn Time --> Up or Down Time for a Tunnel
 ==========================================================================
Interface: Tunnel1, IPv4 NHRP Details
 Type:Spoke, NHRP Peers:1,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
 1 192.168.10.1 100.100.100.100 UP 00:09:34 S

We get a nifty little legend for us that makes this output nice and easy to read. We can clearly see the peers NBMA address, the tunnel address, and whether the peer is up or down. Since this output was taken from a Spoke, you can see the “S” for static under the peer attributes; on the Hub side, this will be a “D” for dynamic.

Okay, so show commands are cool, but how do we see whats really going? Debugs are fun right? — “debug nhrp packet” will give us some good output on the registration process. Here we see the process from the Hub side.

*Nov 28 18:01:41.619: NHRP: Receive Registration Request via Tunnel0 vrf 0, packet size: 92
 *Nov 28 18:01:41.619: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
 *Nov 28 18:01:41.619: shtl: 4(NSAP), sstl: 0(NSAP)
 *Nov 28 18:01:41.619: pktsz: 92 extoff: 52
 *Nov 28 18:01:41.619: (M) flags: "unique nat ", reqid: 65688
 *Nov 28 18:01:41.619: src NBMA: 192.168.80.1
 *Nov 28 18:01:41.619: src protocol: 100.100.100.180, dst protocol: 100.100.100.100
 *Nov 28 18:01:41.619: (C-1) code: no error(0)
 *Nov 28 18:01:41.619: prefix: 32, mtu: 17916, hd_time: 7200
 *Nov 28 18:01:41.619: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0
 *Nov 28 18:01:41.619: NHRP: Send Registration Reply via Tunnel0 vrf 0, packet size: 112
 *Nov 28 18:01:41.619: src: 100.100.100.100, dst: 100.100.100.180
 *Nov 28 18:01:41.619: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
 *Nov 28 18:01:41.619: shtl: 4(NSAP), sstl: 0(NSAP)
 *Nov 28 18:01:41.619: pktsz: 112 extoff: 52
 *Nov 28 18:01:41.619: (M) flags: "unique nat ", reqid: 65688
 *Nov 28 18:01:41.619: src NBMA: 192.168.80.1
 *Nov 28 18:01:41.620: src protocol: 100.100.10.11, dst protocol: 100.100.10.10
 *Nov 28 18:01:41.620: (C-1) code: no error(0)
 *Nov 28 18:01:41.620: prefix: 32, mtu: 17916, hd_time: 7200
 *Nov 28 18:01:41.620: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0

TL;DR — You can see above the in the bolded lines where the interesting parts are (for Humans to read at least). The router receives the request on the tunnel, you can see the source NBMA address (this is the “public” IP on our Spoke router), and see the mappings for the actual “private” tunnel addresses. “Show ip nhrp” can give us a little further verification that the registration has been successful:

100.100.100.180/32 via 100.100.100.180
 Tunnel0 created 00:19:03, expire 01:43:13
 Type: dynamic, Flags: unique registered
 NBMA address: 192.168.80.1

So does it ping!? Sure does, from the hub:

HQ_WWW#ping 100.100.100.180 
Type escape sequence to abort. 
Sending 5, 100-byte ICMP Echos to 100.100.100.180, timeout is 2 seconds: 
!!!!!

One last cool piece to all of this; since we have already setup the tunnels as multi-point GRE, we are already up to snuff for DMVPN “Phase 2.” I’ve gone ahead and added the same basic tunnel/nhrp configurations on the second Spoke router. Now from the Hub we can see two NHRP entries:

100.100.100.180/32 via 100.100.100.180
 Tunnel10 created 00:23:40, expire 01:38:36
 Type: dynamic, Flags: unique registered
 NBMA address: 192.168.80.1
100.100.100.190/32 via 100.100.100.190
 Tunnel10 created 00:00:10, expire 01:59:50
 Type: dynamic, Flags: unique registered used
 NBMA address: 192.168.90.1

It gets a little cooler though! At first glance, we have no spoke to spoke connectivity, here is some output on Spoke “1,” it shows that we only have the NHRP entry for the Hub router:

100.100.100.100/32 via 100.100.100.100
 Tunnel10 created 00:22:38, never expire
 Type: static, Flags: used
 NBMA address: 192.168.10.1

Once we start to send some traffic to our Spoke “2” site, some magic starts to happen though! Below we can see that a traceroute from Spoke “1” goes “directly” to Spoke “2” — this is all due to the magic of NHRP. The NHS already knows about the second Spoke site, and it shares this information with Spoke “1.” Spoke “1” can then build its NHRP entry for Spoke “2” and forward traffic directly across the NBMA network. This spoke-to-spoke tunnel will stay “alive,” for a time determined by the “nhrp holdtime” which defaults to 2 hours.

Branch_1#traceroute 100.100.100.190
 Type escape sequence to abort.
 Tracing the route to 100.100.100.190
 VRF info: (vrf in name/id, vrf out name/id)
 1 100.100.100.190 1 msec 1 msec *

At the moment this is a little less than impressive since we are just pinging from tunnel IP to tunnel IP, but do not fear, shortly we will be adding on some dynamic routing and IPSEC on top of our lovely basic DMVPN configuration!

Lastly, here are all the relevant configurations for the Hub and two Spokes that were referenced in this post:

Hub:

hostname HQ_WWW
 !
 interface Tunnel0
 ip address 100.100.100.100 255.255.255.0
 ip nhrp network-id 1
 tunnel source e0/0
 tunnel mode gre multipoint
 !
 interface e0/0
 description To WWW
 ip address 192.168.10.1 255.255.255.252
 !
 ip route 0.0.0.0 0.0.0.0 192.168.10.2

Spoke1:

hostname Branch_1
 !
 interface Tunnel0
 ip address 100.100.100.180 255.255.255.0
 ip nhrp map 100.100.100.100 192.168.10.1
 ip nhrp nhs 100.100.100.100
 ip nhrp network-id 1
 tunnel source e0/0
 tunnel mode gre multipoint
 tunnel key 1
 !
 interface e0/0
 description To WWW
 ip address 192.168.80.1 255.255.255.252
 !
 ip route 0.0.0.0 0.0.0.0 192.168.80.2

Spoke2:

hostname Branch_2
 !
 interface Tunnel0
 ip address 100.100.100.190 255.255.255.0
 ip nhrp map 100.100.100.100 192.168.10.1
 ip nhrp nhs 100.100.100.100
 ip nhrp network-id 1
 tunnel source e0/0
 tunnel mode gre multipoint
 tunnel key 1
 !
 interface e0/0
 description To WWW
 ip address 192.168.90.1 255.255.255.252
 !
 ip route 0.0.0.0 0.0.0.0 192.168.90.2

I guess I have a blog now?

Well, here it is, this is my very first blog post, and I think that its fitting that this is taking place on Thanksgiving. The reason I say that (and not just because of how awesomely cliché it sounds) is that I want to thank my best friend and mentor for helping push me to create this blog. He has been a terrible nag, so I figured it was about time I shut him up (that’s the part I’m thankful for)!

I’m hoping that this will be a great place for me to document some of my studies and some of the cooler technologies or issues that I run across in the course of my career. As the lovely “About” page mentions, I am currently working on CCIE number two — Service Provider. I suspect that there will be a fair amount of cool IOS-XR and L2VPN type things posted — particularly since I don’t get to play with these kinds of things in my day job. Other than that, some things I am currently plotting to write about include some PfR, DMVPN, and maybe some ZBFW. Hopefully some of you lovely people out there in the intertubes will find this useful! Stick around and drop me a comment and let me know what you think.