Guest Post! WTF Are all those Checkboxes? (ACI L3 Outs) – Part 2 of ???

My friend and colleague Mr. Jason Banker recently ran into some good times with the mysteries of the ACI L3 Out Checkbox Madness! He Slack’d me and told me he’d found some clowns blog post about it (yours truly) and that some updates and additional information was needed, so he kindly volunteered some time to help out! Without further ado here is Jason’s Checkbox Madness:


 

As we continue to deploy fabrics we always joke about these damn routing checkboxes shooting us in the foot.  We play with different scenarios in the lab to ensure we understand how these pesky boxes work and what other options we have for future deployments.   The scenario here was to use get different OSPF areas connected to the same border leaf using ACI as the transit.  This scenario brings up some certain challenges and hopefully my testing will help others understand it a little better as well.

Design:

We have two external routers coming into a border leaf on ACI, two L3Outs (required because of multiple areas), one is Area 0 (backbone) and one is Area 1.  Here is the breakdown of routes on each router:

External Router 1 (Area 0):

  • Loopback0: 2.2.2.2/32
  • Loopback1: 4.4.4.4/32
  • Transits: 192.168.0.0/29

External Router 2 (Area 1):

  • Loopback0: 3.3.3.3/32
  • Loopback1: 5.5.5.5/32
  • Transits: 172.16.0.0/29

 

Using ACI as a transit we want routes from Area 0 to be imported into Area 1 and vice versa across the two L3Outs.   We will skip the build of the L3Out portion but I want to focus on those pesky checkboxes again.  Whenever I build an L3Out my network EPG usually looks something like this:

By default, “External Subnets for the External EPG” is checked (this checkbox simply enforces policy on this L3out and contracts are applied to the specific subnet) and I am using 0.0.0.0/0 network as a catch-all.  Moving along with the defaults, I show full adjacency:

 

As well as a full routing table within ACI, receiving the networks above in the fabric as expected:

Note:  Anything received from area 0 is shown as backbone and everything from area 1 is 0.0.0.1.

 

Now if look at the routing table for External Router 1 we see no routes across the fabric being received from External Router 2.

 

Let me check my OSPF Neighbors:

 

So, we have no OSPF routes but we have a neighbor relationship.  Let’s go check External Router 2:

 

So, we are showing some OSPF routes but they are only the loopbacks of the ACI Fabric (Area 0), not what we are necessarily looking for.  ACI blocks transit routes between different L3Outs unless permitted by policy via an OSPF area filter-list (to verify ssh to the border leaf and run “show route-map”).  Let’s go look at the Network EPG checkboxes again and see if we can get routing to occur between OSPF Areas across the fabric.

 

As we showed earlier we are using a catch-all 0.0.0.0/0 with “External Subnets for the External EPG”. If we select the “!” on the upper right portion of the screen we will receive the properties of this screen:

 

Based on what this screen states, “Export Route Control Subnet” – controls the export route direction and “Import Route Control Subnet” – controls the import direction.  This sounds like what we need to get routing to traverse the fabric.  Let’s go ahead and select them for Area 0 but before we can select import there is another configuration we need to apply before we can get import to not be grayed out.  If we go back to the top of the L3Out in the navigation pane we need to select the “Route Control Enforcement:” import checkbox:

 

Now if we go back to the Network EPG we should have both options available to us:

 

Now let’s see if we have any changes within our routing table:

 

We still have no change in the table.  Remember that we are using a catch all 0.0.0.0/0? This would require us to also select the aggregate export and import features on the subnet/network epg we have created for Area 0 and Area 1:

 

Time to verify:

 

This looks great.  Now we will verify External Router 2:

 

If we want to take it a step further we can do network specific routes instead of a catch-all:

 

Router 2:

 

Verify Router 2 is receiving 2.2.2.2/32 from Router 1:

 

Now we can send 3.3.3.3/32 from Router 2 into Router 1:

 

Router 1:

 

As I stated earlier these checkboxes are updating route-maps and prefix-lists within ACI.  Prior to us selecting the import/export feature our route-maps had a deny all so no routes would traverse areas.  Upon selecting these checkboxes we can see the change:

 

Instead of giving you route-map blah I will try and breakdown the map for you just focusing on export route-map. Prior to import/export our route-map looked like this:

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 1

  Match clauses:

    tag: 4294967295

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9998

  Match clauses:

    ospf-area: backbone

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9999

  Match clauses:

    ospf-area: 0.0.0.1

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, permit, sequence 10000

  Match clauses:

  Set clauses:

route-map exp-ctx-2392064-set-external-tag, permit, sequence 2

  Match clauses:

  Set clauses:

    tag 4294967295

route-map imp-ctx-bgp-st-interleak-2392064, deny, sequence 1

  Match clauses:

    tag: 4294967295

  Set clauses:

route-map imp-ctx-bgp-st-interleak-2392064, permit, sequence 10000

  Match clauses:

  Set clauses:

 

You can see that we had “deny” for backbone and area 0.0.0.1 preventing us from using the fabric as a transit.  After we selected import/export features our route-map is updated as such (just focusing on the export route-map):

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 1

  Match clauses:

    tag: 4294967295

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, permit, sequence 9801

  Match clauses:

    ip address prefix-lists: IPv4-ospf-rt2392064--0-dst-rtpfx

    ipv6 address prefix-lists: IPv6-deny-all

    ospf-area: backbone

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, permit, sequence 9802

  Match clauses:

    ip address prefix-lists: IPv4-ospf-rt2392064--1-dst-rtpfx

    ipv6 address prefix-lists: IPv6-deny-all

    ospf-area: 0.0.0.1

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9998

  Match clauses:

    ospf-area: backbone

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, deny, sequence 9999

  Match clauses:

    ospf-area: 0.0.0.1

  Set clauses:

route-map exp-ctx-2392064-deny-external-tag, permit, sequence 10000

  Match clauses:

  Set clauses:

route-map exp-ctx-2392064-set-external-tag, permit, sequence 2

  Match clauses:

  Set clauses:

    tag 4294967295

route-map imp-ctx-bgp-st-interleak-2392064, deny, sequence 1

  Match clauses:

    tag: 4294967295

  Set clauses:

route-map imp-ctx-bgp-st-interleak-2392064, permit, sequence 10000

  Match clauses:

  Set clauses:

Now our route-map has been updated with prefix-lists to allow our traffic across areas, we will look at the prefix-list itself:

 

Leaf-103# show ip prefix-list IPv4-ospf-rt2392064--0-dst-rtpfx

ip prefix-list IPv4-ospf-rt2392064--0-dst-rtpfx: 1 entries

   seq 1 permit 0.0.0.0/0 le 32

 

The 0.0.0.0/0 catch-all has been added and our routes can traverse the fabric.  I suggest you also peak at the import route-maps and see what is happening under the hood there as well.

 

 

 

 

Advertisements

WTF Are all those Checkboxes? (ACI L3 Outs) – Part 1 of ???

Here is a screenshot of some of said checkboxes in case you don’t know what the hell I’m going on about:

 

wtf-checkboxes

If you still don’t know what these things are, these are part of an L3 Out in ACI, specifically these are options that are configurable on the “subnet” of a “network” on an L3 Out. Essentially these control the import/export of prefixes for the L3 out. Since I literally always forget which one does what I figured other people probably do too,  let’s try and go through each of them and figure it out.

First off, lets outline our really really simple topology we’ll be using for this. I will be working with two leafs, and two virtual routers (CSR1000v) connected via two vPCs to a pair of UCS Fabric Interconnects. My CSRs each live in a single VLAN (two total VLANs, one per CSR) that is plumbed through UCS up into ACI (no VMM stuff, just a good ol’ fashion VLAN piped up to the leafs). So more or less I have two routers connected via vPC to my leafs, and I’m routing on a VLAN between everything. Clear as mud?

I’m going to mostly ignore the actual setup of the L3 Out itself, as I want to focus on these damn check boxes. For now, we will just start with our L3 Out “CSR-1” which is a simple OSPF area 0 L3 Out, our adjacencies between this L3 Out and CSR-1 (the router, not the L3 Out) are already up and look good.

screen-shot-2016-09-21-at-1-42-17-pm

Okay great, so we have an L3 Out, we’ve got some adjacencies now what? Well first things first, this is a “regular” area in OSPF, so we should be exchanging routes between the CSR and ACI (both CSR and ACI should be advertising their loopbacks – CSR-1 and CSR-2 are 1.1.1.1 and 2.2.2.2 respectively, and Leaf103 and Leaf104 are 3.3.3.3 and 4.4.4.4 respectively), so lets take a look on CSR-1:

CSR-1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
 a - application route
 + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

3.0.0.0/32 is subnetted, 1 subnets
O 3.3.3.3 [110/2] via 10.1.1.2, 00:00:08, GigabitEthernet2
 4.0.0.0/32 is subnetted, 1 subnets
O 4.4.4.4 [110/2] via 10.1.1.3, 00:00:08, GigabitEthernet2
O E2 192.168.1.0/24 [110/20] via 10.1.1.3, 00:02:21, GigabitEthernet2
                    [110/20] via 10.1.1.2, 00:02:22, GigabitEthernet2

Okay cool, so obviously OSPF is up as we are learning about the loopbacks of the Leafs. Good start. How does it look on the ACI side (note you can do all the stuff I will be doing in the GUI if you’d like but in the interest of not taking up a million pages of screenshots I’ll use the CLI)?

Leaf103# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

Leaf103#

So that’s not really what we are wanting to see is it…? We have relationships up, and it’s obviously working since the CSR gets the routes from ACI, so what gives? (Before you think it’s a stub area or something like that it’s not, this has to do with check boxes remember!?) Well out of habit I created the External EPG (under “Network” in the L3 Out) with a subnet of 0.0.0.0/0 — why? Because usually I like to make ACI a stub area and just take a default route and be on with my life (which requires no check box foo). So what does this have to do with anything? Well lets take a look at those check boxes a little closer, we have the following options under the “Scope” heading:

  • Export Route Control Subnet
  • Import Route Control Subnet (currently grayed out)
  • External Subnets for the External EPG (checked by default)
  • Shared Route Control Subnet
  • Shared Security Import Subnet

So before we can have any traffic flowing between the CSR and ACI, we probably need to figure out why ACI is NOT getting any routes from the CSR (note right now I have one of the CSRs shut down so we are focusing just on CSR-1 at the moment). Lets take a look at that mysterious little “i”  button in the top right of the screen… it sometimes has pretty good information.

screen-shot-2016-09-22-at-8-06-21-am

 

Okay so lots of words. Think we know what an IP is, but lets take a bit closer look at that one to start with… “subnet IP address to be imported from the outside into the fabric.” Okay, so that basically sounds like it may be used in route filtering, lets keep reading… “contracts associated with its parent […] are applied to the subnet.” Alright so we know have an idea that the subnet(s) configured under the “Network” in the L3 Out have at least two purposes 1) defining routes that can be imported into the fabric, and 2) this subnet is related to contracts somehow. Since we still don’t have a route from the CSR in the fabric, lets dig a bit more as to why…

Per the information page it mentions that this subnet matches a route to be imported into the fabric, well I just put 0.0.0.0/0 which would match nothing but a default route right? So let’s try and add a subnet to the “Network” (which is a bit of a confusing name for the parent folder) that corresponds with the loopback on CSR-1:

screen-shot-2016-09-22-at-8-11-50-am

 

Alrighty, subnet added (with only the default “External Subnets” box checked). Lets see if we get a route on the leaf:

 

Leaf103# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

Leaf103#

 

No? Well now we are matching an exact prefix, at least in theory, but what about those damn check boxes? If you’ll notice, there’s a box that is grayed out for “Import Route Control Subnet.” That sounds vaguely helpful right? The note from the “i” page says “Controls the import route direction.” Yeah, that seems kinda like what we want, but why is it grayed out, and more importantly how do we un-grey it out? To do that we have to take a look at the parent L3 Out object.

l3out-import

 

Back up on the top-level of the L3 Out (parent object), there are two boxes for “Route Control Enforcement” one for Import and one for Export. As far as I know Export is always checked, but Import is not checked by default. If we want to bring routes into the fabric, we will need to check this box, so I’m going to go ahead and do just that. Do note that Import control is not supported for EIGRP. With the Import Route Control box checked, our “Import Route Control Subnet” box is now check-able for our subnet, so I’ve gone ahead and ticked that box as well. In theory, we will now get the loopback from CSR-1 installed in the route table on our leaf switches… (drum roll please!):

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

1.1.1.1/32, ubest/mbest: 1/0
 *via 10.1.1.5, vlan73, [110/5], 00:00:01, ospf-default, intra
Leaf3#

Well look at that, sure enough we now install the route for the loopback. Now what if I decided that I didn’t want a /32 loopback on my CSR and did something like this:

CSR-1#show run int lo0
Building configuration...

Current configuration : 63 bytes
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.255
end

Carl-Test-1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
CSR-1(config)#int lo0
CSR-1(config-if)#ip add 1.1.1.1 255.255.255.0
CSR-1(config-if)#ip ospf network point-to-point
CSR-1(config-if)#end

So now my loopback is a /24, and I’ve added ospf network type of point-to-point to ensure OSPF doesn’t just insert the loopback IP as a /32 into the database. Lets pop back over and see what the leaf things about this:

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

Leaf3#

The leaf sure doesn’t think that’s very cool. When we first started looking at the list of check boxes, there was a whole other section that I left out, there are three boxes in the “Aggregate” section.

  • Aggregate Export
  • Aggregate Import
  • Aggregate Shared Routes

The information page is again our friend here, the very first line in the “Aggregate” section states: “when aggregation is not set, the subnets are matched exactly.” Which perfectly explains why when we changed our loopback IP we lost the route in ACI. So knowing that we can summarize, I’m going to go ahead and delete the 1.1.1.1/32 subnet from the network, and enable “Import Route Control Subnet” as well as “Aggregate Import,” and then lets take a look at the routing table on the leaf again:

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

1.1.1.0/24, ubest/mbest: 1/0
 *via 10.1.1.5, vlan73, [110/5], 00:00:16, ospf-default, intra
Leaf3#

Bam! Alrighty so if you were paying attention earlier you may have noticed that there was a 192.168.1.0/24 prefix in the route table on the CSR, that subnet lives in this same VRF in ACI and contains an Ubuntu VM, so lets pop over to that and see if we can ping through to our loopback.

screen-shot-2016-09-22-at-9-08-37-am

Well it starts out looking pretty good… we can ping our default gateway, but we can’t ping our loopback. We can however ping the loopback from a leaf. Remember that box checked by default? “External Subnets for the External EPG”? Turns out there’s a bit more to that! By default in ACI VRFs are in “enforced” mode, which just means that you need to have contracts in place to permit traffic to flow between EPGs. This subnet we’ve been talking about is in fact an EPG, it is a prefix-based EPG. That little tick box that is set by default simply says “hey this subnet that I’m defining is now part of the EPG “ExternalEPG” (the name we gave our “network”).” Okay cool so it’s an EPG, and we need contracts but where do those go…

contracts

In our “Network” (what is basically an EPG!), there is a tab on the main Policy pane for contracts. If you open that tab you will have options for provide and consume. I’ve added a simple permit any contract between the EPG containing my VM and this EPG in our L3 Out, and another drum roll….

screen-shot-2016-09-22-at-9-21-02-am

Awesome! So far we’ve covered the main Import Route Control check box, Import Route Control Subnet, and External Subnets for the External EPG “Scope” check boxes, as well as the Aggregate Import check box. We’ve still got quite a few check boxes to figure out though, so stay tuned for some more!