WTF Are all those Checkboxes? (ACI L3 Outs) – Part 1 of ???

Here is a screenshot of some of said checkboxes in case you don’t know what the hell I’m going on about:



If you still don’t know what these things are, these are part of an L3 Out in ACI, specifically these are options that are configurable on the “subnet” of a “network” on an L3 Out. Essentially these control the import/export of prefixes for the L3 out. Since I literally always forget which one does what I figured other people probably do too,  let’s try and go through each of them and figure it out.

First off, lets outline our really really simple topology we’ll be using for this. I will be working with two leafs, and two virtual routers (CSR1000v) connected via two vPCs to a pair of UCS Fabric Interconnects. My CSRs each live in a single VLAN (two total VLANs, one per CSR) that is plumbed through UCS up into ACI (no VMM stuff, just a good ol’ fashion VLAN piped up to the leafs). So more or less I have two routers connected via vPC to my leafs, and I’m routing on a VLAN between everything. Clear as mud?

I’m going to mostly ignore the actual setup of the L3 Out itself, as I want to focus on these damn check boxes. For now, we will just start with our L3 Out “CSR-1” which is a simple OSPF area 0 L3 Out, our adjacencies between this L3 Out and CSR-1 (the router, not the L3 Out) are already up and look good.


Okay great, so we have an L3 Out, we’ve got some adjacencies now what? Well first things first, this is a “regular” area in OSPF, so we should be exchanging routes between the CSR and ACI (both CSR and ACI should be advertising their loopbacks – CSR-1 and CSR-2 are and respectively, and Leaf103 and Leaf104 are and respectively), so lets take a look on CSR-1:

CSR-1#sh ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
 D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
 N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
 E1 - OSPF external type 1, E2 - OSPF external type 2
 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
 ia - IS-IS inter area, * - candidate default, U - per-user static route
 o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
 a - application route
 + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set is subnetted, 1 subnets
O [110/2] via, 00:00:08, GigabitEthernet2 is subnetted, 1 subnets
O [110/2] via, 00:00:08, GigabitEthernet2
O E2 [110/20] via, 00:02:21, GigabitEthernet2
                    [110/20] via, 00:02:22, GigabitEthernet2

Okay cool, so obviously OSPF is up as we are learning about the loopbacks of the Leafs. Good start. How does it look on the ACI side (note you can do all the stuff I will be doing in the GUI if you’d like but in the interest of not taking up a million pages of screenshots I’ll use the CLI)?

Leaf103# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>


So that’s not really what we are wanting to see is it…? We have relationships up, and it’s obviously working since the CSR gets the routes from ACI, so what gives? (Before you think it’s a stub area or something like that it’s not, this has to do with check boxes remember!?) Well out of habit I created the External EPG (under “Network” in the L3 Out) with a subnet of — why? Because usually I like to make ACI a stub area and just take a default route and be on with my life (which requires no check box foo). So what does this have to do with anything? Well lets take a look at those check boxes a little closer, we have the following options under the “Scope” heading:

  • Export Route Control Subnet
  • Import Route Control Subnet (currently grayed out)
  • External Subnets for the External EPG (checked by default)
  • Shared Route Control Subnet
  • Shared Security Import Subnet

So before we can have any traffic flowing between the CSR and ACI, we probably need to figure out why ACI is NOT getting any routes from the CSR (note right now I have one of the CSRs shut down so we are focusing just on CSR-1 at the moment). Lets take a look at that mysterious little “i”  button in the top right of the screen… it sometimes has pretty good information.



Okay so lots of words. Think we know what an IP is, but lets take a bit closer look at that one to start with… “subnet IP address to be imported from the outside into the fabric.” Okay, so that basically sounds like it may be used in route filtering, lets keep reading… “contracts associated with its parent […] are applied to the subnet.” Alright so we know have an idea that the subnet(s) configured under the “Network” in the L3 Out have at least two purposes 1) defining routes that can be imported into the fabric, and 2) this subnet is related to contracts somehow. Since we still don’t have a route from the CSR in the fabric, lets dig a bit more as to why…

Per the information page it mentions that this subnet matches a route to be imported into the fabric, well I just put which would match nothing but a default route right? So let’s try and add a subnet to the “Network” (which is a bit of a confusing name for the parent folder) that corresponds with the loopback on CSR-1:



Alrighty, subnet added (with only the default “External Subnets” box checked). Lets see if we get a route on the leaf:


Leaf103# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>



No? Well now we are matching an exact prefix, at least in theory, but what about those damn check boxes? If you’ll notice, there’s a box that is grayed out for “Import Route Control Subnet.” That sounds vaguely helpful right? The note from the “i” page says “Controls the import route direction.” Yeah, that seems kinda like what we want, but why is it grayed out, and more importantly how do we un-grey it out? To do that we have to take a look at the parent L3 Out object.



Back up on the top-level of the L3 Out (parent object), there are two boxes for “Route Control Enforcement” one for Import and one for Export. As far as I know Export is always checked, but Import is not checked by default. If we want to bring routes into the fabric, we will need to check this box, so I’m going to go ahead and do just that. Do note that Import control is not supported for EIGRP. With the Import Route Control box checked, our “Import Route Control Subnet” box is now check-able for our subnet, so I’ve gone ahead and ticked that box as well. In theory, we will now get the loopback from CSR-1 installed in the route table on our leaf switches… (drum roll please!):

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>, ubest/mbest: 1/0
 *via, vlan73, [110/5], 00:00:01, ospf-default, intra

Well look at that, sure enough we now install the route for the loopback. Now what if I decided that I didn’t want a /32 loopback on my CSR and did something like this:

CSR-1#show run int lo0
Building configuration...

Current configuration : 63 bytes
interface Loopback0
 ip address

Carl-Test-1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
CSR-1(config)#int lo0
CSR-1(config-if)#ip add
CSR-1(config-if)#ip ospf network point-to-point

So now my loopback is a /24, and I’ve added ospf network type of point-to-point to ensure OSPF doesn’t just insert the loopback IP as a /32 into the database. Lets pop back over and see what the leaf things about this:

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>


The leaf sure doesn’t think that’s very cool. When we first started looking at the list of check boxes, there was a whole other section that I left out, there are three boxes in the “Aggregate” section.

  • Aggregate Export
  • Aggregate Import
  • Aggregate Shared Routes

The information page is again our friend here, the very first line in the “Aggregate” section states: “when aggregation is not set, the subnets are matched exactly.” Which perfectly explains why when we changed our loopback IP we lost the route in ACI. So knowing that we can summarize, I’m going to go ahead and delete the subnet from the network, and enable “Import Route Control Subnet” as well as “Aggregate Import,” and then lets take a look at the routing table on the leaf again:

Leaf3# show ip route ospf vrf Carl-Testbed:Carl-Testbed
IP Route Table for VRF "Carl-Testbed:Carl-Testbed"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>, ubest/mbest: 1/0
 *via, vlan73, [110/5], 00:00:16, ospf-default, intra

Bam! Alrighty so if you were paying attention earlier you may have noticed that there was a prefix in the route table on the CSR, that subnet lives in this same VRF in ACI and contains an Ubuntu VM, so lets pop over to that and see if we can ping through to our loopback.


Well it starts out looking pretty good… we can ping our default gateway, but we can’t ping our loopback. We can however ping the loopback from a leaf. Remember that box checked by default? “External Subnets for the External EPG”? Turns out there’s a bit more to that! By default in ACI VRFs are in “enforced” mode, which just means that you need to have contracts in place to permit traffic to flow between EPGs. This subnet we’ve been talking about is in fact an EPG, it is a prefix-based EPG. That little tick box that is set by default simply says “hey this subnet that I’m defining is now part of the EPG “ExternalEPG” (the name we gave our “network”).” Okay cool so it’s an EPG, and we need contracts but where do those go…


In our “Network” (what is basically an EPG!), there is a tab on the main Policy pane for contracts. If you open that tab you will have options for provide and consume. I’ve added a simple permit any contract between the EPG containing my VM and this EPG in our L3 Out, and another drum roll….


Awesome! So far we’ve covered the main Import Route Control check box, Import Route Control Subnet, and External Subnets for the External EPG “Scope” check boxes, as well as the Aggregate Import check box. We’ve still got quite a few check boxes to figure out though, so stay tuned for some more!



ACI Fabric Access Policy Best Practices

I’ve been meaning to write this post for quite a while now as I think this is one of the most important pieces to a solid ACI deployment — having well-defined fabric access policies. If you screw this up early in your ACI deployment you can always fix it but it can be a big pain in the ass. Over the course of the last two years(ish) and a bunch of ACI deployments this is the best strategy I’ve seen; big thanks to my friends @highspeedsnow and @therealaciguy for coming up with the guts of this “design” (or whatever you wanna call it).

Before we dive in, lets review some basics by going over the relevant bits we will discussing:

Interface Policy Groups:

  • An Interface Policy Group is basically a folder that contains L1/L2 traits that you would like to apply to a port or group of ports.
  • Interface Policy Groups come in three main flavors: access, port-channel, and virtual port-channel (vPC).
  • Access Policy Groups can be reused as many times as you want (this will hopefully be more clear as we keep going if it’s not already), whereas port-channel and vPC Policy Groups cannot be reused. PC/vPC Policy Groups aren’t reusable as they are kind of the “interface port-channel” config from traditional NX-OS — what I mean by that is interfaces get associated the Policy Group, and then become part of the port-channel — obviously if we want to have unique port-channels we use unique port-channel IDs in a traditional switch, its much the same in ACI.
  • By themselves, Interface Policy Groups do pretty much nothing! We’ll need to apply the Interface Policy Group to an Interface Selector, don’t worry it all seems kinda crazy and like lots of work at first, but its all quite organized and logical.

Interface Profiles:

  • An Interface Profile by itself kinda doesn’t really do anything either. It’s really a container for Interface Selectors.
  • Interface Profiles ultimately get tied to Switch Profiles — I swear this will all start to make sense shortly 🙂

Interface Selectors:

  • Interface Selectors tie an Interface Policy Group to a port or ports.
  • An Interface Selector is a child object of an Interface Profile.
  • Note that the Interface Selector “selects” a port (via port ID), but not a switch — i.e. Interface Policy Group “Carls-Access-Port” gets associated to port 1/1, but that doesn’t tell us which Switch…

Switch Profiles:

  • Switch Profiles tie all the stuff we’ve talked about together and actually associates things to a specific switch or switches.

Okay so now that we’ve gotten that out-of-the-way, lets jump right into how I like to design this and why. We’ll go in the same order outlined above.

Interface Policy Groups:
Screen Shot 2016-07-24 at 3.26.29 PM

  • As outlined above, only access-port Interface Policy Groups are reusable, so I always create a “Standard-Acess” Policy Group for my access-ports.
  • You’re probably used to having port-channels/vPCs be identified by a number — don’t do that! The name field for the Interface Policy Group accepts numbers so you could easily just name your PC/vPC “1”, “2”, “3”, etc., but its a PITA later as ACI will automatically allocate a PC ID and it will almost certainly not be what you entered!
  • Do yourself a favor and name the PC/vPC policy groups logically. In the above picture I have “F5-PR-A” and “F5-PR-B” — these represent F5 Prod A and B (nice and easy right?). Often times we end up naming the Interface Policy Groups after the hostname of the server they are connecting to.
  • You’ll need Interface Policy Groups for your L3 Outs as well, so I always just name those “L3-Out-XXXX” again you can see I have some L3 Outs to ASAs and a “Subifs” Policy Group as well for the primary L3 Out.
  • Note that in 2.0+ you will see a folder for both “Leaf Policy Groups” and “Spine Policy Groups” – if you are on code prior to 2.0 you will simply have the parent “Policy Groups” folder.

Interface Profiles:
Screen Shot 2016-07-24 at 3.51.05 PM

  • So far nothing too exciting — really just some common sense naming stuff, but this is where it starts to get a little more interesting.
  • My first few times through I was building out Interface Profiles that were specific to a function — my logic was if I have an Interface Profile for lets say my F5 Load Balancers, and I eventually expanded so much I had to add more F5s I could simply deploy that Interface Profile again to the next set of Leaf switches that would connect the new additional F5s. Good logic I guess, but I don’t advocate it!
  • My current strategy is to create an Interface Profile per Leaf switch and per PAIR of Leaf switches (vPC pairs) — this should be pretty obvious in the pic above.

Interface Selectors:
Screen Shot 2016-07-24 at 3.51.21 PM

  • This is perhaps the most important piece to my overall strategy — I always create an Interface Selector PER PORT. So even though we could do a range of ports I always always always do a single port at a time.
  • At first glance that doesn’t make a ton of sense right? Why create more work (building individual Interface Selectors) when we can simply have a range of lets say 10 ports that are all access-ports that all have the same Interface Policy Group attached to them? The answer is simple and pretty obvious (maybe not unless you’ve been burned by this before which is why I’m writing this!) — if I want to make a change to port 1/3, but I’ve previously created an Interface Selector that spans from port 1 to port 10, I would have to delete that Interface Selector (causing some faults and things to happen), create a new Interface Selector for port 1/3, and then another two Interface Selectors for ports 1/1-1/2 and 1/4-1/10. Not cool!
  • Another less drastic example would be if I had a port-channel that was 8 total ports and I created an Interface Selector for ports 1/1-1/4 in my Interface Profile L101-L102 — what happens if I decide that an 80G vPC to my QA A10 is overkill and I want to reclaim some of those ports? You guessed it, gotta delete, rebuild the previous Interface Selectors for the port-channel, then a new one for whatever else I want to do.
  • So the moral of the story here is to be granular as it will save you a lot of hassle in the future.
  • Finally, I always just name these nice and simply “Port-1”, “Port-40”, etc. — keeps it easy to read, and my thinking is that the EPG and/or Interface Policy Group should be the real descriptor (or hey, just use the description field that is available pretty much everywhere in ACI!).

Switch Profiles:
Screen Shot 2016-07-24 at 3.51.35 PM

  • Okay bringing it all together!! So as you can see the Interface Profiles I created mapped to individual switches and vPC pairs of switches (in name only, because they aren’t actually “hooked” to a switch yet), and as you can see I’ve created Switch Profiles that have exactly the same naming convention.
  • And of course, as the name implies, this is where we actually tie all of the interface level stuff to an actual switch. Logically, each Switch Profile is associated to the switches in the name — i.e. Switch Profile “L101-L102” is associated to the vPC switch pair of nodes 101 and 102.

In closing, some of this may seem like extra work, and I suppose it is. That being said blowing all this config in  via the API is SUPER simple (check out the ACI Power Deployment Tool, a Python library I wrote to help folks get started w/ ACI/ReST). Hopefully this all makes sense and you can see the value in being nice and granular with your naming/deployment strategy – it will help you out a ton later down the road!