ACI – Initial Impressions

I’ve written a fair bit about ACI, having attended as many Cisco roadshow/classes as possible on the subject. I learned a ton about ACI and the theory, the underlying infrastructure bits, I learned a bit about how to navigate the GUI, and I’ve gone on to talk about it with lots of cool customers. I’ve really enjoyed it so far. I believe in the overall direction that Cisco is trying to go with ACI — VxLAN underlay, Hypervisor Agnosticism, Media Agnosticism (bare metal and/or virtual), and the heavy use of Broadcom chips to keep the cost down, with a bit of Cisco ASIC magic sprinkled in to be able to do some other cool things. Since I really do like where all of this is going I’ve been preaching about VxLAN/9ks and even ACI to customers and people within my company. Apparently to some effect, as I was able to convince my company to acquire an ACI lab! I wanted to write a bit about my first impressions outside of the guidance of Cisco classes and slide decks. This will be a brief post as I want to get back to the lab and start churning out some material, so here we go!

First things first — we all know that ACI is API and GUI driven, but once you get out on your own and don’t have a lab guide to follow it is seriously overwhelming! Theres a CLI on the 9ks, but its a weird pseudo CLI that is actually just a sort of front end for the Linux kernel. It’s there, you can certainly validate some things, but you can’t just fall back to typing “switchport mode access” etc., you’re pretty well forced into the GUI/API. I think longer term this won’t be such a big deal, but it for sure is a difficult adjustment to make.

It’s VERY object driven. I’ve heard that, I’ve read that, I knew it was going to be like that, but it really really is. I’ve not had any real UCS experience, so I was starting out in this world from scratch. I think for the UCS junkies out there this will be pretty familiar. I don’t have any particular gripe about how object driven it is — in fact I think it’s actually a good thing; orderly and hierarchical. What has been difficult for me is understanding all of the object interactions. If you change something in a bridge domain, it obviously has lots of impacts. Something I would love to see is a way to highlight a particular configuration and to immediately be able to see what objects reference that. Perhaps more importantly, if you go to delete an object, it would be great to know if it’s still being referenced elsewhere. I suspect some of these improvements will get ironed out as the platform matures.

Abstraction is great, but not ALL the time. I think this is maybe just a product of the learning curve, but I’ve been getting frustrated with simple tasks mostly it seems. Setting up NTP… I think I set it up right… the NTP server is reachable, but it’s still throwing (cryptic) faults at me? What gives? I don’t know because I don’t have my normal show commands. Setting up a routed interface… do I really have to have 10 different things setup? It seems that way. This kind of thing sucks for one-off configurations, but the repeatability and reusability will be awesome for larger deployments.

So far I think the time I’ve spent on the lab, getting it setup and tested, has been pretty evenly divided between being frustrated and being excited. There is clearly a learning curve at play here, it’s also a big shift from the traditional way we configure networks, but its all for the best I think. As ACI progresses and more people adopt it and more blog posts are written and more how-tos are created everything will get easier. I’m looking forward to trying to crank out some posts and videos over the next few months to try to help others as they dive into this.



Spanning-tree is hard! (for Nexus 93128s)

This is a blog post about how I thought I lost my mind this weekend and forgot how to do the spanning-trees.

The project couldn’t have been any simpler — a single 6509 core (I know, I know… but dual Sups at least) connected to some Nexus 93128 switches for the servers to land on. The 9ks function as the default gateway for the servers, and have vPCs to servers where applicable. A static default route (no licensing for dynamic routing… again I know…) pointed back to the 6k. The routing was happening over a VLAN since there was a requirement for some L2 between the 6k/9ks while servers migrated. Easy right?

Physical connectivity to each 9k was a pair of 10g ports from the SUP-2Ts, these connections landed on a 40g QSFP port with the QSFP->SFP+ adapter module. Not super relevant, but I can say I’ve implemented “40g” now 🙂

The uplink was a simple vPC, trunking a native VLAN and the VLAN that the routing is happening on. vPC Peer-Keepalive was up and happy, the Peer-Link was up and happy, and everything was looking good. Once the uplinks were connected, the 6k saw the 9ks as CDP neighbors and things were trucking along… but…. the next hop for the default route was not reachable. So at this point I’m like okay, I suck and fat fingered a VLAN or left it shutdown or something, so I go through and check it all out. L2 VLAN is created on both the 9ks and the 6k, the L3 SVI is created and is up/up on both sides. The 9ks are looking good — can ping the standby IP, and both of the real IPs there, but still no dice on pinging the 6k. So I take a look at the ARP table for my next hop and see that I’m getting an incomplete entry. At this point its pretty obvious that it’s a L2 thing, so I double and triple check that the VLANs are created and all that looks good. Next, I check spanning-tree and see that everything is operating in the same mode, and that the 6k is set up to be the root for this VLAN I’m sending between the boxes. At first glance, the 6k side seems cool — he knows he’s the root for this VLAN and the ports/port-channel are all up so I moved on…

Over on the shiny new 9ks though things are not so awesome though! The 9ks apparently thought because they were new and cool that they needed to be the center of the universe, and showed that they were in fact the root for this VLAN. Well that sure ain’t cool. Interestingly, for the native VLAN though, the 9ks seem to recognize that the 6k is the root. Weird. Configs look fine however, and just to be sure the 9ks REALLY aren’t the root, I bumped the priority for that VLAN to the max. Bump the vPC, and things are still weird. 9ks still think they are the root (well only one of them obviously, but the point being that they stills on’t see the 6k as root). Jumping back over to the 6k, and taking closer inspection I see that the 6k port-channel that goes to the 9ks is in blocking state for that VLAN. Okay…. furthermore it says its in blocking state and lists “P2p Dispute.”

After starring at this and shutting and un-shutting the ports for a few minutes to see if its going to magically start working I say screw it and kill the second 9k, and whittle things down to a single trunk port without any port-channel/vPC to muddy the waters. Same exact results… okay what the hell is happening here. Now I’m starting to wonder if there is something silly going on with the QSFP ports/adapters, so I change things up to a routed link and try to ping across. 100% success for however many pings I want to throw at it. So this yet again confirms that L1 is good to go. At this point I used the phone a friend option thinking I’m just blind and or completely forgot how to do anything with spanning-tree. Thankfully my awesome co-worker picked up and we hopped on a webex and after about 30 minutes he verified that I was in fact not losing my mind! Big relief, but obviously things were still broken.

As a final test before calling TAC to cry we moved the 6k/9k connectivity off the QSFP ports and onto a copper port on the 9k (96 10g copper ports on the 93128TX). Boom — 9ks see the 6k as root and everything starts working as expected.

Turns out there is a nifty feature (not a bug of course) on the 93128 running this particular version of NX-OS that the 40g ports just decide that they don’t want to receive PVST BDPUs…. which of course caused the 6k to not be seen as root, and the 9k to advertise itself as root, causing the dispute on the 6k side and moving that port to a blocking state.

While troubleshooting this one my good friend Google was not very helpful, so I figured I’d write about it. Hopefully if you’ve found this post I can save ya some time 🙂

Bug ID: CSCup33000
Description: PVST BPDUs dropped on Nexus 9300 40G ports