Tidbit: The basics are Kinda Important, see also: Being Dumb with napalm-ansible

Hello friends!

This is a tiny post to remind you that the basics (and the obvious stuff!) is kinda important, or so it turns out! I was messing around using napalm-ansible to push templatized (templetized? template-ized?) configurations to some devices. Everything was going great until it wasn’t. Using Ansible template my configurations were looking good, but for whatever reason napalm-ansible kept timing out and leaving me with a nasty-gram like this:

"msg": "cannot install config: Search pattern never detected in send_command_expect: [>##]\\s*$"

Well then… that isn’t ideal clearly. The extra fun part was that if I ran the exact same Playbook again immediately after failure it would complete and everything would be great. I hate when you reboot a switch (or anything) and the thing you were troubleshooting works… this is kinda how this felt for me. So determined to get to the bottom of it I cloned my CSR a bajillion times and started testing.

I found this GitHub issue: https://github.com/ktbyers/netmiko/issues/555 where Kirk suggested setting the global delay factor. As I understand it this is there to basically just delay timeouts for things so that if the underlying napalm config merge was taking a long time for one or more configs we could gain some buffer time. I did this and originally set it to 2 as Kirk suggested, then I tried 4, and when that didn’t work I tried 20. Needless to say that took a *really* long time, but still eventually failed 😦

Doing as Kirk suggested in that same issue and manually doing the config replace didn’t really work since 1) I wasn’t doing a config replace, and 2) my configuration template did not contain management access stuff, so doing a merge would gut my connectivity (because it was a CSR I still had access via console but still).

Eventually, I realized I was just being a noob and not paying attention to the little obvious stuff we always overlook (because you would think it would just work). Turns out that the router I was poking had no public interweb access — why would this matter you may ask yourself? Of course if napalm/ansible can get to it, shouldn’t life be all rainbows and puppy dogs? Oh, one would think, but yours truly was doing a dumb thing and setting not one, but FOUR NTP servers to a DNS name. Yeah… without internet access that whole resolving DNS thing doesn’t work out so well. I have no idea what the timeouts were (and w/ the global delay set to 20 or whatever I used it took forever to timeout), but it clearly was angering things! Flipping those NTP servers to dummy IPs immediately solved my problem, duh 🙂

This has been your friendly public service reminder that it’s always something obvious and simple!



Full Stack Engineers – Deep Like a Papercut?

The term “full stack engineer” has come up quite a bit lately on blogs and podcasts and the like and I figured I’d throw in my $0.02 on the matter. First though, I think it’s probably worth a super quick review of my “career path” (if you can call it that) since I think that probably has a lot to do with shaping my opinion on the term/idea (duh).

I started out in the Air Force and basically started doing network stuff right away. Strictly network stuff, there were server guys, and desktop guys, and long haul circuit guys, but I was just a network guy. My first real job after the Air Force I worked in healthcare for a large company that owns a whole slew of hospitals across the US. I was just a network guy again. I did like at least a million network refreshes on hospitals of 50 beds to 500 beds. Then I jumped over to a large VAR and for the most part was still just a network guy, but started to slowly touch more security (ASAs mostly) and wireless (5508s and the like).

So basically up till I jumped to the VAR I did nothing but route/switch network stuff. Like nothing at all, and I really liked that. I got to nerd out pretty hard and learn a whole bunch of crap that I don’t remember now about the OSPF database and spanning-tree and who the hell knows what else. That was a big point of pride of me being the “expert” (lol or whatever) on the network. Frankly at that time I scoffed at the idea of generalizing or getting wider w/out staying deep (beer was/is continuing to help me get wider regardless of my opinion on the matter). As I moved onto the VAR though and got into customer facing situations more so, or at least more often, than in the Air Force or at an Enterprise, and I got tasked with doing more things that were outside of my comfort zone I had to broaden out at least a little bit. That was good I think because I was still the “network guy,” but at least I could begin to have intelligent conversations about the firewall and how it should interact with the network.

As I moved into pre-sales and then into whatever the hell it is I am doing now I’ve gotten broader and broader, but have always been able to relate back to my core base in networking. I think my point I’m trying to get at is I don’t think its good advice (personal opinion here) to try to start wide and stay wide. I think you’ve got to be able to have an area of expertise that you can start from, and expand out from there — in my case nowadays that s learning more about virtualization, programming, and storage. These things touch all of the data center networking that I do most, and as I learn to converse about these other silos better I get better at my core strength because I’m able to bridge silos and help translate requirements and configurations between teams.

I have a love/hate with the idea of a “full stack engineer.” I think that you need to have a strong foundational knowledge in one or a few (but can’t get too wide IMO) areas, and then just start asking questions like crazy about the other things/silos that touch/interact with whatever your core area is. I think if you can do that and just never stop asking you will be setup for success in whatever you do.