Tidbit: MacOS Sierra and APICs (SSH)

I upgraded my laptop to MacOS Sierra last night and then was greeted by a fun new error logging into an APIC:

MacBook-Pro:~ Carl$ ssh -l carl 1.1.1.1
Unable to negotiate with 1.1.1.1 port 22: no matching host key type found. Their offer: ssh-dss

Boo! Seems bad yeah? Well it is. Please see this lovely bug here. So more or less this is not impactful, just not so secure. OpenSSH 7.0 deprecated ssh-dss due to its “inherit weakness.” I am not all up to speed with fancy crypto stuff so I will just believe them. In the meantime, until ACI gets on the gravy train with newer/better key exchange algorithms I’ve still got to SSH to stuff. A quick jaunt around Google and the answer is pretty apparent, you can simply define the key exchange algorithm in your SSH line straight from your terminal like so:

MacBook-Pro:~ Carl$ ssh -oHostKeyAlgorithms=+ssh-dss -l carl 1.1.1.1

Kinda a PITA to type huh? If you want to be super lazy, you can edit your ssh config file to always use ssh-dss for a particular host (or * for all hosts if you are feeling frisky) like so:

Host 1.1.1.1

HostkeyAlgorithms +ssh-dss

That file may or may not exist for you, but should live in ~/.ssh/ – this should just use the ssh-dss algorithm as a last resort as far as I understand.

I’ll update this if I find anything else, or if newer versions of code support new algorithms. For what its worth this was tested on ACI 2.0(1o) and macOS Sierra 10.12.

Tidbit: ACI Fabric Certificate and NTP

Quick post here for a fun little issue I ran into this week.

I’ve been building out a greenfield ACI deployment with some Nexus 6ks “north” of the fabric acting as a core. The 6ks and the fabric is all greenfield at this point, no connectivity to the rest of the customers network or the internet or anything. I configured the fabric to use the 6ks as the NTP server, with the intent that eventually the 6ks will point to a low stratum clock of some kind. Set this up, was happy NTP was synced and the NTP faults went away in the fabric and I went to grab some lunch. When I came back none of the APICs could see each other (controller state unavailable), yet I could ping in band on each APIC to any other APIC…

Well, the NTP configuration worked, but since the 6ks were out of the box and didn’t have their clocks set their time was the default — something in 2001. Normally that’s probably just an annoyance — “hey! my clock is way off!”… not this time! The APICs synced up with the 6ks and then their time was set to whatever time/date in 2001 as well. What I didn’t know then was that there is a certificate on the APICs (automagically there from Cisco) that has a “not valid before” date… and, when NTP synced up it flipped the clock on the APIC to a date that was before the validity of the cert. This caused the whole APIC cluster to fail because the controllers couldn’t build the cluster because this certificate was deemed invalid! I should note here that even though I foobared the entire APIC cluster, the fabric was totally fine and passing traffic and all that was good — just couldn’t change the fabric since the cluster was messed up.

You can verify the health of the cluster with the “acidiag avread” command — this will list out all the APICs in the cluster. The output isn’t very friendly to read, but you should be able to see all your APICs that way.

The “show controller” command is much more friendly:

apic1# show controller
Fabric Name : ACI-Pod-1
Operational Size : 1
Cluster Size : 3
Time Difference : -27031964
Fabric Security Mode : permissive

ID Address In-Band IPv4 Address In-Band IPv6 Address OOB IPv4 Address OOB IPv6 Address Version Fl
ags Serial Number Health
 ---- --------------- --------------- ------------------------- --------------- ------------------------- ------------ --
--- ---------------- ----------------
 1* x.x.x.x 0.0.0.0 0.0.0.0 x.x.x.x 0.0.0.0 1.2(2g) cr
v- xxxxxxxxxxx fully-fit


Flags - c:Commissioned | r:Registered | v:Valid Certificate | a:Approved

From this output you can see the controller and some important information – in band IP (TEP IP in the infra tenant), any in band management IPs, the version, serial number, and importantly the flags. In the above output we can see we have the flags “c”, “r”, and “v” — for this particular scenario the one I care about is that valid certificate flag. If you do like I did and foobar the NTP beyond when the cert is valid this would be something to look for.

Finally, and perhaps easiest of all you can use the “acidiag verifyapic” command to very easily check the certificate status as well as the dates that the certificate is valid for:

apic1# acidiag verifyapic
openssl_check: certificate details
subject= CN=xxxxxxxxxxx,serialNumber=PID:APIC-SERVER-M1 SN:xxxxxxxxxxx
issuer= CN=Cisco Manufacturing CA,O=Cisco Systems
notBefore=Mar 21 03:21:13 2015 GMT
notAfter=Mar 21 03:31:13 2025 GMT
openssl_check: passed
ssh_check: passed
all_checks: passed

So to fix all of this hot mess the date/time had to be set manually on the 6k, but then the time was too far out of range for the APICs to get synced up so we had to actually manually set the time on the APIC. You can do that like so:

dbgtoken

login root

date MMDDhhmmYYYY

Note that you will need the debug token and internal access (TAC) to get this fixed.