Packer and NAT

I’ve recently been doing a lot of interesting stuff with HashiCorp tools, specifically Packer and Terraform. If you’re not aware of these tools you can check them out: here for Packer, and here for Terraform. I hope to be writing a fair bit about these tools and some of the other stuff Hashicorp is up to in the next little bit since there is some very cool stuff going on here!

Before I go any further, two notes: 1) I am not an expert on these toolsets so bear with me :)! 2) I’m doing some things with these tools that is maybe a smidge outside of their intended purpose — they are generally speaking “systems” tools not “networking” tools but there is obviously tons of bleed over between “systems” and “networking” so those lines are a bit blurred, and I’m just going to do cool stuff with whatever tools I can get my hands on… so buckle up!

I’m currently using Packer to roll Cisco CSR1000v and someUbuntu baseline images for some lab scenarios (hopefully more on what the actual labs are in the future too — even cooler stuff there). Packer basically consumes a .json file that describes the type of image you are building, what assets you are using to create said image, and all sorts of other relevant data. I won’t get too much into that as the documentation is generally pretty solid. In my case, I’ve been using VMware Vsphere to actually create the images — Packer basically deploys a VM based on my .json file off to my vCenter, and then exports that image via OVFTool (under the covers, you don’t even have to know about OVFTool which is kinda cool all by itself). In a vacuum this works fantastically well. Here is a simple example of how that may look (apologies for bad WordPress formatting):

{
  "builders": [
    {
      "vm_name": "ignw_netops_front_end",
      "type": "vmware-iso",
      "guest_os_type": "other", 
      "format": "vmx",
      "iso_url": "assets/csr1000v-universalk9.03.12.03.S.154-2.S3-std.iso",
      "iso_checksum_type": "md5",
      "iso_checksum": "a92df894bd57af6cd33b715b6cf0ffe8",
      "remote_host": "{{user `esxi_host`}}",
      "remote_datastore": "{{user `esxi_datastore`}}",
      "remote_username": "{{user `esxi_user`}}",
      "remote_password": "{{user `esxi_password`}}",
      "remote_type": "esx5",
      "vnc_disable_password": true,
      "disk_size": "8192",
      "disk_type_id": "zeroedthick",
      "disk_adapter_type": "scsi",
      "vmx_data": {
        "memsize": "8192",
        "numvcpus": "2",
        "ethernet0.virtualDev": "vmxnet3",
        "ethernet0.networkName": "{{user `csr_prtgrp`}}",
      },
     "boot_wait": "4m",
     "keep_registered": true
    }
  ]
}

This shouldn’t look *too* crazy even if you’ve never used Packer. Basically, we’re pointing to the CSR1000v ISO (yes I know that is an older version!) and to where we want to deploy it (ESX blah), and configuring some basic VM settings (disk, CPU, memory, ethernet0). The double squiggly brackets are me bringing in variables from a .json file — a handy way to templatize some stuff you’ll use over and over again. So, so far so good!

Before the above can work though, there is one more little hack that Packer needs — we need to enable the “Guest IP Hack” — check out Nick Charlton’s blog post on that here (no affiliation, just a super helpful blog post!). Here is a super ugly (but functional!) expect script that you can run prior to your Packer builds to enable that on your ESX host:

#!/usr/bin/expect

set username [lindex $argv 0];
set host [lindex $argv 1];
set password [lindex $argv 2];

spawn ssh -oStrictHostKeyChecking=no $username@$host
expect "*assword: "
send "$password\r"
sleep 1
expect "*:~] "
send "esxcli system settings advanced set -o /Net/GuestIPHack -i 1\r"
expect "*:~] "
exit

Here is where things get/got interesting. With all that out-of-the-way Packer can now build your VM, and interrogate VMware to get the IP address (allegedly?). Packer will connect to the VM (via the IP address that it learns about from VMware) to validate all is well and to execute any additional commands that you’ve asked it to (install packages, configure stuff, etc. — none of this shown here). My CSR has no IP address (ok it does in some stuff I omitted, but not a *reachable* IP address), so Packer does some interesting stuff in the logs:

2018/04/29 21:44:30 ui: ==> carl_csr: Waiting for SSH to become available...
2018/04/29 21:44:30 packer: 2018/04/29 21:44:30 [DEBUG] Opening new ssh session
2018/04/29 21:44:30 packer: 2018/04/29 21:44:30 [DEBUG] starting remote command: esxcli --formatter csv network vm list
2018/04/29 21:44:30 packer: 2018/04/29 21:44:30 [DEBUG] Opening new ssh session
2018/04/29 21:44:30 packer: 2018/04/29 21:44:30 [DEBUG] starting remote command: esxcli --formatter csv network vm port list -w 200228
2018/04/29 21:44:32 packer: 2018/04/29 21:44:32 [INFO] Attempting SSH connection...
2018/04/29 21:44:32 packer: 2018/04/29 21:44:32 [DEBUG] reconnecting to TCP connection for SSH
2018/04/29 21:44:32 packer: 2018/04/29 21:44:32 [DEBUG] handshaking with SSH
2018/04/29 21:44:33 packer: 2018/04/29 21:44:33 [DEBUG] handshake complete!
2018/04/29 21:44:33 packer: 2018/04/29 21:44:33 [INFO] no local agent socket, will not connect agent
2018/04/29 21:44:33 ui: ==> carl_csr: Connected to SSH!
2018/04/29 21:44:33 packer: 2018/04/29 21:44:33 Running the provision hook
2018/04/29 21:44:33 ui: ==> carl_csr: Forcibly halting virtual machine...

Weird! It seems that Packer just decided that its VNC connection (what it uses to do the initial provisioning, again not shown, just take my word for it I guess) is good enough and it’ll turn down the router and export it. All good… for now….

Here comes the wonky part of what I’m doing (and something that kinda goes against “normal” usage of these tools) is that I’m rolling these images behind a NAT. And “these” images includes things that are *not* a CSR, like regular Ubuntu boxes. Packer has access to the ESX box(es), but no direct access to the VM(s) once it(they) is(are) brought up. This basically breaks Packer because it can never complete its tasks because it can’t connect. I’m not totally clear *why* Packer is perfectly happy to not connect via SSH to the CSR, but it is very clear that it does want to connect to Ubuntu boxes via SSH as we can see here in the log output:

2018/04/29 22:01:12 packer: 2018/04/29 22:01:12 Connection refused when connecting to: X.X.X.X
2018/04/29 22:01:12 packer: 2018/04/29 22:01:12 [DEBUG] Error getting SSH address: No interface on the VM has an IP address ready
2018/04/29 22:01:14 ui error: ==> carl_jenkins: Timeout waiting for SSH.
2018/04/29 22:01:14 packer: 2018/04/29 22:01:14 [DEBUG] SSH wait cancelled. Exiting loop.
2018/04/29 22:01:14 ui: ==> carl_jenkins: Step "StepConnect" failed

A bit of searching around seemed to indicate that you can basically provide SSH data in your .json file so that Packer knows credentials and IPs and all that fun stuff. So I added the following:

 "ssh_username": "carl",
 "ssh_password": "carl",
 "ssh_wait_timeout": "1m",
 "ssh_port": 1234,
 "ssh_host": "10.10.10.10",

Running it again, it looks like nothing has changed….

2018/04/29 22:01:12 packer: 2018/04/29 22:01:12 Connection refused when connecting to: X.X.X.X
2018/04/29 22:01:12 packer: 2018/04/29 22:01:12 [DEBUG] Error getting SSH address: No interface on the VM has an IP address ready 
2018/04/29 22:01:14 ui error: ==> carl_jenkins: Timeout waiting for SSH. 
2018/04/29 22:01:14 packer: 
2018/04/29 22:01:14 [DEBUG] SSH wait cancelled. Exiting loop. 
2018/04/29 22:01:14 ui: ==> carl_jenkins: Step "StepConnect" failed

Weirdly it doesn’t even seem to try to connect to the IP that I provided, just the IP that it gleaned from ESX. The “fix” is super obvious in retrospect, however the documentation does not make it immediately clear (at least to me, maybe I’m slow?! It is Sunday afternoon and I do have a beer handy after all!): add the “communicator” argument to the .json file. In

"communicator": "ssh",

This kinda blew my mind since it was REALLY obviously already using SSH… and I had already passed the right info to it… but whatever, it’s working now in my silly non-standard setup. And the log shows what we would hope it should:

2018/04/29 22:13:12 packer: 2018/04/29 22:13:12 [DEBUG] TCP connection to SSH ip/port failed: dial tcp 10.10.10.10:1234: connect: connection refused
2018/04/29 22:13:17 packer: 2018/04/29 22:13:17 [INFO] Attempting SSH connection...
2018/04/29 22:13:17 packer: 2018/04/29 22:13:17 [DEBUG] reconnecting to TCP connection for SSH
2018/04/29 22:13:17 packer: 2018/04/29 22:13:17 [DEBUG] handshaking with SSH
2018/04/29 22:13:17 packer: 2018/04/29 22:13:17 [DEBUG] handshake complete!
2018/04/29 22:13:17 packer: 2018/04/29 22:13:17 [INFO] no local agent socket, will not connect agent
2018/04/29 22:13:17 ui: ==> carl_jenkins: Connected to SSH!

Hopefully this will save somebody some headache later!

If this was interesting to you check out IGNW (disclaimer, they pay my bills!) to come work with me or see what other cool stuff we’re up to!

Advertisements

Python (relative) Imports and Unittests – Derp!

Been far too long since I wrote something. Spent way longer than I feel good about talking about derping around getting my ass kicked by some obvious shit so figured I should write it up as a reminder for me and to hopefully save some anonymous Internet person some heartache! Anyway, I’ve been meaning to put on my big boy pants and start actually writing test cases for all the Python code I write… I’ve been terrible at this and figured it’s time to get my life together with it. Rather than have a bunch of code that won’t really matter, I’ll just use ridiculously dumb examples here to get the issue and the “fix” (not being dumb!) across.

A pretty standard structure for a python project would look something like this:

mycoolproject/
├── mycoolproject
│   ├── __init__.py
│   ├── my_module_1.py
│   └── my_module_2.py
└── tests
 ├── __init__.py
 └── test_basics.py

You can see that we have our “project” — in this case named “mycoolproject” and within that directory we have a folder for our tests and a folder for the actual project code. In the root directory for our project we would probably have other stuff like a setup.py or a requirements.txt or whatever, but those things don’t matter for us at the moment. One last bit — we need those “__init__.py” files in there to make this a “project” — from the real Python docs:

“The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.”

You can find that doc here if you want to read some more about it.

Ok so now that we have the overall gist of things down, let’s take a look at our ridiculously over simplified “module”. Here is the contents of our file “my_module_1.py”:

my_string = 'whoa, this is so kewl'
print(my_string)

Pretty serious code 🙂

Our other script “my_module_2.py” is also pretty simple, but this one refers back to the first module to use the variable “my_string” — we’ll get to why this tripped me up in a bit:

import my_module_1
my_new_string = f'carl said: {my_module_1.my_string}'
print(f'carl said: {my_module_1.my_string}')

Alrighty, so if we run “my_module_1.py” it will simply print “whoa, this is so kewl” — no surprise there. If we run “my_module_2.py” it will *also* print “whoa, this is so kewl” because it is importing/loading the first script, and then of course it will also print “carl said: whoa, this is so kewl” as expected. Great, so our super fancy project works as desired. Now, because we are trying to be better about testing, let’s write a super simple test to validate this code works as expected.

In our test folder, we’ll create a new script called “test_basics.py” that looks like this:

import unittest
import my_module_1


class TestMe(unittest.TestCase):
   def test_stuff(self):
       assert my_module_1.my_string == 'whoa, this is so kewl'


if __name__ == '__main__':
    unittest.main()

At the top we’ll import the unittest library to use for our testing, and we’ll also import our script “my_module_1” so that we can validate (assert) that our variable “my_string” is actual equal to what we think it should be (“whoa, this is so kewl”).

So let’s go ahead and run our test suite and see what happens:

Carls-MacBook-Pro-2:mycoolproject Carl$ python3 -m unittest tests/test_basics.py
E
======================================================================
ERROR: test_basics (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_basics
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
 module = __import__(module_name)
 File "/Users/Carl/Desktop/mycoolproject/tests/test_basics.py", line 2, in <module>
 import my_module_1
ModuleNotFoundError: No module named 'my_module_1'


----------------------------------------------------------------------
Ran 1 test in 0.000s

 

Well… not ideal eh? Obviously we have some kind of import error since Python is complaining it can’t find our module “my_module_1”. What gives? Well, we know that Python is mad at the line where we import “my_module_1” so obviously we need to start there. We also know that our modules actually run fine within their directory — they do exactly what we think they should. So with this information we can understand that Python — when ran from the tests directory has no idea how and where to find the module we are trying to run. This makes sense because when you think about it Python will search for modules in the local folder and the system path(s) — we can see where Python is looking by importing sys and printing out the path, let’s see what that looks like in our tests folder:

Carls-MacBook-Pro-2:tests Carl$ python3
 Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)
 [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import sys
 >>> sys.path
 ['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages']
 >>>

Ok, so we know it’s looking in the normal paths, and that very first entry (”) shows us its going to look for stuff locally too, but obviously nowhere to be seen is the module we’ve been building. So somehow we need to tell Python to look there, what happens if we ask Python to import “my_module_1” *from* “mycoolproject” like so:

from mycoolproject import my_module_1

Let’s run it and see what happens:

Carls-MacBook-Pro-2:mycoolproject Carl$ python3 -m unittest tests/test_basics.py
whoa, this is so kewl
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Hey that seems a lot better huh? Up to this point everything has been super straight forward and if you’ve done any amount of Python stuff you’ll be more than familiar with import errors as you’ve undoubtedly forgotten to import something and had this happen to you. The next bit is where I got tripped up… let’s add a quick test case to test our other Python file:

import unittest
from mycoolproject import my_module_1
from mycoolproject import my_module_2


class TestMe(unittest.TestCase):
   def test_stuff(self):
       assert my_module_1.my_string == 'whoa, this is so kewl'

  def test_other_stuff(self):
      assert my_module_2.my_new_string == 'carl said: whoa, this is so kewl'


if __name__ == '__main__':
    unittest.main()

Pretty straight forward stuff here too — we simply imported the other module and added a test case to assert that the string is what we think it should be. So what happens if we run our unit tests again?

Carls-MacBook-Pro-2:mycoolproject Carl$ python3 -m unittest tests/test_basics.py
whoa, this is so kewl
E
======================================================================
ERROR: test_basics (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_basics
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
 module = __import__(module_name)
 File "/Users/Carl/Desktop/mycoolproject/tests/test_basics.py", line 3, in <module>
 from mycoolproject import my_module_2
 File "/Users/Carl/Desktop/mycoolproject/mycoolproject/my_module_2.py", line 1, in <module>
 import my_module_1
ModuleNotFoundError: No module named 'my_module_1'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

Annnnnnd we’re back to not working. So what gives? We get kinda the same error as before — complaining about import errors and whatnot. Not cool, especially since we basically did the same thing we did for “my_module_2” as we already did (and got working) for “my_module_1”. This time we can see that Python is upset about not finding a module called “my_module_1” — pretty ridiculous given the fact that importing that module is the second line of our test file right? BUT this is not failing *in* our test file — and here is what tripped me up. The issue is (and I’ll probably butcher the exact technical reasoning for this but you can check out this super handy SO post) that Python is confused about where that module is because the path for the execution (via unittest) is not *in* the same folder as the modules themselves. So we can address this by ensuring that the imports in our modules are not relative, but instead fully qualified if you will. Changing our “my_module_2” file to look like this:

from mycoolproject import my_module_1

my_new_string = f'carl said: {my_module_1.my_string}'
print(my_new_string)

Instead of importing from our local file we are now specifying the project that we are importing from, running our test again we get the following (good) results:

Carls-MacBook-Pro-2:mycoolproject Carl$ python3 -m unittest tests/test_basics.py
whoa, this is so kewl
carl said: whoa, this is so kewl
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

TL;DR — pay attention to your imports, easy thing to fix but easy to miss it, run it locally and have everything run great and then be dumb like me and get angry at tests for not behaving the way you think they should 🙂