Using Packer to build custom bootable images on the Catalyst Cloud

This tutorial shows you how to use Packer to build custom bootable images on the Catalyst Cloud. Packer is an open source tool developed by Hashicorp for creating machine images for multiple platforms from a single source configuration.

Packer makes use of a builders and provisioners to create custom bootable images.

Builders

Packer supports a number of builders for different target platforms including Amazon EC2 AMI images, VirtualBox and VMware. When building images for the Catalyst Cloud we will be using the OpenStack builder.

Provisioners

Provisioners provide a way to configure a base image such that a new custom image can be created. Many provisioners are available including shell provisoners and provisioners that use DevOps tools like Ansible, Puppet and Chef.

Setup

This tutorial assumes that you have sourced an openrc file, as described at Source an OpenStack RC file. This is required in order for the OpenStack builder to interact with the Catalyst Cloud image service.

You will also need an appropriate security group to allow SSH access for the temporary build machine that packer will create.

Next we need to install packer, packer is a single go binary so this it a simple process:

$ wget https://releases.hashicorp.com/packer/0.10.1/packer_0.10.1_linux_amd64.zip
$ unzip packer_0.10.1_linux_amd64.zip
$ ./packer --version
0.10.1

Create a packer template file

Templates are JSON files that configure the builders and provisioners that we will use to create our custom image.

In this example we will create a basic template that can be invoked with the packer build command. It will create an instance in the Catalyst cloud, and once running copy a script to it and run that script using SSH. Once the script is finished running it will create a new Catalyst Cloud image that includes the changes we have made. Once this process is complete it will cleanup after itself so that only the new image remains.

In this example we will be using the shell provisioner to update the packages on an Ubuntu 16.04 machine to the latest versions. We will then build a golang application called ssllabs-scan from source.

{
  "builders": [{
    "type": "openstack",
    "ssh_username": "ubuntu",
    "image_name": "ubuntu1604_packer_test_1",
    "source_image": "49fb1409-c88e-4750-a394-56ddea80231d",
    "flavor": "c1.c1r1",
    "security_groups": ["example-sg"],
    "floating_ip_pool": "public-net"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "sudo apt-get update",
      "sudo apt-get upgrade -y",
      "sudo apt-get install -y golang-go make",
      "git clone https://github.com/ssllabs/ssllabs-scan",
      "cd /home/ubuntu/ssllabs-scan/",
      "make"
    ]
  }]
}

Building an image

Now we can build a new image called ubuntu1604_packer_test_1 using this template:

$ ./packer build domain-check-packer.json
openstack output will be in this color.

==> openstack: Discovering enabled extensions...
==> openstack: Loading flavor: c1.c1r1
    openstack: Verified flavor. ID: 28153197-6690-4485-9dbc-fc24489b0683
==> openstack: Creating temporary keypair: packer 57c659c0-081a-3bef-2bdb-6cdba8fdaaf3 ...
==> openstack: Created temporary keypair: packer 57c659c0-081a-3bef-2bdb-6cdba8fdaaf3
==> openstack: Launching server...
    openstack: Server ID: e9655fb3-e239-4f4b-80e3-5476f098a132
==> openstack: Waiting for server to become ready...
==> openstack: Creating floating IP...
    openstack: Pool: public-net
    openstack: Created floating IP: 150.242.41.201
==> openstack: Associating floating IP with server...
    openstack: IP: 150.242.41.201
    openstack: Added floating IP 150.242.41.201 to instance!
==> openstack: Waiting for SSH to become available...
==> openstack: Connected to SSH!
==> openstack: Provisioning with shell script: /tmp/packer-shell905865588
    openstack: sudo: unable to resolve host ubuntu1604-domain-check-packer
    openstack: Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]

... Much truncation of apt output

    openstack: Setting up golang-1.6-src (1.6.2-0ubuntu5~16.04) ...
    openstack: Setting up golang-1.6-go (1.6.2-0ubuntu5~16.04) ...
    openstack: Setting up golang-src (2:1.6-1ubuntu4) ...
    openstack: Setting up golang-go (2:1.6-1ubuntu4) ...
    openstack: Setting up libalgorithm-diff-perl (1.19.03-1) ...
    openstack: Setting up libalgorithm-diff-xs-perl (0.04-4build1) ...
    openstack: Setting up libalgorithm-merge-perl (0.08-3) ...
    openstack: Setting up libfile-fcntllock-perl (0.22-3) ...
    openstack: Setting up manpages-dev (4.04-2) ...
    openstack: Setting up pkg-config (0.29.1-0ubuntu1) ...
    openstack: Setting up golang-1.6-race-detector-runtime (0.0+svn252922-0ubuntu1) ...
    openstack: Setting up golang-race-detector-runtime (2:1.6-1ubuntu4) ...
    openstack: Processing triggers for libc-bin (2.23-0ubuntu3) ...
    openstack: Cloning into 'ssllabs-scan'...
    openstack: go build ssllabs-scan.go
==> openstack: Stopping server: e9655fb3-e239-4f4b-80e3-5476f098a132 ...
    openstack: Waiting for server to stop: e9655fb3-e239-4f4b-80e3-5476f098a132 ...
==> openstack: Creating the image: ubuntu1604_domain_check_packer
    openstack: Image: e81c38a0-6fbf-4f62-b873-79af33e4f246
==> openstack: Waiting for image ubuntu1604_domain_check_packer (image id: e81c38a0-6fbf-4f62-b873-79af33e4f246) to become ready...
==> openstack: Deleted temporary floating IP 150.242.41.201
==> openstack: Terminating the source server: e9655fb3-e239-4f4b-80e3-5476f098a132 ...
==> openstack: Deleting temporary keypair: packer 57c659c0-081a-3bef-2bdb-6cdba8fdaaf3 ...
Build 'openstack' finished.

==> Builds finished. The artifacts of successful builds are:
--> openstack: An image was created: e81c38a0-6fbf-4f62-b873-79af33e4f246

Note

The process of building a new image takes some time, now would be a good time to make a cup of tea.

Booting an image

Once the packer build command is complete our newly build image should be available:

$ openstack image show e81c38a0-6fbf-4f62-b873-79af33e4f246
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                         |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | 1abfc6cac5c989e5ad364c1fe0effbde                                                                                                                                                              |
| container_format | bare                                                                                                                                                                                          |
| created_at       | 2016-08-31T04:21:14Z                                                                                                                                                                          |
| disk_format      | raw                                                                                                                                                                                           |
| file             | /v2/images/e81c38a0-6fbf-4f62-b873-79af33e4f246/file                                                                                                                                          |
| id               | e81c38a0-6fbf-4f62-b873-79af33e4f246                                                                                                                                                          |
| min_disk         | 10                                                                                                                                                                                            |
| min_ram          | 1                                                                                                                                                                                             |
| name             | ubuntu1604_domain_check_packer                                                                                                                                                                |
| owner            | 0cb6b9b744594a619b0b7340f424858b                                                                                                                                                              |
| properties       | base_image_ref='49fb1409-c88e-4750-a394-56ddea80231d', direct_url='rbd://b0849a66-357e-4428-a84c-f5ccd277c076/images/e81c38a0-6fbf-4f62-b873-79af33e4f246/snap', image_location='snapshot',   |
|                  | image_state='available', image_type='image', instance_uuid='e9655fb3-e239-4f4b-80e3-5476f098a132', kernel_id='None', owner_id='0cb6b9b744594a619b0b7340f424858b', ramdisk_id='None',          |
|                  | user_id='8c1914eda99d406195674864f2846d45'                                                                                                                                                    |
| protected        | False                                                                                                                                                                                         |
| schema           | /v2/schemas/image                                                                                                                                                                             |
| size             | 10737418240                                                                                                                                                                                   |
| status           | active                                                                                                                                                                                        |
| tags             |                                                                                                                                                                                               |
| updated_at       | 2016-08-31T04:34:21Z                                                                                                                                                                          |
| virtual_size     | None                                                                                                                                                                                          |
| visibility       | private                                                                                                                                                                                       |
+------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Lets boot this image and verify we can invoke the ssllabs-scan application we installed in the image:

$ openstack server create --flavor c1.c1r1 --image e81c38a0-6fbf-4f62-b873-79af33e4f246 --key-name example-key \
--security-group default --security-group example-sg --nic net-id=097a6779-ca20-4017-913e-715c7a2c9967 ssl-scan
+--------------------------------------+-----------------------------------------------------------------------+
| Field                                | Value                                                                 |
+--------------------------------------+-----------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                |
| OS-EXT-AZ:availability_zone          |                                                                       |
| OS-EXT-STS:power_state               | NOSTATE                                                               |
| OS-EXT-STS:task_state                | scheduling                                                            |
| OS-EXT-STS:vm_state                  | building                                                              |
| OS-SRV-USG:launched_at               | None                                                                  |
| OS-SRV-USG:terminated_at             | None                                                                  |
| accessIPv4                           |                                                                       |
| accessIPv6                           |                                                                       |
| addresses                            |                                                                       |
| adminPass                            | XXXXXXXXXXXXXXX                                                       |
| config_drive                         |                                                                       |
| created                              | 2016-08-31T04:50:36Z                                                  |
| flavor                               | c1.c1r1 (28153197-6690-4485-9dbc-fc24489b0683)                        |
| hostId                               |                                                                       |
| id                                   | 79d4e503-205d-4c40-a7d1-a2003844e3fd                                  |
| image                                | ubuntu1604_domain_check_packer (e81c38a0-6fbf-4f62-b873-79af33e4f246) |
| key_name                             | example-key                                                           |
| name                                 | ssl-scan                                                              |
| os-extended-volumes:volumes_attached | []                                                                    |
| progress                             | 0                                                                     |
| project_id                           | 0cb6b9b744594a619b0b7340f424858b                                      |
| properties                           |                                                                       |
| security_groups                      | [{u'name': u'default'}, {u'name': u'example-sg'}]                     |
| status                               | BUILD                                                                 |
| updated                              | 2016-08-31T04:50:36Z                                                  |
| user_id                              | 8c1914eda99d406195674864f2846d45                                      |
+--------------------------------------+-----------------------------------------------------------------------+
$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 |
+--------------------------------------+---------------------+------------------+--------------------------------------+
| a316c6b9-80ba-46ec-9b0a-1838eb6ce78a | 150.242.43.231      | None             | None                                 |
+--------------------------------------+---------------------+------------------+--------------------------------------+
$ openstack server add floating ip ssl-scan 150.242.43.231
$ ssh ubuntu@150.242.43.231
The authenticity of host '150.242.43.231 (150.242.43.231)' can't be established.
ECDSA key fingerprint is 47:db:dc:21:14:d1:ea:03:52:70:0c:2f:6d:a6:82:74.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '150.242.43.231' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-31-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

9 packages can be updated.
7 updates are security updates.


ubuntu@ssl-scan:~$ ls
ssllabs-scan
ubuntu@ssl-scan:~$ ssllabs-scan/ssllabs-scan -version
ssllabs-scan v1.3.0 (stable $Id: 81cb03888c46dd07fb4d97acffa6768b692efa49 $)
API location: https://api.ssllabs.com/api/v2

Using Packer with Windows on the Catalyst Cloud

Note

At this time, due to a known issue in the Catalyst Cloud, it is not possible to deploy a Windows image using packer directly fom the publicly available Windows image.

In order to overcome this limitation it is necessary to deploy a new temporary Windows instance in the Catalyst Cloud. When launching this instance it is required to say Yes to Create New Volume when selecting the Instance Source.

Once the image has booted succesfully, take a snapshot of it. This new snapshot can now be used as the source image for your packer build. It is not necessary to keep the temporary windows instance once the snapshot has been successfully taken.

It is possible to use Packer to create custom Windows images, this requires some changes in approach as the tools and connection details are those typical of Windows technologies.

The first change is in the builders section of the packer build file. Here we need to add the settings to specify the connection type and the credentails to use on this connection.

Below is an example of the new communicator settings. These make use of the Windows Remote Management feature, this uses the WS-Management Protocol, which is based on SOAP (Simple Object Access Protocol).

"builders": [{
    ...

    "communicator": "winrm",
    "winrm_username": "Administrator",
    "winrm_password": "uUteQ419EPFUMoE4zaTE",

    ...
}],

Setting "communicator" to "winrm" is mandatory in order for this to work as expected. The username is required but it does not have to be Administrator, though for a Windows instance it makes sense to have a known administration account.

The other important change is the creation of userdata script that is run by the builders section of the build file. The purpose of the this userdata section is to configure the WinRM access and define the user so that packer is able to connect to the instance once it has been created.

The reference to the userdata script needs to be added to the builders section and provide the location of the script that needs to be run.

"builders": [{
    ...

    "user_data_file": "./userdata_setup.ps1",

    ...
}],

The userdata itself is a Windows command-line/PowerShell script that configures various settings required to allow remote connectivity via WinRM.

#ps1_sysnative
wmic UserAccount set PasswordExpires=False
net user Administrator uUteQ419EPFUMoE4zaTE
cmd /C netsh advfirewall set allprofiles state off
winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="500"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/client/auth '@{Basic="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'
net stop winrm
net start winrm

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine -Force

Warning

The userdata script disables the windows firewall and also sets the Administrator password using plain text which means it could be recovered from the file system.

These two points present a huge risk and should both be addressed to prevent any subsequent compromise of security.

Once the userdata file has been created and the packer build file edited accordingly simply run the packer build command as discussed above.

./packer build windows-build-file.json