Deploying Highly Available instances with Keepalived

This tutorial assumes you have installed the OpenStack command line tools and sourced an openrc file, as explained at Command line interface (CLI). We also assume that you have uploaded a ssh key as explained at Uploading an SSH key.

Introduction

In this tutorial you will learn how to deploy a highly available instance pair using VRRP. This tutorial is largely based from a blog post by Aaron O’Rosen with modifications appropriate for Catalysts cloud. Networks and names have been kept largely compatible with the source material. Additionally information about configuring allowed_address_pairs in heat was sourced from this post.

We will be using two different methods to setup this stack. Initially we will use the openstack command line tool to complete the setup manually. We will then replicate the manual configuration using a heat template to instantiate the same stack automatically.

Note

Due to some functionality not yet being available in the openstack command line tool it is still necessary to use the neutron & nova tools for certain operations.

Virtual Router Redundancy Protocol

VRRP provides hardware redundancy and automatic failover for routers. It allows specifying a virtual router which maps to two or more physical routers. Individual VRRP router instances share an IP address but at any time, only one of the instances is the master (active), the other instances are backups and will not respond using the virtual address. If the master fails, one of the backups is elected as the new master and will begin to respond on the virtual address.

Instances use priorities from 1 (lowest) through 255 (highest), devices running VRRP dynamically elect master and backup routers based on their respective priorities. Only the router that is acting as the master sends out VRRP advertisements at any given point in time. The master router sends advertisements to backup routers at regular intervals (default 1 second). If a backup router does not receive an advertisement for a set period, the backup router with the next highest priority takes over as master and begins forwarding packets.

VRRP instances communicate using packets with multicast IP address 224.0.0.18 and IP protocol number 112. The protocol is defined in RFC3768.

Note

There is an extension to VRRP that uses IPSEC-AH (IP protocol 51) for integrity (see http://www.keepalived.org/draft-ietf-vrrp-ipsecah-spec-00.txt). This tutorial will demostrate using standard VRRP. See this article for more information on securing VRRP.

Warning

A neutron bug that causes DHCP agents to reject previously held DHCP leases for instances after they have been migrated to another node has been discovered. The DHCP agent responds with a DHCPNAK message to instances trying to renew their leases after the agent has been migrated. The DHCPNAK message forces the instances to release the IP addresses assigned to them by the DHCP agent and re-negotiate the DHCP handshake process all over again. A keepalived master instance that receives a DHCPNAK message when attempting to renew the lease for the IP addresses on its primary interface would release all IP addresses including the virtual IP addresses associated with the interface, and this would not be noticed by the keepalived backup instance because the failure time is much shorter than the keepalived advertisement interval thus leaving the virtual IP address unmanaged and unattached to any of the keepalived instances until the keepalived daemon on the master instance is restarted.

While we work on getting this bug fixed, we would recommend that users create new keepalived instances and update existing keepalived instances to use fixed IP on their interfaces.

Allowed Address Pairs

Allowed Address Pairs is a Neutron Extension that extends the port attribute to enable you to specify arbitrary mac_address/ip_address(cidr) pairs that are allowed to pass through a port regardless of the subnet associated with the network.

Let’s double check that this extension is available on the Catalyst Cloud:

$ openstack extension list --network -c Name -c Alias
+-----------------------------------------------+-----------------------+
| Name                                          | Alias                 |
+-----------------------------------------------+-----------------------+
| DNS Integration                               | dns-integration       |
| Neutron L3 Configurable external gateway mode | ext-gw-mode           |
| Port Binding                                  | binding               |
| agent                                         | agent                 |
| Subnet Allocation                             | subnet_allocation     |
| L3 Agent Scheduler                            | l3_agent_scheduler    |
| Neutron external network                      | external-net          |
| Neutron Service Flavors                       | flavors               |
| Network MTU                                   | net-mtu               |
| Quota management support                      | quotas                |
| HA Router extension                           | l3-ha                 |
| Provider Network                              | provider              |
| Multi Provider Network                        | multi-provider        |
| VPN service                                   | vpnaas                |
| Neutron Extra Route                           | extraroute            |
| Neutron Extra DHCP opts                       | extra_dhcp_opt        |
| Neutron Service Type Management               | service-type          |
| security-group                                | security-group        |
| DHCP Agent Scheduler                          | dhcp_agent_scheduler  |
| RBAC Policies                                 | rbac-policies         |
| Neutron L3 Router                             | router                |
| Allowed Address Pairs                         | allowed-address-pairs |
| Distributed Virtual Router                    | dvr                   |
+-----------------------------------------------+-----------------------+

As you can see, the Allowed Address Pairs extension is available.

Clone Orchestration Git Repository

Before we start let’s checkout the https://github.com/catalyst/catalystcloud-orchestration git repository. We will be using some scripts and heat templates from this repository in this tutorial.

$ git clone https://github.com/catalyst/catalystcloud-orchestration.git && ORCHESTRATION_DIR="$(pwd)/catalystcloud-orchestration" && echo $ORCHESTRATION_DIR

Network Setup

Let’s create a network called vrrp-net where we can run our highly available hosts:

$ openstack network create vrrp-net
+-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| admin_state_up  | UP                                   |
| headers         |                                      |
| id              | 98ec34ba-b25e-4720-ae5e-ab7a87fadc51 |
| mtu             | 0                                    |
| name            | vrrp-net                             |
| project_id      | <PROJECT_ID>                         |
| router:external | Internal                             |
| shared          | False                                |
| status          | ACTIVE                               |
| subnets         |                                      |
+-----------------+--------------------------------------+

Now let’s set up a subnet of the network we have just created. We are going to do this so we can use part of the vrrp-net as a dynamically assigned pool of addresses and reserve the rest of the addresses for manual assignment. In this case the pool addresses are in the range 2-200 while the remainder of the /24 will be statically assigned.

$ openstack subnet create --network vrrp-net --allocation-pool start=10.0.0.2,end=10.0.0.200 --subnet-range 10.0.0.0/24 vrrp-subnet
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| allocation_pools  | 10.0.0.2-10.0.0.200                  |
| cidr              | 10.0.0.0/24                          |
| dns_nameservers   |                                      |
| enable_dhcp       | True                                 |
| gateway_ip        | 10.0.0.1                             |
| headers           |                                      |
| host_routes       |                                      |
| id                | cd376d6f-42f4-46c2-8988-717b2f642af4 |
| ip_version        | 4                                    |
| ipv6_address_mode | None                                 |
| ipv6_ra_mode      | None                                 |
| name              | vrrp-subnet                          |
| network_id        | 98ec34ba-b25e-4720-ae5e-ab7a87fadc51 |
| project_id        | <PROJECT_ID>                         |
| subnetpool_id     | None                                 |
+-------------------+--------------------------------------+

Next we will create a router, we will give this router an interface on our new subnet and we will set its gateway as our public network:

$ openstack router create vrrp-router
+-----------------------+--------------------------------------+
| Field                 | Value                                |
+-----------------------+--------------------------------------+
| admin_state_up        | UP                                   |
| external_gateway_info | null                                 |
| headers               |                                      |
| id                    | 78701fa1-e831-4987-a26e-3c24a245294c |
| name                  | vrrp-router                          |
| project_id            | <PROJECT_ID>                         |
| routes                |                                      |
| status                | ACTIVE                               |
+-----------------------+--------------------------------------+


$ openstack router add subnet vrrp-router vrrp-subnet

$ neutron router-gateway-set vrrp-router public-net
Set gateway for router vrrp-router

Note

  • The previous command uses the old neutron command rather than the openstack command as setting router gateways is not yet implemented in the new client.
  • If you look at the ports created at this point using the openstack port list -c ID -c 'Fixed IP Addresses' command you will notice three interfaces have been created. The ip 10.0.0.1 is the gateway address while 10.0.0.2 and 10.0.0.3 provide DHCP for this network.
  • Note the DNS nameservers, gateway address, subnet mask and allocation pool of the subnet from the openstack subnet create command.

Next we will create ports with a fixed IP for our new keepalived instances:

To find the correct subnet and network ID use the following commands

$ VRRP_SUBNET_ID=$( openstack subnet show vrrp-subnet -f value -c id ) && echo $VRRP_SUBNET_ID
cd376d6f-42f4-46c2-8988-717b2f642af4

$ VRRP_NET_ID=$( openstack network show vrrp-net -f value -c id ) && echo $VRRP_NET_ID
98ec34ba-b25e-4720-ae5e-ab7a87fadc51

Then create the ports with your preferred IP addresses

$ openstack port create --fixed-ip subnet=$VRRP_SUBNET_ID,ip-address=10.0.0.4 --network $VRRP_NET_ID vrrp_master_server_port

$ openstack port create --fixed-ip subnet=$VRRP_SUBNET_ID,ip-address=10.0.0.5 --network $VRRP_NET_ID vrrp_backup_server_port

Security Group Setup

Now we will create the vrrp-sec-group security group with rules to allow http, ssh and icmp ingres:

$ openstack security group create --description 'VRRP security group' vrrp-sec-group
+-------------+---------------------------------------------------------------------------------+
| Field       | Value                                                                           |
+-------------+---------------------------------------------------------------------------------+
| description | VRRP security group                                                             |
| headers     |                                                                                 |
| id          | f981cd9a-14fe-4aff-bccf-a60957a32023                                            |
| name        | vrrp-sec-group                                                                  |
| project_id  | <PROJECT_ID>                                                                    |
| rules       | direction='egress', ethertype='IPv4', id='8aa56d18-4b41-4d4a-9cf4-1c2ca0ce2f4e' |
|             | direction='egress', ethertype='IPv6', id='bc6ab597-e58f-405c-9b2a-685be0091d26' |
+-------------+---------------------------------------------------------------------------------+

$ openstack security group rule create --ingress --protocol icmp vrrp-sec-group
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| headers           |                                      |
| id                | 269fe470-9d8c-40d8-ba4a-5261a4ff757a |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | <PROJECT_ID>                         |
| protocol          | icmp                                 |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | f981cd9a-14fe-4aff-bccf-a60957a32023 |
+-------------------+--------------------------------------+


$ openstack security group rule create --ingress --protocol tcp --dst-port 80 vrrp-sec-group
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| headers           |                                      |
| id                | 15e64147-1ed1-41f3-8dc1-a827877f726e |
| port_range_max    | 80                                   |
| port_range_min    | 80                                   |
| project_id        | <PROJECT_ID>                         |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | f981cd9a-14fe-4aff-bccf-a60957a32023 |
+-------------------+--------------------------------------+

$ openstack security group rule create --ingress --protocol tcp --dst-port 22 vrrp-sec-group
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| headers           |                                      |
| id                | fb2a36d9-bb6e-4c73-b86f-0a45728e6872 |
| port_range_max    | 22                                   |
| port_range_min    | 22                                   |
| project_id        | <PROJECT_ID>                         |
| protocol          | tcp                                  |
| remote_group_id   | None                                 |
| remote_ip_prefix  | 0.0.0.0/0                            |
| security_group_id | f981cd9a-14fe-4aff-bccf-a60957a32023 |
+-------------------+--------------------------------------+

Next we will add a rule to allow our Keepalived instances to communicate with each other via VRRP broadcasts:

$ openstack security group rule create --protocol 112 --src-group vrrp-sec-group vrrp-sec-group
+-------------------+--------------------------------------+
| Field             | Value                                |
+-------------------+--------------------------------------+
| direction         | ingress                              |
| ethertype         | IPv4                                 |
| headers           |                                      |
| id                | 3645d670-9cb5-4194-8352-16c5304f1e19 |
| port_range_max    | None                                 |
| port_range_min    | None                                 |
| project_id        | <PROJECT_ID>                         |
| protocol          | 112                                  |
| remote_group_id   | f981cd9a-14fe-4aff-bccf-a60957a32023 |
| remote_ip_prefix  | None                                 |
| security_group_id | f981cd9a-14fe-4aff-bccf-a60957a32023 |
+-------------------+--------------------------------------+

Instance Creation

The next step is to boot two instances where we will run Keepalived and Apache. We will be using the Ubuntu 14.04 image and c1.c1r1 flavour. We will assign these instances to the vrrp-sec-group security group. We will also provide the name of our SSH key so we can login to these machines via SSH once they are created:

Note

You will need to substitute the name of your SSH key.

To find the correct IDs you can use the following commands:

$ VRRP_IMAGE_ID=$( openstack image show ubuntu-14.04-x86_64 -f value -c id ) && echo $VRRP_IMAGE_ID
cab9f3f4-a3a5-488b-885e-892873c15f53

$ VRRP_FLAVOR_ID=$( openstack flavor show c1.c1r1 -f value -c id ) && echo $VRRP_FLAVOR_ID
28153197-6690-4485-9dbc-fc24489b0683

$ VRRP_NET_ID=$( openstack network show vrrp-net -f value -c id ) && echo $VRRP_NET_ID
98ec34ba-b25e-4720-ae5e-ab7a87fadc51

$ VRRP_MASTER_PORT=$(openstack port show vrrp_master_server_port -f value -c id) && echo $VRRP_MASTER_PORT
8f1997e4-fd12-41df-9fb9-d4605e5157d8

$ VRRP_BACKUP_PORT=$(openstack port show vrrp_backup_server_port -f value -c id) && echo $VRRP_BACKUP_PORT
1736183d-8beb-4131-bb60-eb447bcb18f4


$ openstack keypair list
+------------------+-------------------------------------------------+
| Name             | Fingerprint                                     |
+------------------+-------------------------------------------------+
| vrrp-demo-key    | <SSH_KEY_FINGERPRINT>                           |
+------------------+-------------------------------------------------+

We will be passing a script to our instance boot command using the --user-data flag. This script sets up Keepalived and Apache on our master and backup instances. This saves us having to execute these commands manually. This script is located in the git repository you cloned previously at Clone Orchestration Git Repository.

$ cat "$ORCHESTRATION_DIR/hot/ubuntu-14.04/vrrp-basic/vrrp-setup.sh"
#!/bin/bash

HOSTNAME=$(hostname)

if [ "$HOSTNAME" == "vrrp-master" ]; then
    KEEPALIVED_STATE='MASTER'
    KEEPALIVED_PRIORITY=100
elif [ "$HOSTNAME" == "vrrp-backup" ]; then
    KEEPALIVED_STATE='BACKUP'
    KEEPALIVED_PRIORITY=50
else
    echo "invalid hostname $HOSTNAME for install script $0";
    exit 1;
fi

IP=$(ip addr | grep inet | grep eth0 | grep -v secondary | awk '{ print $2 }' | awk -F'/' '{ print $1 }')

echo "$IP $HOSTNAME" >> /etc/hosts

apt-get update
apt-get -y install keepalived

echo "auto eth0
      iface eth0 inet static
         address $IP
         netmask 255.255.255.0
         broadcast 10.0.0.255
         gateway   10.0.0.1 " > /etc/network/interfaces.d/eth0.cfg

apt-get -y --purge remove resolvconf

echo "nameserver 202.78.247.197
      nameserver 202.78.247.198
      nameserver 202.78.247.199
      search openstacklocal" > /etc/resolv.conf

service networking reload

echo "vrrp_instance vrrp_group_1 {
    state $KEEPALIVED_STATE
    interface eth0
    virtual_router_id 1
    priority $KEEPALIVED_PRIORITY
    authentication {
        auth_type PASS
        auth_pass password
    }
    virtual_ipaddress {
        10.0.0.201/24 brd 10.0.0.255 dev eth0
    }
}" > /etc/keepalived/keepalived.conf

apt-get -y install apache2
echo "$HOSTNAME" > /var/www/html/index.html
service keepalived restart

Let’s run the boot command (you will need to substitute your SSH key name and path to the vrrp-setup.sh script):

$ openstack server create --image $VRRP_IMAGE_ID --flavor $VRRP_FLAVOR_ID --nic port-id=$VRRP_MASTER_PORT \
--security-group vrrp-sec-group --user-data vrrp-setup.sh --key-name vrrp-demo-key vrrp-master
+--------------------------------------+------------------------------------------------------------+
| Field                                | Value                                                      |
+--------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                     |
| OS-EXT-AZ:availability_zone          |                                                            |
| OS-EXT-STS:power_state               | NOSTATE                                                    |
| OS-EXT-STS:task_state                | scheduling                                                 |
| OS-EXT-STS:vm_state                  | building                                                   |
| OS-SRV-USG:launched_at               | None                                                       |
| OS-SRV-USG:terminated_at             | None                                                       |
| accessIPv4                           |                                                            |
| accessIPv6                           |                                                            |
| addresses                            |                                                            |
| adminPass                            | <ADMIN_PASSWD>                                             |
| config_drive                         |                                                            |
| created                              | 2016-09-08T03:39:09Z                                       |
| flavor                               | c1.c1r1 (28153197-6690-4485-9dbc-fc24489b0683)             |
| hostId                               |                                                            |
| id                                   | <INSTANCE_ID>                                              |
| image                                | ubuntu-14.04-x86_64 (cab9f3f4-a3a5-488b-885e-892873c15f53) |
| key_name                             | vrrp-demo-key                                              |
| name                                 | vrrp-master                                                |
| os-extended-volumes:volumes_attached | []                                                         |
| progress                             | 0                                                          |
| project_id                           | <PROJECT_ID>                                               |
| properties                           |                                                            |
| security_groups                      | [{u'name': u'vrrp-sec-group'}]                             |
| status                               | BUILD                                                      |
| updated                              | 2016-09-08T03:39:10Z                                       |
| user_id                              | <USER_ID>                                                  |
+--------------------------------------+------------------------------------------------------------+


$ openstack server create --image $VRRP_IMAGE_ID --flavor $VRRP_FLAVOR_ID --nic port-id=$VRRP_BACKUP_PORT \
--security-group vrrp-sec-group --user-data vrrp-setup.sh --key-name vrrp-demo-key vrrp-backup

Let’s check the instances have been created:

$ openstack server list
+---------------------------------+-------------------------+---------+---------------------------------+------------------------------+
| ID                              | Name                    | Status  | Networks                        | Image Name                   |
+---------------------------------+-------------------------+---------+---------------------------------+------------------------------+
| d920fa78-a463-4e17-90de-        | vrrp-backup             | ACTIVE  | vrrp-net=10.0.0.4               | ubuntu-14.04-x86_64          |
| d3167b97a4a3                    |                         |         |                                 |                              |
| ffebb72c-                       | vrrp-master             | ACTIVE  | vrrp-net=10.0.0.5               | ubuntu-14.04-x86_64          |
| 54f7-4a25-a8a9-d164259f8fa5     |                         |         |                                 |                              |
+---------------------------------+-------------------------+---------+---------------------------------+------------------------------+

Virtual Address Setup

The next step is to create the IP address that will be used by our virtual router:

$ openstack port create --network vrrp-net --fixed-ip ip-address=10.0.0.201 vrrp-port
+-----------------------+---------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                       |
+-----------------------+---------------------------------------------------------------------------------------------+
| admin_state_up        | UP                                                                                          |
| allowed_address_pairs |                                                                                             |
| binding_vnic_type     | normal                                                                                      |
| device_id             |                                                                                             |
| device_owner          |                                                                                             |
| dns_assignment        | fqdn='host-10-0-0-201.openstacklocal.', hostname='host-10-0-0-201', ip_address='10.0.0.201' |
| dns_name              |                                                                                             |
| fixed_ips             | ip_address='10.0.0.201', subnet_id='cd376d6f-42f4-46c2-8988-717b2f642af4'                   |
| headers               |                                                                                             |
| id                    | aa70e5e7-0dcb-4de2-8ccc-033b1851fa01                                                        |
| mac_address           | fa:16:3e:36:8e:f2                                                                           |
| name                  | vrrp-port                                                                                   |
| network_id            | 98ec34ba-b25e-4720-ae5e-ab7a87fadc51                                                        |
| project_id            | <PROJECT_ID>                                                                                |
| security_groups       | 87426623-b895-4fa8-bf1b-b3ea6f074328                                                        |
| status                | DOWN                                                                                        |
+-----------------------+---------------------------------------------------------------------------------------------+

Now we need to create a floating IP and point it to our virtual router IP using its port ID:

$ VRRP_VR_PORT_ID=$(openstack port list | grep 10.0.0.201 | awk '{ print $2 }') && echo $VRRP_VR_PORT_ID

$ openstack floating ip create --port $VRRP_VR_PORT_ID public-net
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.201                           |
| floating_ip_address | 150.242.40.101                       |
| floating_network_id | 849ab1e9-7ac5-4618-8801-e6176fbbcf30 |
| headers             |                                      |
| id                  | a4209b06-74f5-4ea8-91c0-832ff75aa612 |
| port_id             | deb027b4-0c3a-4c9e-a5e3-282313e1cb3c |
| project_id          | <PROJECT_ID>                         |
| router_id           | f14dfd59-0bb4-4bc2-b01b-c808d33bc775 |
| status              | DOWN                                 |
+---------------------+--------------------------------------+

Next up we update the ports associated with each instance to allow the virtual router IP as an allowed-address-pair. This will allow them to send traffic using this address.

$ VRRP_MASTER_PORT=$(openstack port list | grep '10.0.0.4' | awk '{ print $2 }') && echo $VRRP_MASTER_PORT
8f1997e4-fd12-41df-9fb9-d4605e5157d8

$ VRRP_BACKUP_PORT=$(openstack port list | grep '10.0.0.5' | awk '{ print $2 }') && echo $VRRP_BACKUP_PORT
1736183d-8beb-4131-bb60-eb447bcb18f4

$ neutron port-update $VRRP_MASTER_PORT --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 8f1997e4-fd12-41df-9fb9-d4605e5157d8

$ neutron port-update $VRRP_BACKUP_PORT --allowed_address_pairs list=true type=dict ip_address=10.0.0.201
Updated port: 1736183d-8beb-4131-bb60-eb447bcb18f4

Check that the virtual router address is associated with this port under allowed_address_pairs:

$ openstack port show $VRRP_MASTER_PORT
+-----------------------+---------------------------------------------------------------------------------+
| Field                 | Value                                                                           |
+-----------------------+---------------------------------------------------------------------------------+
| admin_state_up        | True                                                                            |
| allowed_address_pairs | {"ip_address": "10.0.0.201", "mac_address": "fa:16:3e:f7:af:bf"}                |
| binding:vnic_type     | normal                                                                          |
| device_id             | ebd4b72f-6fcf-4e1d-ad7d-507b944f86df                                            |
| device_owner          | compute:nz-por-1a                                                               |
| extra_dhcp_opts       |                                                                                 |
| fixed_ips             | {"subnet_id": "7c3ca3d4-70a2-4fdd-be9e-4b6bd1eef537", "ip_address": "10.0.0.4"} |
| id                    | 8f1997e4-fd12-41df-9fb9-d4605e5157d8                                            |
| mac_address           | fa:16:3e:f7:af:bf                                                               |
| name                  |                                                                                 |
| network_id            | 617ff618-9da6-4c47-ab3f-527fe5413ea8                                            |
| security_groups       | 3d50882c-c8b8-4c39-9758-390593a5774b                                            |
| status                | ACTIVE                                                                          |
| tenant_id             | 0cb6b9b744594a619b0b7340f424858b                                                |
+-----------------------+---------------------------------------------------------------------------------+

We should now have a stack that looks something like this:

../_images/vrrp-network.png

Updating Existing VRRP Instances To Use Fixed IP

To update existing VRRP instances to use fixed IP on their interfaces, obtain the port ID of the instances and update the port:

$ VRRP_SUBNET_ID=$( openstack subnet show vrrp-subnet -f value -c id ) && echo $VRRP_SUBNET_ID
cd376d6f-42f4-46c2-8988-717b2f642af4

$ VRRP_NET_ID=$( openstack network show vrrp-net -f value -c id ) && echo $VRRP_NET_ID
98ec34ba-b25e-4720-ae5e-ab7a87fadc51

$ VRRP_MASTER_ID=$(openstack server list | grep 'vrrp-master' | awk '{print $2}') && echo $VRRP_MASTER_ID
ffebb72c-54f7-4a25-a8a9-d164259f8fa5

$ VRRP_MASTER_PORT=$(openstack port list --server $VRRP_MASTER_ID | grep '10.0.0.4' | awk '{print $2}') && echo $VRRP_MASTER_PORT
8f1997e4-fd12-41df-9fb9-d4605e5157d8

$ neutron port-update $VRRP_MASTER_PORT --request-format=json --fixed-ips type=dict list=true subnet_id=$VRRP_SUBNET_ID,ip_address=10.0.0.4

$ VRRP_BACKUP_ID=$(openstack server list | grep 'vrrp-backup' | awk '{print $2}') && echo $VRRP_BACKUP_ID
d920fa78-a463-4e17-90de-d3167b97a4a3

$ VRRP_BACKUP_PORT=$(openstack port list --server $VRRP_BACKUP_ID | grep '10.0.0.5' | awk '{print $2}') && echo $VRRP_BACKUP_PORT

$ neutron port-update $VRRP_BACKUP_PORT --request-format=json --fixed-ips type=dict list=true subnet_id=$VRRP_SUBNET_ID,ip_address=10.0.0.5

Then login to the instances and edit their network interfaces and resolv.conf files:

$ sudo vi /etc/network/interfaces.d/eth0.cfg
auto eth0
iface eth0 inet static
   address 10.0.0.4
   netmask 255.255.255.0
   broadcast 10.0.0.255
   gateway  10.0.0.1

$ sudo apt-get -y --purge remove resolvconf

$ sudo vi /etc/resolv.conf
nameserver 202.78.247.197
nameserver 202.78.247.198
nameserver 202.78.247.199
search openstacklocal

$ sudo service networking reload

VRRP Testing

We should now have a working VRRP setup so let’s try it out! We should be able to curl the floating IP associated with our virtual router:

$ VRRP_FLOATING_IP=$(openstack floating ip list | grep 10.0.0.201 | awk '{ print $4 }') && echo $VRRP_FLOATING_IP
150.242.40.121
$ curl $VRRP_FLOATING_IP
vrrp-master

As you can see we are hitting the master instance. Let’s take down the port the virtual router address is configured on on the master to test that we failover to the backup:

$ os port set $VRRP_MASTER_PORT --disable

Curl again:

$ curl $VRRP_FLOATING_IP
vrrp-backup

Instance Access

If we want to take a closer look at what is happening when we switch between VRRP hosts we need to SSH to the instances. We won’t use the floating IP associated with our virtual router, as that will be switching between instances which will make our SSH client unhappy. Consequently, we will assign a floating IP to each instance for SSH access.

$ openstack floating ip create --port $VRRP_MASTER_PORT public-net
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.4                             |
| floating_ip_address | 150.242.42.223                       |
| floating_network_id | 849ab1e9-7ac5-4618-8801-e6176fbbcf30 |
| headers             |                                      |
| id                  | c83a90df-0797-4407-822d-824dc4cbcf01 |
| port_id             | e4ff861b-eb5f-4ec9-95fb-e6ede74b0ad5 |
| project_id          | <PROJECT_ID>                         |
| router_id           | f14dfd59-0bb4-4bc2-b01b-c808d33bc775 |
| status              | DOWN                                 |
+---------------------+--------------------------------------+


$ openstack floating ip create --port $VRRP_BACKUP_PORT public-net
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    | 10.0.0.5                             |
| floating_ip_address | 150.242.42.226                       |
| floating_network_id | 849ab1e9-7ac5-4618-8801-e6176fbbcf30 |
| headers             |                                      |
| id                  | f75bca89-a590-44d0-8a1c-0b5e1c3f18a9 |
| port_id             | a32870b3-e70b-49e6-882d-d4f5332f181b |
| project_id          | <PROJECT_ID>                         |
| router_id           | f14dfd59-0bb4-4bc2-b01b-c808d33bc775 |
| status              | DOWN                                 |
+---------------------+--------------------------------------+

Now we can SSH to our instances. We will connect using the default ubuntu user that is configured on Ubuntu cloud images. You will need to substitute the correct floating IP address.

You can tail syslog in order to see what keepalived is doing. For example, here we can see the backup instance switch from backup to master state:

$ tail -f /var/log/syslog
Aug 26 05:17:47 vrrp-backup kernel: [ 4807.732605] IPVS: ipvs loaded.
Aug 26 05:17:47 vrrp-backup Keepalived_vrrp[2980]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 05:17:47 vrrp-backup Keepalived_vrrp[2980]: Configuration is using : 60109 Bytes
Aug 26 05:17:47 vrrp-backup Keepalived_healthcheckers[2979]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 26 05:17:47 vrrp-backup Keepalived_healthcheckers[2979]: Configuration is using : 4408 Bytes
Aug 26 05:17:47 vrrp-backup Keepalived_vrrp[2980]: Using LinkWatch kernel netlink reflector...
Aug 26 05:17:47 vrrp-backup Keepalived_vrrp[2980]: VRRP_Instance(vrrp_group_1) Entering BACKUP STATE
Aug 26 05:17:47 vrrp-backup Keepalived_healthcheckers[2979]: Using LinkWatch kernel netlink reflector...
Aug 26 05:22:21 vrrp-backup Keepalived_vrrp[2980]: VRRP_Instance(vrrp_group_1) Transition to MASTER STATE
Aug 26 05:22:22 vrrp-backup Keepalived_vrrp[2980]: VRRP_Instance(vrrp_group_1) Entering MASTER STATE

You can also watch the VRRP traffic on the wire with this command:

$ sudo tcpdump -n -i eth0 proto 112
05:28:23.651795 IP 10.0.0.5 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 1s, length 20
05:28:24.652909 IP 10.0.0.5 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 1s, length 20

You can see the VRRP advertisements every second.

If you bring the master port back up at this point you will be able to see the master node switch from the backup instance to the master instance:

$ neutron port-update $VRRP_MASTER_PORT --admin_state_up=True
Updated port: 8f1997e4-fd12-41df-9fb9-d4605e5157d8

on vrrp-backup:

$ sudo tcpdump -n -i eth0 proto 112
05:30:11.773655 IP 10.0.0.5 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype simple, intvl 1s, length 20
05:30:11.774311 IP 10.0.0.4 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype simple, intvl 1s, length 20
05:30:12.775156 IP 10.0.0.4 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype simple, intvl 1s, length 20

At this point we have successfully setup Keepalived with automatic failover between instances. If this is all that you require for your setup so you can stop here.

Resource Cleanup

At this point many people will want to cleanup the OpenStack resources we have been using in this tutorial. Running the following commands should remove all networks, routers, ports, security groups and instances. Note that the order you delete resources is important.

# delete the instances
$ openstack server delete vrrp-master
$ openstack server delete vrrp-backup

# delete instance ports
$ for port_id in $(openstack port list | grep 10.0.0 | grep -v 10.0.0.1 | awk '{ print $2 }'); do openstack port delete $port_id; done

# delete router interface
$ neutron router-interface-delete vrrp-router $(neutron subnet-list | grep vrrp-subnet | awk '{ print $2 }')
Removed interface from router vrrp-router.

# delete router
$ openstack router delete vrrp-router

# delete subnet
$ openstack subnet delete vrrp-subnet

# delete network
$ openstack network delete vrrp-net

# delete security group
$ neutron security-group-delete vrrp-sec-group

Setup Using HEAT Templates

Up to this point in this tutorial we have been using the Nova and Neutron command line clients to setup our system. We have needed to run a large number of different commands in the right order. It would be nice if we could define the entire setup in one configuration file and ask OpenStack to create that setup based on our blueprint.

OpenStack provides just such an orchestration system which is known as heat. In this section we will run heat in order to recreate the stack we have created manually using a single command.

It is beyond the scope of this tutorial to explain the syntax of writing heat templates, thus we will make use of a predefined example from the cloud-orchestration repository. For more information on writing heat templates please consult the documentation at Cloud orchestration.

That said, there are a number of parts of the heat template we should have a look at in more detail. The template is located in the catalystcloud-orchestration repository we cloned earlier.

$ cat "$ORCHESTRATION_DIR/hot/ubuntu-14.04/vrrp-basic/vrrp.yaml"

The first thing to note is the Security Group rule for VRRP traffic:

- direction: ingress
  protocol: 112
  remote_group_id:
  remote_mode: remote_group_id

Note that the remote_mode is set to remote_group_id and remote_group_id is not set. If no value is set then the rule uses the current security group (heat documentation).

The next code block demonstrates how to configure the port and floating IP that will be shared between the VRRP instances.

vrrp_shared_port:
  type: OS::Neutron::Port
  properties:
    network_id: { get_resource: private_net }
    fixed_ips:
      - ip_address: { get_param: vrrp_shared_ip }

vrrp_shared_floating_ip:
  type: OS::Neutron::FloatingIP
  properties:
    floating_network_id: { get_param: public_net_id }
    port_id: { get_resource: vrrp_shared_port }
  depends_on: router_interface

Finally, let’s take a look at the Server and Port definition for an instance:

vrrp_master_server:
  type: OS::Nova::Server
  properties:
    name: vrrp-master
    image: { get_param: image }
    flavor: { get_param: servers_flavor }
    key_name: { get_param: key_name }
    user_data_format: RAW
    networks:
      - port: { get_resource: vrrp_master_server_port }
    user_data:
      get_file: vrrp-setup.sh

vrrp_master_server_port:
  type: OS::Neutron::Port
  properties:
    network_id: { get_resource: private_net }
    allowed_address_pairs:
      - ip_address: { get_param: vrrp_shared_ip }
    fixed_ips:
      - subnet_id: { get_resource: private_subnet }
        ip_address: 10.0.0.4
    security_groups:
       - { get_resource: vrrp_secgroup }

Note the line user_data_format: RAW in the server properties; this is required so that cloud init will setup the ubuntu user correctly (see this blog post for details).

The allowed_address_pairs section associates the shared VRRP address with the instance port. We are explicitly setting the port IP address to 10.0.0.4. This is not required, we are doing it in order to stay consistent with the manual configuration. If we do not set it we cannot control which IPs are assigned to instances and which are assigned for DCHP. If we don’t set these the assigned addresses will be inconsistent across heat invocations.

This configuration is mirrored for the backup instance.

Building the VRRP Stack using HEAT Templates

Before we start, check that the template is valid:

$ openstack orchestration template validate -t $ORCHESTRATION_DIR/hot/ubuntu-14.04/vrrp-basic/vrrp.yaml

This command will echo the yaml if it succeeds and will return an error if it does not. Assuming the template validates let’s build a stack

$ openstack stack create -t $ORCHESTRATION_DIR/hot/ubuntu-14.04/vrrp-basic/vrrp.yaml vrrp-stack
+---------------------+---------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                             |
+---------------------+---------------------------------------------------------------------------------------------------+
| id                  | d5096a5e-4934-490e-822b-d5831fbf57d5                                                              |
| stack_name          | vrrp-stack                                                                                        |
| description         | HOT template for building a Keepalived/Apache VRRP stack in the Catalyst Cloud (nz-por-1) region. |
|                     |                                                                                                   |
| creation_time       | 2016-09-18T23:57:33Z                                                                              |
| updated_time        | None                                                                                              |
| stack_status        | CREATE_IN_PROGRESS                                                                                |
| stack_status_reason | Stack CREATE started                                                                              |
+---------------------+---------------------------------------------------------------------------------------------------+

As you can see the creation is in progress. You can use the openstack stack event list or openstack stack resource list commands to check the progress of creation process:

 $ openstack stack event list vrrp-stack
 2016-09-19 03:20:05Z [vrrp-stack]: CREATE_IN_PROGRESS  Stack CREATE started
 2016-09-19 03:20:06Z [private_net]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:07Z [vrrp_secgroup]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:09Z [router]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:11Z [private_net]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:11Z [vrrp_secgroup]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:11Z [router]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:11Z [private_subnet]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:14Z [private_subnet]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:14Z [vrrp_master_server_port]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:16Z [vrrp_backup_server_port]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:18Z [vrrp_shared_port]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:19Z [router_interface]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:22Z [vrrp_master_server_port]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:22Z [vrrp_backup_server_port]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:22Z [vrrp_shared_port]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:23Z [router_interface]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:24Z [vrrp_master_server_floating_ip]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:25Z [vrrp_backup_server_floating_ip]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:27Z [vrrp_shared_floating_ip]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:28Z [vrrp_master_server]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:20:31Z [vrrp_master_server_floating_ip]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:31Z [vrrp_backup_server_floating_ip]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:32Z [vrrp_shared_floating_ip]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:43Z [vrrp_master_server]: CREATE_COMPLETE  state changed
 2016-09-19 03:20:44Z [vrrp_backup_server]: CREATE_IN_PROGRESS  state changed
 2016-09-19 03:21:06Z [vrrp_backup_server]: CREATE_COMPLETE  state changed
 2016-09-19 03:21:06Z [vrrp-stack]: CREATE_COMPLETE  Stack CREATE completed successfully


$ openstack stack resource list -c resource_name -c resource_type -c resource_status  vrrp-stack
+--------------------------------+------------------------------+-----------------+
| resource_name                  | resource_type                | resource_status |
+--------------------------------+------------------------------+-----------------+
| vrrp_backup_server_port        | OS::Neutron::Port            | CREATE_COMPLETE |
| vrrp_backup_server_floating_ip | OS::Neutron::FloatingIP      | CREATE_COMPLETE |
| vrrp_master_server             | OS::Nova::Server             | CREATE_COMPLETE |
| router_interface               | OS::Neutron::RouterInterface | CREATE_COMPLETE |
| vrrp_master_server_port        | OS::Neutron::Port            | CREATE_COMPLETE |
| vrrp_master_server_floating_ip | OS::Neutron::FloatingIP      | CREATE_COMPLETE |
| vrrp_secgroup                  | OS::Neutron::SecurityGroup   | CREATE_COMPLETE |
| private_subnet                 | OS::Neutron::Subnet          | CREATE_COMPLETE |
| private_net                    | OS::Neutron::Net             | CREATE_COMPLETE |
| router                         | OS::Neutron::Router          | CREATE_COMPLETE |
| vrrp_backup_server             | OS::Nova::Server             | CREATE_COMPLETE |
| vrrp_shared_floating_ip        | OS::Neutron::FloatingIP      | CREATE_COMPLETE |
| vrrp_shared_port               | OS::Neutron::Port            | CREATE_COMPLETE |
+--------------------------------+------------------------------+-----------------+

If you prefer to create this stack in the Wellington region you can modify the appropriate parameters on the command line:

$ OS_REGION_NAME=nz_wlg_2
$ heat stack-create vrrp-stack --template-file $ORCHESTRATION_DIR/hot/ubuntu-14.04/vrrp-basic/vrrp.yaml /
--parameters "public_net_id=e0ba6b88-5360-492c-9c3d-119948356fd3;private_net_dns_servers=202.78.240.213,202.78.240.214,202.78.240.215"

The stack-show and resource-list commands are useful commands for viewing the state of your stack. Give them a go:

$ openstack stack show vrrp-stack
$ openstack stack resource list vrrp-stack

Once all resources in your stack are in the CREATE_COMPLETE state you are ready to re-run the tests as described under VRRP Testing. The neturon floatingip-list command will give you the IP addresses and port IDs you need:

$ openstack floating ip list

If you wish you can SSH to the master and backup instances as described under Instance Access.

Once satisfied with the configuration we can cleanup and get back to our original state:

$ openstack stack delete vrrp-stack
Are you sure you want to delete this stack(s) [y/N]? y

This ends the tutorial on setting up hot swap VRRP instances in the Catalyst Cloud.