In this tutorial you will learn how to deploy a high performance computing (HPC) cluster on the Catalyst Cloud using elasticluster and SLURM.
ElastiCluster is an open source tool to create and manage compute clusters on cloud infrastructures. The project was originally created by the Grid Computing Competence Center from the University of Zurich.
SLURM is a highly scalable cluster management and resource manager, used by many of the world’s supercomputers and computer clusters (it is the workload manager on about 60% of the TOP500 supercomputers).
The following video outlines what you will learn in this tutorial. It shows a SLURM HPC cluster being deployed automatically by ElastiCluster on the Catalyst Cloud, a data set being uploaded, the cluster being scaled on demand from 2 to 10 nodes, the execution of an embarrassingly parallel job, the results being downloaded, and finally, the cluster being destroyed.
Warning
This tutorial assumes you are starting with a blank project and using your VPC only for ElastiCluster. You may need to adjust things (e.g. create a dedicated elasticluster security group), if you are doing this in a shared VPC.
Install Python development tools:
sudo apt-get install python-dev virtualenv gcc
Create a virtual environment to install the software:
cd ~
virtualenv elasticluster
source elasticluster/bin/activate
Install Elasticluster on the virtual environment:
git clone https://github.com/gc3-uzh-ch/elasticluster src
cd src
pip install -e .
pip install ndg-httpsclient
Install the Catalyst Cloud OpenStack client tools:
pip install python-keystoneclient python-novaclient python-cinderclient \
python-glanceclient python-ceilometerclient python-heatclient \
python-neutronclient python-swiftclient python-openstackclient
Create template configuration files for ElastiCluster:
elasticluster list-templates 1> /dev/null 2>&1
Edit the ElastiCluster configuration file (~/.elasticluster/config). A sample configuration file compatible with the Catalyst Cloud is provided below:
[cloud/catalyst]
provider=openstack
auth_url=auth_url
username=username
password=password
project_name=projectname
request_floating_ip=True
[login/ubuntu]
image_user=ubuntu
image_user_sudo=root
image_sudo=True
user_key_name=elasticluster
user_key_private=~/elasticluster/id_rsa
user_key_public=~/elasticluster/id_rsa.pub
[setup/slurm]
provider=ansible
frontend_groups=slurm_master
compute_groups=slurm_worker
[cluster/slurm]
cloud=catalyst
login=ubuntu
setup_provider=slurm
security_group=default
# Ubuntu image - Use the ID from running (as of June 2018, 18.04 doesn't work):
# openstack image show -c id ubuntu-16.04-x86_64
image_id=<image UUID>
# Use the correct network UUID from: openstack network list
network_ids=<network UUID>
flavor=c1.c1r1
frontend_nodes=1
compute_nodes=2
ssh_to=frontend
[cluster/slurm/frontend]
# The frontend shares /home via NFS to the compute nodes.
boot_disk_type=b1.standard
boot_disk_size=50
[cluster/slurm/compute]
# Use whatever flavour you'd like to use for your compute nodes.
flavor=c1.c16r64
Create SSH keys for ElastiCluster (no passphrase):
ssh-keygen -t rsa -b 4096 -f ~/elasticluster/id_rsa
Source your openrc file, as explained on Installation on Linux and Mac.
Allow ElastiCluster to connect to instances over SSH (if the nova command fails, then try the openstack command):
nova secgroup-add-group-rule default default tcp 22 22
or
openstack security group rule create --dst-port 22 default
The following commands are provided as examples of how to use ElastiCluster to create and interact with a simple SLURM cluster. For more information on ElastiCluster, please refer to https://elasticluster.readthedocs.io/en/latest/.
Deploy a SLURM cluster on the cloud using the configuration provided:
elasticluster start slurm -n cluster
List information about the cluster:
elasticluster list-nodes cluster
Connect to the front-end node of the SLURM cluster over SSH:
elasticluster ssh cluster
Connect to the front-end node of the SLURM cluster over SFTP, to upload (put file-name) or download (get file-name) data sets:
elasticluster sftp cluster
Grow the cluster to 10 nodes (add another 8 nodes):
elasticluster resize cluster -a 8:compute
Terminate (destroy) the cluster:
elasticluster stop cluster
Connect to the front-end node of the SLURM cluster over SSH as described on the previous section.
The following example demonstrates how to create a simple, embarrassingly parallel workload job that will trigger four tasks and write its output to results.txt.
#!/bin/bash
#
#SBATCH --job-name=test
#SBATCH --output=results.txt
#
#SBATCH --ntasks=4
#SBATCH --time=10:00
#SBATCH --mem-per-cpu=100
srun hostname
srun printenv SLURM_PROCID
srun sleep 15
Submit a job:
sbatch job.sh
List the jobs in the queue:
squeue
There is an options to use elasticluster with server group anti-affinity groups to ensure best load distribution in Openstack cluster. To use this feature clone elasticluster from the repository shown below, this is a temporary step until the feature gets merged upstream.
git clone https://github.com/flashvoid/elasticluster --branch=feature/openstack-aaf
cd src
pip install -e .
pip install ndg-httpsclient
And then set anti_affinity_group_prefix property in [cloud/catalyst] section.
[cloud/catalyst]
provider=openstack
auth_url=auth_url
username=username
password=password
project_name=projectname
request_floating_ip=True
anti_affinity_group_prefix=elasticluster
[login/ubuntu]
image_user=ubuntu
image_user_sudo=root
image_sudo=True
user_key_name=elasticluster
user_key_private=~/elasticluster/id_rsa
user_key_public=~/elasticluster/id_rsa.pub
[setup/slurm]
provider=ansible
frontend_groups=slurm_master
compute_groups=slurm_worker
[cluster/slurm]
cloud=catalyst
login=ubuntu
setup_provider=slurm
security_group=default
# Ubuntu image - Use the ID from running (as of June 2018, 18.04 doesn't work):
# openstack image show -c id ubuntu-16.04-x86_64
image_id=<image UUID>
# Use the correct network UUID from: openstack network list
network_ids=<network UUID>
flavor=c1.c1r1
frontend_nodes=1
compute_nodes=2
ssh_to=frontend
[cluster/slurm/frontend]
# The frontend shares /home via NFS to the compute nodes.
boot_disk_type=b1.standard
boot_disk_size=50
[cluster/slurm/compute]
# Use whatever flavour you'd like to use for your compute nodes.
flavor=c1.c16r64