Containers🔗
Containers lets you define your own environment and makes your work portable and mostly reproducible on any HPC system that supports it. We provide the option to Apptainer (and open fork of Singularity) which intentionally brings in more information from the system to make containers work well for HPC environments.
Apptainer has extensive documentation for helping you build and use containers https://apptainer.org/docs As of the upgrade to Apptainer it is now possible to also build your containers directly on the cluster login nodes, but you can also easily install and use Apptainer on your own linux machine (or a linux VM or via WSL2) and transfer the container images to the cluster.
Our main repository of containers can be found on the clusters under /apps/containers/
and on github https://github.com/c3se/containers/ which can be used as-is or as a basis and inspiration for new container definitions.
Building containers🔗
For complete instructions see https://apptainer.org/docs.
Building a container can be done on your own linux machine, or directly on the login nodes.
A simple definition file based on the miniconda docker image and a local requirements.txt
file could be:
Bootstrap: docker
From: continuumio/miniconda3:4.12.0
%files
requirements.txt
%post
/opt/conda/bin/conda install -y --file requirements.txt
and can be built using
apptainer build my_container.sif my_recipe.def
You can find many some examples definition files on https://github.com/c3se/containers/
Extending images at /apps/containers🔗
Many times you may want more packages than what is contained under /apps/containers
.
In those cases you can bootstrap from these images and add what is missing in the %post step.
For example: Adding a package to an existing miniconda container image.
With recipe file my_recipe.def
Bootstrap: localimage
From: /apps/containers/Conda/miniconda-4.12.0.sif
%post
conda install -y python-dateutil
and building
apptainer build my_container.sif my_recipe.def
Hint: use
apptainer shell --fakeroot --writable-tmpfs /apps/containers/path-to-local-image.sif
to make temporary changes to the localimage and figure out what to put in the recipe.
Using overlays for persistent storage🔗
As of the update to Apptainer we no longer recommend using overlays.
If you still want to use them note:
- Using overlays stored on Mimer does not work due to filesystem limitations.
- An overlay will generally only work in conjunction with the exact original container and nothing else.
- Whenever using overlays add the ':ro' to make them read-only. E.g.
apptainer exec --overlay overlay.img:ro container.sif my_command
- Whenever writing to overlays
- The permissions need to be correct in base container with
chmod
in recipe or--fix-perms
to Apptainer build command. - In Apptainer
--fakeroot
is needed to make changes to overlay image
- The permissions need to be correct in base container with
Using containers in jobs🔗
Using the image in a job is straight forward, and requires no special steps:
#!/bin/bash
#SBATCH -n 1
#SBATCH -t 0:30:00
#SBATCH -A **your-project** -p **your-partition**
echo "Outside of apptainer, host python version:"
python --version
apptainer exec ~/ubuntu.img echo "This is from inside a container. Check python version:"
apptainer exec ~/ubuntu.img python --version
GPU🔗
To access the GPU, Apptainer exposes the system NVidia drivers and libraries into the container space using via the --nv
flag, e.g:
apptainer exec --nv my_image.sif my_gpu_app
Note: We have configured this option on by default on all nodes with GPUs.
When running graphical applications that need 3D acceleration on the GUI machines, you need to combine this with VirtualGL which needs to be installed into the container image:
apptainer exec --nv my_image.sif vglrun my_gui_app
On Alvis, the flag '--nv' is configured to be used by default.
Using modules inside your container🔗
If you need to import additional paths into your container using the APPTAINERENV_
prefix.
This is in particular useful with the PATH
and LD_LIBRARY_PATH
which are for technical reasons cleared inside the container environment.
module load MATLAB
export APPTAINERENV_PATH=$PATH
export APPTAINERENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
apptainer exec ~/ubuntu.sif matlab -nodesktop -r "disp('hello world');"
However, note that is it very easy to break other software inside your container by importing the host's PATH
and LD_LIBRARY_PATH
into your container.
In addition, any system library that the software depends on needs to be installed in your container.
E.g. you can not start MATLAB if there is no X11 installed, which is typically not done when setting up a small, lean, Apptainer image.
Thus, if possible, strive to call modules from outside your container unless you have a special need, e.g:
apptainer exec ~/ubuntu.sif run_my_program simulation.inp
module load MATLAB
matlab < post_process_results.m
Online resources🔗
- Apptainer documentation https://apptainer.org/docs
- Nvidia's NGC catalogue including an up-to-date plethora of HPC/AI/Visualization container images verified by Nvidia: https://ngc.nvidia.com
- Nvidia's HPC Container Maker: https://github.com/NVIDIA/hpc-container-maker
- Open Containers Image Specifications: https://github.com/opencontainers/image-spec
- Docker's library of container images: https://hub.docker.com/
- Docker Docs: https://docs.docker.com/
- Docker multi-stage build: https://docs.docker.com/develop/develop-images/multistage-build/