Singularity

Singularity is a container system for HPC that lets you define your own environment and makes your work portable and reproducible on any HPC that supports it. Singularity has extensive documentation and examples for helping you installing, setting up containers http://singularity.lbl.gov/

Note: creating and modifying containers must be done outside the cluster on your own Linux machine where you have root access. Please see our build instructions for more information.

Our main repository of containers can be found on the clusters under /apps/containers/, but many users will also want to build their own customized containers.

Deprecated: Some older images, and examples recipes are accessible under /apps/Singularity/.

GPU

To access the GPU, you can use the --nv option when running your container, e.g:

singularity exec --nv my_image.img  my_gpu_app

When running graphical applications that need 3D acceleration on the GUI machines, you need to combine this with VirtualGL:

singularity exec --nv my_image.img  vglrun my_gui_app

Using containers in jobs

Using the image in a job is straight forward, and requires no special steps:

#!/bin/bash
#SBATCH -n 1
#SBATCH -t 0:30:00
#SBATCH -A **your-project** -p hebbe

echo "Outside of singularity, host python version:"
python --version
singularity exec ~/ubuntu.img echo "This is from inside a singularity. Check python version:"
singularity exec ~/ubuntu.img python --version

Using modules inside your container

If you need to import additional paths into your container using the SINGULARITYENV_ prefix. This is in particular useful with the PATH and LD_LIBRARY_PATH which are for technical reasons cleared inside the container environment.

module load MATLAB
export SINGULARITYENV_PATH=$PATH
export SINGULARITYENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
singularity exec ~/ubuntu.simg matlab -nodesktop -r "disp('hello world');"

However, note that is it very easy to break other software inside your container by importing the host's PATH and LD_LIBRARY_PATH into your container. In addition, any system library that the software depends on needs to be installed in your container. E.g. you can not start MATLAB if there is no X11 installed, which is typically not done when setting up a small, lean, Singularity image. Thus, if possible, strive to call modules from outside your container unless you a special need, e.g:

singularity exec ~/ubuntu.simg run_my_program simulation.inp
module load MATLAB
matlab < post_process_results.m

Using overlays for persistent storage

A singularity container is by default immutable, that is, the container will not allow internal modifications while running. If you try to update or add a file in the container image you will get a "Read-only file system" error or similar.

singularity shell myimage.sif
Singularity> echo date >> /myapp.sh
bash: /myapp.sh: Read-only file system

Prototyping and developing often requies one to install an additional package, or make changes to files inside a container. In these cases the --overlay option can be used. Once we know the changes we can later make them persistent by rebuilding the SIF-image, or bind-mount the affected files, or, if we see fit, continue to use the overlay.

An overlay is a directory or file system that "over lays" the immutable filesystem inside your container. The overlay will receive and persist your changes, thus avoiding changing the read-only parts of the container. Creating and preparing the overlays is a bit tricky but using them is simple. At Alvis we have prepared ready-to-go overlays of different sizes at /app/containers/overlay_<size>.img.

Example: Using the 1G overlay on myimage.sif.

cp /apps/containers/overlay_1G.img ~/dev_1G_overlay.img
singularity shell --overlay ~/dev_1G_overlay.img myimage.sif
Singularity> echo date >> /myapp.sh
Singularity>

It is now possible to write to /myapp.sh.

You can also use an overlay to add additional software on top of an existing container. Example: Installing tensorflow into centrally installed anaconda container

cp /apps/containers/overlay_1G.img ~/tf_anaconda_2021.05.img
singularity shell --overlay ~/tf_anaconda_2021.05.img /apps/containers/Conda/anaconda-2021.05.sif
Singularity> conda install tensorflow

Note the such an overlay will likely only work in conjunction with the exact original singularity container and nothing else.

You can not use --overlay from multiple hosts at the same time

If you attempt to use --overlay from multiple hosts at the same time you will get a FATAL error as the overlay option does not support concurrent processes. Only a single writer is supported at one time. If you want to run your container with an overlay on several hosts (or in multiple separate processes within a host) you need to make the overlay read-only by appending ro to the overlay name.

singularity exec --overlay ~/dev_1G_overlay.img:ro myimage.sif