¶ Apptainer
Why is it worth using containers?
- Isolation: Containers provide isolation for applications, which means each application runs in its own environment. This allows applications to run on the same server without interfering with each other.
- Easy Deployment: Containers are easy to deploy and can run on different platforms without changing the code. This allows for quick migration of applications from one server to another or deployment in the cloud.
- Scalability: Containers enable easy scaling of applications. Multiple instances of containers can be launched to handle a larger number of users.
- Security: Containers are secure because applications run in isolation. In the event that a container is compromised, only that specific application is affected, while the rest of the system remains unaffected.
- Dependency Management: Containers allow for easy management of dependencies, meaning different versions of libraries and tools can be used in different containers without impacting other applications.
When using containers in a cluster, the main advantage is the ability to set up any required environment. This means you are not dependent on the configuration set up in the cluster. You can use any system and tools that may be needed.
This is an alternative to using modules on the cluster, which allows you to utilize any software, not just the pre-built modules.
¶ How to create container images
Apptainer runs containers from SIF image files.
To obtain this file, we mainly have two options:
- We can download a pre-built container image.
- Create our own image.
¶ Option 1 (Downloading a pre-built container image)
Apptainer allows you to download images from sources such as:
- Docker Hub (docker://)
- OCI registry (oras://)
- Container Library (library://)
- Singularity Hub (shub://)
To download a specific container image, you can use the following command:
apptainer pull [options] [target file <URI>
In the above command, if the destination file name is not specified, it will be automatically assigned.
When downloading any OCI-compatible container image, it will be automatically converted to a SIF file.
Example usage of the command:
apptainer pull docker://alpine
apptainer pull alpine.sif docker://alpine
Important information!
When working on the Bem2 cluster and performing image download and conversion, it is recommended to use RAM memory, as the standard memory is network-mounted, which significantly increases the execution time of these operations.
You can follow the above recommendation by setting the following environment variables for the apptainerpull command:
APPTAINER_TMPDIR=/dev/shm/$SLURM_TASK_PID APPTAINER_CACHEDIR=/dev/shm/$SLURM_TASK_PID apptainer pull [options] [target file] <URI>
More informations on page of project Apptainer.
¶ Option 2 (creating your own image)
The second option is to create your own definition file. To get a better understanding of how to create such files, you can refer to the documentation on the project's website Apptainer and informations about Building containers..
You can also take advantage of the option to convert a Dockerfile into an Apptainer definition file using the spython.
Once you have the definition file, you can convert it into a container image using the following command:
apptainer build [options] <target file> <build specyfication>
Usage example:
apptainer build alpine.sif alpine.def
Where alpine.def is the definition file.
Containers should be build in users' own environment and then transfer the resulting SIF file (for example, using scp) or upload the image to a repository from which it can be downloaded.
A possible option is to use the apptainer image in Docker to avoid installing the apptainer software in an environment where Docker is already present. Here is an example of an image that can be used: https://github.com/kaczmarj/apptainer-in-docker.
On the cluster, due to security reasons, users do not have the privilege to escalate permissions. Therefore, it is important to ensure that when building a container, it is created in a way that allows it to be used with the user's level of permissions.
¶ Flag --sandbox
The --sandbox option create a container in a form of directory instead of a SIF file.
To make persistent changes in the sandbox container, it needs to be launched with the --writable flag. This allows for modifications to the contents of the container, similar to a traditional file system.
After making the desired changes in the sandbox container, it can be converted back to a SIF file using the apptainer build command. This process will save the modified container to a SIF file, which can be further utilized.
For more information on using the --sandbox and --writable options, you can refer to the Apptainer project documentation.
¶ Launching images
To run the image, you can use the following command:
apptainer shell [options] <container>
apptainer exec [options] <container> <command>
The first command allows you to open an interactive command line in the container.
The second command allows you to execute a command in the container, which is useful for running batch tasks.
More informations on website Apptainer project.
¶ Mounting paths
Apptainer automatically binds the following paths:
$HOME
/sys:/sys
/proc:/proc
/tmp:/tmp
/var/tmp:/var/tmp
/etc/resolv.conf:/etc/resolv.conf
/etc/passwd:/etc/passwd
$PWD
It follows that by default, full access to files in the user's home directory in the container is provided (unless the apptainer.conf configuration has been modified), allowing for the use of both relative and absolute paths. The absolute path is /home/$USER, and relative paths refer to the current working directory.
¶ Configuration of the apptainer.conf file in relation to bind paths
The apptainer.conf file contains configurations, including which paths will be automatically bound. This file should be owned by the root user, meaning only users with the appropriate permissions can edit it (administrators).
To check the current configuration, you can use the following command
cat /etc/apptainer/apptainer.conf
It's important to note that this can only be done after allocating resources and while working in cluster mode.
The lines indicating which paths will be bound look as follows
mount <mount> == yes
To prevent the configured bindings in the apptainer.conf file from being executed, you should use the flag:
--no-mount <mount>
The exception is the home directory binding. To prevent the home directory from being bound, there are special flags described below.
¶ Disabling the binding of the $HOME directory.
To prevent binding the home directory, we have mainly two options:
- The first option is to use the flag:
--no-home
This flag binds the current working directory without binding the home directory. However, if the current working directory is the home directory, it will be bound.
- The second option is to use the flag:
--containall
This flag prevents the binding of $HOME and instead creates a dummy bind mount at the $HOME point. So, if there are files located in the image under the path /home/$USER, the --containall flag will hide all of them.
¶ Mounting additional paths
¶ --bind
To mount additional paths, you can use the --bind option when running the container.
--bind <host_paths:contianer_paths>
Where the portion before ":" describes the source directory on the host, and the path after ":" describes the target mount point in the container. If the ":" and the path after it are not provided, the path will be assigned the same value in the container. The --bind flag allows for mounting multiple paths at once by separating them with ",".
Example usage:
apptainer shell --bind /opt,/data:/mnt my_container.sif
# or
apptainer exec --bind /data:/mnt my_container.sif ls /mnt
¶ Environment variable
The same can be achieved by setting an appropriate environment variable.
# export APPTAINER_BIND=<"host_paths:contianer_paths">
export APPTAINER_BIND="/data:/mnt"
¶ --mount
This is a more advanced option that allows for managing path bindings, but this option also only supports mounting paths through bindings.
--mount type=bind,src=<source>,dst=<dest>
Some of the features include:
(The "\\" at the end of a line indicates continuation of the command on the next line
- Binding multiple paths to different locations at once. Multiple use of the --mount flag in a single command.
Example:
apptainer shell --mount type=bind,src=/opt,dst=/opt \
--mount type=bind,src=/data,dst=/data \
my_contianer.sif
- Binding paths in read-only mode.
Example:
$ apptainer exec \
--mount type=bind,source=/data,dest=/mnt,ro \
my_contianer.sif touch /mnt/test
- Binding paths to files with characters that --bind cannot handle (e.g.,
.
*Example
$ apptainer run \
--mount type=bind,src=/my:path,dst=/mnt \
my_contianer.sif
¶ Adding paths for Lustre scratch directories.
To add paths for $SCRATCH and $TMPDIR, you can use the following flags:
--mount type=bind,src=$SCRATCH,dst=$SCRATCH --mount type=bind,src=$TMPDIR,dst=$TMPDIR
Use example:
apptainer shell --mount type=bind,src=$SCRATCH,dst=$SCRATCH --mount type=bind,src=$TMPDIR,dst=$TMPDIR moj_kontener.sif
Adding these paths allows you to utilize Lustre resources by using these variables inside the container. For more detailed information on mounting paths, please refer to the Apptainer project documentation.
¶ Fakeroot
This option only works in the "root-mapped user namespace" mode for security reasons, so many options that use this flag are not available.
A more detailed explanation can be found on the Apptainer projec website.
¶ Persistent Overlays
This is an additional option that allows creating "persistent overlays," which means overlaying a directory or a writable file system on a read-only container, giving the illusion of write access. However, due to the disabled support for the --fakeroot flag, this option is very limited.
A more detailed explanation can be found on the Apptainer projec website.
¶ Using GPU with Apptainer
To use graphics cards with Apptainer, you need to use the --nv flag. Instructions on how to access graphics cards can be found in the documentation at man.e-science.pl in informations about GPU resource allocation oraz w instrukcji o nodes with GPUs on the BEM 2 cluster
¶ Examples
¶ Alpine as an interactive task
Assuming our current working directory is /home/$USER, you can do it as follows:
You need to run an interactive job.
srun -c 2 -N 1 --mem=4gb --time=00:30:00 --pty /bin/bash
INFO
Detailed information about resource allocation can be found this.
Next, you need to issue the command:
APPTAINER_TMPDIR=/dev/shm/$SLURM_TASK_PID APPTAINER_CACHEDIR=/dev/shm/$SLURM_TASK_PID apptainer pull docker://alpine
If the available memory is not sufficient, the process will be killed (you will see the message "Killed"). Therefore, when using different images, you should allocate the appropriate amount of memory.
Will be created file alpine_latest.sif.
To execute commands inside the container, you can use the following command:
apptainer shell alpine_latest.sif
¶ Hashcat container with GPU as batch task
To request GPU resources allocation, the user must have the necessary access rights granted. To obtain them, please send an email to kdm@wcss.pl.
Assuming that our current working directory is /home/$USER, to execute a batch job, you should follow these steps:
We create a script with the name create_sif.sh
#!/bin/bash
#SBATCH -N1 # number of nodes
#SBATCH -c3 # number of cores
#SBATCH --mem=40gb #amount of RAM
#SBATCH --time=00:30:00 # time in format hh:mm:ss
#SBATCH --job-name=apptainer-hashcat-sif-test #name
APPTAINER_TMPDIR=/dev/shm/$SLURM_TASK_PID APPTAINER_CACHEDIR=/dev/shm/$SLURM_TASK_PID apptainer pull docker://dizcza/docker-hashcat:latest
Then run the batch job with:
sbatch create_sif.sh
After executing the job, the file named "docker-hashcat_latest.sif" should appear in your home directory.
¶ Creating all the necessary additional files
In this example, it is a file containing a hash named "hash.txt" and a dictionary file named "dictionary.txt" containing potential passwords.
You can create them using the following commands:
touch hash.txt
touch slownik.txt
Next, you need to edit the files using any text editor (e.g., nano) and place some sample content in them.
- hash.txt:
b6b0d451bbf6fed658659a9e7e5598fe
- dictionary.txt:
rockyou
help
funforyou
¶ To run the batch job that starts the container and executes the script:
You need to create a script file named hashtest.sh
#!/bin/bash
#SBATCH -N1 # liczba węzłów
#SBATCH -c2 # ilość rdzeni
#SBATCH --mem=4gb # ilość pamięci RAM
#SBATCH --time=00:05:00 # czas w formacie hh:mm:ss
#SBATCH --job-name=alpine-test #nazwa
#SBATCH -p tesla # partycja
#SBATCH -q student-gpu # qos
#SBATCH -A student-gpu # account
#SBATCH --gres=gpu:tesla:1 #gpu:${nazwa_zasobu}:${ilość}
apptainer exec --nv docker-hashcat_latest.sif hashcat -m 0 hash.txt slownik.txt
The parameters in the above script are for examples only. The user should adjust the parameters to his individual needs. Information about the values of the qos and account parameters that should be used can be found in the e-mail confirming access to GPU resources.
Detailed information on resource allocation can be found here tutaj.
Detailed information on GPU allocation can be found here tutaj.
Then, to run the batch job, issue the command:
sbatch hashtest.sh