Task arrays are helpful if you want to run the same script with different parameters (so-called parameter sweep). Scripts will be running simultaneously on different nodes and the results are saved in different files.
Array jobs can be submited using --array=<indexes> option. The indexes specification identifies what array index values should be used. Multiple values may be specified using a comma separated list and/or a range of values with a - separator. For example, --array=0-15 or --array=0,6,16-32. The minimum index value is 0.
A step function can also be specified with a suffix containing a colon and number. For example --array=0-15:4" is equivalent to --array=0,4,8,12.
The array of tasks does not have to be iterated by 1. The iterator can be specified after
:. An example array from 1 to 10 with the interator of 2 will be written as--array=1-10:2.
An example script (arrayjob.sh) that writes a randomly generated number from 1 to 50:
#!/bin/bash
#SBATCH -N1
#SBATCH -c5
#SBATCH --mem=250
#SBATCH -t1
# Generate a random number in the range 1-50
rand=`shuf -i 1-50 -n 1`
# Write the generated number to a file
echo "The generated number is: ${rand}"
Submit the task to the queue with the command
abcd@ui: ~>sbatch --array=1-3 arrayjob.sh
The script execution results in 3 files
abcd@ui: ~>ls
slurm-581784_1.out slurm-581784_2.out slurm-581784_3.out
abcd@ui: ~>more slurm-*
::::::::::::::::::::::::
slurm-581784_1.out
::::::::::::::::::::::::
The generated number is: 47
::::::::::::::::::::::::
slurm-581784_2.out
::::::::::::::::::::::::
Generated number is: 37
::::::::::::::::::::::::
slurm-581784_3.out
::::::::::::::::::::::::
The generated number is: 32
Example script (argument.py) using Python:
import sys
print(int(sys.argv[1])**2)
The script (arrayjob.sh) used to run the task:
#!/bin/bash
#SBATCH -N1
#SBATCH -c5
#SBATCH --mem=250
#SBATCH -t1
#SBATCH --array=1-5 # list of subtask IDs
module load Python
python3 argument.py $SLURM_ARRAY_TASK_ID
and submit the task to the queue by using the command
abcd@ui: ~>sbatch arrayjob.sh
This job will produce 5 output files with the default name format JOBID_TASKID.out, and each file will contain the square of the number corresponding to the subtask ID. In general, the array does not have to be iterated by one. The iterator can be specified using the : character. A sample array from 1 to 10 with an iterator of value 2 would be written as --array=1-10:2.
The array task has several additional environmental variables.
| Zmienna | Opis |
|---|---|
SLURM_ARRAY_JOB_ID |
JOBID of the first task in the array |
SLURM_ARRAY_TASK_ID |
a number equal to the subtask index |
SLURM_ARRAY_TASK_COUNT |
total number of tasks inside the array |
SLURM_ARRAY_TASK_MAX |
index of the last task in the array |
SLURM_ARRAY_TASK_MIN |
index of the first task in the array |
In the example given earlier with the argument SLURM_ARRAY_JOB_ID is equal to the value 177 which also corresponds to the entry 177_1. The value of the variable SLURM_JOB_ID for subsequent tasks in the array will be assigned in the form of an array-free task index, i.e.
The value of SLURM_ARRAY_TASK_MIN shall be equal to 1, SLURM_ARRAY_TASK_MAX = 9, and SLURM_ARRAY_TASK_COUNT = 5.