On the WCSS supercomputers, there are various SLURM partitions, which can also be understood as job queues. Partitions group selected computing resources and each of them has its own set of restrictions, such as job size limit, job time limit, groups (services) that have access to them, etc. Access to partitions is granted based on membership in the appropriate services (or more precisely, Linux groups corresponding to these services).
Below are tables containing information about the currently available partitions.
Maximum amounts of resources per job
The values given in the columns "Number of CPUs per node", "Number of GPUs per node", "Memory" and "Available TMPDIR" define the maximum amounts of these resources per node for one job.
The TMPDIR directories
Depending on the selected partition and the type of task - single or multi-node, default types of TMPDIR directories are assigned (see the "Available TMPDIR" column). More information on the Temporary disk space for computations (TMPDIR) page.
Automatic partition choice based on the job duration
To automatically assign a job to the appropriatebem2-cpu-shortorbem2-cpu-normalpartition based on the declared job duration, use the--partition bem2-cpuoption when submitting the job (this is an additional requeuing mechanism andbem2-cpuis not formally a SLURM partition).
Limitations and requirements
- service with CPU hours required (check withservice-balance --check-cpu)
| Partition | Node Count | CPU Model | Number of CPUs per node | GPU Model | Number of GPUs per node | Memory | Max Job Time | Available TMPDIR (quotab) |
|---|---|---|---|---|---|---|---|---|
| bem2-cpu-short | 487 | Intel(R) Xeon(R) Platinum 8268 | 48 | - | - | 177G/357Ga | 3-00:00:00 |
|
| bem2-cpu-normal | 336 | Intel(R) Xeon(R) Platinum 8268 | 48 | - | - | 177G/357Ga | 21-00:00:00 |
|
a two types of nodes with different amounts of memory available
b maximum TMPDIR occupancy: Lustre - no limit on TMPDIR occupancy; SHM - maximum TMPDIR capacity equal to the amount of memory on the node, LOCAL - available TMPDIR space per node given in brackets
* default TMPDIR space for single-node tasks
** default TMPDIR space for multi-node tasks
Automatic partition choice based on the job duration
To automatically assign a job to the appropriatelem-gpu-shortorlem-gpu-normalpartition based on the declared job duration, use the--partition lem-gpuoption when submitting the job (this is an additional requeuing mechanism andlem-gpuis not formally a SLURM partition).
Limitations and requirements
- service with CPU and GPU hours required (check withservice-balance --check-cpu --check-gpu)
| Partition | Node Count | CPU Model | Number of CPUs per node | GPU Model | Number of GPUs per node | Memory | Max Job Time | Available TMPDIR (quotab) |
|---|---|---|---|---|---|---|---|---|
| lem-gpu-short | 74 | Intel(R) Xeon(R) Platinum 8462Y+ | 64 | NVIDIA H100-94GB | 4 | 996G | 3-00:00:00 |
|
| lem-gpu-normal | 52 | Intel(R) Xeon(R) Platinum 8462Y+ | 64 | NVIDIA H100-94GB | 4 | 996G | 7-00:00:00 |
|
| tesla | 2 | Intel(R) Xeon(R) Gold 6126 | 24 | NVIDIA Tesla P100-16GB | 2 | 117G | 7-00:00:00 |
|
b maximum TMPDIR occupancy: Lustre - no limit on TMPDIR occupancy; SHM - maximum TMPDIR capacity equal to the amount of memory on the node, LOCAL - available TMPDIR space per node given in brackets
* default TMPDIR space for single-node tasks
** default TMPDIR space for multi-node tasks
Startup script
To start an interactive session use the commands:
-sub-interactivefor the bem2-cpu-interactive partition
-sub-interactive-lem-gpufor the lem-gpu-interactive partition
Limitations and requirements
- ability to start only ansruninteractive task (you cannot assign a task usingsbatch)
More limitations for the selected interactive partition can be found in the "Additional limitations" column
| Partition | Node Count | CPU Model | Number of CPUs per node | GPU Model | Number of GPUs per node | Memory | Max Job Time | Available TMPDIR (quotab) | Additional limitations |
|---|---|---|---|---|---|---|---|---|---|
| bem2-cpu-interactive | 2 | Intel(R) Xeon(R) Platinum 8268 | 96 | - | - | 177G | 06:00:00 |
|
|
| lem-gpu-interactive | 1 | Intel(R) Xeon(R) Platinum 8462Y+ | 64 | NVIDIA H100-94GB | 4 | 996G | 06:00:00 |
|
|
b maximum TMPDIR occupancy: Lustre - no limit on TMPDIR occupancy; SHM - maximum TMPDIR capacity equal to the amount of memory on the node, LOCAL - available TMPDIR space per node given in brackets
* default TMPDIR space for single-node tasks
** default TMPDIR space for multi-node tasks
The following partitions are available only to PLGRID infrastructure users
To submit a job in the following partitions, use the-p <PARTITION>option, where<PARTITION>is the name of one of the partitions given in the table below
| Partition | Node Count | CPU Model | Number of CPUs per node | GPU Model | Number of GPUs per node | Memory | Max Job Time | Available TMPDIR (quotab) |
|---|---|---|---|---|---|---|---|---|
| plgrid-short | 32 | Intel(R) Xeon(R) Platinum 8268 | 48 | - | - | 177G | 1-00:00:00 |
|
| plgrid | 32 | Intel(R) Xeon(R) Platinum 8268 | 48 | - | - | 187G | 3-00:00:00 |
|
| plgrid-long | 32 | Intel(R) Xeon(R) Platinum 8268 | 48 | - | - | 177G | 1-00:00:00 |
|
b maximum TMPDIR occupancy: Lustre - no limit on TMPDIR occupancy; SHM - maximum TMPDIR capacity equal to the amount of memory on the node, LOCAL - available TMPDIR space per node given in brackets
* default TMPDIR space for single-node tasks
** default TMPDIR space for multi-node tasks
To obtain information about currently available nodes on individual partitions, use the check-partitions command:
$ check-partitions
PARTITION TIMELIMIT NODES(A/I)
bem2-cpu-short 3-00:00:00 412/53
bem2-cpu-normal 21-00:00:00 308/12
bem2-cpu-interactive 6:00:00 1/1
lem-cpu-short 3-00:00:00 0/171
lem-cpu-normal 21-00:00:00 0/128
lem-cpu-interactive 6:00:00 0/1
lem-gpu-short 3-00:00:00 14/47
lem-gpu-normal 7-00:00:00 13/27
lem-gpu-interactive 6:00:00 0/1
staff-bem2-cpu infinite 1/0
staff-lem-cpu infinite 1/0
staff-lem-gpu infinite 1/0
where:
More details
Information about a specific partition can be obtained using the commandscontrol show partition <PARTITION_NAME>
ATTENTION!
Lack of information about a given partition means that it is not available to the user.