PD-info command;For each Service "Process on a Supercomputer", a dedicated long-term Storage Space for user data, labeled as PD (pl. Przestrzeń Dyskowa), is available.
On Supercomputer, a unique PD directory is assigned to each Service, in which all users of this Serice may store their data and directly share with other users of this Service.
One can access the PD directories by logging to supercomputer, for example using the ssh protocol, or copy the data directly using the scp protocol (host address ui.wcss.pl). Each PD directory has a unique path that can be viewed using the PD-info command.
For example, after finished calculations, user may move the obtained bulky data to the PD directory (possible to include within the sbatch script) and other co-worker could access the data directly.
The PD directories belong to one of the dedicated Lustre file systems. Rules for an efficient and safe usage the Lustre file systems, including the PD storage space, are described at Lustre File system.
The PD directory is not intended for temporary files used during computations!
For such temporary files a dedicated directoryTMPDIR = /lustre/tmp/slurm/$SLURM_JOB_IDis created for the duration of the computational job (see Section Temporary disk space for computations).
PD-info command;/home/$USER;* For Services where no disk space was specified during the application and for all Services started before October 2023 (i.e., before the introduction of the new user and service management system, E-SCIENCE.PL).
Basic information about the available PD directories can be obtained using the PD-info command:"
$ PD-info
----------------------------------------------------------------
1) SERVICE NAME: "xxx"
SERVICE ID: "hpc-xxx-xxxxxxxxx"
QUOTA: 200.0 GB
USED: 69.6 kB
PD DIRECTORY: /lustre/pd01/hpc-xxx-xxxxxxxxx
LUSTRE PROJID: 229
LUSTRE FS: /lustre/pd01
----------------------------------------------------------------
2) SERVICE NAME: "xxx"
SERVICE ID: "hpc-xxx-xxxxxxxxx"
QUOTA: 200.0 GB
USED: 60.4 kB
PD DIRECTORY: /lustre/pd01/hpc-xxx-xxxxxxxxx
LUSTRE PROJID: 24
LUSTRE FS: /lustre/pd01
where
SERVICE NAME - name of the Service;SERVICE ID - a unique identifier of the Service, based on which the PD directory are created;QUOTA - available amount of disk space for the PD directory, given in units kB, MB, GB, and TB;USED - currently used amount of disk space in the PD directory, given in units kB, MB, GB, and TB;PD DIRECTORY - full path to the PD directory;LUSTRE PROJID - a unique project number used exclusively by the Lustre file system, by which the Lustre file system classifies files. Using the given LUSTRE PROJID number, one can, for example, display more information about disk quotas using the command lfs quota -p <LUSTRE PROJID>;Paths to PD directories are based on the Service identifier (assigned at the time of Service creation and are not the same as the Service name), for example hpc-login-xxxxxxxxxxx Users can check the identifiers of their Services on https://users.e-science.pl in the 'Services' tab.
User might belong to multiple HPC Services, each having a separate directory for PD. Therefore, there is no single default path to the PD directory similar to the home directory variable $HOME.
The PD directories are accessible from the Supercomputer on the following machines:
- Access server (
ui.wcss.pl);- Computational nodes - it is possible to directly transfer files between the user's home directory (
$HOME), the temporary directory assigned for a specific computational task ($TMPDIR), and PD;
Current disk usage of the PD directory can be checked using the PD-info command or directly using the lfs quota command dedicated to the Lustre file system:
$ lfs quota -p <LUSTRE PROJID> <FS_PATH>
where
<LUSTRE PROJID> - a unique Lustre project number that can be obtained with the help of PD-info command;<FS_PATH> - path to the mounted Lustre filesystem, to which the selected PD directory belongs to: /lustre/pd01 lub /lustre/pd02;The
lfs quotacommand uses binary prefix bases, i.e., k=2^10, M=2^20, G=2^30, and T=2^40, not decimal ones, i.e., k=10^3, M=10^6, G=10^9 oraz T=10^12.
To facilitate access to the PD directory without the need to provide the absolute path to the PD directory each time, it is possible to create a symbolic link to the PD directory using the ln -s command.
For example:
$ ln -s /lustre/pd01/hpc-xxx-xxxxxxxxxxx ~/storage_space-hpc-xxx
$ ls -l ~/storage_space-hpc-xxx
lrwxrwxrwx 1 teosps kdm 34 Jan 17 11:03 ~/storage_space-hpc-xxx -> /lustre/pd01/hpc-xxx-xxxxxxxxxxx
It will create a symbolic link-directory ~/storage_space-hpc-xxx that will lead directly to the directory /lustre/pd01/hpc-xxx-xxxxxxxxxxx. Using such symbolic links allows for a quick navigatation to the PD directory through cd command or move files to PD using the cp command. Moreover, it is possible to create several separate symbolic links to different PD directories.
Executing the
cdcommand to a directory that is a symbolic link to another directory will perform this operation correctly. However, for the user, it show that they are in the home directory (eg.~/storage_space-hpc-xxx) and not the PD directory at the Lustre filesystem (eg./lustre/pd01/hpc-xxx-xxxxxxxxxxx). This is due to the character of thecdcommand itself, which by default respects logical directory paths (rather than physical ones). To directly go to the physical directory using a symbolic link, use thecd -Pcommand. Thepwdcommand works in a similar way.For example:
$ cd ~/storage_space-hpc-xxx $ pwd /home/login/storage_space-hpc-xxx $ pwd -P /lustre/pd01/hpc-xxx-xxxxxxxxxxx $ cd -P ~/storage_space-hpc-xxx $ pwd /lustre/pd01/hpc-xxx-xxxxxxxxxxx
Amount of the disk space for the PD directory is requested during submission of an application for this Service via https://e-science.pl webpage.
In the event that the allocated amount of disk space for the PD directory proves to be insufficient for the HPC Service's needs, the Service Manager can submit a request to increase computational resources (including additional disk space for the PD directory) through the E-SCIENCE.PL webpage.
Submission of the applications for Services (as well as modifications of the currently available ones) requires an active scientific affiliation being linked to the Service Manager's E-SCIENCE.PL account (instruction on adding scientific affiliation using the https://users.e-science.pl/ portal).
The procedure for submitting a request for resource increase is as follows:
The Service Manager should log in to https://e-science.pl.
Access the user panel by clicking on the button with the name located at the top of the page.
Go to the Application processed ("Wnioski realizowane") tab and in the category Application/Request regarding computing hours ("Wnioski dot. godzin obliczeniowych") select the HPC Service for which the resource increase is to be requested. After selecting the service, basic data regarding previously requested resources will be displayed.
Click the "Zwiększ zasoby" button, which will lead to a new form.
Enter the additional disk space in the "Dodatkowa przestrzeń dyskowa [GB]" cell and make any additional changes if necessary.
Submit the request by clicking the "Wyślij" button.
The Service Manager will be notified of the application's consideration results via email.