Ptolemy Documentation

Ptolemy is an HPE Apollo 6500 Linux Cluster with 8 nodes equipped with 8 NVIDIA A100 GPUs per node. It uses NVIDIA/Mellanox HDR100 InfiniBand interconnect.

The Ptolemy Cluster consists of the following nodes:

Nodes and Specifications

Node Name	Node Type	Cores/Node, CPU Type	Memory/Node, Configuration	GPUs/Node
ptolemy-login-[1-2]	Login	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	512 GB (16x 32GB 2Rx4 PC4-3200AA-R)	N/A
ptolemy-dtn-1	Data Transfer	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	512 GB (16x 32GB 2Rx4 PC4-3200AA-R)	N/A
ptolemy-devel-[1-2]	Development	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	512 GB (16x 32GB 2Rx4 PC4-3200AA-R)	1x NVIDIA A100 (80GB) mig=7
ptolemy-gpu-[01-04]	Compute (A100 GPU)	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	1 TB (16x 64GB DDR-4 Dual Rank 3200MHz)	8x NVIDIA A100 (80GB) mig=1
ptolemy-gpu-[05-06]	Compute (A100 GPU)	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	1 TB (16x 64GB DDR-4 Dual Rank 3200MHz)	8x NVIDIA A100 (80GB) mig=2
ptolemy-gpu-[07-08]	Compute (A100 GPU)	128 cores (2x64 core 2.00GHz AMD EPYC Milan 7713)	1 TB (16x 64GB DDR-4 Dual Rank 3200MHz)	8x NVIDIA A100 (80GB) mig=7

Partitions and Limits

* Default Partition

Partition	TotalNodes	Nodes	MaxNodes (Per Job)	MaxTime	DefMemPerCPU	Allowed Qos	DeviceName
gpu-a100	4	ptolemy-gpu-[01-04]	QoS limited	QoS limited	7686	ALL	a100
gpu-a100-mig2	2	ptolemy-gpu-[05-06]	QoS limited	QoS limited	7686	ALL	a100_3g.40gb
gpu-a100-mig7 *	2	ptolemy-gpu-[07-08]	QoS limited	QoS limited	7686	ALL	a100_1g.10gb
development	2	ptolemy-devel-[1-2]	QoS limited	QoS limited	3827	ALL	NA
service	1	ptolemy-dtn-1	QoS limited	QoS limited	3827	ALL	NA

QoS's and Limits

QoS	Priority	MaxNodes	MaxTime	Notes
normal	20	1	48 Hours	Default QoS, Limits cpu=128 gres/gpu:a100=4 gres/gpu:a100_1g.10gb=14 gres/gpu:a100_3g.40gb=8 mem=1T

The following storage spaces are available with intended purpose.
homes - /home/$USER : home directory, quota, backed up, no scrub
work - /work/$CLUSTER : working storage, quota, not backed up, no scrub
scratch - /scratch/$CLUSTER : scratch storage, no quota, not backed up, scrubbed, see /scratch/$CLUSTER/README
reference - /reference : for reference data set, no quota, backed up, no scrub, by request only, see the README Only the login and dtn nodes can be accessed from outside of the Mississippi State HPC2 Network. They login nodes and DTN nodes can be accessed by connecting via ssh:

ssh <UserID>@Ptolemy-login.arc.msstate.edu 
ssh <UserID>@Ptolemy-dtn.arc.msstate.edu

OpenSSH or PuTTY implementations of ssh are recommended. When you log in, you will be on the a login node. The login node is a shared resource among all users that are currently logged in to the system. Please do NOT run computationally or memory intensive tasks on the login node, this will negatively impact performance for all other users on the system. See the Slurm section for instructions on how to run such tasks on compute nodes. Globus Online can be used to transfer data to and from the Ptolemy cluster. The DTN nodes can also be accessed from outside of the Mississippi State HPC2 network via a single globus endpoint:

MsStateHPC#Ptolemy-dtn

For small amounts of data that need to be transferred to user home directories, the scp command can be used. This command copies files between hosts on a network. It uses ssh for data transfer, and uses the same authentication and provides the same security as ssh. SCP will ask for passwords as well as two-factor authentication codes.

To copy a file from a remote host to local host:

$ scp  <username>@<remotehost>:/path/to/file.txt  /local/directory/

To copy a file from a local host to a remote host:

$ scp  /path/to/file.txt  <username>@<remotehost>:/remote/directory/

To copy a directory from a remote host to local host:

$ scp  -r  <username>@<remotehost>:/remote/directory  /local/directory

To copy a directory from a local host to a remote host:

$ scp  -r  /local/directory  <username>@<remotehost>:/remote/directory

On this cluster, only certain nodes are reachable from the internet. Any software packages, libraries, or datasets needed for jobs or software development can be downloaded on the login nodes, devel nodes, or dtn nodes. The compute nodes of the cluster are on a private network, and they are unreachable from the internet. Ptolemy uses LMOD as an environment module system. For a guide on how to use LMOD to set up the programming environment, please refer to the official LMOD User Guide.

Ptolemy uses a heirarchy based on the Compilers and MPI implementations. Software in the Core tree is built using the default system compilers. Software built against a specific compiler is available only after that compiler module has been loaded. Software built against a specific MPI implementation is available only after that MPI module has been loaded. Information on available modules can be found with the "module avail" and "module spider" commands:

$ module spider quantum-espresso-gpu

------------------------------------------------------------------------------------------------------------------
  quantum-espresso-gpu:
------------------------------------------------------------------------------------------------------------------
     Versions:
        quantum-espresso-gpu/develop

------------------------------------------------------------------------------------------------------------------
  For detailed information about a specific "quantum-espresso-gpu" package (including how to load the modules) use the module's full name.
  Note that names that have a trailing (E) are extensions provided by other modules.
  For example:

     $ module spider quantum-espresso-gpu/develop
------------------------------------------------------------------------------------------------------------------

On Ptolemy, the parallel filesystem is mounted as /work. All projects have their own directory located at /work/<projectname>/, and have quotas that are set by HPC2. This filesystem is considered a 'temporary' or 'scratch' filesystem.

It is important that users submit and run jobs from their respective /work directories instead of their home directories. The /home filesystem is not designed or configured for high performance use, nor does it have much space on it. Home directories will run out of space quickly on parallel jobs and will cause jobs to fail. After useful data is generated from supercompute jobs, it is recommended that users transfer this data to a more long-term storage location.

/reference/ptolemy is the location for reference datasets. It is populated by HPC2 upon request. Ptolemy compute nodes provide up to 5TB of local disk space at /local/scratch that may be used for temporary storage. However, data stored in these locations is not backed up, and is susceptible to data loss due to disk failure or corruption. Each job sets up a unique local space available only with the job script via the environmental $TMPDIR variable:

TMPDIR=/local/scratch/$SLURM_JOB_USER/$SLURM_JOB_ID

You can use this for any scratch disk space you need, or if you plan to compute on an existing large data set (such as a sequence assembly job) it might be beneficial to copy all your input data to this space at the beginning of your job, and then do all your computation on $TMPDIR. You must copy any output data you need to keep back to permanent storage before the job ends, since $TMPDIR will be erased upon job exit. The following example shows how to copy data in, and then run from $TMPDIR:

 #!/bin/bash -l

 #SBATCH --job-name="TMPDIR example"
 #SBATCH --partition=ptolemy
 #SBATCH --account=AccountName
 #SBATCH --nodes=1
 #SBATCH --ntasks=48
 #SBATCH --time=08:00:00

 # Always good practice to reset environment when you start
 module purge

 # start staging data to the job temporary directory in $TMPDIR
 MYDIR=`pwd`
 /bin/cp -r $MYDIR $TMPDIR/
 cd $TMPDIR

 # add regular job commands like module load
 # and commands to launch scientific software

 # copy output data off of local scratch
 /bin/cp -r output $MYDIR/output

$TMPDIR is defined as the above directory at the beginning of every job, before the job scripts are executed. Users must overwrite this definition inside of the batch script itself if TMPDIR needs to be set to a different location. On each login node, we have a utility named Arbiter which regulates activity by monitoring and limiting resource consumption via cgroups. Users are limited to using 4 cpu cores and 50GB of memory at time while their status is "normal". When a user uses more than half of their cap for 10 minutes, they are sent a warning email and penalized by having their usage caps reduced. Each violation of this usage policy results in an occurance which raises the penalty level. A user's occurance level will drop by 1 for every 3 hours that they go without triggering another usage violation. The following table outlines the penalty/status levels that we currently have defined.

Status	CPU Cap	Memory Cap	Penalty Timeout
Normal	4 Cores	50 GB	N/A
Penalty1	3 Cores	40 GB	30 Minutes
Penalty2	2 Cores	25 GB	1 Hour
Penalty3	1 Cores	15 GB	2 Hours
Penalty4	0.2 Cores	5 GB	4 Hours

Certain programs, such as compilers and build utilities are whitelisted. These whitelisted programs will not cause the user to be penalized. Login nodes can still be freely used to build and test software. The purpose of Arbiter is to identify and limit computationally intensive jobs which should be run on the compute nodes instead of the login and development nodes. Ptolemy uses the Slurm Workload Manager as a scheduler and resource manager. For a guide on how to use the Slurm system to submit and run jobs on this cluster, please refer to the official Slurm Quickstart Guide.

Slurm has three primary job allocation commands which accept almost identical options:

• SBATCH Submits a job runscript for later execution (batch mode)
• SALLOC Creates a job allocation and starts a shell to use it (interactive mode)
• SRUN Creates a job allocation and launches the job step (typically an MPI job)

The salloc command is configured to have the default functionality on Ptolemy. The salloc command allocates resources for the job, but spawns a shell on the login node with various Slurm environment variables set. Job steps can be launched from the salloc shell with the srun command.

Example salloc usage:

 user_name@Ptolemy-login-1 ~$ salloc -A account_name
 salloc: Pending job allocation 527990
 salloc: job 527990 queued and waiting for resources
 salloc: Granted job allocation 527990
 salloc: Waiting for resource configuration
 salloc: Nodes Ptolemy-gpu-02 are ready for job

 user_name@Ptolemy-login-1 ~$ srun hostname
 
 Ptolemy-gpu-02.arc.MsState.Edu

The srun command can be used to launch an interactive shell on an allocated node or set of nodes. Simply specify the --pty option while launching a shell (such as bash) with srun. It is also recommended to set the wallclock limit along with the number of nodes and processors needed for the interactive shell.

Example interactive shell:

 user_name@Ptolemy-login-1 ~$ srun -A account_name --pty --preserve-env bash
 srun: job 527987 queued and waiting for resources
 srun: job 527987 has been allocated resources

 user_name@Ptolemy-gpu-02 ~$  hostname
 
 Ptolemy-gpu-02.arc.MsState.Edu

When running batch jobs, it is necessary to interact with the job queue. It is usually helpful to be able to see information about the system, the queue, the nodes, and your job. This can be accomplished a set of important commands:

• SQUEUE Displays information about jobs in the scheduling queue.
• SJSTAT Displays short summary of running jobs and scheduling pool data.
• SHOWUSERJOBS Displays short summary of jobs by user and account, along with a summary of node state.
• SHOWPARTITIONS Displays short summary and current state of the available partitions.
• SSTAT Displays information about specific jobs.
• SINFO Reports system status (nodes, queues, etc).
• SACCT Displays accounting information from the Slurm database.

Each of these commands has a variety of functions, options, and filters that refine the information returned and displayed. Users can customize filtering, sorting, and output format using command line options or environment variables. Please consult the man page of each command or the Slurm Documentation for more information on using these commands.

The default walltime is 15 minutes. Any jobs that do not specify a walltime will be terminated 15 minutes after starting.
The default allocation is 1 node. Any jobs that do not specify the number of nodes will run on one node.
The default number of tasks is 1 core. Any jobs that do not specify the number of tasks will run on only 1 core.

To see which accounts you are on, along with valid QoS's for that account, use the following command:

 $ sacctmgr show associations where user=$USER format=account%20,qos%50

Currently, the ptolemy partition is not set to assign users exclusive nodes by default. Users will only get the amount of cores specified per node and will leave the rest of the cores on the nodes unallocated and available for other users' jobs.

Slurm allocates all of a node's memory by default, so in order to take advantage of nodesharing, users must specify the memory required per node for their jobs using the --mem option in their runscript or srun command. Specifying a memory limit with the --mem option will ensure that user jobs are allocated the amount specified. For example, if a user's job only needs 150 GB of memory per node, the user must specify the following sbatch directive:

 $ srun -n 10 -N 2 --mem=150G ./example_program

If a user requests 10 cores and 50 GB of memory for one job, along with 10 cores and 50 GB of memory for a second job, then both of these jobs may run on the same node. The same principle would also work for jobs owned by two different users.

In order to disallow sharing the remainder of the cores while running on less than 48 cores, users must specify the "--exclusive" option in their runscripts or in their salloc/srun commands:

 $ srun -n 10 -N 1 --exclusive ./example_program

The gpu and bigmem partitions will give users exclusive nodes by default. Job dependencies are used to defer the start of a job until the specified dependencies have been satisfied. They are specified with the --dependency option to sbatch or swarm in the format:

 sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> ...

Dependency types:

after:jobid[:jobid...]	job can begin after the specified jobs have started
afterany:jobid[:jobid...]	job can begin after the specified jobs have terminated
afternotok:jobid[:jobid...]	job can begin after the specified jobs have failed
afterok:jobid[:jobid...]	job can begin after the specified jobs have run to completion with an exit code of zero
singleton	jobs can begin execution after all previously launched jobs with the same name and user have ended

Job dependencies are useful in setting up job pipelines. If a particular job needs a dataset downloaded before it can run, this must be submit as two jobs: the first job downloads the dataset on the DTN nodes in the service partition and the second job operates on the dataset retrieved by the first job. To set up pipelines using job dependencies the most useful types are afterany, afterok and singleton. The simplest way is to use the afterok dependency for single consecutive jobs. For example:

 $ sbatch job1.sh
11254323

 $ sbatch --dependency=afterok:11254323 job2.sh

Now when job1 ends with an exit code of zero, job2 will become eligible for scheduling. However, if job1 fails (ends with a non-zero exit code), job2 will not be scheduled but will remain in the queue and needs to be canceled manually. As an alternative, the afterany dependency can be used and checking for successful execution of the prerequisites can be done in the jobscript itself. Containers are a portable method for running software on separate machines in a reproducible manner. Ptolemy utilizes apptainer. The packages can be loaded into a user's environment with one of the following commands:

module load apptainer/1.0.2

Apptainer is configured on Ptolemy such that users do not have to define additional environment variables to have access to their working folders in the container. However, users wishing to utilize the "remote build" features will need to unset the APPTAINER_BIND variable.

Many containers are available on Ptolemy, and inquiries about accessing existing containers or adding new containers may be submitted by emailing help@hpc.msstate.edu. Python is ubiquitous in computing. It is extremely configurable, and as such it requires some planning to utilize appropriately. The recommended method for interacting with Python or Conda is through environments.

Create an Environment in Python:

python3 -m venv $MYDIR/python-env
source $MYDIR/python-env/bin/activate
pip3 install --upgrade pip
pip3 install matplotlib scikit-learn torch keras tensorflow

Accessing the Environment:

source /reference/ptolemy/class/me8213/python-env/bin/activate

Creating an Environment in Conda:

module purge
module load miniconda3/24.3.0
export CONDA_PKGS_DIRS=/tmp/$USER
mkdir $CONDA_PKGS_DIRS
conda create --prefix $MYDIR/conda-environments/env-name numpy pandas matplotlib seaborn scikit-learn ipykernel
source activate $MYDIR/conda-environments/env-name
python -m ipykernel install --name env-name --display-name "env-name" --user
conda deactivate

Accessing the Environment:

source /reference/ptolemy/class/me8213/python-env/bin/activate

OnDemand is a user-friendly front-end interface for access to the Ptolemy Cluster resources. Review the Ptolemy Open On Demand documentation for more information.