SLURM

Why

Slurm (Simple Linux Utility for Resource Management) is a job scheduler. Basically use this if too many people are trying to use a system at the same time and its crashing or you want to distribute resources more fairly.

Job Config

Here are most of the configurable parameters (soon to include NVIDIA MPS)

• nodes number of nodes (if you don't know, just use 1)
• ntasks the number of tasks
• cpu-per-task the number of CPU cores per tasks
• mem-per-cpu the amount of ram per CPU core
• mem the amount of ram (does NOT depend on CPU core count)

Here is an example script

#!/bin/bash
#SBATCH --job-name=my_test
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4          # Request 4 CPU cores
#SBATCH --mem-per-cpu=2G           # Request 2GB of RAM per core (total 8GB)

# Your application command here
./program

Installation/Config

First you need to install the "slurm" packages and dependencies so run

sudo apt update
sudo apt install slurm-wlm slurmd slurmctld munge -y

Then you need to verify that there are MUNGE keys, which there should already be but run the following to check

sudo test -f /etc/munge/munge.key && echo "Key exists" || echo "Key is missing"

For this next part, you are going to need the systems hostname, CPU core count, and total RAM. Now create the file /etc/slurm/slurm.conf and write the following in it (replacing the temps with the mentioned data)

ClusterName=localcluster
SlurmctldHost=<your-hostname>

# Authentication
AuthType=auth/munge
CredType=cred/munge

# Process tracking and resource limits
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity
SelectTypeParameters=CR_Core_Memory

# Paths and logging
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmctldPidFile=/run/slurmctld.pid
SlurmdPidFile=/run/slurmd.pid
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdLogFile=/var/log/slurmd.log

# Node definition
NodeName=<your-hostname> CPUs=<CPU-cores> RealMemory=<Total-RAM> State=UNKNOWN

# Partition definition
PartitionName=debug Nodes=<your-hostname> Default=YES MaxTime=INFINITE State=UP

Also create the file /etc/slurm/cgroup.conf and write

CgroupAutomount=yes
ConstrainCores=yes
ConstrainRAMSpace=yes

The last config step is to create the need directories for slurm's processes (state and spool) and transfer their ownership to slurm. To do that run the following

sudo mkdir -p /var/lib/slurm/slurmd /var/lib/slurm/slurmctld
sudo chown slurm:slurm /var/lib/slurm/slurmd /var/lib/slurm/slurmctld

Now enable the slurm controller/compute deamons

sudo systemctl enable slurmctld slurmd
sudo systemctl start slurmctld slurmd

Basic Testing

To check the install/to make sure it's running

systemctl status slurmctld
systemctl status slurmd
sinfo

If sinfo shows the node as down or drain, you can set it to idle by

sudo scontrol update nodename=localhost state=idle

If sinfo fails to connect or the node is unresponsive, you can check the log by

sudo tail -f /var/log/slurmctld.log