00 - NVIDIA Kickstart workshop: Empire AI Alpha Slurm Cheat Sheet

Run these commands from the Alpha login node. Replace 12345 with your actual job ID.

Workshop-tested resources

Partition:    alpha
Account:      xx_micalves_workshop
QoS:          priority
Reservation:  nvidia_workshop
Node:         alphagpu[11,16]
MIG GPU:      gpu:1g.10gb:1

These values are workshop-specific. For other work, use the allocation details from the cluster administrators.

The three Slurm commands

Access the cluster

To access alpha you are expected to have set up your account, once set up you can ssh to alpha:

ssh <YOUR_USER_NAME>@alpha1.empireai.edu

Copy the workshop to your home directory

Run this once on the Alpha login node. Each participant should work from their own copy rather than the shared workshop folder.

cd "$HOME"
cp -r /projects/workshops/micalves_workshop/empire-ai-pytorch-workshop .
cd "$HOME/empire-ai-pytorch-workshop"
mkdir -p logs runs

logs/ must exist before running sbatch: Slurm creates the standard-output and error files before the job script begins.

Quick preflight

sacctmgr show assoc where user="$USER" format=User,Account%30,Partition,QOS%60
squeue -u "$USER"

# Confirm a workshop MIG GPU can be allocated.
srun -p alpha -A xx_micalves_workshop \
  --reservation=nvidia_workshop --nodelist=alphagpu[11,16] --nodes=1 \
  --gres=gpu:1g.10gb:1 nvidia-smi -L

Find files and text on Alpha

Use the standard tools installed on the cluster:

# Find filenames below the current directory.
find . -type f -name '*.sbatch'

# Search text recursively. -n prints line numbers; -I skips binary files.
grep -RInI 'apptainer' .

# Search one log file while it is being written.
grep -n 'device=cuda' logs/shapes-pytorch-*.out

Batch job with sbatch

Create a file named my_job.sbatch:

#!/bin/bash
#SBATCH --job-name=my-job
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err
#SBATCH --partition=alpha
#SBATCH --account=xx_micalves_workshop
#SBATCH --qos=priority
#SBATCH --reservation=nvidia_workshop
#SBATCH --nodelist=alphagpu[11,16]
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:1g.10gb:1
#SBATCH --mem=16G
#SBATCH --time=00:15:00

set -euo pipefail
hostname
nvidia-smi
python3 my_program.py

mkdir -p logs                    # Required before sbatch; Slurm opens logs first.
sbatch my_job.sbatch
squeue -u "$USER"
squeue -j 12345
scontrol show job 12345
tail -f logs/my-job-12345.out
tail -f logs/my-job-12345.err
sacct -j 12345 --format=JobID,JobName,State,Elapsed,ExitCode,AllocTRES
scancel 12345

PD means pending, R means running, CD means completed, and F means failed. For a pending reason, use scontrol show job 12345.

Interactive allocation with salloc

salloc --job-name=interactive \
  --partition=alpha --account=xx_micalves_workshop --qos=priority \
  --reservation=nvidia_workshop --nodelist=alphagpu[11,16] \
  --nodes=1 --ntasks=1 --cpus-per-task=4 \
  --gres=gpu:1g.10gb:1 --mem=16G --time=00:15:00

# After allocation is granted, enter the compute node.
srun --pty bash -l
nvidia-smi
python3 my_program.py

# Exit the compute node, then release the allocation.
exit
scancel "$SLURM_JOB_ID"

Do not run GPU work on the login node. salloc only reserves resources; srun is what enters the allocated compute node.

One-command interactive srun

# Open a shell directly on the requested GPU.
srun -p alpha -A xx_micalves_workshop \
  --reservation=nvidia_workshop --nodelist=alphagpu[11,16] --nodes=1 \
  --gres=gpu:1g.10gb:1 --pty bash -l

# Or run a single test command directly.
srun -p alpha -A xx_micalves_workshop \
  --reservation=nvidia_workshop --nodelist=alphagpu[11,16] --nodes=1 \
  --gres=gpu:1g.10gb:1 nvidia-smi -L

GPU requests: MIG versus full GPU

MIG is an isolated partition of a physical GPU. It looks like one CUDA device, but has less memory and compute than a full H100. Reduce batch size if necessary. Do not combine a full-GPU request with a MIG request unless the administrators explicitly document that configuration.

nvidia-smi -L
echo "$CUDA_VISIBLE_DEVICES"
python3 -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

Run a container with Apptainer

The workshop image:

/projects/workshops/micalves_workshop/empire-ai-pytorch-workshop
/containers/pytorch-2.4.1-cuda12.4-runtime.sif

The --nv flag exposes the allocated NVIDIA GPU and compatible host driver libraries inside the container. The temporary directory is needed on this cluster when optional FUSE mounting helpers are unavailable.

srun -p alpha -A xx_micalves_workshop \
  --reservation=nvidia_workshop --nodelist=alphagpu11 \
  --gres=gpu:1g.10gb:1 --pty bash -lc '
    module load apptainer/1.1.9
    export APPTAINER_TMPDIR="/tmp/$USER/apptainer-$SLURM_JOB_ID"
    mkdir -p "$APPTAINER_TMPDIR"
    apptainer exec --nv \
      $HOME/empire-ai-pytorch-workshop/containers/pytorch-2.4.1-cuda12.4-runtime.sif \
      python3 -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
  '

In a batch script:

module load apptainer/1.1.9
export APPTAINER_TMPDIR="/tmp/$USER/apptainer-$SLURM_JOB_ID"
mkdir -p "$APPTAINER_TMPDIR"

apptainer exec --nv --bind "$PWD:/workspace" \
  $HOME/empire-ai-pytorch-workshop/containers/pytorch-2.4.1-cuda12.4-runtime.sif \
  bash -lc 'cd /workspace && python3 src/train_shapes.py'

The bind mount makes the host directory visible as /workspace, so checkpoints and logs remain after the container exits.

Run without a container

# Use only when the node provides the needed software/environment.
module load YOUR_PYTHON_OR_CUDA_MODULE
python3 my_program.py

If PyTorch imports on the login node but not inside a job, you are using different Python environments. Use the Apptainer image, or set up an environment that is accessible from the compute node.

CPU-only job

Remove the GPU request:

#!/bin/bash
#SBATCH --job-name=cpu-test
#SBATCH --output=logs/%x-%j.out
#SBATCH --error=logs/%x-%j.err
#SBATCH --partition=alpha
#SBATCH --account=xx_micalves_workshop
#SBATCH --qos=priority
#SBATCH --reservation=nvidia_workshop
#SBATCH --nodelist=alphagpu[11,16]
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --time=00:05:00

python3 cpu_program.py

Use a GPU reservation for CPU-only work only when the workshop administrator intends it.

Useful options

Command-line options override matching SBATCH directives:

sbatch --time=00:30:00 --job-name=longer-test my_job.sbatch

Useful environment variables

echo "$SLURM_JOB_ID"
echo "$SLURM_JOB_NODELIST"
echo "$SLURM_CPUS_PER_TASK"
echo "$CUDA_VISIBLE_DEVICES"
scontrol show hostnames "$SLURM_JOB_NODELIST"

Common messages

Workshop fast path

cd "$HOME"
cp -a /projects/workshops/micalves_workshop/empire-ai-pytorch-workshop .
cd "$HOME/empire-ai-pytorch-workshop"
mkdir -p logs runs
sbatch slurm/job_gpu.sbatch
squeue -u "$USER"

See README.md for the complete PyTorch training walkthrough. Cluster documentation: https://empireai.freshdesk.com/support/solutions