01 - NVIDIA Kickstart workshop: PyTorch + Slurm

This workshop trains a small convolutional network on generated 32x32 images. No dataset download is needed. Training runs in a PyTorch Apptainer image on an Alpha H100 1g.10gb MIG slice, and writes logs, metrics, and a checkpoint to your own copy of the workshop.

0. Access the cluster

To access alpha you are expected to have set up your account, once set up you can ssh to alpha:

ssh <YOUR_USER_NAME>@alpha1.empireai.edu

1. Copy the workshop to your home directory

Run this once on the Alpha login node. Each participant should work from their own copy rather than the shared workshop folder.

cd "$HOME"
cp -r /projects/workshops/micalves_workshop/empire-ai-pytorch-workshop .
cd "$HOME/empire-ai-pytorch-workshop"
mkdir -p logs runs

logs/ must exist before running sbatch: Slurm creates the standard-output and error files before the job script begins.

2. Use the validated shared image

The tested workshop image is stored on project storage:

/projects/workshops/micalves_workshop/empire-ai-pytorch-workshop
/containers/pytorch-2.4.1-cuda12.4-runtime.sif

Participants need a personal copy of code, logs, and results, the container is also included in each users home directory.

The SIF contains Python, PyTorch, CUDA userspace libraries, and dependencies. The GPU driver remains on the compute node. The apptainer exec --nv command exposes the assigned NVIDIA GPU and compatible driver libraries inside the container.

3. Build the PyTorch SIF image (DO NOT RUN THIS STEP)

Note: Ideally, when generating an image, we recommend users to utilize an interactive node.

Build only when you need a new image. The temporary build directory must be on node-local /tmp, avoiding extended-attribute limitations of home/project storage.

cd "$HOME/empire-ai-pytorch-workshop"
module load apptainer/1.1.9

unset SINGULARITY_TMPDIR SINGULARITY_CACHEDIR
export APPTAINER_CACHEDIR="$HOME/.apptainer/cache"
export APPTAINER_TMPDIR="/tmp/$USER/apptainer-build"
mkdir -p "$APPTAINER_CACHEDIR" "$APPTAINER_TMPDIR" containers

apptainer pull containers/pytorch-2.4.1-cuda12.4-runtime.sif \
  docker://pytorch/pytorch:2.4.1-cuda12.4-cudnn9-runtime

Why these variables matter:

• APPTAINER_CACHEDIR keeps downloaded registry layers in home storage so later pulls can reuse them.
• APPTAINER_TMPDIR must use local /tmp. Apptainer expands and packages the image there; home and project filesystems can reject the extended-attribute operations used during unpacking.
• The final SIF remains in containers/, where it can be reused by later jobs.

The large PyTorch image may stay quiet at Creating SIF file... for 10–20 minutes while it is compressed. Let it complete. Verify the resulting file:

ls -lh containers/pytorch-2.4.1-cuda12.4-runtime.sif

4. Submit the GPU training job

Before submitting the full training job, run this one-command smoke test. It uses the same account, reservation, Alpha node, MIG GPU profile, Apptainer module, image, and NVIDIA GPU passthrough as the job script. It proves that Slurm can grant the requested GPU and that PyTorch sees CUDA inside the container.

srun -p alpha -A xx_micalves_workshop \
  --reservation=nvidia_workshop \
  --nodelist=alphagpu[11,16] \
  --nodes=1 \
  --gres=gpu:1g.10gb:1 \
  --pty bash -lc '
    module load apptainer/1.1.9
    export APPTAINER_TMPDIR="/tmp/$USER/apptainer-${SLURM_JOB_ID}"
    mkdir -p "$APPTAINER_TMPDIR"
    apptainer exec --nv \
      $HOME/empire-ai-pytorch-workshop/containers/pytorch-2.4.1-cuda12.4-runtime.sif \
      python3 -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"
  '

Expected result: a PyTorch version such as 2.4.1+cu124, then True, then NVIDIA H100 80GB HBM3 MIG 1g.10gb.

The workshop job is slurm/job_gpu.sbatch. It is configured for the Alpha workshop allocation:

#SBATCH --partition=alpha
#SBATCH --account=xx_micalves_workshop
#SBATCH --qos=priority
#SBATCH --reservation=nvidia_workshop
#SBATCH --nodelist=alphagpu[11,16]
#SBATCH --nodes=1
#SBATCH --gres=gpu:1g.10gb:1

Submit from the workshop root:

cd "$HOME/empire-ai-pytorch-workshop"
sbatch slurm/job_gpu.sbatch

sbatch prints a job ID, for example Submitted batch job 12345. Record this ID because it identifies the queue entry, logs, metrics, checkpoint, and accounting record.

The job requests one node, one task, four CPUs, 16 GB RAM, 15 minutes, and one H100 gpu:1g.10gb MIG slice. The reservation, account, QoS, node, and MIG request match the validated workshop test. Use xx_micalves_workshop with priority, not the separate cuny account.

Inside the job, the script loads apptainer/1.1.9, creates a compute-node-local temporary directory, opens the shared SIF with --nv, binds your home-directory workshop copy to /workspace, trains the model, and writes a separate run directory.

5. Monitor, inspect, and cancel jobs

# Your queued and running jobs
squeue -u "$USER"

# One job, including its queue state/reason
squeue -j 12345
scontrol show job 12345

# Follow standard output and error while the job runs
tail -f logs/shapes-pytorch-12345.out
tail -f logs/shapes-pytorch-12345.err

# Accounting summary after completion
sacct -j 12345 --format=JobID,JobName,State,Elapsed,ExitCode,AllocTRES

# Cancel a queued or running job
scancel 12345

Common states: PD means pending (use scontrol show job for the reason), R means running, and CD means completed. F or a nonzero ExitCode in sacct means the job failed; start with its .err file, then its .out file.

6. Find results

For job ID 12345, the job creates:

logs/shapes-pytorch-12345.out     # Slurm standard output; GPU and training details
logs/shapes-pytorch-12345.err     # Slurm standard error
runs/shapes-12345/metrics.jsonl   # per-epoch loss and accuracy
runs/shapes-12345/best.pt         # best validation checkpoint

Summarize a completed run again at any time:

python3 $HOME/empire-ai-pytorch-workshop/tools/summarize_run.py runs/shapes-12345

7. Interactive GPU session

Use an interactive allocation when you want to test a command before submitting a batch job:

salloc --job-name=shapes-interactive \
  --partition=alpha --account=xx_micalves_workshop --qos=priority \
  --reservation=nvidia_workshop \
  --nodelist=alphagpu[11,16] \
  --nodes=1 --ntasks=1 --cpus-per-task=4 --gres=gpu:1g.10gb:1 --mem=16G --time=00:15:00

# Run this command once you get allocated into an interactive job
srun --pty bash -l
cd "$HOME/empire-ai-pytorch-workshop"
module load apptainer/1.1.9
export APPTAINER_TMPDIR="/tmp/$USER/apptainer-${SLURM_JOB_ID}"
mkdir -p "$APPTAINER_TMPDIR"
apptainer exec --nv --bind "$PWD:/workspace" \
  $HOME/empire-ai-pytorch-workshop/containers/pytorch-2.4.1-cuda12.4-runtime.sif \
  python3 -c "import torch; print(torch.__version__); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))"

# Leave the node, then release the allocation.
exit
scancel "$SLURM_JOB_ID"

8. Workshop Wrap-Up

By the end of this workshop, you should be able to finish a training run and gather the evidence needed to reproduce it. Take a moment to answer the following:

1. What job ID did sbatch return?
2. What status did the job show in squeue before it started running?
3. Which GPU name appears in the standard-output log?
4. Where can you find metrics.jsonl and best.pt for your run?
5. If a subsequent job failed, what would be your first point of check?

9. Other helpful information

Expected Apptainer messages

On this cluster, the following messages are normal for a SIF image and can be ignored when the command continues and produces the expected PyTorch output:

INFO: squashfuse not found, will not be able to mount SIF
INFO: fuse2fs not found, will not be able to mount EXT3 filesystems
INFO: Converting SIF file to temporary sandbox...
INFO: underlay of /etc/localtime required more than 50 ... bind mounts
INFO: underlay of /usr/bin/nvidia-smi required more than 50 ... bind mounts

Apptainer falls back to converting the read-only SIF to a temporary sandbox because the optional FUSE helpers are unavailable. The underlay notices describe the host files that are being made visible in the container. They are informational; the successful smoke test prints the PyTorch version, True for CUDA, and the H100 MIG device after them.

Errors that need action

Slurm options used by this workshop

Only change these after checking the cluster allocation rules. More CPU, GPU, memory, or wall time can affect scheduling and account limits.

Files

• src/train_shapes.py: PyTorch training application.
• slurm/job_gpu.sbatch: the single supported GPU training job.
• tools/summarize_run.py: reads and summarizes metrics.
• containers/: location for your built SIF image.

Cluster documentation: https://empireai.freshdesk.com/support/solutions