Empire AI Alpha Job Submission and QoS Overview

Created by Cesar Arias, Modified on Fri, 22 May at 2:36 AM by Cesar Arias

ALPHA GPU ENVIRONMENT

Empire AI Alpha — Job Submission and QoS Overview

This guide explains how to submit jobs on Alpha, how Alpha QoS tiers affect cost and scheduling behavior, and which submission pattern to use for interactive work, quick validation, production training, and longer-running jobs.

Related articles

Upcoming partition change: Alpha currently uses institutional partitions. In a future update, that model is expected to transition toward an alpha hardware-tier partition using the QoS model described here. During the transition, both the current GPU partitions and the future alpha partition may be available in parallel.

Mixed-architecture note: Alpha is an x86_64 GPU environment. If part of your workflow also runs on Grace ARM nodes, environments and compiled packages may need to be rebuilt. Use the Alpha and Grace mixed-architecture guidance article when moving software between Alpha and Grace.

Environment overview

Alpha is the general GPU environment for H100/H200-class workflows, interactive GPU debugging, model training, inference, and batch experimentation. The current GPU pool is 192 GPUs across 24 GPU nodes, 8 GPUs per node.

Current vs future pattern

The current submission pattern on Alpha is to use your institutional partition and select the QoS tier that matches the job type. Over time, those institutional partitions are expected to transition toward an alpha hardware-tier partition, but the basic job structure will stay the same.

Workflow	Current pattern	Future pattern
Batch job	`sbatch --partition=<institutional_alpha_partition> ... job.sh`	`sbatch --partition=alpha --qos=<tier> ... job.sh`
Interactive shell with direct launch	`srun --partition=<institutional_alpha_partition> --gres=gpu:1 --time=01:00:00 --pty bash`	`srun --partition=alpha --qos=interactive --gres=gpu:1 --time=01:00:00 --pty bash`
Interactive allocation first	`salloc --partition=<institutional_alpha_partition> --gres=gpu:1 --time=01:00:00`	`salloc --partition=alpha --qos=interactive --gres=gpu:1 --time=01:00:00`

Example syntax note: In command examples, ... stands in for additional optional Slurm flags such as --time, --nodes, --cpus-per-task, --mem, or --requeue. Remove ... and replace it with the extra options you need, or omit it entirely if you do not need any additional flags.

QoS tiers

Every Alpha job will run under a QoS tier. The tier controls scheduling priority, wall-time limit, GPU count limit, and SU billing behavior.

QoS	Best for	Wall time	GPU limit	SU factor
test	Quick validation and script checks	2 hours	8 GPUs, 1 node	0.5x
interactive	Live GPU shell for debugging and setup	2 hours	4 GPUs, 1 session	1.0x
standard	Default production jobs	48 hours	32 GPUs	1.0x
long	Longer cost-sensitive runs	7 days	32 GPUs	0.5x
priority	Deadline-driven urgent work	24 hours	64 GPUs	2.0x
burst	System-assigned overflow mode after allocation exhaustion	7 days	32 GPUs	Free / system-assigned

Every tier has a wall-time limit. No QoS tier gives unlimited runtime. If your training run may exceed the tier wall-time limit, use checkpointing and consider --requeue with a warning signal so you can resume cleanly on a later run.

Interactive methods

On Alpha, interactive work can start with either srun or salloc. Both are valid, but they are useful in slightly different situations.

Command	What it does	Best for
`srun --pty bash`	Requests resources and immediately launches an interactive shell on the compute node	Quick GPU debugging sessions and short interactive checks
`salloc`	Requests an allocation first, then lets you run one or more commands inside that allocation	Workflows where you want more control after the allocation starts, such as launching multiple commands or mixing shell work with explicit `srun` steps

srun and salloc examples

Direct interactive shell with srun

srun --partition=<institutional_alpha_partition> --qos=interactive --gres=gpu:1 --time=00:10:00 --pty bash

Check the interactive shell

nvidia-smi && echo interactive works && exit

Allocate first with salloc

salloc --partition=<institutional_alpha_partition> --qos=interactive --gres=gpu:1 --time=00:10:00

Then launch work inside the allocation

srun --pty bash
nvidia-smi
python smoke_test.py

GPU request patterns

Use --gres=gpu:N when any Alpha GPU is acceptable. If you specifically need H200 GPUs, use the full GRES name for H200.

GPU type	Node range	VRAM	Request pattern
H100	`alphagpu01-alphagpu18`	80 GB	`--gres=gpu:N` or `--gres=gpu:nvidia_h100_80gb_hbm3:N`
H200	`alphagpu19-alphagpu24`	141 GB	`--gres=gpu:nvidia_h200:N`

Example jobs

The examples below illustrate Alpha submission workflows and can be adapted to match your partition, job size, and runtime needs.

Interactive examples

Interactive GPU shell with srun

srun --partition=<institutional_alpha_partition> --qos=interactive --gres=gpu:1 --time=00:10:00 --pty bash

Interactive GPU shell with salloc + srun

salloc --partition=<institutional_alpha_partition> --qos=interactive --gres=gpu:1 --time=00:10:00
srun --pty bash

Interactive smoke test

nvidia-smi && echo interactive works && exit

Production training examples

Single-node training job

sbatch --partition=<institutional_alpha_partition> --qos=standard --requeue --signal=B:SIGTERM@900 --gres=gpu:8 --time=24:00:00 train.sh

Multi-node training job

sbatch --partition=<institutional_alpha_partition> --qos=standard --requeue --signal=B:SIGTERM@900 --nodes=4 --gres=gpu:8 --time=2-00:00:00 train.sh

Long-running cost-sensitive job

sbatch --partition=<institutional_alpha_partition> --qos=long --requeue --signal=B:SIGTERM@900 --gres=gpu:8 --time=5-00:00:00 train.sh

Priority urgent run

sbatch --partition=<institutional_alpha_partition> --qos=priority --requeue --signal=B:SIGTERM@900 --gres=gpu:16 --time=12:00:00 train.sh

H200, arrays, and troubleshooting examples

Explicit H200 request

sbatch --partition=<institutional_alpha_partition> --qos=standard --gres=gpu:nvidia_h200:1 --time=04:00:00 train.sh

H200 test job

sbatch --partition=<institutional_alpha_partition> --qos=test --gres=gpu:nvidia_h200:1 --time=00:30:00 --wrap="nvidia-smi; python smoke_test.py"

Array job

sbatch --partition=<institutional_alpha_partition> --qos=standard --array=1-100 --gres=gpu:1 --time=04:00:00 sweep.sh

Check your associations

sacctmgr show assoc user=$USER format=Partition%15,Account%20,QOS%60

Check completed and running jobs

sacct -u $USER -S today
squeue -u $USER

Check scheduling and fairshare

sprio -j <jobid>
sshare -u $USER -l

Requeue and checkpointing

If a run may exceed the wall-time limit, the recommended pattern is to combine checkpointing in your application with Slurm requeue eligibility and a warning signal before time-limit expiry. This is especially useful for standard, long, priority, and burst workflows.

Flag	What it does
`--requeue`	Makes the batch job eligible to be requeued
`--signal=B:SIGTERM@900`	Sends a warning signal to the batch shell 900 seconds before wall time
`--signal=SIGTERM@900`	Sends the warning signal directly to job steps when that pattern is needed

A useful mental model is that Slurm can restart the batch script, but it does not save model state for you. Your application still needs to save and reload checkpoints.

How to choose

If you need...	Use...	Why
A quick script validation	`test`	Low cost, short wall time, intended for checking that a job works
A live shell on a GPU node	`interactive` with `srun --pty bash` or `salloc` followed by `srun`	Best for debugging and quick interactive work
A normal training or inference job	`standard`	The default production tier for most work
A cheaper but slower-to-start long run	`long`	Lower SU cost with lower scheduling priority
An urgent deadline-driven run	`priority`	Highest scheduling priority at higher SU cost

SU billing

SU = GPUs × Hours × SU Rate. For example, 4 GPUs for 10 hours on standard costs 40 SU, while the same job on priority costs 80 SU and on long costs 20 SU.

Scenario	Calculation	Cost
4 GPUs × 10h on standard	4 × 10 × 1.0	40 SU
8 GPUs × 2h on test	8 × 2 × 0.5	8 SU
8 GPUs × 24h on long	8 × 24 × 0.5	96 SU
8 GPUs × 12h on priority	8 × 12 × 2.0	192 SU

Monitoring

squeue -u $USER shows your currently queued and running jobs.
sacct -u $USER --format=JobID%10,JobName%20,QOS%12,AllocTRES%35,Elapsed,State -S today shows your recent job history from today, including job name, QoS, allocated resources, elapsed time, and state.
sacctmgr show assoc where user=$USER format=Account%20,QOS%60 lists the Slurm accounts and QoS tiers associated with your user.
sshare -u $USER -Ul shows your fairshare usage and priority information.
scancel JOBID cancels a job; replace JOBID with the numeric job ID you want to stop.