Empire AI Beta — Job Submission and QoS Overview

Created by Izabel Cavassim, Modified on Wed, 19 Aug at 12:56 PM by Izabel Cavassim

Beta GPU ENVIRONMENT

Empire AI Beta — Job Submission and QoS Overview

This guide explains how to submit jobs on Beta, how Beta QoS tiers affect cost and scheduling behavior, and which submission pattern to use for interactive work, quick validation, production training, and longer-running jobs.

Related articles

No institutional partition in beta: Differently than Alpha, Beta does not use institutional partitions.

Environment overview

Beta serves as the primary GPU environment for B200-class workflows, including model training, large language model development, and other AI applications. The present GPU pool consists of 4 racks, each rack housing 72 nodes equipped with 4 GPUs per node, amounting to a total of 288 GPUs.

Find User Accounts & QoS tiers

Each user is associated with one or more accounts, which are established based on the projects they are part of. QoS tiers are assigned based on the user’s account and institutional affiliation. When submitting jobs, users must specify the account to be charged for the job. Run the following command on the login node to view your available accounts, partitions, and QoS tiers.

sacctmgr show assoc user=$USER format=Account%25,Partition%20,QOS%60

Throughout this document, "YOUR_ACCOUNT" refers to the specific account associated with your user.

QoS pattern

The current submission pattern on Beta is to use "beta" partition followed by account (Project's code, check above) and select the QoS tier that matches the job type.

Workflow	Pattern
Batch job	`sbatch --qos=<tier> ... job.sh`
Interactive shell with direct launch	`srun --partition=beta --account=YOUR_ACCOUNT --qos=interactive --gres=gpu:1 --time=01:00:00 --pty bash`
Interactive allocation first	`salloc --partition=beta --qos=interactive --gres=gpu:1 --time=01:00:00`

Example syntax note: In command examples, ... stands in for additional optional Slurm flags such as --time, --nodes, --cpus-per-task, --mem, or --requeue. Remove ... and replace it with the extra options you need, or omit it entirely if you do not need any additional flags.

QoS tiers

Every Beta job will run under a QoS tier. The tier controls scheduling priority, wall-time limit, GPU count limit, and SU billing behavior. Note that differently than Alpha, the base rate for Beta is 2 SU per GPU-hour (see link for more details).

QoS	Best for	Wall time	GPU limit	SU factor
test	Quick validation and script checks	2 hours	4 GPUs, 1 node	1.0x
interactive	Live GPU shell for debugging and setup	2 hours	4 GPUs, 1 session	2.0x
standard	Default production jobs	48 hours	36 GPUs	2.0x
long	Longer cost-sensitive runs	7 days	36 GPUs	1.0x
priority	Deadline-driven urgent work	24 hours	72 GPUs	4.0x
burst	System-assigned overflow mode after allocation exhaustion	7 days	32 GPUs	Free / system-assigned

Every tier has a wall-time limit. No QoS tier gives unlimited runtime. If your training run may exceed the tier wall-time limit, use checkpointing and consider --requeue with a warning signal so you can resume cleanly on a later run.

Interactive methods

On Beta, interactive work can start with either srun or salloc. Both are valid, but they are useful in slightly different situations.

Command	What it does	Best for
`srun --pty bash`	Requests resources and immediately launches an interactive shell on the compute node	Quick GPU debugging sessions and short interactive checks
`salloc`	Requests an allocation first, then lets you run one or more commands inside that allocation	Workflows where you want more control after the allocation starts, such as launching multiple commands or mixing shell work with explicit `srun` steps

srun and salloc examples

Direct interactive shell with srun

srun --partition=beta --qos=interactive --gres=gpu:1 --time=00:10:00 --pty bash

Check the interactive shell

nvidia-smi && echo interactive works && exit

Allocate first with salloc

salloc --partition=beta --qos=interactive --gres=gpu:1 --time=00:10:00

Then launch work inside the allocation

srun --pty bash
nvidia-smi
python smoke_test.py

GPU request patterns

Given that Beta has only one node type, use --gres=gpu:N when requesting for GPUs

GPU type	VRAM	Request pattern
B200	80 GB	`--gres=gpu:N`

Example jobs

The examples below illustrate Beta submission workflows and can be adapted to match your partition, job size, and runtime needs.

Interactive examples

Interactive GPU shell with srun

srun --partition=beta --account=YOUR_ACCOUNT --qos=interactive --gres=gpu:1 --time=00:10:00 --pty bash
nvidia-smi; hostname; exit

Interactive GPU shell with salloc + srun

salloc --partition=beta --account=YOUR_ACOUNT --qos=interactive --gres=gpu:1 --time=00:10:00
srun --pty bash

Interactive smoke test

nvidia-smi && echo interactive works && exit

Production training examples

Single-node training job

sbatch --partition=beta --account=YOUR_ACCOUNT --qos=standard --requeue --signal=B:SIGTERM@900 --gres=gpu:8 --time=24:00:00 train.sh

Multi-node training job

sbatch --partition=beta --qos=standard --requeue --signal=B:SIGTERM@900 --nodes=4 --gres=gpu:8 --time=2-00:00:00 train.sh

Long-running cost-sensitive job

sbatch --partition=beta --qos=long --requeue --signal=B:SIGTERM@900 --gres=gpu:8 --time=5-00:00:00 train.sh

Priority urgent run

sbatch --partition=beta --qos=priority --requeue --signal=B:SIGTERM@900 --gres=gpu:16 --time=12:00:00 train.sh

troubleshooting examples

B200 test job

sbatch --partition=beta --qos=test --gres=gpu:1 --time=00:30:00 --wrap="nvidia-smi; python smoke_test.py"

Array job

sbatch --partition=beta --qos=standard --array=1-100 --gres=gpu:1 --time=04:00:00 sweep.sh

Check your associations

sacctmgr show assoc user=$USER format=Partition%15,Account%20,QOS%60

Check completed and running jobs

sacct -u $USER -S today
squeue -u $USER

Check scheduling and fairshare

sprio -j <jobid>
sshare -u $USER -l

Requeue and checkpointing

If a run may exceed the wall-time limit, the recommended pattern is to combine checkpointing in your application with Slurm requeue eligibility and a warning signal before time-limit expiry. This is especially useful for standard, long, priority, and burst workflows.

Flag	What it does
`--requeue`	Makes the batch job eligible to be requeued
`--signal=B:SIGTERM@900`	Sends a warning signal to the batch shell 900 seconds before wall time
`--signal=SIGTERM@900`	Sends the warning signal directly to job steps when that pattern is needed

A useful mental model is that Slurm can restart the batch script, but it does not save model state for you. Your application still needs to save and reload checkpoints.

How to choose

If you need...	Use...	Why
A quick script validation	`test`	Low cost, short wall time, intended for checking that a job works
A live shell on a GPU node	`interactive` with `srun --pty bash` or `salloc` followed by `srun`	Best for debugging and quick interactive work
A normal training or inference job	`standard`	The default production tier for most work
A cheaper but slower-to-start long run	`long`	Lower SU cost with lower scheduling priority
An urgent deadline-driven run	`priority`	Highest scheduling priority at higher SU cost

SU billing

SU = GPUs × Hours × SU . For example, 4 GPUs for 10 hours on standard costs 80 SU, while the same job on priority costs 160 SU and on long costs 40 SU.

Scenario	Calculation	Cost
4 GPUs × 10h on standard	4 × 10 × 2.0	80 SU
8 GPUs × 2h on test	8 × 2 × 1.0	16 SU
8 GPUs × 24h on long	8 × 24 × 1.0	192 SU
8 GPUs × 12h on priority	8 × 12 × 4.0	384 SU

Monitoring

squeue -u $USER shows your currently queued and running jobs.
sacct -u $USER --format=JobID%10,JobName%20,QOS%12,AllocTRES%35,Elapsed,State -S today shows your recent job history from today, including job name, QoS, allocated resources, elapsed time, and state.
sacctmgr show assoc where user=$USER format=Account%20,QOS%60 lists the Slurm accounts and QoS tiers associated with your user.
sshare -u $USER -Ul shows your fairshare usage and priority information.
scancel JOBID cancels a job; replace JOBID with the numeric job ID you want to stop.