The following table contains information on how to troubleshoot QoS and scheduling errors
| Error / Message | What it usually means | What the customer should check | Support action / information to collect |
|---|---|---|---|
QOSMinGRES | The QoS requires the job to request a minimum amount/type of GRES, usually a GPU. | Check the --gres / --gpus request in the job script. | Ask for the job script and requested GPU type/count. |
QOSMaxGRESPerJob | The job requests more GPUs/GRES than the QoS allows per job. | Reduce the GPU request or use an appropriate QoS. | Check requested vs. QoS GPU limit. |
QOSMaxGRESPerUser | The user has reached the maximum GPU/GRES allocation allowed by the QoS. | Check whether they already have running/allocated GPU jobs. | Check user’s active jobs and QoS limits. |
QOSGrpGRES | The QoS/account has reached its group GPU/GRES limit. | Usually nothing to change in the job script. | Check current group/account GPU usage and limit. |
QOSMaxWallDurationPerJobLimit | Requested wall time exceeds the maximum allowed by the QoS. | Check --time / #SBATCH --time. | Confirm the QoS maximum wall time. |
QOSMaxTRESPU | User has exceeded the QoS limit for a trackable resource (TRES), such as GPUs or CPUs. | Check currently running jobs/resources. | Review user’s resource usage and QoS limits. |
QOSMaxTRESPerJob | This individual job requests more resources than the QoS permits. | Reduce CPU/GPU/memory/time request as appropriate. | Compare request against QoS limits. |
QOSGrpTRES | The account/group has reached its aggregate resource limit. | Usually wait for resources/jobs to finish. | Check account-wide resource consumption. |
QOSJobLimit | User/QoS has reached the maximum number of simultaneously running jobs. | Check running jobs with squeue. | Verify job count and QoS limit. |
QOSSubmitJobLimit | User/QoS has reached the maximum number of submitted jobs. | Cancel unnecessary pending jobs or wait. | Check submitted/pending job count. |
AssocMaxJobsLimit | The user’s account association limits the number of running jobs. | Check currently running jobs. | Check account association limits. |
AssocMaxSubmitJobLimit | The account association limits how many jobs can be submitted. | Reduce/cancel unnecessary pending jobs. | Check association submission limit. |
AssocMaxTRESPU | User has exceeded an account-association resource limit. | Check current resource usage. | Review account association and TRES limits. |
AssocGrpTRES | The account/group has reached its aggregate resource limit. | Wait for resources to become available. | Check account-wide usage and limits. |
Invalid account | The specified account is invalid or unavailable to the user. | Run sacctmgr show assoc user=$USER or use Empire AI’s account guidance. | Verify user’s account association. |
Invalid qos specification | The requested QoS does not exist or isn’t available to the user/account/partition. | Check available QoS options. | Verify account ↔ partition ↔ QoS relationship. |
Invalid partition name | The requested partition doesn’t exist or isn’t accessible. | Check the partition name. | Verify available partitions and user access. |
Invalid generic resource (gres) specification | The requested GPU/GRES type or syntax is invalid. | Check --gres / --gpus syntax and GPU type. | Verify available GRES on the partition. |
PartitionNodeLimit | The partition limits how many nodes the job can request. | Reduce --nodes. | Verify partition node limit. |
PartitionTimeLimit | Requested wall time exceeds the partition’s limit. | Reduce --time. | Verify partition time limit. |
Resources (PENDING) | The job is valid but requested resources aren’t currently available. | Usually wait; check squeue. | Determine whether this is normal capacity contention. |
Priority (PENDING) | Other jobs currently have higher scheduling priority. | Usually wait. | Check fair-share/QoS/priority if unusually delayed. |
Dependency (PENDING) | The job is waiting for another job/event to complete. | Check --dependency. | Verify dependency chain if unexpected. |
Reservation (PENDING) | The job is waiting for a reservation. | Check whether a reservation was requested. | Verify reservation status if unexpected. |
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article