Institutional allocations and job scheduling/preemption

Changes to be made 2/10/24:  (1)The number of nodes in the priority partition reduced from 12 to 6 to provide more access for non-priority users and because most jobs are only using 1 node. (2) Fair-share scheduling enabled on all partitions again for more equitable access.

Each member institution of the Empire AI consortium has an equal allocation of time, but there are many possible ways of accomplishing this.  Since Alpha was loaned as a gift to the consortium we have some additional flexibility and are using it as an opportunity to explore approaches to scheduling for adoption on future systems.  

The current approach is that each week users from one institution have priority access to 6 of the 13 compute nodes (the other 7 nodes remain accessible to all). 

• Jobs from the priority institution will be started on the dedicated  ahead of jobs from other institutions.
• A running job from a non-priority institution will be preempted (cancelled and requeued) if a priority job needs its resources.  This happens after a delay of a few minutes.
• Jobs from non-priority institutions will be started if resources are available and scheduled using a fair-share algorithm.
• If you don't want a preempted job to be requeued, please submit it with the --no-requeue SLURM option.

In addition to ensuring each institution can access its share of the resources, this schedule aims to avoid long wait times for large jobs for the priority institution while also trying to minimize under utilized resources.

The priority institution is selected at 00:00am on Mondays round robin from the following list (alphabetic order) 

Note: Due to an extended outage the week of 04/14/25 (SUNY priority), the following week (starting 4/21/25) will also be a SUNY priority week.