Skip to main content
ENES
Bring a workflow →

Queues and priorities

When several jobs request more GPU than is available, the platform queues them and admits them in order. A queued job does not fail: it waits until resources are available.

How to know why it is waiting​

In the lab panel, the job shows a pending status with a condition that indicates the reason: "waiting for quota" (there is no free GPU or memory in the size) or "not admitted" (it requests something outside the budget). In Studio, the lab card shows the rejection if it is due to quota.

Send a job to the queue​

Add the queue label to the job. Your administrator tells you your lab's queue; the usual name is cola-investigacion:

metadata:
  labels:
    kueue.x-k8s.io/queue-name: cola-investigacion

Priority​

An urgent job can carry a higher priority class. If your installation's policy allows it, a high-priority job preempts a normal-priority one, which goes back to the queue and resumes later. Use priorities with judgment: a preempted job loses any progress not saved in a checkpoint.

metadata:
  labels:
    kueue.x-k8s.io/queue-name: cola-investigacion
    kueue.x-k8s.io/priority-class: prioridad-critica

Recommendations​

  • Save checkpoints to the lab volume every few epochs.
  • Do not request more GPU than the job uses: a job that requests 2 and uses 1 blocks another one.
  • If a job has been in the queue longer than expected, check whether your own notebook has the GPU.

Reference in the Kueue documentation.

Next step​

Last updated on