Training

A training run is launched as a lab job (PyTorch or TensorFlow), not inside the notebook. That way it runs without you keeping the browser open, it is recorded and it can be queued.

The GPU rule

The GPU in your size is a shared budget. If your notebook has it reserved, the training job waits. The usual sequence:

1. Save your script and data on the notebook volume.
2. Stop the notebook to free the GPU.
3. Launch the training job pointing to the same volume.
4. When it finishes, resume the notebook: the model and the metrics are in your folder.

A minimal PyTorch job

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: entrena-fugas
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
              command: ["python", "/workspace/train.py"]
              resources:
                limits: { nvidia.com/gpu: 1 }
              volumeMounts:
                - { name: ws, mountPath: /workspace }
          volumes:
            - name: ws
              persistentVolumeClaim: { claimName: workspace-mi-notebook }

Apply the file from the notebook or from the lab panel. The volume name is your notebook's (it appears in Volumes).

Track progress

In the lab panel, Training lists the jobs with status and logs. If the job stays waiting, see Queues and Sizes and quota.

For advanced options (several workers, checkpoints), see the Kubeflow Trainer documentation.

Next step

• Experiments