Skip to content

NVIDIA GPU on Talos

GPU support has two ownership layers. Talos supplies the signed kernel modules and container toolkit in its immutable image. Flux installs NVIDIA GPU Operator components that discover the device and expose it to Kubernetes.

Prepare the host during installation

The Talos Image Factory schematic contains:

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/nvidia-container-toolkit-production
      - siderolabs/nvidia-open-gpu-kernel-modules-production

The installed image uses the same schematic and Talos version:

factory.talos.dev/metal-installer/6698d6f136c5bb37ca8bb8482c9084305084da0a5ead1f4dcae760796f8ab3a2:v1.13.6

The initial controlplane.yaml also declares the required modules:

machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset

These settings were applied as part of the fresh installation, together with the single-node scheduling and storage configuration. No post-install Talos upgrade or machine patch is required. See Talos Linux for the complete procedure.

Verify the Talos layer

Check the installed extensions:

talosctl get extensions

The expected names are:

nvidia-container-toolkit-production
nvidia-open-gpu-kernel-modules-production
schematic
modules.dep

The schematic entry must report:

6698d6f136c5bb37ca8bb8482c9084305084da0a5ead1f4dcae760796f8ab3a2

Check the live kernel modules:

talosctl get modules | grep nvidia

The module list must include nvidia, nvidia_uvm, nvidia_drm, and nvidia_modeset. Do not continue to the operator layer if either the extensions or modules are absent.

Install the Kubernetes layer through Flux

The GPU Operator Helm release is declared under:

gitops/infrastructure/controllers/base/gpu-operator
gitops/infrastructure/controllers/lab/gpu-operator

The lab overlay configures the ownership boundary:

driver:
  enabled: false

toolkit:
  enabled: false

hostPaths:
  driverInstallDir: /usr/local

Driver and toolkit installation are disabled because Talos already owns them. The operator supplies Node Feature Discovery, GPU Feature Discovery, the device plugin, validation, and DCGM metrics.

Flux activates the release through the infrastructure-controllers Kustomization. Verify reconciliation and the namespace workloads:

flux get kustomization infrastructure-controllers
flux get helmrelease gpu-operator --namespace gpu-operator
kubectl get pods --namespace gpu-operator

The CUDA validator may remain as a completed pod. The operator, discovery, device-plugin, validator, and DCGM exporter workloads should otherwise be running and ready.

Verify Kubernetes GPU capacity

kubectl get nodes -o json | \
  jq '.items[] | {
    name: .metadata.name,
    gpus: .status.allocatable["nvidia.com/gpu"]
  }'

The current node advertises one nvidia.com/gpu resource. The GPU labels added by discovery can be inspected with:

kubectl get node --show-labels | tr ',' '\n' | grep nvidia.com

Run an end-to-end CUDA test

Resource advertisement proves discovery; a CUDA workload proves scheduling, runtime injection, device access, and execution together:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        limits:
          nvidia.com/gpu: 1

Apply the pod and inspect its output:

kubectl apply -f cuda-vectoradd.yaml
kubectl logs pod/cuda-vectoradd

A successful run ends with:

Test PASSED
Done

Remove the validation pod afterward:

kubectl delete -f cuda-vectoradd.yaml

Upgrade rule

Future Talos upgrades must use an Image Factory installer that contains the same NVIDIA extensions for the target Talos release. Upgrading with the default installer would remove the host capabilities even though the GPU Operator manifests remain present in Kubernetes.

References