{"record":{"id":"88a306e14b380db7","repo":"hashicorp/nomad","slug":"requested-docker-runtime-q-was-not-found","errorCode":null,"errorMessage":"requested docker runtime %q was not found","messagePattern":"requested docker runtime %q was not found","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":1067,"sourceCode":"\n\t// create the config block that will later be consumed by go-dockerclient\n\tconfig := &containerapi.Config{\n\t\tImage:      imageID,\n\t\tEntrypoint: driverConfig.Entrypoint,\n\t\tHostname:   driverConfig.Hostname,\n\t\tUser:       task.User,\n\t\tTty:        driverConfig.TTY,\n\t\tOpenStdin:  driverConfig.Interactive,\n\t}\n\n\tif driverConfig.WorkDir != \"\" {\n\t\tconfig.WorkingDir = driverConfig.WorkDir\n\t}\n\n\tcontainerRuntime := driverConfig.Runtime\n\tif _, ok := task.DeviceEnv[nvidiaVisibleDevices]; ok {\n\t\tif !d.gpuRuntime {\n\t\t\treturn c, fmt.Errorf(\"requested docker runtime %q was not found\", d.config.GPURuntimeName)\n\t\t}\n\t\tif containerRuntime != \"\" && containerRuntime != d.config.GPURuntimeName {\n\t\t\treturn c, fmt.Errorf(\"conflicting runtime requests: gpu runtime %q conflicts with task runtime %q\", d.config.GPURuntimeName, containerRuntime)\n\t\t}\n\t\tcontainerRuntime = d.config.GPURuntimeName\n\t}\n\tif _, ok := d.config.allowRuntimes[containerRuntime]; !ok && containerRuntime != \"\" {\n\t\treturn c, fmt.Errorf(\"requested runtime %q is not allowed\", containerRuntime)\n\t}\n\n\t// Validate isolation modes on windows\n\tif runtime.GOOS != \"windows\" {\n\t\tif driverConfig.Isolation != \"\" {\n\t\t\treturn c, fmt.Errorf(\"Failed to create container configuration, cannot use isolation mode \\\"%s\\\" on %s\", driverConfig.Isolation, runtime.GOOS)\n\t\t}\n\t} else {\n\t\tif driverConfig.Isolation == \"\" {\n\t\t\tdriverConfig.Isolation = windowsIsolationModeHyperV","sourceCodeStart":1049,"sourceCodeEnd":1085,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1049-L1085","documentation":"If the task requests GPU devices (nvidia.visibledevices present in DeviceEnv) the container must run under the NVIDIA GPU docker runtime. When the driver's gpuRuntime flag is false — the configured GPU runtime (default `nvidia`) is not present in the Docker daemon's runtime list — container creation fails with this error.","triggerScenarios":"A GPU task is placed on a client whose Docker daemon lacks the `nvidia` runtime: NVIDIA container toolkit not installed, daemon.json not updated with runtimes.nvidia, or Docker not restarted after installing nvidia-container-toolkit.","commonSituations":"New GPU client provisioned without installing nvidia-container-toolkit; Docker upgraded and runtime config lost; typo in runtime name in daemon.json; Nomad fingerprinted the GPU but docker plugin re-probed runtimes and found none.","solutions":["Install nvidia-container-toolkit and register the runtime: `nvidia-ctk runtime configure --runtime=docker`, then restart Docker.","Verify `docker info | grep -A2 Runtimes` shows the nvidia runtime; if not, add runtimes.nvidia to /etc/docker/daemon.json and restart Docker.","Check the plugin's gpu_runtime option matches the runtime name actually registered in Docker.","Keep the task off GPU-less clients with a device constraint, or remove the GPU device request if GPU is not needed."],"exampleFix":"// /etc/docker/daemon.json\n// before\n{}\n// after\n{\"runtimes\": {\"nvidia\": {\"path\": \"nvidia-container-runtime\", \"runtimeArgs\": []}}}","handlingStrategy":"validation","validationCode":"const { execSync } = require('child_process');\nfunction nvidiaRuntimeAvailable() {\n  const info = execSync('docker info --format \"{{json.Runtimes}}\"').toString();\n  return Object.keys(JSON.parse(info)).includes('nvidia');\n}","typeGuard":null,"tryCatchPattern":"try {\n  await driver.StartTask(gpuTask);\n} catch (err) {\n  if (/was not found.*runtime/.test(err.message)) {\n    // install nvidia-container-toolkit, register runtime, restart docker, reschedule\n    console.error('GPU runtime missing on client:', err.message);\n  }\n  throw err;\n}","preventionTips":["Bake nvidia-container-toolkit + daemon.json into the GPU client image.","Fingerprint-check `docker info` runtimes during client provisioning.","Constrain GPU jobs to clients verified to expose the nvidia runtime.","Restart Docker after any nvidia runtime reconfiguration."],"tags":["docker","gpu","nvidia","runtime"],"backgroundTag":"docker-runtime-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}