{"record":{"id":"edc7659fa8dff781","repo":"hashicorp/nomad","slug":"conflicting-runtime-requests-gpu-runtime-q-confl","errorCode":null,"errorMessage":"conflicting runtime requests: gpu runtime %q conflicts with task runtime %q","messagePattern":"conflicting runtime requests: gpu runtime %q conflicts with task runtime %q","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/driver.go","lineNumber":1070,"sourceCode":"\t\tImage:      imageID,\n\t\tEntrypoint: driverConfig.Entrypoint,\n\t\tHostname:   driverConfig.Hostname,\n\t\tUser:       task.User,\n\t\tTty:        driverConfig.TTY,\n\t\tOpenStdin:  driverConfig.Interactive,\n\t}\n\n\tif driverConfig.WorkDir != \"\" {\n\t\tconfig.WorkingDir = driverConfig.WorkDir\n\t}\n\n\tcontainerRuntime := driverConfig.Runtime\n\tif _, ok := task.DeviceEnv[nvidiaVisibleDevices]; ok {\n\t\tif !d.gpuRuntime {\n\t\t\treturn c, fmt.Errorf(\"requested docker runtime %q was not found\", d.config.GPURuntimeName)\n\t\t}\n\t\tif containerRuntime != \"\" && containerRuntime != d.config.GPURuntimeName {\n\t\t\treturn c, fmt.Errorf(\"conflicting runtime requests: gpu runtime %q conflicts with task runtime %q\", d.config.GPURuntimeName, containerRuntime)\n\t\t}\n\t\tcontainerRuntime = d.config.GPURuntimeName\n\t}\n\tif _, ok := d.config.allowRuntimes[containerRuntime]; !ok && containerRuntime != \"\" {\n\t\treturn c, fmt.Errorf(\"requested runtime %q is not allowed\", containerRuntime)\n\t}\n\n\t// Validate isolation modes on windows\n\tif runtime.GOOS != \"windows\" {\n\t\tif driverConfig.Isolation != \"\" {\n\t\t\treturn c, fmt.Errorf(\"Failed to create container configuration, cannot use isolation mode \\\"%s\\\" on %s\", driverConfig.Isolation, runtime.GOOS)\n\t\t}\n\t} else {\n\t\tif driverConfig.Isolation == \"\" {\n\t\t\tdriverConfig.Isolation = windowsIsolationModeHyperV\n\t\t}\n\t\tif !slices.Contains(windowsIsolationModes, driverConfig.Isolation) {\n\t\t\treturn c, fmt.Errorf(\"Unsupported isolation mode \\\"%s\\\"\", driverConfig.Isolation)","sourceCodeStart":1052,"sourceCodeEnd":1088,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/driver.go#L1052-L1088","documentation":"When a task requests both GPU devices and an explicit task-level runtime, the two must agree. If driverConfig.Runtime is set and differs from the GPU runtime name, Nomad cannot satisfy both and fails container configuration. This prevents silently overriding the NVIDIA runtime needed for GPU access.","triggerScenarios":"Job sets runtime = \"runc\" (or any non-nvidia runtime) in the docker task config while also requesting a GPU device (device nvidia/gpu), causing the task runtime to conflict with d.config.GPURuntimeName.","commonSituations":"Shared task template that pins `runtime = \"runc\"` reused for GPU jobs; copy-paste of a CPU job spec with an explicit runtime, then adding a GPU device stanza.","solutions":["Remove the explicit `runtime` option from the GPU task's docker config and let Nomad select the GPU runtime automatically.","Set the task runtime equal to the GPU runtime name (e.g. `runtime = \"nvidia\"`).","Align the plugin's gpu_runtime config and job runtime value so they match."],"exampleFix":"// task docker config\n// before\nruntime = \"runc\"\ndevices = [{name = \"gpu\"}]\n// after\ndevices = [{name = \"gpu\"}]  // omit runtime; Nomad uses gpu runtime","handlingStrategy":"validation","validationCode":"function validateRuntimeConflict(taskConfig, gpuRuntimeName = 'nvidia') {\n  const hasGpu = (taskConfig.Devices || []).some(d => d.Name && d.Name.includes('gpu'));\n  if (hasGpu && taskConfig.Runtime && taskConfig.Runtime !== gpuRuntimeName) {\n    throw new Error(`runtime ${taskConfig.Runtime} conflicts with gpu runtime ${gpuRuntimeName}`);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  await client.jobs.submit(job);\n} catch (err) {\n  if (/conflicting runtime requests/.test(err.message)) {\n    console.error('Drop the explicit runtime option on GPU tasks');\n  }\n  throw err;\n}","preventionTips":["Never pin `runtime` in shared docker task templates that also serve GPU jobs.","Centralize GPU job templates with the runtime option omitted.","Keep plugin gpu_runtime name and job runtime values aligned if both are set."],"tags":["docker","gpu","nvidia","config-conflict"],"backgroundTag":"config-conflict","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}