conflicting runtime requests: gpu runtime %q conflicts with

Error message

conflicting runtime requests: gpu runtime %q conflicts with task runtime %q

What it means

When a task requests both GPU devices and an explicit task-level runtime, the two must agree. If driverConfig.Runtime is set and differs from the GPU runtime name, Nomad cannot satisfy both and fails container configuration. This prevents silently overriding the NVIDIA runtime needed for GPU access.

Source

Thrown at drivers/docker/driver.go:1070

		Image:      imageID,
		Entrypoint: driverConfig.Entrypoint,
		Hostname:   driverConfig.Hostname,
		User:       task.User,
		Tty:        driverConfig.TTY,
		OpenStdin:  driverConfig.Interactive,
	}

	if driverConfig.WorkDir != "" {
		config.WorkingDir = driverConfig.WorkDir
	}

	containerRuntime := driverConfig.Runtime
	if _, ok := task.DeviceEnv[nvidiaVisibleDevices]; ok {
		if !d.gpuRuntime {
			return c, fmt.Errorf("requested docker runtime %q was not found", d.config.GPURuntimeName)
		}
		if containerRuntime != "" && containerRuntime != d.config.GPURuntimeName {
			return c, fmt.Errorf("conflicting runtime requests: gpu runtime %q conflicts with task runtime %q", d.config.GPURuntimeName, containerRuntime)
		}
		containerRuntime = d.config.GPURuntimeName
	}
	if _, ok := d.config.allowRuntimes[containerRuntime]; !ok && containerRuntime != "" {
		return c, fmt.Errorf("requested runtime %q is not allowed", containerRuntime)
	}

	// Validate isolation modes on windows
	if runtime.GOOS != "windows" {
		if driverConfig.Isolation != "" {
			return c, fmt.Errorf("Failed to create container configuration, cannot use isolation mode \"%s\" on %s", driverConfig.Isolation, runtime.GOOS)
		}
	} else {
		if driverConfig.Isolation == "" {
			driverConfig.Isolation = windowsIsolationModeHyperV
		}
		if !slices.Contains(windowsIsolationModes, driverConfig.Isolation) {
			return c, fmt.Errorf("Unsupported isolation mode \"%s\"", driverConfig.Isolation)

View on GitHub (pinned to 482b49bf1a)

Solutions

Remove the explicit `runtime` option from the GPU task's docker config and let Nomad select the GPU runtime automatically.
Set the task runtime equal to the GPU runtime name (e.g. `runtime = "nvidia"`).
Align the plugin's gpu_runtime config and job runtime value so they match.

Example fix

// task docker config
// before
runtime = "runc"
devices = [{name = "gpu"}]
// after
devices = [{name = "gpu"}]  // omit runtime; Nomad uses gpu runtime

Defensive patterns

Strategy: validation

Validate before calling

function validateRuntimeConflict(taskConfig, gpuRuntimeName = 'nvidia') {
  const hasGpu = (taskConfig.Devices || []).some(d => d.Name && d.Name.includes('gpu'));
  if (hasGpu && taskConfig.Runtime && taskConfig.Runtime !== gpuRuntimeName) {
    throw new Error(`runtime ${taskConfig.Runtime} conflicts with gpu runtime ${gpuRuntimeName}`);
  }
}

Try / catch

try {
  await client.jobs.submit(job);
} catch (err) {
  if (/conflicting runtime requests/.test(err.message)) {
    console.error('Drop the explicit runtime option on GPU tasks');
  }
  throw err;
}

Prevention

Never pin `runtime` in shared docker task templates that also serve GPU jobs.
Centralize GPU job templates with the runtime option omitted.
Keep plugin gpu_runtime name and job runtime values aligned if both are set.

When it happens

Trigger: Job sets runtime = "runc" (or any non-nvidia runtime) in the docker task config while also requesting a GPU device (device nvidia/gpu), causing the task runtime to conflict with d.config.GPURuntimeName.

Common situations: Shared task template that pins `runtime = "runc"` reused for GPU jobs; copy-paste of a CPU job spec with an explicit runtime, then adding a GPU device stanza.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/edc7659fa8dff781. Report an issue: GitHub.