hashicorp/nomad · error

requested docker runtime %q was not found

Error message

requested docker runtime %q was not found

What it means

If the task requests GPU devices (nvidia.visibledevices present in DeviceEnv) the container must run under the NVIDIA GPU docker runtime. When the driver's gpuRuntime flag is false — the configured GPU runtime (default `nvidia`) is not present in the Docker daemon's runtime list — container creation fails with this error.

Source

Thrown at drivers/docker/driver.go:1067

	// create the config block that will later be consumed by go-dockerclient
	config := &containerapi.Config{
		Image:      imageID,
		Entrypoint: driverConfig.Entrypoint,
		Hostname:   driverConfig.Hostname,
		User:       task.User,
		Tty:        driverConfig.TTY,
		OpenStdin:  driverConfig.Interactive,
	}

	if driverConfig.WorkDir != "" {
		config.WorkingDir = driverConfig.WorkDir
	}

	containerRuntime := driverConfig.Runtime
	if _, ok := task.DeviceEnv[nvidiaVisibleDevices]; ok {
		if !d.gpuRuntime {
			return c, fmt.Errorf("requested docker runtime %q was not found", d.config.GPURuntimeName)
		}
		if containerRuntime != "" && containerRuntime != d.config.GPURuntimeName {
			return c, fmt.Errorf("conflicting runtime requests: gpu runtime %q conflicts with task runtime %q", d.config.GPURuntimeName, containerRuntime)
		}
		containerRuntime = d.config.GPURuntimeName
	}
	if _, ok := d.config.allowRuntimes[containerRuntime]; !ok && containerRuntime != "" {
		return c, fmt.Errorf("requested runtime %q is not allowed", containerRuntime)
	}

	// Validate isolation modes on windows
	if runtime.GOOS != "windows" {
		if driverConfig.Isolation != "" {
			return c, fmt.Errorf("Failed to create container configuration, cannot use isolation mode \"%s\" on %s", driverConfig.Isolation, runtime.GOOS)
		}
	} else {
		if driverConfig.Isolation == "" {
			driverConfig.Isolation = windowsIsolationModeHyperV

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Install nvidia-container-toolkit and register the runtime: `nvidia-ctk runtime configure --runtime=docker`, then restart Docker.
2. Verify `docker info | grep -A2 Runtimes` shows the nvidia runtime; if not, add runtimes.nvidia to /etc/docker/daemon.json and restart Docker.
3. Check the plugin's gpu_runtime option matches the runtime name actually registered in Docker.
4. Keep the task off GPU-less clients with a device constraint, or remove the GPU device request if GPU is not needed.

Example fix

// /etc/docker/daemon.json
// before
{}
// after
{"runtimes": {"nvidia": {"path": "nvidia-container-runtime", "runtimeArgs": []}}}

const { execSync } = require('child_process');
function nvidiaRuntimeAvailable() {
  const info = execSync('docker info --format "{{json.Runtimes}}"').toString();
  return Object.keys(JSON.parse(info)).includes('nvidia');
}

try {
  await driver.StartTask(gpuTask);
} catch (err) {
  if (/was not found.*runtime/.test(err.message)) {
    // install nvidia-container-toolkit, register runtime, restart docker, reschedule
    console.error('GPU runtime missing on client:', err.message);
  }
  throw err;
}

When it happens

Trigger: A GPU task is placed on a client whose Docker daemon lacks the `nvidia` runtime: NVIDIA container toolkit not installed, daemon.json not updated with runtimes.nvidia, or Docker not restarted after installing nvidia-container-toolkit.

Common situations: New GPU client provisioned without installing nvidia-container-toolkit; Docker upgraded and runtime config lost; typo in runtime name in daemon.json; Nomad fingerprinted the GPU but docker plugin re-probed runtimes and found none.

Related errors

• conflicting runtime requests: gpu runtime %q conflicts with
• requested runtime %q is not allowed
• operation on unknown device(s) "%s/%s/%s" (%v): %v
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/88a306e14b380db7. Report an issue: GitHub.