hashicorp/nomad · error

could not validate task driver capabilities: %v

Error message

could not validate task driver capabilities: %v

What it means

Before using CSI volumes, the hook validates that each task's driver advertises CSI capability support via GetTaskDriverCapabilities. This error means the capability query to the task driver itself failed, so Nomad cannot determine whether the driver can mount CSI configs, and prerun aborts.

Source

Thrown at client/allocrunner/csi_hook.go:230

}

type volumePublishResult struct {
	request        *structs.VolumeRequest // the request from the jobspec
	volume         *structs.CSIVolume     // the volume we get back from the server
	publishContext map[string]string      // populated after claim if provided by plugin
	stub           *state.CSIVolumeStub   // populated from volume, plugin, or stub
}

// validateTasksSupportCSI verifies that at least one task in the group uses a
// task driver that supports CSI. This prevents us from publishing CSI volumes
// only to find out once we get to the taskrunner/volume_hook that no task can
// mount them.
func (c *csiHook) validateTasksSupportCSI(tg *structs.TaskGroup) error {

	for _, task := range tg.Tasks {
		caps, err := c.allocRunnerShim.GetTaskDriverCapabilities(task.Name)
		if err != nil {
			return fmt.Errorf("could not validate task driver capabilities: %v", err)
		}

		if caps.MountConfigs == drivers.MountConfigSupportNone {
			continue
		}

		return nil
	}

	return fmt.Errorf("no task supports CSI")
}

// restoreMounts tries to restore the mount info from the local client state and
// then verifies it with the plugin. If the volume is already mounted, we don't
// want to re-run the claim and mount workflow again. This lets us tolerate
// restarting clients even on disconnected nodes.
func (c *csiHook) restoreMounts(results map[string]*volumePublishResult) error {
	stubs, err := c.allocRunnerShim.GetCSIVolumes()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the driver is loaded and healthy: nomad node status <node> (driver health/attributes)
2. Restart the Nomad client agent so the driver plugin re-handshakes
3. Check client logs for driver plugin startup errors and fix the driver config/plugin install
4. Upgrade the external driver plugin (e.g. podman) to a version that implements driver capabilities, and keep Nomad client/plugin versions compatible

Example fix

// before
# nomad node status: docker driver = unhealthy
// after
$ sudo systemctl restart nomad
$ nomad node status <node>  # docker: healthy, capabilities detected

const node = await nomad.get('node/self');
for (const driver of ['docker', 'exec']) {
  const d = node.Drivers?.[driver];
  if (!d?.Detected || !d?.Healthy) throw new Error(`driver ${driver} not healthy`);
}

function driverIsHealthy(d) {
  return !!d && typeof d === 'object' && d.Detected === true && d.Healthy === true;
}

try {
  await runJob(job);
} catch (err) {
  if (String(err).includes('task driver capabilities')) {
    await restartNomadAgent(nodeId); // re-handshake driver plugins
    await waitForDriverHealthy(nodeId, 'docker');
    await runJob(job);
  } else throw err;
}

When it happens

Trigger: csiHook.validateTasksSupportCSI (called from Prerun) -> c.allocRunnerShim.GetTaskDriverCapabilities(task.Name) returns an error: the driver plugin is not running, failed to initialize, or the capability RPC failed.

Common situations: Docker/exec/podman driver plugin crashed or failed to start on the client; driver missing from the client's driver configuration; Nomad client version newer than driver plugin (missing Capabilities RPC during mixed-version upgrades); plugin handshake failure after node restart.

Related errors

• controller attach volume: %v
• controller validate volume: %v
• controller detach volume: %v
• controller create volume: %v
• controller expand volume: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a0a20872a4527b48. Report an issue: GitHub.