hashicorp/nomad · error

mounting volumes: %w

Error message

mounting volumes: %w

What it means

In CSI hook Prerun, after claims succeed, mountVolumes asks the CSI node plugin to stage/publish the volume onto the host for the allocation. This error wraps failures during that staging/mount operation, commonly reported as CSI NodeStageVolume/NodePublishVolume errors.

Source

Thrown at client/allocrunner/csi_hook.go:136

				stub: &state.CSIVolumeStub{
					VolumeID: volumeRequest.VolumeID(c.alloc.Name)},
			}
		}
	}

	err := c.restoreMounts(c.volumeResults)
	if err != nil {
		return fmt.Errorf("restoring mounts: %w", err)
	}

	err = c.claimVolumes(c.volumeResults)
	if err != nil {
		return fmt.Errorf("claiming volumes: %w", err)
	}

	err = c.mountVolumes(c.volumeResults)
	if err != nil {
		return fmt.Errorf("mounting volumes: %w", err)
	}

	// make the mounts available to the taskrunner's volume_hook
	mounts := helper.ConvertMap(c.volumeResults,
		func(result *volumePublishResult) *csimanager.MountInfo {
			return result.stub.MountInfo
		})
	c.hookResources.SetCSIMounts(mounts)

	// persist the published mount info so we can restore on client restarts
	stubs := helper.ConvertMap(c.volumeResults,
		func(result *volumePublishResult) *state.CSIVolumeStub {
			return result.stub
		})
	c.allocRunnerShim.SetCSIVolumes(stubs)

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check nomad alloc status <alloc> events and node plugin logs for the underlying mount/stage error
2. Verify the volume is attached and the device is visible on the node (lsblk / plugin tools)
3. Ensure required filesystem tools and kernel modules are installed and mount_flags match what the plugin supports
4. Restart the CSI node plugin daemon on the node, then reschedule the allocation

Example fix

// before
mount_flags = ["noatime", "zstd"]   # plugin doesn't support zstd
// after
mount_flags = ["noatime"]
$ nomad alloc stop <alloc>  # reschedules and remounts successfully

// pre-check on target node before scheduling
const node = await nomad.get(`node/${nodeId}`);
if (!node.Drivers?.csi?.Healthy) throw new Error('CSI node plugin unhealthy');
// confirm requested mount flags are in the plugin's supported flags
if (!supportedMountFlags.every(f => pluginMountFlags.includes(f)))
  throw new Error('unsupported mount_flags for plugin');

try {
  await runJob(job);
} catch (err) {
  if (String(err).includes('mounting volumes')) {
    await restartCsiNodePlugin(nodeId); // e.g. nomad/systemd restart
    await retry(() => runJob(job), { retries: 2, backoffMs: 10000 });
  } else throw err;
}

When it happens

Trigger: csiHook.Prerun() -> c.mountVolumes(c.volumeResults) returns an error: node plugin cannot stage the volume (device missing, filesystem mkfs/mount failure, unsupported mount flags) or the node plugin RPC fails.

Common situations: Storage device not attached/visible on the node; missing filesystem tools (mkfs.ext4, xfs) or mount flags not supported by the plugin; node plugin pod/process crashed; kernel lacks required modules; host path conflicts from a previous mount.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/15f887aa40f9f304. Report an issue: GitHub.