hashicorp/nomad · error

restoring mounts: %w

Error message

restoring mounts: %w

What it means

During CSI hook Prerun, Nomad replays previously published CSI mounts for the allocation by calling restoreMounts, which asks the CSI plugin for the node's published volumes. The error wraps any failure of that restoration step, so the allocation fails prerun rather than silently losing its volume attachments.

Source

Thrown at client/allocrunner/csi_hook.go:126

	// everything that's been done so far.

	c.volumeResultsLock.Lock()
	defer c.volumeResultsLock.Unlock()

	// Initially, populate the result map with all of the requests
	for alias, volumeRequest := range tg.Volumes {
		if volumeRequest.Type == structs.VolumeTypeCSI {
			c.volumeResults[alias] = &volumePublishResult{
				request: volumeRequest,
				stub: &state.CSIVolumeStub{
					VolumeID: volumeRequest.VolumeID(c.alloc.Name)},
			}
		}
	}

	err := c.restoreMounts(c.volumeResults)
	if err != nil {
		return fmt.Errorf("restoring mounts: %w", err)
	}

	err = c.claimVolumes(c.volumeResults)
	if err != nil {
		return fmt.Errorf("claiming volumes: %w", err)
	}

	err = c.mountVolumes(c.volumeResults)
	if err != nil {
		return fmt.Errorf("mounting volumes: %w", err)
	}

	// make the mounts available to the taskrunner's volume_hook
	mounts := helper.ConvertMap(c.volumeResults,
		func(result *volumePublishResult) *csimanager.MountInfo {
			return result.stub.MountInfo
		})
	c.hookResources.SetCSIMounts(mounts)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the CSI plugin is healthy on the node (nomad plugin status / nomad node status) and restart it if down
2. Check connectivity between the client and the storage backend/controller
3. Inspect nomad alloc status <alloc> events for the underlying CSI RPC error and address it (e.g. re-publish the volume)
4. If the volume state is stale, nomad volume detach/deregister and re-claim, then reschedule the allocation

Example fix

// before
$ nomad alloc status 7f3a  -> "restoring mounts: could not find volume publish context"
// after
$ nomad volume detach <vol_id> <node_id>
$ nomad volume status <vol_id>   # confirm no stale claims
$ nomad job start <job>          # reschedule and re-publish

// before deploying the job, verify plugin and volume state
const plugins = await nomad.get('plugins').catch(() => []);
if (!plugins.some(p => p.Name === 'aws-ebs-csi')) throw new Error('CSI plugin not running on node');
await nomad.get(`volume/csi/${volumeId}`); // throws early if volume missing

try {
  await scheduleAllocation();
} catch (err) {
  if (String(err).includes('restoring mounts')) {
    await nomad.volumeDetach(volumeId, nodeId).catch(() => {});
    await retry(() => scheduleAllocation(), { retries: 3, backoffMs: 5000 });
  } else throw err;
}

When it happens

Trigger: csiHook.Prerun() -> c.restoreMounts(c.volumeResults) returns an error, typically because the CSI controller/node RPC fails, the plugin is not running on the node, or the volume is no longer published.

Common situations: CSI plugin (controller) not running or crashed on the node; CSI volume was garbage-collected on the storage backend while the alloc was stopped; storage cluster unreachable; Nomad/CSI plugin version mismatch changing publish semantics.

Related errors

• claiming volumes: %w
• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/742a80217d028756. Report an issue: GitHub.