{"record":{"id":"742a80217d028756","repo":"hashicorp/nomad","slug":"restoring-mounts-w","errorCode":null,"errorMessage":"restoring mounts: %w","messagePattern":"restoring mounts: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/csi_hook.go","lineNumber":126,"sourceCode":"\t// everything that's been done so far.\n\n\tc.volumeResultsLock.Lock()\n\tdefer c.volumeResultsLock.Unlock()\n\n\t// Initially, populate the result map with all of the requests\n\tfor alias, volumeRequest := range tg.Volumes {\n\t\tif volumeRequest.Type == structs.VolumeTypeCSI {\n\t\t\tc.volumeResults[alias] = &volumePublishResult{\n\t\t\t\trequest: volumeRequest,\n\t\t\t\tstub: &state.CSIVolumeStub{\n\t\t\t\t\tVolumeID: volumeRequest.VolumeID(c.alloc.Name)},\n\t\t\t}\n\t\t}\n\t}\n\n\terr := c.restoreMounts(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"restoring mounts: %w\", err)\n\t}\n\n\terr = c.claimVolumes(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"claiming volumes: %w\", err)\n\t}\n\n\terr = c.mountVolumes(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"mounting volumes: %w\", err)\n\t}\n\n\t// make the mounts available to the taskrunner's volume_hook\n\tmounts := helper.ConvertMap(c.volumeResults,\n\t\tfunc(result *volumePublishResult) *csimanager.MountInfo {\n\t\t\treturn result.stub.MountInfo\n\t\t})\n\tc.hookResources.SetCSIMounts(mounts)","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/csi_hook.go#L108-L144","documentation":"During CSI hook Prerun, Nomad replays previously published CSI mounts for the allocation by calling restoreMounts, which asks the CSI plugin for the node's published volumes. The error wraps any failure of that restoration step, so the allocation fails prerun rather than silently losing its volume attachments.","triggerScenarios":"csiHook.Prerun() -> c.restoreMounts(c.volumeResults) returns an error, typically because the CSI controller/node RPC fails, the plugin is not running on the node, or the volume is no longer published.","commonSituations":"CSI plugin (controller) not running or crashed on the node; CSI volume was garbage-collected on the storage backend while the alloc was stopped; storage cluster unreachable; Nomad/CSI plugin version mismatch changing publish semantics.","solutions":["Verify the CSI plugin is healthy on the node (nomad plugin status / nomad node status) and restart it if down","Check connectivity between the client and the storage backend/controller","Inspect nomad alloc status <alloc> events for the underlying CSI RPC error and address it (e.g. re-publish the volume)","If the volume state is stale, nomad volume detach/deregister and re-claim, then reschedule the allocation"],"exampleFix":"// before\n$ nomad alloc status 7f3a  -> \"restoring mounts: could not find volume publish context\"\n// after\n$ nomad volume detach <vol_id> <node_id>\n$ nomad volume status <vol_id>   # confirm no stale claims\n$ nomad job start <job>          # reschedule and re-publish","handlingStrategy":"retry","validationCode":"// before deploying the job, verify plugin and volume state\nconst plugins = await nomad.get('plugins').catch(() => []);\nif (!plugins.some(p => p.Name === 'aws-ebs-csi')) throw new Error('CSI plugin not running on node');\nawait nomad.get(`volume/csi/${volumeId}`); // throws early if volume missing","typeGuard":null,"tryCatchPattern":"try {\n  await scheduleAllocation();\n} catch (err) {\n  if (String(err).includes('restoring mounts')) {\n    await nomad.volumeDetach(volumeId, nodeId).catch(() => {});\n    await retry(() => scheduleAllocation(), { retries: 3, backoffMs: 5000 });\n  } else throw err;\n}","preventionTips":["Keep CSI controller and node plugins running on every node that schedules CSI volumes","Check nomad volume status before rescheduling stopped allocs","Pin Nomad and CSI plugin versions to compatible releases","Monitor plugin health and storage backend reachability"],"tags":["csi","storage","nomad","volumes"],"backgroundTag":"csi-volume-restore-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}