{"record":{"id":"15f887aa40f9f304","repo":"hashicorp/nomad","slug":"mounting-volumes-w","errorCode":null,"errorMessage":"mounting volumes: %w","messagePattern":"mounting volumes: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/allocrunner/csi_hook.go","lineNumber":136,"sourceCode":"\t\t\t\tstub: &state.CSIVolumeStub{\n\t\t\t\t\tVolumeID: volumeRequest.VolumeID(c.alloc.Name)},\n\t\t\t}\n\t\t}\n\t}\n\n\terr := c.restoreMounts(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"restoring mounts: %w\", err)\n\t}\n\n\terr = c.claimVolumes(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"claiming volumes: %w\", err)\n\t}\n\n\terr = c.mountVolumes(c.volumeResults)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"mounting volumes: %w\", err)\n\t}\n\n\t// make the mounts available to the taskrunner's volume_hook\n\tmounts := helper.ConvertMap(c.volumeResults,\n\t\tfunc(result *volumePublishResult) *csimanager.MountInfo {\n\t\t\treturn result.stub.MountInfo\n\t\t})\n\tc.hookResources.SetCSIMounts(mounts)\n\n\t// persist the published mount info so we can restore on client restarts\n\tstubs := helper.ConvertMap(c.volumeResults,\n\t\tfunc(result *volumePublishResult) *state.CSIVolumeStub {\n\t\t\treturn result.stub\n\t\t})\n\tc.allocRunnerShim.SetCSIVolumes(stubs)\n\n\treturn nil\n}","sourceCodeStart":118,"sourceCodeEnd":154,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/allocrunner/csi_hook.go#L118-L154","documentation":"In CSI hook Prerun, after claims succeed, mountVolumes asks the CSI node plugin to stage/publish the volume onto the host for the allocation. This error wraps failures during that staging/mount operation, commonly reported as CSI NodeStageVolume/NodePublishVolume errors.","triggerScenarios":"csiHook.Prerun() -> c.mountVolumes(c.volumeResults) returns an error: node plugin cannot stage the volume (device missing, filesystem mkfs/mount failure, unsupported mount flags) or the node plugin RPC fails.","commonSituations":"Storage device not attached/visible on the node; missing filesystem tools (mkfs.ext4, xfs) or mount flags not supported by the plugin; node plugin pod/process crashed; kernel lacks required modules; host path conflicts from a previous mount.","solutions":["Check nomad alloc status <alloc> events and node plugin logs for the underlying mount/stage error","Verify the volume is attached and the device is visible on the node (lsblk / plugin tools)","Ensure required filesystem tools and kernel modules are installed and mount_flags match what the plugin supports","Restart the CSI node plugin daemon on the node, then reschedule the allocation"],"exampleFix":"// before\nmount_flags = [\"noatime\", \"zstd\"]   # plugin doesn't support zstd\n// after\nmount_flags = [\"noatime\"]\n$ nomad alloc stop <alloc>  # reschedules and remounts successfully","handlingStrategy":"validation","validationCode":"// pre-check on target node before scheduling\nconst node = await nomad.get(`node/${nodeId}`);\nif (!node.Drivers?.csi?.Healthy) throw new Error('CSI node plugin unhealthy');\n// confirm requested mount flags are in the plugin's supported flags\nif (!supportedMountFlags.every(f => pluginMountFlags.includes(f)))\n  throw new Error('unsupported mount_flags for plugin');","typeGuard":null,"tryCatchPattern":"try {\n  await runJob(job);\n} catch (err) {\n  if (String(err).includes('mounting volumes')) {\n    await restartCsiNodePlugin(nodeId); // e.g. nomad/systemd restart\n    await retry(() => runJob(job), { retries: 2, backoffMs: 10000 });\n  } else throw err;\n}","preventionTips":["Install required filesystem tools (e2fsprogs, xfsprogs) and kernel modules on clients","Only use mount_flags the CSI plugin declares as supported","Monitor CSI node plugin daemon health on all storage-capable nodes","Test volume staging manually (plugin CLI) before production jobs"],"tags":["csi","storage","mount","nomad"],"backgroundTag":"csi-volume-mount-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}