{"record":{"id":"3a0eaaffdfa980b5","repo":"hashicorp/nomad","slug":"volume-q-is-already-published-on-another-node-and","errorCode":null,"errorMessage":"volume %q is already published on another node and does not have MULTI_NODE volume capability: %v","messagePattern":"volume %q is already published on another node and does not have MULTI_NODE volume capability: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugins/csi/client.go","lineNumber":328,"sourceCode":"\t}\n\n\tpbrequest := req.ToCSIRepresentation()\n\tresp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.NotFound:\n\t\t\terr = fmt.Errorf(\"volume %q or node %q could not be found: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.AlreadyExists:\n\t\t\terr = fmt.Errorf(\n\t\t\t\t\"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.ResourceExhausted:\n\t\t\terr = fmt.Errorf(\"node %q has reached the maximum allowable number of attached volumes: %v\",\n\t\t\t\treq.NodeID, err)\n\t\tcase codes.FailedPrecondition:\n\t\t\terr = fmt.Errorf(\"volume %q is already published on another node and does not have MULTI_NODE volume capability: %v\",\n\t\t\t\treq.ExternalID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\treturn &ControllerPublishVolumeResponse{\n\t\tPublishContext: maps.Clone(resp.PublishContext),\n\t}, nil\n}\n\nfunc (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn nil, err\n\t}\n\terr := req.Validate()\n\tif err != nil {","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L310-L346","documentation":"Raised in ControllerPublishVolume when the CSI controller returns gRPC FailedPrecondition. The volume (req.ExternalID) is already published (CONTROLLER_PUBLISH_VOLUME) on a different node, and the volume was not provisioned with MULTI_NODE (MULTI_NODE_READER_ONLY / MULTI_NODE_SINGLE_WRITER etc.) capability, so it cannot also be attached to req.NodeID. Nomad surfaces this so callers can distinguish an attach-conflict from other publish failures.","triggerScenarios":"c.ControllerPublishVolume() is called for a volume that is still attached to node A while the request targets node B, and the volume's declared capability set lacks MULTI_NODE access modes.","commonSituations":"A task was rescheduled from node A to node B but the old ControllerUnpublishVolume never completed; failover of a stateful workload to a standby node; a single-node RWO volume shared by groups on different clients; the plugin's unpublish was skipped due to a crashed client.","solutions":["Ensure the volume is unpublished from the old node first (verify the old allocation is stopped and ControllerUnpublishVolume succeeded), then retry the publish.","Check the CSI plugin/cluster logs and the Nomad volume checkpoint state for a stale publish on the previous node and force a detach via the storage backend console/CLI.","If the workload genuinely needs multi-node access, recreate the volume with MULTI_NODE capability (and ensure the plugin supports it).","Fix the root cause of missed unpublishes (e.g. node down → use the storage provider's force-detach), and re-run the job."],"exampleFix":"// before: job pinned to a node that changed, old attachment lingers\n//   volume \"vol-123\" is already published on another node ...\n// after: detach first at the provider, then reschedule\n# aws ec2 detach-volume --volume-id vol-123 --instance-id i-old --force\n# nomad volume detach vol-123 i-old\n# nomad job run app.nomad.hcl","handlingStrategy":"validation","validationCode":"// Only one node can hold a single-node-writable volume.\n// Before publishing, verify the volume has no active claim on another node:\nvol, _, err := nomadClient.CSIVolumes().Get(nil, volumeID)\nif err != nil { return err }\nfor _, r := range vol.Reads { if r.NodeID != targetNodeID { /* MULTI_NODE reader ok? */ } }\nif len(vol.WriteAllocs) > 0 {\n    for _, w := range vol.WriteAllocs {\n        if w.NodeID != targetNodeID {\n            return fmt.Errorf(\"volume %s still claimed on node %s; unpublish first\", volumeID, w.NodeID)\n        }\n    }\n}","typeGuard":"func isAttachConflict(err error) bool {\n    return err != nil && strings.Contains(err.Error(),\n        \"already published on another node\")\n}","tryCatchPattern":"err := publishVolume(ctx, req)\nif err != nil {\n    if isAttachConflict(err) {\n        // force-detach at the old node, then retry once\n        _ = unpublishAtOldNode(ctx, volumeID, oldNodeID)\n        err = publishVolume(ctx, req)\n    }\n    return err\n}","preventionTips":["Ensure old allocations fully stop and unpublish completes before rescheduling stateful workloads.","Recreate volumes with MULTI_NODE capability if they must attach to several nodes.","Monitor CSI plugin/node crash events — skipped unpublishes are the usual cause of stale publishes.","Keep volume claim GC enabled so checkpointed claims get cleaned up."],"tags":["csi","grpc","storage","attach-conflict"],"backgroundTag":"volume-already-attached-to-another-node","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}