{"record":{"id":"f4a7561611af38ab","repo":"hashicorp/nomad","slug":"could-not-detach-from-node-w","errorCode":null,"errorMessage":"could not detach from node: %w","messagePattern":"could not detach from node: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/csi_endpoint.go","lineNumber":872,"sourceCode":"\t\tVolumeID:        vol.ID,\n\t\tVolumeNamespace: vol.Namespace,\n\t\tExternalID:      vol.RemoteID(),\n\t\tAllocID:         claim.AllocationID,\n\t\tNodeID:          claim.NodeID,\n\t\tAttachmentMode:  claim.AttachmentMode,\n\t\tAccessMode:      claim.AccessMode,\n\t\tReadOnly:        claim.Mode == structs.CSIVolumeClaimRead,\n\t}\n\terr := v.srv.RPC(\"ClientCSI.NodeDetachVolume\",\n\t\treq, &cstructs.ClientCSINodeDetachVolumeResponse{})\n\tif err != nil {\n\t\t// we should only get this error if the Nomad node disconnects and\n\t\t// is garbage-collected, so at this point we don't have any reason\n\t\t// to operate as though the volume is attached to it.\n\t\t// note: errors.Is cannot be used because the RPC call breaks\n\t\t// error wrapping.\n\t\tif !strings.Contains(err.Error(), structs.ErrUnknownNode.Error()) {\n\t\t\treturn fmt.Errorf(\"could not detach from node: %w\", err)\n\t\t}\n\t}\n\treturn nil\n}\n\n// controllerUnpublishVolume handles the sending RPCs to the Controller plugin\n// to unpublish the volume (detach it from its host). This function should only\n// be called on a copy of the volume.\nfunc (v *CSIVolume) controllerUnpublishVolume(vol *structs.CSIVolume, claim *structs.CSIVolumeClaim) error {\n\tv.logger.Trace(\"controller unpublish\", \"vol\", vol.ID)\n\n\tif !vol.ControllerRequired {\n\t\tclaim.State = structs.CSIVolumeClaimStateReadyToFree\n\t\treturn nil\n\t}\n\n\t// We need a new snapshot after each checkpoint\n\tsnap, err := v.srv.fsm.State().Snapshot()","sourceCodeStart":854,"sourceCodeEnd":890,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/csi_endpoint.go#L854-L890","documentation":"nodeUnpublishVolumeImpl failed to run the controller detach RPC against the client node and wraps the error as \"could not detach from node: <err>\". Notably the code deliberately skips this wrap when the error is ErrUnknownNode (checked by string match because RPC breaks error wrapping), treating a garbage-collected node as \"already detached\". Any other failure reaching the node/plugin during detach surfaces here.","triggerScenarios":"Raised when v.controllerUnpublishVolume / the client RPC fails with an error that does NOT contain structs.ErrUnknownNode: the client node is unreachable, the CSI controller plugin isn't running or errored, the plugin returned a storage-provider failure, or an RPC timeout occurred during ClientCSI.ControllerDetachVolume.","commonSituations":"Client node crashed or network-partitioned mid-unpublish; the CSI controller plugin task is unhealthy so detach RPCs fail; the storage backend (EBS, Ceph, etc.) rejects the detach because the volume isn't attached or credentials are wrong; timeouts from slow cloud APIs under load.","solutions":["Check the client node and controller plugin health: `nomad node status` and `nomad plugin status <plugin-id>`; restart the plugin job if unhealthy.","Inspect the wrapped inner error (text after \"could not detach from node:\") for the plugin's specific failure and address that (credentials, backend API errors, timeouts).","Retry the unpublish — controller detach is designed to be re-run; Nomad checkpoints past-claim state so retries are safe.","If the node is permanently gone, wait for it to be GC'd: once ErrUnknownNode would be returned, unpublish treats it as detached and succeeds."],"exampleFix":"// before: treating every unpublish error as fatal\nif err := client.CSIVolumes().Unpublish(...); err != nil { return err }\n// after: retry transient detach failures with backoff\nif err := client.CSIVolumes().Unpublish(...); err != nil {\n    if strings.Contains(err.Error(), \"could not detach from node\") {\n        return backoff.Retry(unpublish, 3) // node/plugin may recover\n    }\n    return err\n}","handlingStrategy":"retry","validationCode":"node, _, err := client.Nodes().Info(nodeID, nil)\nif err == nil && node != nil && node.Status != \"ready\" {\n    // node down: expect detach failures; wait for recovery or GC\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"could not detach from node\") {\n    // transient (node/plugin unreachable): retry with backoff\n    // note: ErrUnknownNode cases already return nil server-side\n    backoff.Retry(unpublishFunc, expBackoff)\n}","preventionTips":["Keep controller plugin tasks healthy on every node claiming controller-required volumes.","Monitor wrapped inner errors for backend-specific causes (credentials, API limits).","Retry unpublishes — Nomad checkpoints past-claim state so retries are safe.","For permanently lost nodes, wait for node GC so unpublish short-circuits as detached."],"tags":["nomad","csi","detach","network","plugin"],"backgroundTag":"csi-detach-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}