{"record":{"id":"98ab5c90a8f6c367","repo":"hashicorp/nomad","slug":"w-volume-q-or-node-q-could-not-be-found-v","errorCode":null,"errorMessage":"%w: volume %q or node %q could not be found: %v","messagePattern":"%w: volume %q or node %q could not be found: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"plugins/csi/client.go","lineNumber":360,"sourceCode":"\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn nil, err\n\t}\n\terr := req.Validate()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tupbrequest := req.ToCSIRepresentation()\n\t_, err = c.controllerClient.ControllerUnpublishVolume(ctx, upbrequest, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.NotFound:\n\t\t\t// we'll have validated the volume and node *should* exist at the\n\t\t\t// server, so if we get a not-found here it's because we've previously\n\t\t\t// checkpointed. we'll return an error so the caller can log it for\n\t\t\t// diagnostic purposes.\n\t\t\terr = fmt.Errorf(\"%w: volume %q or node %q could not be found: %v\",\n\t\t\t\tstructs.ErrCSIClientRPCIgnorable, req.ExternalID, req.NodeID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\treturn &ControllerUnpublishVolumeResponse{}, nil\n}\n\nfunc (c *client) ControllerValidateCapabilities(ctx context.Context, req *ControllerValidateVolumeRequest, opts ...grpc.CallOption) error {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn err\n\t}\n\tif req.ExternalID == \"\" {\n\t\treturn fmt.Errorf(\"missing volume ID\")\n\t}\n","sourceCodeStart":342,"sourceCodeEnd":378,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L342-L378","documentation":"Raised in ControllerUnpublishVolume when the controller plugin returns gRPC NotFound. Nomad comments that the volume and node were validated to exist, so a NotFound here typically means the volume was previously checkpointed as published but no longer exists at the storage backend. Nomad wraps the error with structs.ErrCSIClientRPCIgnorable so callers can detect that it is safe to ignore (e.g. log for diagnostics and treat the unpublish as complete) via errors.Is.","triggerScenarios":"c.ControllerUnpublishVolume() called for req.ExternalID/req.NodeID and the plugin replies codes.NotFound — the volume was deleted out-of-band at the provider, or the checkpointed attachment no longer exists server-side.","commonSituations":"An operator deleted the volume at the cloud console while Nomad still tracked it; a leaked claim from a node that was destroyed; GC removed the volume while an unpublish was still pending in the checkpoint state.","solutions":["Treat it as ignorable: check errors.Is(err, structs.ErrCSIClientRPCIgnorable) and log the volume/node IDs for diagnostics instead of failing the unpublish flow.","Clean up Nomad's view: deregister or GC the volume (nomad volume deregister <id>) so it stops being checkpointed as published.","If the volume should exist, verify its ID against the storage provider and re-register/recreate it before unpublishing.","Audit for out-of-band deletions (CI scripts, consoles) that bypass Nomad volume lifecycle."],"exampleFix":"// before\nerr := csi.ControllerUnpublishVolume(ctx, req)\nif err != nil {\n    return err // blocks claim GC on already-deleted volumes\n}\n// after\nerr := csi.ControllerUnpublishVolume(ctx, req)\nif err != nil {\n    if errors.Is(err, structs.ErrCSIClientRPCIgnorable) {\n        logger.Warn(\"unpublish not found; ignoring\", \"volume\", req.ExternalID)\n        return nil\n    }\n    return err\n}","handlingStrategy":"type-guard","validationCode":"// Before unpublishing, confirm the volume still exists and which nodes claim it.\nvol, _, err := nomadClient.CSIVolumes().Get(nil, req.ExternalID)\nif err != nil {\n    if err is 404: // already deregistered at Nomad; skip unpublish\n        return nil\n    return err\n}\n// avoid unpublishing for checkpointed volumes that no longer exist at the backend","typeGuard":"// Nomad marks this case with a wrapped sentinel:\nfunc isIgnorableUnpublishNotFound(err error) bool {\n    return errors.Is(err, structs.ErrCSIClientRPCIgnorable)\n}","tryCatchPattern":"_, err := csi.ControllerUnpublishVolume(ctx, req)\nswitch {\ncase err == nil:\n    return nil\ncase errors.Is(err, structs.ErrCSIClientRPCIgnorable):\n    logger.Warn(\"volume/node not found at backend; ignoring\", \"volume\", req.ExternalID, \"node\", req.NodeID)\n    return nil\ndefault:\n    return err\n}","preventionTips":["Always check the ErrCSIClientRPCIgnorable sentinel before treating unpublish failure as fatal.","Prevent out-of-band volume deletion; delete volumes through Nomad (nomad volume deregister -purge only after claims clear).","Keep claim GC enabled so checkpoint state converges when nodes are destroyed.","Log volume/node IDs when ignorable so audits can reconcile backend drift."],"tags":["csi","grpc","not-found","checkpoint"],"backgroundTag":"csi-unpublish-volume-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}