{"record":{"id":"9166c7e715a76154","repo":"hashicorp/nomad","slug":"w-volume-q-could-not-be-found-v","errorCode":null,"errorMessage":"%w: volume %q could not be found: %v","messagePattern":"%w: volume %q could not be found: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"plugins/csi/client.go","lineNumber":849,"sourceCode":"\t\treturn fmt.Errorf(\"missing volumeID\")\n\t}\n\tif stagingTargetPath == \"\" {\n\t\treturn fmt.Errorf(\"missing stagingTargetPath\")\n\t}\n\n\treq := &csipbv1.NodeUnstageVolumeRequest{\n\t\tVolumeId:          volumeID,\n\t\tStagingTargetPath: stagingTargetPath,\n\t}\n\n\t// NodeUnstageVolume's response contains no extra data. If err == nil, we were\n\t// successful.\n\t_, err := c.nodeClient.NodeUnstageVolume(ctx, req, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.NotFound:\n\t\t\terr = fmt.Errorf(\"%w: volume %q could not be found: %v\",\n\t\t\t\tstructs.ErrCSIClientRPCIgnorable, volumeID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"node plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t}\n\n\treturn err\n}\n\nfunc (c *client) NodePublishVolume(ctx context.Context, req *NodePublishVolumeRequest, opts ...grpc.CallOption) error {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn err\n\t}\n\tif err := req.Validate(); err != nil {\n\t\treturn fmt.Errorf(\"validation error: %v\", err)\n\t}\n\n\t// NodePublishVolume's response contains no extra data. If err == nil, we were","sourceCodeStart":831,"sourceCodeEnd":867,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L831-L867","documentation":"When the CSI plugin's NodeUnstageVolume gRPC call returns a NotFound status, Nomad wraps it with structs.ErrCSIClientRPCIgnorable so callers can detect that this failure can safely be ignored/retried. It means the plugin no longer knows about the volume with that ID — in practice the volume is already unstaged or was removed on the plugin side. The %w wrapping makes errors.Is(err, structs.ErrCSIClientRPCIgnorable) work for callers.","triggerScenarios":"The gRPC plugin returns codes.NotFound during NodeUnstageVolume: the volume was already unstaged (unstage after a previous unstage that actually succeeded), the plugin restarted and lost its volume records, or the volume was deleted from the storage backend while a Nomad claim still referenced it.","commonSituations":"Node restarts or plugin crashes losing staging state; garbage collection racing with unstage hooks; CSI plugins that report NotFound for unknown volumes on unstage; Nomad attempting unstage after the task's allocation records were already cleaned up.","solutions":["Treat as benign: check errors.Is(err, structs.ErrCSIClientRPCIgnorable) and continue with claim cleanup","Verify with the plugin that the staging path no longer holds the volume before declaring success","If the volume still exists on the backend, re-check the volume ID being passed to unstage","Check plugin logs to confirm it dropped the volume record legitimately"],"exampleFix":"// before\nif err := client.NodeUnstageVolume(ctx, id, path); err != nil {\n\treturn err\n}\n// after\nif err := client.NodeUnstageVolume(ctx, id, path); err != nil {\n\tif errors.Is(err, structs.ErrCSIClientRPCIgnorable) {\n\t\tlogger.Warn(\"volume not found at unstage, treating as unstaged\", \"error\", err)\n\t\treturn nil\n\t}\n\treturn err\n}","handlingStrategy":"try-catch","validationCode":"// No local check can fully prevent this: it reflects remote plugin state.\n// Optionally verify the volume still exists on the plugin before unstaging:\n_, err := client.ControllerValidateCapabilities(ctx, &csi.ControllerValidateVolumeRequest{ExternalID: volumeID, /* ... */})\nif err != nil {\n\tlogger.Warn(\"volume may already be removed\", \"volume\", volumeID)\n}","typeGuard":"func isCSIIgnorable(err error) bool { return errors.Is(err, structs.ErrCSIClientRPCIgnorable) }","tryCatchPattern":"err := client.NodeUnstageVolume(ctx, volumeID, stagingPath)\nswitch {\ncase err == nil:\n\t// unstaged\ncase errors.Is(err, structs.ErrCSIClientRPCIgnorable):\n\t// plugin returned NotFound: volume already unstaged; safe to ignore\n\tlogger.Info(\"volume already unstaged\", \"volume\", volumeID)\ndefault:\n\treturn err\n}","preventionTips":["Use errors.Is(err, structs.ErrCSIClientRPCIgnorable) to identify ignorable NotFound failures","Make unstage hooks idempotent — treat NotFound as success during claim GC","Reconcile Nomad claims with plugin state after node/plugin restarts","Keep plugin and Nomad versions compatible"],"tags":["csi","go","nomad","grpc","not-found","volume-management"],"backgroundTag":"grpc-not-found","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}