{"record":{"id":"8da384faad2af6d6","repo":"hashicorp/nomad","slug":"w-v","errorCode":null,"errorMessage":"%w: %v","messagePattern":"%w: %v","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"client/pluginmanager/csimanager/volume.go","lineNumber":298,"sourceCode":"\n\tlogger := hclog.FromContext(ctx)\n\tlogger.Trace(\"unpublishing volume\", \"plugin_target_path\", pluginTargetPath)\n\n\t// CSI NodeUnpublishVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\trpcErr := v.plugin.NodeUnpublishVolume(ctx, remoteID, pluginTargetPath,\n\t\tgrpc_retry.WithPerRetryTimeout(DefaultMountActionTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)),\n\t)\n\n\thostTargetPath := v.targetForVolume(v.mountRoot, volID, allocID, usage)\n\tif _, err := os.Stat(hostTargetPath); os.IsNotExist(err) {\n\t\tif rpcErr != nil && strings.Contains(rpcErr.Error(), \"no mount point\") {\n\t\t\t// host target path was already destroyed, nothing to do here.\n\t\t\t// this helps us in the case that a previous GC attempt cleaned\n\t\t\t// up the volume on the node but the controller RPCs failed\n\t\t\trpcErr = fmt.Errorf(\"%w: %v\", structs.ErrCSIClientRPCIgnorable, rpcErr)\n\t\t}\n\t\treturn rpcErr\n\t}\n\n\tlogger.Trace(\"removing host path\", \"host_target_path\", hostTargetPath)\n\n\t// Host Target Path was not cleaned up, attempt to do so here. If it's still\n\t// a mount then removing the dir will fail and we'll return any rpcErr and the\n\t// file error.\n\trmErr := os.Remove(hostTargetPath)\n\tif rmErr != nil {\n\t\treturn combineErrors(rpcErr, rmErr)\n\t}\n\n\t// We successfully removed the directory, return any rpcErrors that were\n\t// encountered, but because we got here, they were probably flaky or was\n\t// cleaned up externally.\n\treturn fmt.Errorf(\"%w: %v\", structs.ErrCSIClientRPCIgnorable, rpcErr)","sourceCodeStart":280,"sourceCodeEnd":316,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/pluginmanager/csimanager/volume.go#L280-L316","documentation":"During unpublishVolume, if the volume's host target path no longer exists on disk AND the controller unmount RPC failed with a \"no mount point\" error, the rpcErr is wrapped with structs.ErrCSIClientRPCIgnorable. This marks the failure as benign — a previous GC attempt already destroyed the volume on the node even though its controller RPCs failed — so callers (GC) can ignore it.","triggerScenarios":"Node-local unmount cleanup runs while the host target path is already gone (prior GC removed it) and the preceding controller UnpublishVolume RPC returned an error mentioning \"no mount point\".","commonSituations":"Repeated GC cycles on the same allocation; a controller that reports 'no mount point' after the node already detached; retrying volume unpublish after a partially completed earlier attempt.","solutions":["Nothing to fix in most cases: treat errors wrapping ErrCSIClientRPCIgnorable as safe to skip during GC","If this appears repeatedly, investigate why prior unmounts left the controller RPC failing (stale attachments at the storage backend)","Verify the volume is fully detached at the storage provider and clean up leftover controller attachments manually","Check for a plugin version that mishandles unpublish of already-removed mounts and upgrade the CSI plugin"],"exampleFix":"// before: treating every unpublish error as fatal\nif err := client.CSI().UnpublishVolume(...); err != nil {\n    return err\n}\n\n// after: ignore ignorable unpublish errors during GC\nif err := client.CSI().UnpublishVolume(...); err != nil {\n    if errors.Is(err, structs.ErrCSIClientRPCIgnorable) {\n        logger.Warn(\"ignorable CSI unpublish error, volume already destroyed\", \"err\", err)\n        return nil\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":"// Nothing to check before the API; the error only occurs after prior GC already removed the mount.\nnull","typeGuard":null,"tryCatchPattern":"err := unpublishVolume(ctx, volID, allocID, usage)\nif err != nil && errors.Is(err, structs.ErrCSIClientRPCIgnorable) {\n\t// host path already destroyed by a previous GC; safe to proceed\n\tlogger.Warn(\"volume already unpublished on node; ignoring\", \"vol\", volID)\n\terr = nil\n}\nreturn err","preventionTips":["Make GC/unpublish handlers idempotent and treat ErrCSIClientRPCIgnorable as success","Use errors.Is against structs.ErrCSIClientRPCIgnorable rather than string matching","Investigate recurring ignorable unpublish errors as symptoms of stale storage-backend attachments","Keep CSI plugins current; older plugins may mis-report 'no mount point' states"],"tags":["csi","nomad","unpublish","garbage-collection","storage"],"backgroundTag":"csi-rpc-ignorable","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}