{"record":{"id":"a41b6088f14d619e","repo":"hashicorp/nomad","slug":"csi-controllerdetachvolume-v","errorCode":null,"errorMessage":"CSI.ControllerDetachVolume: %v","messagePattern":"CSI\\.ControllerDetachVolume: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"client/csi_endpoint.go","lineNumber":180,"sourceCode":"\tcsiReq := req.ToCSIRequest()\n\n\t// Submit the request for a volume to the CSI Plugin.\n\tctx, cancelFn := c.requestContext()\n\tdefer cancelFn()\n\t// CSI ControllerUnpublishVolume errors for timeout, codes.Unavailable and\n\t// codes.ResourceExhausted are retried; all other errors are fatal.\n\t_, err = plugin.ControllerUnpublishVolume(ctx, csiReq,\n\t\tgrpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),\n\t\tgrpc_retry.WithMax(3),\n\t\tgrpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))\n\tif errors.Is(err, nstructs.ErrCSIClientRPCIgnorable) {\n\t\t// if the controller detach previously happened but the server failed to\n\t\t// checkpoint, we'll get an error from the plugin but can safely ignore it.\n\t\tc.c.logger.Debug(\"could not unpublish volume\", \"error\", err)\n\t\treturn nil\n\t}\n\tif err != nil {\n\t\treturn fmt.Errorf(\"CSI.ControllerDetachVolume: %v\", err)\n\t}\n\treturn err\n}\n\nfunc (c *CSI) ControllerCreateVolume(req *structs.ClientCSIControllerCreateVolumeRequest, resp *structs.ClientCSIControllerCreateVolumeResponse) error {\n\tdefer metrics.MeasureSince([]string{\"client\", \"csi_controller\", \"create_volume\"}, time.Now())\n\n\tplugin, err := c.findControllerPlugin(req.PluginID)\n\tif err != nil {\n\t\t// the server's view of the plugin health is stale, so let it know it\n\t\t// should retry with another controller instance\n\t\treturn fmt.Errorf(\"CSI.ControllerCreateVolume: %w: %v\",\n\t\t\tnstructs.ErrCSIClientRPCRetryable, err)\n\t}\n\tdefer plugin.Close()\n\n\tcsiReq, err := req.ToCSIRequest()\n\tif err != nil {","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/client/csi_endpoint.go#L162-L198","documentation":"The plugin's ControllerUnpublishVolume gRPC call failed and the failure was not the ignorable 'already detached' case (which is logged at debug and swallowed). The error surfaces after up to 3 retries, prefixed with CSI.ControllerDetachVolume; it means the node's unpublish checkpoint could not be completed via the controller.","triggerScenarios":"ControllerUnpublishVolume returned a non-retryable gRPC error from the plugin/provider (volume not found, permission error, provider API failure), or transient errors persisted through 3 retries.","commonSituations":"Provider reports the volume is not attached / wrong attachment state, so unpublish fails; expired provider credentials; plugin container unable to reach the storage API; double-detach races leaving inconsistent checkpoint state.","solutions":["Read the wrapped error: if the provider says volume not attached, the state is effectively detached — run `nomad volume detach` or re-run unpublish to reconcile the checkpoint","Check the plugin logs (`nomad alloc logs <plugin-alloc>`) for the underlying gRPC/provider error","Fix provider-side blockers (credentials, region, attachment state) and retry unpublish","As a last resort force-detach at the provider, then let Nomad GC/reconcile the volume claim"],"exampleFix":"// before\n# unpublish keeps failing: attachment stale at provider\n// after\naws ec2 detach-volume --volume-id vol-xxx --instance-id i-yyy --force\nnomad volume detach <vol_id> <node_id>  # reconcile checkpoint","handlingStrategy":"try-catch","validationCode":"// reconcile first: if the provider shows no attachment, skip controller unpublish\nif !providerHasAttachment(volumeExternalID, nodeID) {\n    return nil // already detached; only checkpoint cleanup needed\n}","typeGuard":null,"tryCatchPattern":"err := c.ControllerDetachVolume(req, resp)\nswitch {\ncase err == nil:\n    // detached\ncase isAlreadyDetached(err):\n    // treat as success, reconcile claim state\ncase structs.IsErrRetryable(err):\n    // retry with another controller\ndefault:\n    // inspect plugin logs, fix provider state, force-detach if needed\n}","preventionTips":["Detect 'not attached' provider errors and treat them as idempotent success","Reconcile volume claims periodically to clear stale checkpoints","Keep provider credentials valid on plugin allocations","Force-detach at the provider only after confirming Nomad-side state"],"tags":["csi","grpc","unpublish-volume","detach"],"backgroundTag":"grpc-rpc-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}