{"record":{"id":"1385c98d967a7e56","repo":"hashicorp/nomad","slug":"csi-client-error-retryable","errorCode":null,"errorMessage":"CSI client error (retryable)","messagePattern":"CSI client error \\(retryable\\)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"nomad/structs/errors.go","lineNumber":90,"sourceCode":"\t// duplicates its message so the CLI can match it without importing structs.\n\t// Keep the two in sync.\n\tErrResultPaginatorCreation = errors.New(errResultPaginatorCreation)\n\n\tErrUnknownNode = errors.New(ErrUnknownNodePrefix)\n\n\tErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)\n\tErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)\n\tErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)\n\tErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)\n\tErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)\n\tErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)\n\tErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)\n\tErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)\n\tErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)\n\n\tErrCSIClientRPCIgnorable  = errors.New(\"CSI client error (ignorable)\")\n\tErrCSIClientRPCRetryable  = errors.New(\"CSI client error (retryable)\")\n\tErrCSIVolumeMaxClaims     = errors.New(\"volume max claims reached\")\n\tErrCSIVolumeUnschedulable = errors.New(\"volume is currently unschedulable\")\n\tErrCSIPluginInUse         = errors.New(\"plugin in use\")\n)\n\n// IsErrNoLeader returns whether the error is due to there being no leader.\nfunc IsErrNoLeader(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), errNoLeader)\n}\n\n// IsErrNoRegionPath returns whether the error is due to there being no path to\n// the given region.\nfunc IsErrNoRegionPath(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), errNoRegionPath)\n}\n\n// IsErrTokenNotFound returns whether the error is due to the passed token not\n// being resolvable.\nfunc IsErrTokenNotFound(err error) bool {","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/structs/errors.go#L72-L108","documentation":"ErrCSIClientRPCRetryable marks a CSI controller RPC failure as retryable — typically a per-instance plugin issue (unavailable controller, timeout) where retrying, possibly against another controller instance, may succeed. Because the error is serialized over RPC, client code matches its message string rather than using errors.Is.","triggerScenarios":"CSI.ControllerValidateVolume/AttachVolume/DetachVolume/CreateVolume/ExpandVolume fail and the endpoint wraps the plugin error with fmt.Errorf(\"...: %w: %v\", nstructs.ErrCSIClientRPCRetryable, err); csi_hook.go then string-matches to decide whether to retry the claim.","commonSituations":"CSI controller plugin is temporarily down or restarting; Nomad server connects to a stale plugin instance; storage backend throttles requests; network blip between Nomad client and plugin socket.","solutions":["Simply retry the CSI operation — the error explicitly indicates retryability, ideally after a short backoff","Let Nomad retry with a different controller instance if one is available","Check the CSI plugin/controller health and logs if retries keep failing","Keep code matching via strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error()) because the sentinel does not survive RPC wrapping"],"exampleFix":"// before\nif err := csiAttach(volID, allocID); err != nil {\n    return err // fails the alloc hook permanently\n}\n// after\nif err := csiAttach(volID, allocID); err != nil {\n    if strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error()) {\n        return retry.WithBackoff(func() error { return csiAttach(volID, allocID) })\n    }\n    return err\n}","handlingStrategy":"retry","validationCode":"vol, _, err := client.CSIVolumes().Info(volID, nil)\nif err == nil && vol.ControllersHealthy < 1 {\n    return fmt.Errorf(\"volume %s has no healthy controllers; retry later\", volID)\n}","typeGuard":"func isRetryableCSIErr(err error) bool {\n    return err != nil && strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error())\n}","tryCatchPattern":"err := csiOperation(ctx)\nif isRetryableCSIErr(err) {\n    return retry.Do(func() error { return csiOperation(ctx) },\n        retry.Attempts(3), retry.Delay(2*time.Second))\n}\nreturn err","preventionTips":["Use exponential backoff for retryable CSI errors","Allow Nomad to try alternate controller instances","Alert if retries persist — the plugin/controller is likely genuinely unhealthy"],"tags":["csi","storage","retry","nomad"],"backgroundTag":"csi-plugin-transient-error","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}