hashicorp/nomad · warning

CSI client error (retryable)

Error message

CSI client error (retryable)

What it means

ErrCSIClientRPCRetryable marks a CSI controller RPC failure as retryable — typically a per-instance plugin issue (unavailable controller, timeout) where retrying, possibly against another controller instance, may succeed. Because the error is serialized over RPC, client code matches its message string rather than using errors.Is.

Source

Thrown at nomad/structs/errors.go:90

	// duplicates its message so the CLI can match it without importing structs.
	// Keep the two in sync.
	ErrResultPaginatorCreation = errors.New(errResultPaginatorCreation)

	ErrUnknownNode = errors.New(ErrUnknownNodePrefix)

	ErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)
	ErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)
	ErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)
	ErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)
	ErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)
	ErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)
	ErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)
	ErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)
	ErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)

	ErrCSIClientRPCIgnorable  = errors.New("CSI client error (ignorable)")
	ErrCSIClientRPCRetryable  = errors.New("CSI client error (retryable)")
	ErrCSIVolumeMaxClaims     = errors.New("volume max claims reached")
	ErrCSIVolumeUnschedulable = errors.New("volume is currently unschedulable")
	ErrCSIPluginInUse         = errors.New("plugin in use")
)

// IsErrNoLeader returns whether the error is due to there being no leader.
func IsErrNoLeader(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoLeader)
}

// IsErrNoRegionPath returns whether the error is due to there being no path to
// the given region.
func IsErrNoRegionPath(err error) bool {
	return err != nil && strings.Contains(err.Error(), errNoRegionPath)
}

// IsErrTokenNotFound returns whether the error is due to the passed token not
// being resolvable.
func IsErrTokenNotFound(err error) bool {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Simply retry the CSI operation — the error explicitly indicates retryability, ideally after a short backoff
2. Let Nomad retry with a different controller instance if one is available
3. Check the CSI plugin/controller health and logs if retries keep failing
4. Keep code matching via strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error()) because the sentinel does not survive RPC wrapping

Example fix

// before
if err := csiAttach(volID, allocID); err != nil {
    return err // fails the alloc hook permanently
}
// after
if err := csiAttach(volID, allocID); err != nil {
    if strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error()) {
        return retry.WithBackoff(func() error { return csiAttach(volID, allocID) })
    }
    return err
}

vol, _, err := client.CSIVolumes().Info(volID, nil)
if err == nil && vol.ControllersHealthy < 1 {
    return fmt.Errorf("volume %s has no healthy controllers; retry later", volID)
}

func isRetryableCSIErr(err error) bool {
    return err != nil && strings.Contains(err.Error(), structs.ErrCSIClientRPCRetryable.Error())
}

err := csiOperation(ctx)
if isRetryableCSIErr(err) {
    return retry.Do(func() error { return csiOperation(ctx) },
        retry.Attempts(3), retry.Delay(2*time.Second))
}
return err

When it happens

Trigger: CSI.ControllerValidateVolume/AttachVolume/DetachVolume/CreateVolume/ExpandVolume fail and the endpoint wraps the plugin error with fmt.Errorf("...: %w: %v", nstructs.ErrCSIClientRPCRetryable, err); csi_hook.go then string-matches to decide whether to retry the claim.

Common situations: CSI controller plugin is temporarily down or restarting; Nomad server connects to a stale plugin instance; storage backend throttles requests; network blip between Nomad client and plugin socket.

Related errors

• CSI.ControllerAttachVolume: VolumeID is required
• CSI.ControllerAttachVolume: ClientCSINodeID is required
• CSI.ControllerDetachVolume: VolumeID is required
• CSI.ControllerDetachVolume: ClientCSINodeID is required
• CSI.NodeDetachVolume: PluginID is required

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1385c98d967a7e56. Report an issue: GitHub.