hashicorp/nomad · warning · ErrCSIClientRPCRetryable

%v: %v

Error message

%v: %v

What it means

The CSI controller attach RPC (ClientCSI.ControllerAttachVolume) sent to the plugin returned an error containing "FailedPrecondition", so controllerPublishVolume wraps it with structs.ErrCSIClientRPCRetryable to mark it safe to retry. The plugin is telling Nomad the volume is in a state that doesn't allow attach yet (e.g. being detached, published elsewhere, or still creating).

Source

Thrown at nomad/csi_endpoint.go:593

	cReq := &cstructs.ClientCSIControllerAttachVolumeRequest{
		VolumeID:        vol.RemoteID(),
		ClientCSINodeID: req.ExternalNodeID,
		AttachmentMode:  req.AttachmentMode,
		AccessMode:      req.AccessMode,
		MountOptions:    csiVolumeMountOptions(vol.MountOptions),
		ReadOnly:        req.Claim == structs.CSIVolumeClaimRead,
		Secrets:         vol.Secrets,
		VolumeContext:   vol.Context,
	}
	cReq.PluginID = plug.ID
	cResp := &cstructs.ClientCSIControllerAttachVolumeResponse{}

	err = v.serializedControllerRPC(plug.ID, func() error {
		return v.srv.RPC(method, cReq, cResp)
	})
	if err != nil {
		if strings.Contains(err.Error(), "FailedPrecondition") {
			return fmt.Errorf("%v: %v", structs.ErrCSIClientRPCRetryable, err)
		}
		return err
	}
	resp.PublishContext = cResp.PublishContext
	return nil
}

func (v *CSIVolume) volAndPluginLookup(namespace, volID string) (*structs.CSIPlugin, *structs.CSIVolume, error) {
	state := v.srv.fsm.State()
	vol, err := state.CSIVolumeByID(nil, namespace, volID)
	if err != nil {
		return nil, nil, err
	}
	if vol == nil {
		return nil, nil, fmt.Errorf("volume not found: %s", volID)
	}
	if !vol.ControllerRequired {
		return nil, vol, nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the claim/operation after a short backoff — ErrCSIClientRPCRetryable signals the RPC is retryable and Nomad's internal retry loops will re-run it.
2. Check `nomad volume status <vol-id>` for conflicting claims; release stale claims from the old node if a detach is stuck.
3. Verify the plugin on the client is healthy and up to date; some plugins have known races in attach/detach serialization.
4. If it persists, restart the CSI plugin task on the client node to clear stuck controller state.

Example fix

// caller-side: treat as retryable
err := client.CSIVolumes().Claim(volID, ns, opts, req)
if err != nil && strings.Contains(err.Error(), "FailedPrecondition") {
    time.Sleep(5 * time.Second) // retry with backoff
    return retry()
}

if err != nil && strings.Contains(err.Error(), "FailedPrecondition") {
    time.Sleep(backoff) // ErrCSIClientRPCRetryable: safe to retry
    return retryClaim()
}

When it happens

Trigger: Raised when v.srv.RPC("ClientCSI.ControllerAttachVolume", ...) returns an error whose message contains "FailedPrecondition" (a gRPC status string surfaced by the plugin). Common with plugins that serialize attach/detach per volume: the volume is mid-detach from a previous node, or a conflicting attach is in flight.

Common situations: Rapid rescheduling of a job across nodes while the old detach hasn't completed; EBS/EFS/Azure-disk plugins rejecting attach while the volume is still detaching from another instance; leadership transitions leaving the serialized controller RPC queue in a transient state.

Related errors

• CSI plugin failed to register: %w
• failed to probe plugin: %w
• CSI.ControllerAttachVolume: %w: %v (wraps ErrCSIClientRPCRet
• CSI.ControllerExpandVolume: plugin did not return error or r
• CSI.ControllerDeleteVolume: %w: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/6e3a988378b5f5f2. Report an issue: GitHub.