hashicorp/nomad · error

CSI.ControllerListVolumes: %v

Error message

CSI.ControllerListVolumes: %v

What it means

The CSI plugin's ControllerListVolumes RPC failed after Nomad's retry wrapper (3 attempts, exponential backoff, per-retry timeout CSIPluginRequestTimeout). Only ResourceExhausted-type codes are retried; any other error is returned wrapped as 'CSI.ControllerListVolumes: <err>' and treated as fatal for that RPC.

Source

Thrown at client/csi_endpoint.go:336

		// should retry with another controller instance
		return fmt.Errorf("CSI.ControllerListVolumes: %w: %v",
			nstructs.ErrCSIClientRPCRetryable, err)
	}
	defer plugin.Close()

	csiReq := req.ToCSIRequest()

	ctx, cancelFn := c.requestContext()
	defer cancelFn()

	// CSI ControllerListVolumes errors for timeout, codes.Unavailable and
	// codes.ResourceExhausted are retried; all other errors are fatal.
	cresp, err := plugin.ControllerListVolumes(ctx, csiReq,
		grpc_retry.WithPerRetryTimeout(CSIPluginRequestTimeout),
		grpc_retry.WithMax(3),
		grpc_retry.WithBackoff(grpc_retry.BackoffExponential(100*time.Millisecond)))
	if err != nil {
		return fmt.Errorf("CSI.ControllerListVolumes: %v", err)
	}

	resp.NextToken = cresp.NextToken
	resp.Entries = []*nstructs.CSIVolumeExternalStub{}

	for _, entry := range cresp.Entries {
		if entry.Volume == nil {
			return fmt.Errorf("CSI.ControllerListVolumes: plugin returned an invalid entry")
		}
		vol := &nstructs.CSIVolumeExternalStub{
			ExternalID:    entry.Volume.ExternalVolumeID,
			CapacityBytes: entry.Volume.CapacityBytes,
			VolumeContext: entry.Volume.VolumeContext,
			CloneID:       entry.Volume.ContentSource.CloneID,
			SnapshotID:    entry.Volume.ContentSource.SnapshotID,
		}
		if entry.Status != nil {
			vol.PublishedExternalNodeIDs = entry.Status.PublishedNodeIds

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the controller plugin container/task is running and its socket is reachable from the Nomad client
2. Check the wrapped gRPC code in plugin logs (e.g. Unimplemented means upgrade the plugin; PermissionDenied means fix cloud credentials)
3. Increase CSIPluginRequestTimeout tolerance or scale the backend if ResourceExhausted/throttling persists
4. Re-run the listing after the plugin stabilizes

Example fix

// before: controller plugin job crashed, list fails
job "ebs-controller" { ... count = 0 }
// after
job "ebs-controller" {
  group "controller" {
    task "plugin" { ... }
  }
}

plug, err := apiClient.Plugins().Get("aws-ebs-controller")
if err != nil || plug.ControllersHealthy < 1 {
    return fmt.Errorf("controller not available for ListVolumes")
}
if !plug.ControllerInfo.HasCapability(structs.CSIControllerSupportsListVolumes) {
    return fmt.Errorf("plugin does not implement ListVolumes")
}

func isListVolumesFatal(err error) bool {
    if err == nil { return false }
    var st *status.Status
    if errors.As(err, &st) {
        return st.Code() != codes.ResourceExhausted
    }
    return true
}

err := client.CSI().ControllerListVolumes(req)
if err != nil {
    var st *status.Status
    if errors.As(err, &st) && st.Code() == codes.ResourceExhausted {
        // backend throttled: back off and retry
        time.Sleep(5 * time.Second)
        return retryListVolumes(req)
    }
    if errors.As(err, &st) && st.Code() == codes.Unimplemented {
        return fmt.Errorf("plugin lacks ListVolumes support: %w", err)
    }
    return err
}

When it happens

Trigger: The plugin is unreachable (socket down/crashed), returns gRPC codes like Unavailable, Unimplemented, PermissionDenied, or InvalidArgument; per-attempt timeout exceeded on a slow backend; the cresp nil-handling is not involved here — any non-nil err is wrapped.

Common situations: Enumerating external volumes ('nomad volume status' external view) while the controller is overloaded or restarting; plugin lacks permissions to list volumes in the cloud account; plugin does not implement ListVolumes (Unimplemented); storage backend throttling causes timeouts.

Related errors

• CSI plugin failed to register: %w
• failed to probe plugin: %w
• CSI.ControllerExpandVolume: plugin did not return error or r
• CSI.ControllerDeleteVolume: %v
• %v: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/544763e4c7c082e7. Report an issue: GitHub.