What causes "node plugin returned an error: %v" in hashicorp/nomad?

Check plugin health on the node (`nomad plugin status`, plugin alloc state) and retry once it is running — transient codes like Unavailable often resolve

hashicorp/nomad · error

node plugin returned an error: %v

Error message

node plugin returned an error: %v

What it means

The catch-all branch of NodeExpandVolume error handling in Nomad's CSI client. Any gRPC status code not explicitly mapped (e.g. Unavailable, DeadlineExceeded, Unknown) is surfaced as 'node plugin returned an error'. It is a generic pass-through of the plugin error from the volume expansion RPC.

Source

Thrown at plugins/csi/client.go:953

		code := status.Code(err)
		switch code {
		case codes.InvalidArgument:
			return nil, fmt.Errorf(
				"requested capabilities not compatible with volume %q: %v",
				req.ExternalVolumeID, err)
		case codes.NotFound:
			return nil, fmt.Errorf("%w: volume %q could not be found: %v",
				structs.ErrCSIClientRPCIgnorable, req.ExternalVolumeID, err)
		case codes.FailedPrecondition:
			return nil, fmt.Errorf("volume %q cannot be expanded while in use: %v", req.ExternalVolumeID, err)
		case codes.OutOfRange:
			return nil, fmt.Errorf(
				"unsupported capacity_range for volume %q: %v", req.ExternalVolumeID, err)
		case codes.Internal:
			return nil, fmt.Errorf(
				"node plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		default:
			return nil, fmt.Errorf("node plugin returned an error: %v", err)
		}
	}

	return &NodeExpandVolumeResponse{resp.GetCapacityBytes()}, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Check plugin health on the node (`nomad plugin status`, plugin alloc state) and retry once it is running — transient codes like Unavailable often resolve
Inspect the plugin allocation logs for the underlying error detail (%v contains the original gRPC error)
If persistent, redeploy/upgrade the CSI plugin and confirm the volume's node is correctly registered

Defensive patterns

Strategy: retry

Validate before calling

// shell: ensure the node plugin task is running before expanding
nomad node status <node-id> -verbose | grep -i csi
nomad plugin status

Try / catch

// Go: retry transient plugin failures with backoff
err := retry.Do(func() error {
    _, err := nodeClient.NodeExpandVolume(ctx, req)
    return err
}, retry.Attempts(3), retry.Delay(time.Second))

Prevention

Retry expansion on transient plugin unavailability with capped backoff
Alert on CSI plugin restarts/crash loops
Capture the wrapped %v error and correlate it with plugin alloc logs for diagnosis

When it happens

Trigger: NodeExpandVolume invoked and the plugin returns any non-special-cased gRPC status: plugin temporarily unavailable, context deadline exceeded, unknown plugin error.

Common situations: CSI plugin task restarting or temporarily down on the node; network partition between Nomad client and plugin socket; plugin reporting vendor-specific errors with uncommon codes.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/91ef07c78a9ebb6f. Report an issue: GitHub.