hashicorp/nomad · error

controller plugin returned an internal error, check the plug

Error message

controller plugin returned an internal error, check the plugin allocation logs for more information: %v

What it means

Nomad's ControllerPublishVolume maps any gRPC codes.Internal error from the CSI controller plugin to this message. It indicates the plugin itself hit an unexpected internal condition while processing the publish request; Nomad deliberately does not guess details and directs the operator to the plugin's allocation logs. This is not a Nomad-side bug in most cases — the plugin or storage backend failed internally.

Source

Thrown at plugins/csi/client.go:331

	resp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			err = fmt.Errorf("volume %q or node %q could not be found: %v",
				req.ExternalID, req.NodeID, err)
		case codes.AlreadyExists:
			err = fmt.Errorf(
				"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v",
				req.ExternalID, req.NodeID, err)
		case codes.ResourceExhausted:
			err = fmt.Errorf("node %q has reached the maximum allowable number of attached volumes: %v",
				req.NodeID, err)
		case codes.FailedPrecondition:
			err = fmt.Errorf("volume %q is already published on another node and does not have MULTI_NODE volume capability: %v",
				req.ExternalID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}

	return &ControllerPublishVolumeResponse{
		PublishContext: maps.Clone(resp.PublishContext),
	}, nil
}

func (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}
	err := req.Validate()
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the controller plugin's allocation logs (nomad alloc logs <alloc_id> for the plugin task) — the message explicitly points there for the root cause.
2. Verify plugin ↔ Nomad CSI spec compatibility and upgrade the plugin to a version matching your Nomad release.
3. Check the plugin's storage-backend credentials and connectivity (e.g. cloud API permissions, network egress from the plugin).
4. Restart/redeploy the controller plugin task (nomad job restart or redeploy the plugin job) and retry the publish.

Example fix

// before: internal error surfaces in Nomad with no detail
// after: capture plugin-side detail and assert spec compatibility at registration
# nomad alloc logs -stderr <plugin_alloc_id>
# nomad plugin status <plugin_id>
// ensure plugin csi_spec_version_min/max covers your Nomad version before retrying

// Before publishing, verify plugin health and spec compatibility:
plugin, _, err := nomadClient.CSIPlugins().Get(nil, pluginID)
if err != nil { return err }
healthy := false
for _, c := range plugin.ControllersHealthy == nil || plugin.ControllersHealthy > 0; ; {}
// equivalent check:
if plugin.ControllersHealthy < 1 { return fmt.Errorf("controller plugin %s unhealthy", pluginID) }

func isPluginInternalError(err error) bool {
    return err != nil && strings.Contains(err.Error(),
        "controller plugin returned an internal error")
}

resp, err := csi.ControllerPublishVolume(ctx, req)
if err != nil {
    if isPluginInternalError(err) {
        // backoff-retry: internal plugin errors are often transient
        // (backend API hiccup). Check plugin logs before escalating.
        return retry.WithBackoff(ctx, 3, func() error {
            _, err = csi.ControllerPublishVolume(ctx, req)
            return err
        })
    }
    return err
}

When it happens

Trigger: c.ControllerPublishVolume() receives codes.Internal from the controller plugin RPC — plugin-side panic, unrecoverable backend API error, plugin configuration/version incompatibility, or the plugin's own dependencies failing.

Common situations: Mismatched CSI spec versions between Nomad and the plugin; a plugin container crash-looping or losing connectivity to the cloud storage API; expired cloud credentials causing the backend to return unexpected errors; plugin binary built against an older CSI spec.

Related errors

• CSI.ControllerListVolumes: plugin returned an invalid entry
• node %q has reached the maximum allowable number of attached
• volume %q is already published on another node and does not
• volume %q could not be found: %v
• volume %q snapshot source %q is not compatible with these pa

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ee4938d99695ffd4. Report an issue: GitHub.