{"record":{"id":"ee4938d99695ffd4","repo":"hashicorp/nomad","slug":"controller-plugin-returned-an-internal-error-chec","errorCode":null,"errorMessage":"controller plugin returned an internal error, check the plugin allocation logs for more information: %v","messagePattern":"controller plugin returned an internal error, check the plugin allocation logs for more information: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugins/csi/client.go","lineNumber":331,"sourceCode":"\tresp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.NotFound:\n\t\t\terr = fmt.Errorf(\"volume %q or node %q could not be found: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.AlreadyExists:\n\t\t\terr = fmt.Errorf(\n\t\t\t\t\"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.ResourceExhausted:\n\t\t\terr = fmt.Errorf(\"node %q has reached the maximum allowable number of attached volumes: %v\",\n\t\t\t\treq.NodeID, err)\n\t\tcase codes.FailedPrecondition:\n\t\t\terr = fmt.Errorf(\"volume %q is already published on another node and does not have MULTI_NODE volume capability: %v\",\n\t\t\t\treq.ExternalID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\treturn &ControllerPublishVolumeResponse{\n\t\tPublishContext: maps.Clone(resp.PublishContext),\n\t}, nil\n}\n\nfunc (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn nil, err\n\t}\n\terr := req.Validate()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n","sourceCodeStart":313,"sourceCodeEnd":349,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L313-L349","documentation":"Nomad's ControllerPublishVolume maps any gRPC codes.Internal error from the CSI controller plugin to this message. It indicates the plugin itself hit an unexpected internal condition while processing the publish request; Nomad deliberately does not guess details and directs the operator to the plugin's allocation logs. This is not a Nomad-side bug in most cases — the plugin or storage backend failed internally.","triggerScenarios":"c.ControllerPublishVolume() receives codes.Internal from the controller plugin RPC — plugin-side panic, unrecoverable backend API error, plugin configuration/version incompatibility, or the plugin's own dependencies failing.","commonSituations":"Mismatched CSI spec versions between Nomad and the plugin; a plugin container crash-looping or losing connectivity to the cloud storage API; expired cloud credentials causing the backend to return unexpected errors; plugin binary built against an older CSI spec.","solutions":["Read the controller plugin's allocation logs (nomad alloc logs <alloc_id> for the plugin task) — the message explicitly points there for the root cause.","Verify plugin ↔ Nomad CSI spec compatibility and upgrade the plugin to a version matching your Nomad release.","Check the plugin's storage-backend credentials and connectivity (e.g. cloud API permissions, network egress from the plugin).","Restart/redeploy the controller plugin task (nomad job restart or redeploy the plugin job) and retry the publish."],"exampleFix":"// before: internal error surfaces in Nomad with no detail\n// after: capture plugin-side detail and assert spec compatibility at registration\n# nomad alloc logs -stderr <plugin_alloc_id>\n# nomad plugin status <plugin_id>\n// ensure plugin csi_spec_version_min/max covers your Nomad version before retrying","handlingStrategy":"retry","validationCode":"// Before publishing, verify plugin health and spec compatibility:\nplugin, _, err := nomadClient.CSIPlugins().Get(nil, pluginID)\nif err != nil { return err }\nhealthy := false\nfor _, c := range plugin.ControllersHealthy == nil || plugin.ControllersHealthy > 0; ; {}\n// equivalent check:\nif plugin.ControllersHealthy < 1 { return fmt.Errorf(\"controller plugin %s unhealthy\", pluginID) }","typeGuard":"func isPluginInternalError(err error) bool {\n    return err != nil && strings.Contains(err.Error(),\n        \"controller plugin returned an internal error\")\n}","tryCatchPattern":"resp, err := csi.ControllerPublishVolume(ctx, req)\nif err != nil {\n    if isPluginInternalError(err) {\n        // backoff-retry: internal plugin errors are often transient\n        // (backend API hiccup). Check plugin logs before escalating.\n        return retry.WithBackoff(ctx, 3, func() error {\n            _, err = csi.ControllerPublishVolume(ctx, req)\n            return err\n        })\n    }\n    return err\n}","preventionTips":["Pin plugin versions compatible with your Nomad CSI spec support.","Monitor controller plugin allocation health (nomad plugin status) and auto-restart on crash-loops.","Rotate storage-backend credentials before expiry and verify plugin egress to the backend API.","Nomad already retries RPCs; add your own bounded retry only for publish paths."],"tags":["csi","grpc","internal-error","storage"],"backgroundTag":"csi-plugin-internal-error","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}