{"record":{"id":"a6e242412f5072ac","repo":"hashicorp/nomad","slug":"error-querying-csi-external-volumes-for-plugin-q","errorCode":null,"errorMessage":"Error querying CSI external volumes for plugin %q: %w","messagePattern":"Error querying CSI external volumes for plugin %q: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"command/volume_status_csi.go","lineNumber":101,"sourceCode":"\t\treturn fmt.Errorf(\"Error querying CSI plugins: %w\", err)\n\t}\n\n\tif len(plugins) == 0 {\n\t\treturn nil // No more output if we have no plugins\n\t}\n\n\tvar mErr *multierror.Error\n\tq := &api.QueryOptions{PerPage: 30} // TODO: tune page size\n\nNEXT_PLUGIN:\n\tfor _, plugin := range plugins {\n\t\tif !plugin.ControllerRequired || plugin.ControllersHealthy < 1 {\n\t\t\tcontinue // only controller plugins can support this query\n\t\t}\n\t\tfor {\n\t\t\texternalList, _, err := client.CSIVolumes().ListExternal(plugin.ID, q)\n\t\t\tif err != nil && !errors.Is(err, io.EOF) {\n\t\t\t\tmErr = multierror.Append(mErr, fmt.Errorf(\n\t\t\t\t\t\"Error querying CSI external volumes for plugin %q: %w\", plugin.ID, err))\n\t\t\t\t// we'll stop querying this plugin, but there may be more to\n\t\t\t\t// query, so report and set the error code but move on to the\n\t\t\t\t// next plugin\n\t\t\t\tcontinue NEXT_PLUGIN\n\t\t\t}\n\t\t\tif externalList == nil || len(externalList.Volumes) == 0 {\n\t\t\t\t// several plugins return EOF once you hit the end of the page,\n\t\t\t\t// rather than an empty list\n\t\t\t\tcontinue NEXT_PLUGIN\n\t\t\t}\n\t\t\tc.Ui.Output(\"\") // force a newline\n\t\t\trows := []string{\"External ID|Condition|Nodes\"}\n\t\t\tfor _, v := range externalList.Volumes {\n\t\t\t\tcondition := \"OK\"\n\t\t\t\tif v.IsAbnormal {\n\t\t\t\t\tcondition = fmt.Sprintf(\"Abnormal (%v)\", v.Status)\n\t\t\t\t}","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/volume_status_csi.go#L83-L119","documentation":"When listing external (controller-managed) CSI volumes per plugin via client.CSIVolumes().ListExternal, any error other than io.EOF is appended to a multierror with this message. The loop then skips to the next plugin (NEXT_PLUGIN), so one bad plugin does not abort the whole listing.","triggerScenarios":"csiVolumesList (verbose) iterates plugins where ControllerRequired is true and ControllersHealthy >= 1, and ListExternal(plugin.ID, q) returns a non-EOF error: controller RPC failure, plugin socket down, ACL denial, or timeout on the paginated query.","commonSituations":"A CSI plugin registers as healthy but its controller endpoint is broken (crash-looping controller pod), leaving stale registrations; also tokens lacking volume listing perms. Result is a partial listing plus an aggregated error at exit.","solutions":["Inspect the wrapped error and plugin ID; deregister/redeploy the unhealthy CSI plugin (nomad plugin status, then fix the controller workload)","Check the storage backend/controller reachability from the client node running the plugin","Verify ACL token has volume listing permissions; re-run to see if other plugins listed fine"],"exampleFix":"// controller container crashing\n// before: plugin registered, ListExternal times out\n// after: fix controller deployment, then\nnomad plugin status <plugin-id>  # ControllersHealthy >= 1\nnomad volume status -verbose","handlingStrategy":"fallback","validationCode":"for _, p := range plugins {\n    if p.ControllerRequired && p.ControllersHealthy < 1 {\n        // plugin will be skipped anyway; pre-warn users\n    }\n}","typeGuard":"func controllerCapable(p *api.CSIPluginListStub) bool { return p.ControllerRequired && p.ControllersHealthy >= 1 }","tryCatchPattern":"if err := cmd.Run(); err != nil {\n    if strings.Contains(err.Error(), \"Error querying CSI external volumes\") {\n        // output may be partial: re-check plugin health before trusting results\n    }\n}","preventionTips":["Monitor plugin ControllersHealthy; fix crash-looping controllers proactively","Treat this error as partial-data, not total failure","Deregister stale plugin instances (nomad system gc / plugin deregister)"],"tags":["csi","api","pagination","multierror"],"backgroundTag":"csi-controller-rpc-failure","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}