{"record":{"id":"3d97c11f57c8b6d1","repo":"hashicorp/nomad","slug":"node-q-has-reached-the-maximum-allowable-number-o","errorCode":null,"errorMessage":"node %q has reached the maximum allowable number of attached volumes: %v","messagePattern":"node %q has reached the maximum allowable number of attached volumes: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugins/csi/client.go","lineNumber":325,"sourceCode":"\terr := req.Validate()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tpbrequest := req.ToCSIRepresentation()\n\tresp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)\n\tif err != nil {\n\t\tcode := status.Code(err)\n\t\tswitch code {\n\t\tcase codes.NotFound:\n\t\t\terr = fmt.Errorf(\"volume %q or node %q could not be found: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.AlreadyExists:\n\t\t\terr = fmt.Errorf(\n\t\t\t\t\"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v\",\n\t\t\t\treq.ExternalID, req.NodeID, err)\n\t\tcase codes.ResourceExhausted:\n\t\t\terr = fmt.Errorf(\"node %q has reached the maximum allowable number of attached volumes: %v\",\n\t\t\t\treq.NodeID, err)\n\t\tcase codes.FailedPrecondition:\n\t\t\terr = fmt.Errorf(\"volume %q is already published on another node and does not have MULTI_NODE volume capability: %v\",\n\t\t\t\treq.ExternalID, err)\n\t\tcase codes.Internal:\n\t\t\terr = fmt.Errorf(\"controller plugin returned an internal error, check the plugin allocation logs for more information: %v\", err)\n\t\t}\n\t\treturn nil, err\n\t}\n\n\treturn &ControllerPublishVolumeResponse{\n\t\tPublishContext: maps.Clone(resp.PublishContext),\n\t}, nil\n}\n\nfunc (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {\n\tif err := c.ensureConnected(ctx); err != nil {\n\t\treturn nil, err","sourceCodeStart":307,"sourceCodeEnd":343,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/plugins/csi/client.go#L307-L343","documentation":"This error is produced in Nomad's CSI client wrapper when the storage provider's ControllerPublishVolume RPC returns a gRPC ResourceExhausted status. It means the node identified by req.NodeID has hit the storage backend's hard limit on the number of volumes that can be attached to it. Nomad re-wraps the provider error so the caller knows the publish failure is a capacity problem on that specific node, not a transient RPC fault.","triggerScenarios":"c.ControllerPublishVolume() is called and the CSI controller plugin responds with codes.ResourceExhausted — e.g. an EBS/Azure Disk/Google PD style per-instance attach limit was exceeded when trying to attach one more volume to req.NodeID.","commonSituations":"Clusters scheduling many volume-claiming tasks onto the same client node (e.g. AWS instances limited to ~28 EBS attachments); oversized instance types not used or instance-store volumes counted against the quota; repeated reschedules stacking orphaned attachments on one node.","solutions":["Move workloads (or their volume mounts) to other client nodes that are below the attach limit.","Use larger instance types or a storage backend with a higher per-node attach limit for that node pool.","Reduce per-task volume usage or share one multi-node-writable volume (MULTI_NODE capability) instead of many single-node volumes.","Verify for stale attachments: unpublish/detach leaked volumes from the node (fix leaked checkpointed attachments), then retry the publish."],"exampleFix":"// before: task keeps failing to publish on node-1 after reschedules\n//   node \"i-abc\" has reached the maximum allowable number of attached volumes\n// after: constrain the job to a node pool with a higher attach limit\njob \"app\" {\n  group \"g\" {\n    constraint {\n      attribute = \"${node.datacenter}\"\n      value     = \"dc-high-iops\"\n    }\n  }\n}","handlingStrategy":"validation","validationCode":"// Before scheduling a job that publishes volumes, confirm node attach headroom.\nplugs, err := nomadClient.CSIPlugins().List(nil)\nif err != nil { log.Fatal(err) }\n// Check the node's csi capacity and the provider's per-instance attach limit\n// (e.g. EBS: limits per instance type) before pinning jobs to that node;\n// spread volume claims with a spread stanza instead of packing one node.","typeGuard":"func isVolumeLimitExceeded(err error) bool {\n    return err != nil && strings.Contains(err.Error(),\n        \"maximum allowable number of attached volumes\")\n}","tryCatchPattern":"resp, err := csi.ControllerPublishVolume(ctx, req)\nif err != nil {\n    if isVolumeLimitExceeded(err) {\n        // reschedule onto another node / request a different client\n        return retryOnOtherNode(ctx, req)\n    }\n    return err\n}","preventionTips":["Check per-node attach limits of your storage provider (EBS/Azure/PD) when sizing node pools.","Use Nomad spread constraints so volume-claiming tasks don't pile onto one client.","Run periodic volume status audits to catch leaked/stale attachments consuming slots.","Alert on repeated ControllerPublishVolume failures per node ID."],"tags":["csi","grpc","storage","resource-limit"],"backgroundTag":"csi-node-volume-limit-exceeded","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}