node has reached the maximum allowable number of attached…

Error message

node %q has reached the maximum allowable number of attached volumes: %v

What it means

This error is produced in Nomad's CSI client wrapper when the storage provider's ControllerPublishVolume RPC returns a gRPC ResourceExhausted status. It means the node identified by req.NodeID has hit the storage backend's hard limit on the number of volumes that can be attached to it. Nomad re-wraps the provider error so the caller knows the publish failure is a capacity problem on that specific node, not a transient RPC fault.

Solutions

Move workloads (or their volume mounts) to other client nodes that are below the attach limit.
Use larger instance types or a storage backend with a higher per-node attach limit for that node pool.
Reduce per-task volume usage or share one multi-node-writable volume (MULTI_NODE capability) instead of many single-node volumes.
Verify for stale attachments: unpublish/detach leaked volumes from the node (fix leaked checkpointed attachments), then retry the publish.

Example fix

// before: task keeps failing to publish on node-1 after reschedules
//   node "i-abc" has reached the maximum allowable number of attached volumes
// after: constrain the job to a node pool with a higher attach limit
job "app" {
  group "g" {
    constraint {
      attribute = "${node.datacenter}"
      value     = "dc-high-iops"
    }
  }
}

Defensive patterns

Strategy: validation

Validate before calling

// Before scheduling a job that publishes volumes, confirm node attach headroom.
plugs, err := nomadClient.CSIPlugins().List(nil)
if err != nil { log.Fatal(err) }
// Check the node's csi capacity and the provider's per-instance attach limit
// (e.g. EBS: limits per instance type) before pinning jobs to that node;
// spread volume claims with a spread stanza instead of packing one node.

Type guard

func isVolumeLimitExceeded(err error) bool {
    return err != nil && strings.Contains(err.Error(),
        "maximum allowable number of attached volumes")
}

Try / catch

resp, err := csi.ControllerPublishVolume(ctx, req)
if err != nil {
    if isVolumeLimitExceeded(err) {
        // reschedule onto another node / request a different client
        return retryOnOtherNode(ctx, req)
    }
    return err
}

Prevention

Check per-node attach limits of your storage provider (EBS/Azure/PD) when sizing node pools.
Use Nomad spread constraints so volume-claiming tasks don't pile onto one client.
Run periodic volume status audits to catch leaked/stale attachments consuming slots.
Alert on repeated ControllerPublishVolume failures per node ID.

When it happens

Trigger: c.ControllerPublishVolume() is called and the CSI controller plugin responds with codes.ResourceExhausted — e.g. an EBS/Azure Disk/Google PD style per-instance attach limit was exceeded when trying to attach one more volume to req.NodeID.

Common situations: Clusters scheduling many volume-claiming tasks onto the same client node (e.g. AWS instances limited to ~28 EBS attachments); oversized instance types not used or instance-store volumes counted against the quota; repeated reschedules stacking orphaned attachments on one node.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3d97c11f57c8b6d1. Report an issue: GitHub.

Appendix: source

Thrown at plugins/csi/client.go:325

	err := req.Validate()
	if err != nil {
		return nil, err
	}

	pbrequest := req.ToCSIRepresentation()
	resp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			err = fmt.Errorf("volume %q or node %q could not be found: %v",
				req.ExternalID, req.NodeID, err)
		case codes.AlreadyExists:
			err = fmt.Errorf(
				"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v",
				req.ExternalID, req.NodeID, err)
		case codes.ResourceExhausted:
			err = fmt.Errorf("node %q has reached the maximum allowable number of attached volumes: %v",
				req.NodeID, err)
		case codes.FailedPrecondition:
			err = fmt.Errorf("volume %q is already published on another node and does not have MULTI_NODE volume capability: %v",
				req.ExternalID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}

	return &ControllerPublishVolumeResponse{
		PublishContext: maps.Clone(resp.PublishContext),
	}, nil
}

func (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err

View on GitHub (pinned to 482b49bf1a)