node %q has reached the maximum allowable number of attached

Error message

node %q has reached the maximum allowable number of attached volumes: %v

What it means

This error is produced in Nomad's CSI client wrapper when the storage provider's ControllerPublishVolume RPC returns a gRPC ResourceExhausted status. It means the node identified by req.NodeID has hit the storage backend's hard limit on the number of volumes that can be attached to it. Nomad re-wraps the provider error so the caller knows the publish failure is a capacity problem on that specific node, not a transient RPC fault.

Source

Thrown at plugins/csi/client.go:325

	err := req.Validate()
	if err != nil {
		return nil, err
	}

	pbrequest := req.ToCSIRepresentation()
	resp, err := c.controllerClient.ControllerPublishVolume(ctx, pbrequest, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.NotFound:
			err = fmt.Errorf("volume %q or node %q could not be found: %v",
				req.ExternalID, req.NodeID, err)
		case codes.AlreadyExists:
			err = fmt.Errorf(
				"volume %q is already published at node %q but with capabilities or a read_only setting incompatible with this request: %v",
				req.ExternalID, req.NodeID, err)
		case codes.ResourceExhausted:
			err = fmt.Errorf("node %q has reached the maximum allowable number of attached volumes: %v",
				req.NodeID, err)
		case codes.FailedPrecondition:
			err = fmt.Errorf("volume %q is already published on another node and does not have MULTI_NODE volume capability: %v",
				req.ExternalID, err)
		case codes.Internal:
			err = fmt.Errorf("controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
		return nil, err
	}

	return &ControllerPublishVolumeResponse{
		PublishContext: maps.Clone(resp.PublishContext),
	}, nil
}

func (c *client) ControllerUnpublishVolume(ctx context.Context, req *ControllerUnpublishVolumeRequest, opts ...grpc.CallOption) (*ControllerUnpublishVolumeResponse, error) {
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err

View on GitHub (pinned to 482b49bf1a)

Solutions

Move workloads (or their volume mounts) to other client nodes that are below the attach limit.
Use larger instance types or a storage backend with a higher per-node attach limit for that node pool.
Reduce per-task volume usage or share one multi-node-writable volume (MULTI_NODE capability) instead of many single-node volumes.
Verify for stale attachments: unpublish/detach leaked volumes from the node (fix leaked checkpointed attachments), then retry the publish.

Example fix

// before: task keeps failing to publish on node-1 after reschedules
//   node "i-abc" has reached the maximum allowable number of attached volumes
// after: constrain the job to a node pool with a higher attach limit
job "app" {
  group "g" {
    constraint {
      attribute = "${node.datacenter}"
      value     = "dc-high-iops"
    }
  }
}

Defensive patterns

Strategy: validation

Validate before calling

// Before scheduling a job that publishes volumes, confirm node attach headroom.
plugs, err := nomadClient.CSIPlugins().List(nil)
if err != nil { log.Fatal(err) }
// Check the node's csi capacity and the provider's per-instance attach limit
// (e.g. EBS: limits per instance type) before pinning jobs to that node;
// spread volume claims with a spread stanza instead of packing one node.

Type guard

func isVolumeLimitExceeded(err error) bool {
    return err != nil && strings.Contains(err.Error(),
        "maximum allowable number of attached volumes")
}

Try / catch

resp, err := csi.ControllerPublishVolume(ctx, req)
if err != nil {
    if isVolumeLimitExceeded(err) {
        // reschedule onto another node / request a different client
        return retryOnOtherNode(ctx, req)
    }
    return err
}

Prevention

Check per-node attach limits of your storage provider (EBS/Azure/PD) when sizing node pools.
Use Nomad spread constraints so volume-claiming tasks don't pile onto one client.
Run periodic volume status audits to catch leaked/stale attachments consuming slots.
Alert on repeated ControllerPublishVolume failures per node ID.

When it happens

Trigger: c.ControllerPublishVolume() is called and the CSI controller plugin responds with codes.ResourceExhausted — e.g. an EBS/Azure Disk/Google PD style per-instance attach limit was exceeded when trying to attach one more volume to req.NodeID.

Common situations: Clusters scheduling many volume-claiming tasks onto the same client node (e.g. AWS instances limited to ~28 EBS attachments); oversized instance types not used or instance-store volumes counted against the quota; repeated reschedules stacking orphaned attachments on one node.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/3d97c11f57c8b6d1. Report an issue: GitHub.