What causes "volume %q cannot be expanded while in use: %v" in hashicorp/nomad?

Stop or migrate the allocations using the volume (drain the node or stop the job) so the volume is detached, then retry NodeExpandVolume

hashicorp/nomad · warning

volume %q cannot be expanded while in use: %v

Error message

volume %q cannot be expanded while in use: %v

What it means

Nomad's CSI client wraps gRPC status errors returned by the CSI node plugin during NodeExpandVolume. When the plugin replies with codes.FailedPrecondition, the client converts it to this message: the volume cannot be expanded because it is currently in use (mounted/attached). It signals a precondition violation of the CSI spec rather than a transient failure.

Source

Thrown at plugins/csi/client.go:945

	}
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}

	exReq := req.ToCSIRepresentation()
	resp, err := c.nodeClient.NodeExpandVolume(ctx, exReq, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.InvalidArgument:
			return nil, fmt.Errorf(
				"requested capabilities not compatible with volume %q: %v",
				req.ExternalVolumeID, err)
		case codes.NotFound:
			return nil, fmt.Errorf("%w: volume %q could not be found: %v",
				structs.ErrCSIClientRPCIgnorable, req.ExternalVolumeID, err)
		case codes.FailedPrecondition:
			return nil, fmt.Errorf("volume %q cannot be expanded while in use: %v", req.ExternalVolumeID, err)
		case codes.OutOfRange:
			return nil, fmt.Errorf(
				"unsupported capacity_range for volume %q: %v", req.ExternalVolumeID, err)
		case codes.Internal:
			return nil, fmt.Errorf(
				"node plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		default:
			return nil, fmt.Errorf("node plugin returned an error: %v", err)
		}
	}

	return &NodeExpandVolumeResponse{resp.GetCapacityBytes()}, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Stop or migrate the allocations using the volume (drain the node or stop the job) so the volume is detached, then retry NodeExpandVolume
Verify the storage provider/plugin supports online (in-use) volume expansion; if so, enable it in the plugin config
Check plugin allocation logs for the underlying FailedPrecondition detail to confirm which mount/attachment blocks expansion

Example fix

// before: expanding while job runs
nomad volume status my-vol // in use
nomad node expand ...
// after: stop consumers first
nomad job stop fs-job
nomad system gc
nomad volume expand my-volume -capacity-bytes=20G

Defensive patterns

Strategy: retry

Validate before calling

// shell pre-check before expanding
nomad volume status <volume-id>   # ensure 'Allocations' is empty / volume not claimed
nomad job inspect <job> | jq '.TaskGroups[].Volumes'

Try / catch

// Go: retry after detaching
_, err := client.Nodes().NodeExpandVolume(...)
if err != nil && strings.Contains(err.Error(), "cannot be expanded while in use") {
    // stop/drain consumers, then retry with backoff
}

Prevention

Check volume claim state (`nomad volume status`) before resizing
Stop or reschedule jobs consuming the volume prior to expansion
Prefer storage providers/plugins that support online volume expansion if zero-downtime resize is required

When it happens

Trigger: Calling NodeExpandVolume on a volume whose node plugin reports FailedPrecondition — typically because the volume is attached/mounted by a running allocation at expand time.

Common situations: Expanding a Nomad CSI volume while an allocation still has it mounted; online expansion unsupported by the storage provider; forgetting to stop the job (or reschedule it off the node) before resizing.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e3029e9369a961b4. Report an issue: GitHub.