What causes "volume %q is in use: %v" in hashicorp/nomad?

Stop the jobs/allocations using the volume (`nomad job stop`) and wait for claims to be released.

volume %q is in use: %v

Error message

volume %q is in use: %v

What it means

ControllerDeleteVolume got gRPC code FailedPrecondition from the CSI plugin: the volume req.ExternalVolumeID cannot be deleted because it is still in use — attachments/staged/published state exists on nodes. Nomad wraps it as "volume is in use".

Source

Thrown at plugins/csi/client.go:508

	return NewListVolumesResponse(resp), nil
}

func (c *client) ControllerDeleteVolume(ctx context.Context, req *ControllerDeleteVolumeRequest, opts ...grpc.CallOption) error {
	if err := c.ensureConnected(ctx); err != nil {
		return err
	}

	err := req.Validate()
	if err != nil {
		return err
	}
	creq := req.ToCSIRepresentation()
	_, err = c.controllerClient.DeleteVolume(ctx, creq, opts...)
	if err != nil {
		code := status.Code(err)
		switch code {
		case codes.FailedPrecondition:
			return fmt.Errorf("volume %q is in use: %v", req.ExternalVolumeID, err)
		case codes.Internal:
			return fmt.Errorf(
				"controller plugin returned an internal error, check the plugin allocation logs for more information: %v", err)
		}
	}
	return err
}

func (c *client) ControllerExpandVolume(ctx context.Context, req *ControllerExpandVolumeRequest, opts ...grpc.CallOption) (*ControllerExpandVolumeResponse, error) {
	if err := req.Validate(); err != nil {
		return nil, err
	}
	if err := c.ensureConnected(ctx); err != nil {
		return nil, err
	}

	exReq := req.ToCSIRepresentation()
	resp, err := c.controllerClient.ControllerExpandVolume(ctx, exReq, opts...)

View on GitHub (pinned to 482b49bf1a)

Solutions

Stop the jobs/allocations using the volume (`nomad job stop`) and wait for claims to be released.
Run `nomad volume status <id>` to find claimed nodes and wait for unstage/unpublish, or force-release claims with `nomad volume detach` (Nomad >=1.1).
If a node is gone, deregister the node or use `nomad volume detach <vol> <node_id>` to clear the stale attachment.
Retry ControllerDeleteVolume after all claims are cleared.

Example fix

// before (delete while allocations still claim it)
nomad volume delete <volume_id>
// after
nomad job stop <job_using_volume>
nomad volume detach <volume_id> <node_id>
nomad volume delete <volume_id>

Defensive patterns

Strategy: validation

Validate before calling

// Ensure no Nomad claims remain before deleting
vs, err := client.Volumes().Get(volumeID, nil)
if err != nil { return err }
if len(vs.CondensedConstraints) > 0 || len(vs.Reads) > 0 || len(vs.Writes) > 0 {
    return fmt.Errorf("volume %s still has claims; stop consumers first", volumeID)
}

Try / catch

err := client.ControllerDeleteVolume(ctx, req)
if err != nil && strings.Contains(err.Error(), "is in use") {
    // release claims: stop jobs / nomad volume detach, then retry delete
}

Prevention

Always stop the consuming job before deleting its volumes.
Use `nomad volume status` to confirm zero claims before deletion.
Clean up stale node attachments with `nomad volume detach` when nodes are lost.
Automate volume lifecycle so deletes follow claim release, not precede it.

When it happens

Trigger: Calling ControllerDeleteVolume while the volume is still attached or staged on one or more nodes — the plugin returns codes.FailedPrecondition per the CSI spec. In Nomad this happens when deregistering/deleting a volume that allocations still claim.

Common situations: Deleting a volume used by a running job; orphaned node attachments after a node was lost without unstaging; allocations stuck in a pending state holding the claim; forgotten volume claim from a stopped-but-not-GC'd allocation.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a9f9b0d19c584c0f. Report an issue: GitHub.