hashicorp/nomad · error

could not delete volume %s in use by alloc %s

Error message

could not delete volume %s in use by alloc %s

What it means

deleteHostVolumeTxn refuses to delete a host volume that is still referenced by a non-terminal allocation. It scans allocations using the volume's node and checks each running job's task group volumes via MatchesRequestSource; any live match aborts the delete so in-use storage is never removed from tracking.

Source

Thrown at nomad/state/state_store_host_volumes.go:158

	if err != nil {
		return err
	}
	if obj != nil {
		vol := obj.(*structs.HostVolume)

		// we can't use AllocsByNodeTerminal because we only want to filter out
		// allocs that are client-terminal, not server-terminal
		allocs, err := s.AllocsByNode(nil, vol.NodeID)
		if err != nil {
			return fmt.Errorf("could not query allocs to check for host volume claims: %w", err)
		}
		for _, alloc := range allocs {
			if alloc.ClientTerminalStatus() {
				continue
			}
			for _, volReq := range alloc.Job.LookupTaskGroup(alloc.TaskGroup).Volumes {
				if vol.MatchesRequestSource(volReq, alloc) {
					return fmt.Errorf("could not delete volume %s in use by alloc %s",
						vol.ID, alloc.ID)
				}
			}
		}

		err = s.subtractVolumeFromQuotaUsageTxn(txn, index, vol)
		if err != nil {
			return err
		}

		err = txn.Delete(TableHostVolumes, vol)
		if err != nil {
			return fmt.Errorf("host volume delete: %w", err)
		}
	}

	return nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop or drain the workloads using the volume and wait until all referencing allocations reach a terminal (failed/complete) status, then retry the delete.
2. Check `nomad alloc status` / `nomad job status` for allocs referencing the volume to identify what holds it.
3. Update the job specs to remove the volume reference, redeploy, and delete the volume after new allocs are placed.
4. If the volume truly must go now, migrate workloads to another volume first.

Example fix

// before
stateStore.DeleteHostVolume(idx, "watcher", volID, nil) // fails while alloc running

// after
job.Stop = true
stateStore.UpsertJob(idx, job, nil)
// wait for allocs to be terminal, then:
stateStore.DeleteHostVolume(idx2, "watcher", volID, nil)

allocs, err := s.AllocsByNodeVolume(nil, vol.NodeID, vol.ID, false)
if err != nil {
    return err
}
for _, alloc := range allocs {
    if alloc.ClientTerminalStatus() {
        continue
    }
    for _, volReq := range alloc.Job.LookupTaskGroup(alloc.TaskGroup).Volumes {
        if vol.MatchesRequestSource(volReq, alloc) {
            return fmt.Errorf("volume %s still in use by alloc %s", vol.ID, alloc.ID)
        }
    }
}

func allocIsTerminal(a *structs.Allocation) bool {
    return a != nil && a.ClientTerminalStatus()
}

err := s.DeleteHostVolume(idx, "uuid", volID, nil)
if err != nil && strings.Contains(err.Error(), "in use by alloc") {
    // stop the referencing job, wait for terminal allocs, then retry
}

When it happens

Trigger: Calling DeleteHostVolume (or node-deregistration cleanup via deleteHostVolumesOnNode) while an allocation whose ClientTerminalStatus() is false still requests the volume in its task group's `volumes` block and matches via MatchesRequestSource.

Common situations: Deleting a host volume while jobs using it are still running or pending; stopping a job but deleting the volume before allocs reach terminal state; node drain still in progress with allocations not yet finished.

Related errors

• host volume %s has nonexistent node ID %s
• error parsing: root should be an object
• cannot specify Accessor ID
• network already configured but not found in state
• eval not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0528a627f7cfedf2. Report an issue: GitHub.