{"record":{"id":"0528a627f7cfedf2","repo":"hashicorp/nomad","slug":"could-not-delete-volume-s-in-use-by-alloc-s","errorCode":null,"errorMessage":"could not delete volume %s in use by alloc %s","messagePattern":"could not delete volume (.+?) in use by alloc (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store_host_volumes.go","lineNumber":158,"sourceCode":"\tif err != nil {\n\t\treturn err\n\t}\n\tif obj != nil {\n\t\tvol := obj.(*structs.HostVolume)\n\n\t\t// we can't use AllocsByNodeTerminal because we only want to filter out\n\t\t// allocs that are client-terminal, not server-terminal\n\t\tallocs, err := s.AllocsByNode(nil, vol.NodeID)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"could not query allocs to check for host volume claims: %w\", err)\n\t\t}\n\t\tfor _, alloc := range allocs {\n\t\t\tif alloc.ClientTerminalStatus() {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tfor _, volReq := range alloc.Job.LookupTaskGroup(alloc.TaskGroup).Volumes {\n\t\t\t\tif vol.MatchesRequestSource(volReq, alloc) {\n\t\t\t\t\treturn fmt.Errorf(\"could not delete volume %s in use by alloc %s\",\n\t\t\t\t\t\tvol.ID, alloc.ID)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\terr = s.subtractVolumeFromQuotaUsageTxn(txn, index, vol)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\terr = txn.Delete(TableHostVolumes, vol)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"host volume delete: %w\", err)\n\t\t}\n\t}\n\n\treturn nil\n}","sourceCodeStart":140,"sourceCodeEnd":176,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store_host_volumes.go#L140-L176","documentation":"deleteHostVolumeTxn refuses to delete a host volume that is still referenced by a non-terminal allocation. It scans allocations using the volume's node and checks each running job's task group volumes via MatchesRequestSource; any live match aborts the delete so in-use storage is never removed from tracking.","triggerScenarios":"Calling DeleteHostVolume (or node-deregistration cleanup via deleteHostVolumesOnNode) while an allocation whose ClientTerminalStatus() is false still requests the volume in its task group's `volumes` block and matches via MatchesRequestSource.","commonSituations":"Deleting a host volume while jobs using it are still running or pending; stopping a job but deleting the volume before allocs reach terminal state; node drain still in progress with allocations not yet finished.","solutions":["Stop or drain the workloads using the volume and wait until all referencing allocations reach a terminal (failed/complete) status, then retry the delete.","Check `nomad alloc status` / `nomad job status` for allocs referencing the volume to identify what holds it.","Update the job specs to remove the volume reference, redeploy, and delete the volume after new allocs are placed.","If the volume truly must go now, migrate workloads to another volume first."],"exampleFix":"// before\nstateStore.DeleteHostVolume(idx, \"watcher\", volID, nil) // fails while alloc running\n\n// after\njob.Stop = true\nstateStore.UpsertJob(idx, job, nil)\n// wait for allocs to be terminal, then:\nstateStore.DeleteHostVolume(idx2, \"watcher\", volID, nil)","handlingStrategy":"validation","validationCode":"allocs, err := s.AllocsByNodeVolume(nil, vol.NodeID, vol.ID, false)\nif err != nil {\n    return err\n}\nfor _, alloc := range allocs {\n    if alloc.ClientTerminalStatus() {\n        continue\n    }\n    for _, volReq := range alloc.Job.LookupTaskGroup(alloc.TaskGroup).Volumes {\n        if vol.MatchesRequestSource(volReq, alloc) {\n            return fmt.Errorf(\"volume %s still in use by alloc %s\", vol.ID, alloc.ID)\n        }\n    }\n}","typeGuard":"func allocIsTerminal(a *structs.Allocation) bool {\n    return a != nil && a.ClientTerminalStatus()\n}","tryCatchPattern":"err := s.DeleteHostVolume(idx, \"uuid\", volID, nil)\nif err != nil && strings.Contains(err.Error(), \"in use by alloc\") {\n    // stop the referencing job, wait for terminal allocs, then retry\n}","preventionTips":["Always stop/drain jobs using the volume and confirm allocs are terminal before deleting.","Script the check: query allocs referencing the volume before issuing the delete.","Use drain with migrations so workloads move off before volume removal.","Make delete workflows idempotent and retry on in-use errors with backoff."],"tags":["nomad","state-store","host-volume","in-use-resource"],"backgroundTag":"resource-in-use","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}