{"record":{"id":"00cb4811eec1e774","repo":"hashicorp/nomad","slug":"volume-cannot-be-removed-from-unknown-node-without","errorCode":null,"errorMessage":"volume cannot be removed from unknown node without force=true","messagePattern":"volume cannot be removed from unknown node without force=true","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/host_volume_endpoint.go","lineNumber":712,"sourceCode":"\tvol, err := snap.HostVolumeByID(nil, ns, id, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not query host volume: %w\", err)\n\t}\n\tif vol == nil {\n\t\treturn fmt.Errorf(\"no such volume: %s\", id)\n\t}\n\tif len(vol.Allocations) > 0 {\n\t\tallocIDs := helper.ConvertSlice(vol.Allocations,\n\t\t\tfunc(a *structs.AllocListStub) string { return a.ID })\n\t\treturn fmt.Errorf(\"volume %s in use by allocations: %v\", id, allocIDs)\n\t}\n\n\t// serialize client RPC and raft write per volume ID\n\tindex, err := v.serializeCall(vol.ID, \"delete\", func() (uint64, error) {\n\t\tif err := v.deleteVolume(vol); err != nil {\n\t\t\tif structs.IsErrUnknownNode(err) || structs.IsErrNoNodeConn(err) {\n\t\t\t\tif !args.Force {\n\t\t\t\t\treturn 0, fmt.Errorf(\n\t\t\t\t\t\t\"volume cannot be removed from unknown node without force=true\")\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\treturn 0, err\n\t\t\t}\n\t\t}\n\t\t_, idx, err := v.srv.raftApply(structs.HostVolumeDeleteRequestType, args)\n\t\tif err != nil {\n\t\t\tv.logger.Error(\"raft apply failed\", \"error\", err, \"method\", \"delete\")\n\t\t\treturn 0, err\n\t\t}\n\t\treturn idx, nil\n\t})\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treply.Index = index","sourceCodeStart":694,"sourceCodeEnd":730,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/host_volume_endpoint.go#L694-L730","documentation":"This error comes from Nomad's host-volume DELETE endpoint (nomad/host_volume_endpoint.go:712). Deleting a host volume requires contacting the node that hosts it so the client can release/deregister the volume; when the node is unknown or unreachable (IsErrUnknownNode / IsErrNoNodeConn), the server refuses to delete unless args.Force is set, to avoid silently removing a volume that may still exist on the node.","triggerScenarios":"Calling the HostVolume.Delete RPC (or `nomad volume delete <id>`) for a volume whose registered node has been deregistered, is down, or has no established connection, without setting Force=true in the delete request.","commonSituations":"A node was drained, decommissioned, or its agent crashed and was removed from the cluster, leaving a stale host volume registration; operators then try to clean up the volume via API/CLI and get blocked. Also common after node ID changes or restoring state snapshots.","solutions":["Pass Force=true in the delete request (or run `nomad volume delete -force <volume_id>`) to remove the volume registration without contacting the node.","Re-register/restart the node (start the Nomad agent) so the server can connect and perform a graceful delete, then delete without force.","Deregister the dead node properly (`nomad node status` to find it, then purge/let it reap) if it should no longer exist, and clean up the volume afterward.","Verify the volume's NodeID (`nomad volume status <id>`) — if it points at a node that no longer exists, force delete is the correct operation."],"exampleFix":"// before\ncurl -X DELETE http://localhost:4646/v1/volumes/my-volume?namespace=default\n\n// after\ncurl -X DELETE 'http://localhost:4646/v1/volumes/my-volume?namespace=default&force=true'\n// or CLI: nomad volume delete -force my-volume","handlingStrategy":"validation","validationCode":"vol, _, err := client.HostVolumes().Get(ctx, \"my-volume\", nil)\nif err != nil { /* volume unknown; nothing to delete */ }\nnode, _, err := client.Nodes().Info(vol.NodeID, nil)\nif err == nil && node.Status != \"down\" {\n    // safe graceful delete\n    _, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, false, nil)\n} else {\n    // node unknown/down: force required\n    _, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, true, nil)\n}","typeGuard":"func nodeReachable(n *api.Node) bool {\n\treturn n != nil && n.Status != \"down\" && n.Status != \"disconnected\"\n}","tryCatchPattern":"_, err := client.HostVolumes().Delete(vol.ID, vol.Namespace, false, nil)\nif err != nil && strings.Contains(err.Error(), \"unknown node without force=true\") {\n\t_, err = client.HostVolumes().Delete(vol.ID, vol.Namespace, true, nil)\n}","preventionTips":["Check `nomad volume status <id>` for a stale NodeID before deleting.","Default automation to -force when the target node is deregistered or drained.","Purge dead nodes promptly so stale volume registrations don't accumulate.","Wrap deletes to retry once with force=true after an unknown-node error."],"tags":["nomad","host-volumes","storage","node-connectivity"],"backgroundTag":"stale-node-volume-delete","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}