{"record":{"id":"27eb4f55095e1b1b","repo":"hashicorp/nomad","slug":"volume-delete-failed-s-v","errorCode":null,"errorMessage":"volume delete failed: %s: %v","messagePattern":"volume delete failed: (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2974,"sourceCode":"\t\t}\n\n\t\tvol, ok := existing.(*structs.CSIVolume)\n\t\tif !ok {\n\t\t\treturn fmt.Errorf(\"volume row conversion error: %s\", id)\n\t\t}\n\n\t\t// The common case for a volume deregister is when the volume is\n\t\t// unused, but we can also let an operator intervene in the case where\n\t\t// allocations have been stopped but claims can't be freed because\n\t\t// ex. the plugins have all been removed.\n\t\tif vol.InUse() {\n\t\t\tif !force || !s.volSafeToForce(txn, vol) {\n\t\t\t\treturn fmt.Errorf(\"volume in use: %s\", id)\n\t\t\t}\n\t\t}\n\n\t\tif err = txn.Delete(TableCSIVolumes, existing); err != nil {\n\t\t\treturn fmt.Errorf(\"volume delete failed: %s: %v\", id, err)\n\t\t}\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{TableCSIVolumes, index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn txn.Commit()\n}\n\n// volSafeToForce checks if the any of the remaining allocations\n// are in a non-terminal state.\nfunc (s *StateStore) volSafeToForce(txn Txn, v *structs.CSIVolume) bool {\n\tv = v.Copy()\n\tvol, err := s.csiVolumeDenormalizeTxn(txn, nil, v)\n\tif err != nil {\n\t\treturn false\n\t}","sourceCodeStart":2956,"sourceCodeEnd":2992,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2956-L2992","documentation":"After the in-use check passes, CSIVolumeDeregister deletes the row with txn.Delete(TableCSIVolumes, existing). If that memdb delete fails, this error wraps the volume ID and cause and aborts the transaction. It is a low-level storage failure, not a domain rule.","triggerScenarios":"Volume deregister where the memdb/BoltDB delete of the volume row errors — disk full, I/O failure, or corrupted state store.","commonSituations":"Disk exhaustion on the Nomad server data_dir; BoltDB corruption after unclean shutdown; underlying storage hardware failure.","solutions":["Inspect the wrapped '%v' underlying cause in server logs","Free disk space / fix I/O issues on the server data_dir","Restart the server; restore from a known-good snapshot if corruption persists","Re-run the deregister once the state store is healthy"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Ensure server health before issuing deletes\n_, _, err := client.Status().Leader()\nif err != nil { return fmt.Errorf(\"cluster unhealthy: %w\", err) }","typeGuard":null,"tryCatchPattern":"if err := deregisterVolume(volID); err != nil && strings.Contains(err.Error(), \"volume delete failed\") {\n\t// storage failure: fix disk/state, then retry deregistration\n}","preventionTips":["Monitor data_dir disk usage and I/O health","Use clean server shutdowns to protect BoltDB","Keep regular raft snapshots","Re-run deregistration after storage recovery"],"tags":["go","state-store","csi","nomad","delete"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}