{"record":{"id":"c4b9caab44539cc7","repo":"hashicorp/nomad","slug":"alloc-delete-failed-v","errorCode":null,"errorMessage":"alloc delete failed: %v","messagePattern":"alloc delete failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3791,"sourceCode":"\t\teval := existing.(*structs.Evaluation)\n\n\t\ttuple := structs.NamespacedID{\n\t\t\tID:        eval.JobID,\n\t\t\tNamespace: eval.Namespace,\n\t\t}\n\t\tjobs[tuple] = \"\"\n\t}\n\n\tfor _, alloc := range allocs {\n\t\traw, err := txn.First(\"allocs\", \"id\", alloc)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"alloc lookup failed: %v\", err)\n\t\t}\n\t\tif raw == nil {\n\t\t\tcontinue\n\t\t}\n\t\tif err := txn.Delete(\"allocs\", raw); err != nil {\n\t\t\treturn fmt.Errorf(\"alloc delete failed: %v\", err)\n\t\t}\n\n\t\t// Mark that we have made a successful modification to the allocs\n\t\t// table.\n\t\tallocsTableUpdated = true\n\n\t\tif err := s.deleteServiceRegistrationByAllocIDTxn(txn, index, alloc); err != nil {\n\t\t\treturn fmt.Errorf(\"service registration delete for alloc failed: %v\", err)\n\t\t}\n\t}\n\n\t// Update the indexes\n\tif evalsTableUpdated {\n\t\tif err := txn.Insert(\"index\", &IndexEntry{\"evals\", index}); err != nil {\n\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t}\n\t}\n\tif allocsTableUpdated {","sourceCodeStart":3773,"sourceCodeEnd":3809,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3773-L3809","documentation":"In the batch eval/alloc delete transaction, this error wraps a failure from txn.Delete(\"allocs\", raw) after the alloc was found. The alloc exists but the memdb write transaction cannot remove it, indicating store-level corruption or an invalid transaction state. The whole transaction aborts, so deletions do not commit and subsequent per-alloc service-registration cleanup does not run.","triggerScenarios":"Batch alloc deletion where a specific alloc record cannot be deleted — schema mismatch on the alloc object, corrupted allocs table, or an aborted memdb transaction.","commonSituations":"Snapshot restore corruption; alloc records written under an older Nomad schema being deleted by a newer server; disk corruption during GC.","solutions":["Inspect the wrapped cause for the actual delete failure","Verify/restore state store integrity (clean snapshot, healthy server restart)","Retry the operation once the server is healthy — the transactional design makes re-running safe","If a specific alloc is persistently undeletable, rebuild the server (rejoin as fresh member) to rebuild its tables"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Verify the alloc is still present and server healthy before deleting\nalloc, _, err := client.Allocs().GetAlloc(allocID, nil)\nif err != nil || alloc == nil { return nil }\nh, _, _ := client.Agent().Health(); if !h.Server.Ok { return fmt.Errorf(\"server unhealthy\") }","typeGuard":"func isAllocDeleteStoreError(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"alloc delete failed\")\n}","tryCatchPattern":"if err != nil {\n\tif isAllocDeleteStoreError(err) {\n\t\treturn fmt.Errorf(\"alloc delete aborted atomically; fix server, then retry: %w\", err)\n\t}\n\treturn err\n}","preventionTips":["Restore only validated, known-good snapshots","Avoid mixed-version clusters during alloc GC","Monitor memdb/BoltDB errors in server logs","Rebuild a server that persistently fails table deletes"],"tags":["nomad","state-store","allocs","memdb"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}