{"record":{"id":"9dfc514e3e089dcd","repo":"hashicorp/nomad","slug":"service-registration-delete-for-alloc-failed-v","errorCode":null,"errorMessage":"service registration delete for alloc failed: %v","messagePattern":"service registration delete for alloc failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":3799,"sourceCode":"\n\tfor _, alloc := range allocs {\n\t\traw, err := txn.First(\"allocs\", \"id\", alloc)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"alloc lookup failed: %v\", err)\n\t\t}\n\t\tif raw == nil {\n\t\t\tcontinue\n\t\t}\n\t\tif err := txn.Delete(\"allocs\", raw); err != nil {\n\t\t\treturn fmt.Errorf(\"alloc delete failed: %v\", err)\n\t\t}\n\n\t\t// Mark that we have made a successful modification to the allocs\n\t\t// table.\n\t\tallocsTableUpdated = true\n\n\t\tif err := s.deleteServiceRegistrationByAllocIDTxn(txn, index, alloc); err != nil {\n\t\t\treturn fmt.Errorf(\"service registration delete for alloc failed: %v\", err)\n\t\t}\n\t}\n\n\t// Update the indexes\n\tif evalsTableUpdated {\n\t\tif err := txn.Insert(\"index\", &IndexEntry{\"evals\", index}); err != nil {\n\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t}\n\t}\n\tif allocsTableUpdated {\n\t\tif err := txn.Insert(\"index\", &IndexEntry{\"allocs\", index}); err != nil {\n\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t}\n\t}\n\n\t// TODO: should we really be doing this here?  The only time this will affect the\n\t// status of a job is if it's the last eval and alloc for a client, at which point\n\t// the status of the job will already be \"dead\" from handling the alloc update.","sourceCodeStart":3781,"sourceCodeEnd":3817,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L3781-L3817","documentation":"After successfully deleting each alloc in the batch delete transaction, Nomad also deletes service registrations anchored to that alloc via deleteServiceRegistrationByAllocIDTxn. This error wraps any failure from that helper, aborting the transaction so the alloc deletions do not commit. It keeps the allocs table and the service registrations table consistent within one transaction.","triggerScenarios":"Batch alloc deletion when the service-registrations-by-alloc-id index is corrupted, the service_registrations table write fails, or the registration lookup/delete errors inside the transaction.","commonSituations":"Corrupted service registration index after snapshot restore; servers upgraded across the service-registration schema introduction with partially built indexes; deleting allocs that carry stale registrations on a degraded server.","solutions":["Inspect the wrapped cause from deleteServiceRegistrationByAllocIDTxn for the root failure","Verify the service_registrations table/index integrity against peers or a clean snapshot","Restore a known-good Raft snapshot or replace the degraded server","Re-run the batch delete after the store is healthy"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before deleting allocs, check for service registrations anchored to them\nsvcs, _, err := client.Services().Get(\"*\", nil) // or list registrations per alloc\nif err != nil { return err }\nh, _, _ := client.Agent().Health(); if !h.Server.Ok { return fmt.Errorf(\"server unhealthy\") }","typeGuard":"func isServiceRegDeleteError(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"service registration delete for alloc failed\")\n}","tryCatchPattern":"if err != nil {\n\tif isServiceRegDeleteError(err) {\n\t\treturn fmt.Errorf(\"service registration cleanup failed; verify service_registrations table then retry: %w\", err)\n\t}\n\treturn err\n}","preventionTips":["Complete upgrade migrations so the service_registrations index is fully built","Validate snapshots before restore","Check registration-related server logs during alloc GC","Retry batch deletes only when server health is green"],"tags":["nomad","state-store","service-registration","allocs"],"backgroundTag":"state-store-corruption","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}