{"record":{"id":"913d03aca38b0bba","repo":"hashicorp/nomad","slug":"job-delete-failed-v","errorCode":null,"errorMessage":"job delete failed: %v","messagePattern":"job delete failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2020,"sourceCode":"\t\t\t\tif modified {\n\t\t\t\t\t// Update the modify index\n\t\t\t\t\tpSummary.ModifyIndex = index\n\n\t\t\t\t\t// Insert the summary\n\t\t\t\t\tif err := txn.Insert(\"job_summary\", pSummary); err != nil {\n\t\t\t\t\t\treturn fmt.Errorf(\"job summary insert failed: %v\", err)\n\t\t\t\t\t}\n\t\t\t\t\tif err := txn.Insert(\"index\", &IndexEntry{\"job_summary\", index}); err != nil {\n\t\t\t\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\t// Delete the job\n\tif err := txn.Delete(\"jobs\", existing); err != nil {\n\t\treturn fmt.Errorf(\"job delete failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"jobs\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\t// Delete the job versions\n\tif err := s.deleteJobVersions(index, job, txn); err != nil {\n\t\treturn err\n\t}\n\n\t// Delete job deployments\n\tdeployments, err := s.DeploymentsByJobID(nil, namespace, job.ID, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"deployment lookup for job %s failed: %v\", job.ID, err)\n\t}\n\n\tdeploymentIDs := []string{}\n\tfor _, d := range deployments {","sourceCodeStart":2002,"sourceCodeEnd":2038,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2002-L2038","documentation":"DeleteJobTxn (nomad/state/state_store.go:2020) wraps a failure from txn.Delete(\"jobs\", existing) when removing the job row. The deletion failed inside the transaction, so the whole deregistration aborts and rolls back; nothing else (versions, deployments) is cleaned up.","triggerScenarios":"Calling DeleteJob/DeleteJobTxn when txn.Delete on the jobs table errors for the existing job object — e.g. MemDB internal error or a write conflict within the txn.","commonSituations":"Server-side in-memory DB problems while applying Job.Deregister; corrupted job raw entries from a bad restore; resource exhaustion on heavily loaded servers.","solutions":["Retry the deregistration; the txn rolled back so state is consistent","Check server logs for the wrapped %v root cause","Restart the Nomad server agent to rebuild state from Raft snapshots","Free resources / reduce load if errors correlate with server memory pressure"],"exampleFix":"// before\nerr := s.DeleteJobTxn(idx, ns, jobID, txn)\nreturn err\n// after\nif err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {\n    if strings.Contains(err.Error(), \"job delete failed\") {\n        return retryWithBackoff(func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })\n    }\n    return err\n}","handlingStrategy":"retry","validationCode":"// confirm the job exists and is stoppable before deregister\njob, err := stateStore.JobByID(nil, ns, jobID)\nif err != nil { return err }\nif job == nil { return nil } // avoids failed delete paths entirely","typeGuard":"func isDeletable(j interface{}) bool { j2, ok := j.(*structs.Job); return ok && j2 != nil && j2.ID != \"\" }","tryCatchPattern":"if err := s.DeleteJobTxn(idx, ns, jobID, txn); err != nil {\n    if strings.Contains(err.Error(), \"job delete failed\") {\n        return retry(3, backoff, func() error { return s.DeleteJobTxn(idx, ns, jobID, txn) })\n    }\n    return err\n}","preventionTips":["Keep servers under memory limits; delete failures correlate with resource exhaustion","Retry deregistrations; failed txns are atomic","Inspect wrapped %v errors in server logs","Verify state after any snapshot restore"],"tags":["state-store","raft","delete"],"backgroundTag":"memdb-transaction-delete-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}