{"record":{"id":"e1123b05a068eb04","repo":"hashicorp/nomad","slug":"deleting-job-scaling-events-failed-v","errorCode":null,"errorMessage":"deleting job scaling events failed: %v","messagePattern":"deleting job scaling events failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2114,"sourceCode":"\n\t// Delete the job submission\n\tif err := s.deleteJobSubmission(job, txn); err != nil {\n\t\treturn fmt.Errorf(\"deleting job submission failed: %v\", err)\n\t}\n\n\t// Delete any remaining job scaling policies\n\tif err := s.deleteJobScalingPolicies(index, job, txn); err != nil {\n\t\treturn fmt.Errorf(\"deleting job scaling policies failed: %v\", err)\n\t}\n\n\t// Delete any job recommendations\n\tif err := s.deleteRecommendationsByJob(index, txn, job); err != nil {\n\t\treturn fmt.Errorf(\"deleting job recommendatons failed: %v\", err)\n\t}\n\n\t// Delete the scaling events\n\tif _, err = txn.DeleteAll(\"scaling_event\", \"id\", namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"deleting job scaling events failed: %v\", err)\n\t}\n\n\t// Delete task group volume claims\n\tif err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"deleting job volume claims failed: %v\", err)\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{\"scaling_event\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// deleteJobScalingPolicies deletes any scaling policies associated with the job\nfunc (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {\n\titer, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)\n\tif err != nil {","sourceCodeStart":2096,"sourceCodeEnd":2132,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2096-L2132","documentation":"Nomad's StateStore wraps every underlying memdb transaction error when deleting all scaling events for a job inside deleteJobScaling (part of the job-deletion transaction). The raw memdb error is preserved via %v so the operator can see the root cause (e.g. the transaction was already aborted or the table write failed). If this fires, the whole job deletion txn is rolled back.","triggerScenarios":"Calling job deregistration (Job.Unregister / state store DeleteJob) when the memdb txn.DeleteAll on table \"scaling_event\" fails — typically because the surrounding transaction was already in an error/aborted state (e.g. a prior statement in the same txn failed), memory pressure/allocation failure in memdb, or table corruption.","commonSituations":"Deleting jobs whose scaling events exist; usually surfaced as a cascade after an earlier failure in the same deleteJob transaction rather than a standalone problem. Also seen in tests or tooling that drive StateStore directly with a closed or aborted txn.","solutions":["Inspect the wrapped %v cause in the log line — fix the root memdb/txn error it reports.","Retry the job deletion; memdb errors from a transient aborted txn resolve on a fresh transaction.","Check that no earlier error in the deleteJob path was swallowed, leaving the txn aborted before this DeleteAll.","If persistent, verify Raft/state store health (restore from backup or resync from a healthy peer)."],"exampleFix":"// before (caller ignoring wrapped cause)\nif _, err = txn.DeleteAll(\"scaling_event\", \"id\", namespace, jobID); err != nil {\n\treturn fmt.Errorf(\"deleting job scaling events failed: %v\", err)\n}\n// after (caller-side: ensure txn is valid and surface cause)\nif err := txn.Abort(); err != nil { /* txn already broken */ }\ntxn = s.db.Txn(true)\nif _, err = txn.DeleteAll(\"scaling_event\", \"id\", namespace, jobID); err != nil {\n\treturn fmt.Errorf(\"deleting job scaling events failed: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"// before calling job deregistration, verify job and txn state\nif job == nil || job.ID == \"\" || job.Namespace == \"\" {\n\treturn fmt.Errorf(\"job ID and namespace are required\")\n}\nif txn == nil {\n\treturn fmt.Errorf(\"job deletion requires a writable transaction\")\n}","typeGuard":"func isTxnStateErr(err error) bool {\n\treturn err != nil && strings.Contains(err.Error(), \"transaction\")\n}","tryCatchPattern":"err := state.DeleteJob(index, namespace, jobID)\nif err != nil {\n\tif strings.Contains(err.Error(), \"deleting job scaling events\") {\n\t\t// memdb txns are all-or-nothing: a fresh txn is safe to retry\n\t\ttime.Sleep(backoff)\n\t\terr = state.DeleteJob(index, namespace, jobID)\n\t}\n}","preventionTips":["Always retry state store mutations on a fresh transaction — memdb txns are atomic and abort as a unit.","Check the wrapped %v cause in logs before assuming the scaling_events table is the problem.","Do not reuse a transaction after any prior statement returned an error.","Monitor Raft/state store health so transient memdb failures are caught early."],"tags":["nomad","state-store","memdb","job-deletion"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}