{"record":{"id":"abef02058254ccd3","repo":"hashicorp/nomad","slug":"deleting-scaling-policy-failed-v","errorCode":null,"errorMessage":"deleting scaling policy failed: %v","messagePattern":"deleting scaling policy failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2150,"sourceCode":"\tif err != nil {\n\t\treturn fmt.Errorf(\"getting job scaling policies for deletion failed: %v\", err)\n\t}\n\n\t// Put them into a slice so there are no safety concerns while actually\n\t// performing the deletes\n\tpolicies := []any{}\n\tfor {\n\t\traw := iter.Next()\n\t\tif raw == nil {\n\t\t\tbreak\n\t\t}\n\t\tpolicies = append(policies, raw)\n\t}\n\n\t// Do the deletes\n\tfor _, p := range policies {\n\t\tif err := txn.Delete(\"scaling_policy\", p); err != nil {\n\t\t\treturn fmt.Errorf(\"deleting scaling policy failed: %v\", err)\n\t\t}\n\t}\n\n\tif len(policies) > 0 {\n\t\tif err := txn.Insert(\"index\", &IndexEntry{\"scaling_policy\", index}); err != nil {\n\t\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (s *StateStore) deleteJobSubmission(job *structs.Job, txn *txn) error {\n\t// find submissions associated with job\n\tremove := *set.NewHashSet[*structs.JobSubmission, string](s.config.JobTrackedVersions)\n\n\titer, err := txn.Get(\"job_submission\", \"id_prefix\", job.Namespace, job.ID)\n\tif err != nil {\n\t\treturn err","sourceCodeStart":2132,"sourceCodeEnd":2168,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2132-L2168","documentation":"After collecting a job's scaling policies into a slice, deleteJobScalingPolicies deletes each one with txn.Delete on the \"scaling_policy\" table. This wrapped error means one of those row deletions failed inside memdb, aborting the job-deletion transaction.","triggerScenarios":"Job deregistration when txn.Delete(\"scaling_policy\", p) returns an error for one of the collected policies — memdb internal error, aborted transaction, or a row that vanished mid-transaction causing a delete mismatch.","commonSituations":"Concurrent modification of scaling policies while a job is being deleted; state store churn under load; tests that delete policies between the list and delete phases.","solutions":["Read the wrapped cause for the specific failing policy.","Retry the job deletion; the list-then-delete snapshot is rebuilt on retry.","Avoid concurrent policy writes against the same job while deregistering it.","If persistent, verify the scaling_policy table's integrity in the state store."],"exampleFix":"// before\nif err := txn.Delete(\"scaling_policy\", p); err != nil {\n\treturn fmt.Errorf(\"deleting scaling policy failed: %v\", err)\n}\n// after (caller retrying with a fresh transaction)\nfor _, p := range policies {\n\tif err := txn.Delete(\"scaling_policy\", p); err != nil {\n\t\treturn fmt.Errorf(\"deleting scaling policy failed: %w\", err)\n\t}\n}","handlingStrategy":"retry","validationCode":"// confirm policies are readable before deregistering\n_, err := state.ScalingPoliciesByJob(nil, ns, jobID)\nif err != nil {\n\treturn err\n}","typeGuard":null,"tryCatchPattern":"err := state.DeleteJob(index, ns, jobID)\nif err != nil && strings.Contains(err.Error(), \"deleting scaling policy\") {\n\ttime.Sleep(backoff)\n\treturn state.DeleteJob(index, ns, jobID)\n}","preventionTips":["Avoid concurrent scaling-policy writes against a job being deleted.","Retry deregistration — list-then-delete is recomputed atomically per txn.","Surface the wrapped cause (%w) so callers can distinguish transient vs structural errors.","Check for orphaned scaling_policy rows after failed deletions."],"tags":["nomad","state-store","scaling-policies","job-deletion"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}