{"record":{"id":"a3a295f76a8c278e","repo":"hashicorp/nomad","slug":"deleting-job-volume-claims-failed-v","errorCode":null,"errorMessage":"deleting job volume claims failed: %v","messagePattern":"deleting job volume claims failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2119,"sourceCode":"\n\t// Delete any remaining job scaling policies\n\tif err := s.deleteJobScalingPolicies(index, job, txn); err != nil {\n\t\treturn fmt.Errorf(\"deleting job scaling policies failed: %v\", err)\n\t}\n\n\t// Delete any job recommendations\n\tif err := s.deleteRecommendationsByJob(index, txn, job); err != nil {\n\t\treturn fmt.Errorf(\"deleting job recommendatons failed: %v\", err)\n\t}\n\n\t// Delete the scaling events\n\tif _, err = txn.DeleteAll(\"scaling_event\", \"id\", namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"deleting job scaling events failed: %v\", err)\n\t}\n\n\t// Delete task group volume claims\n\tif err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {\n\t\treturn fmt.Errorf(\"deleting job volume claims failed: %v\", err)\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{\"scaling_event\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// deleteJobScalingPolicies deletes any scaling policies associated with the job\nfunc (s *StateStore) deleteJobScalingPolicies(index uint64, job *structs.Job, txn *txn) error {\n\titer, err := s.ScalingPoliciesByJobTxn(nil, job.Namespace, job.ID, txn)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"getting job scaling policies for deletion failed: %v\", err)\n\t}\n\n\t// Put them into a slice so there are no safety concerns while actually\n\t// performing the deletes","sourceCodeStart":2101,"sourceCodeEnd":2137,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2101-L2137","documentation":"The StateStore wraps errors from deleteTaskGroupHostVolumeClaimByNamespaceAndJob, which removes task-group host volume claims for the job being deleted. This wrapped error means the volume-claim cleanup step of job deletion failed inside the same memdb transaction, aborting the overall job deletion.","triggerScenarios":"Job deregistration (DeleteJob path) when the nested deleteTaskGroupHostVolumeClaimByNamespaceAndJob call fails — usually an inner memdb delete/lookup error (aborted txn, missing index table entry, or allocation failure) on the host volume claim tables.","commonSituations":"Clusters where jobs use host volumes with dynamic claims; frequently a downstream symptom of an earlier failed statement in the same delete transaction. Also hit when custom tooling writes directly to the state store and leaves it inconsistent.","solutions":["Read the wrapped inner error message for the real cause (it names the failing claim delete/lookup).","Retry the job deregistration so the deletion runs in a fresh, valid transaction.","Verify host-volume/CSI claim table consistency (nomad volume status) and remove orphaned claims if present.","If persistent, check Raft store health and consider restoring state from a snapshot."],"exampleFix":"// before\nif err = s.deleteTaskGroupHostVolumeClaimByNamespaceAndJob(index, txn, namespace, jobID); err != nil {\n\treturn fmt.Errorf(\"deleting job volume claims failed: %v\", err)\n}\n// after (caller: retry deletion on a fresh txn)\nerr := s.StateStore.DeleteJob(index, namespace, jobID)\nif err != nil {\n\ttime.Sleep(backoff)\n\terr = s.StateStore.DeleteJob(index, namespace, jobID) // fresh txn\n}","handlingStrategy":"retry","validationCode":"// verify claims can be listed before deletion\n_, err := state.hostVolumeClaimsByNamespaceJobTxn(nil, namespace, jobID)\nif err != nil {\n\treturn fmt.Errorf(\"cannot read claims for job %s/%s: %w\", namespace, jobID, err)\n}","typeGuard":null,"tryCatchPattern":"err := state.DeleteJob(index, ns, jobID)\nif err != nil && strings.Contains(err.Error(), \"volume claims\") {\n\t// inner cause is preserved; retry with a fresh transaction\n\ttime.Sleep(backoff)\n\terr = state.DeleteJob(index, ns, jobID)\n}","preventionTips":["Retry job deregistration before investigating — claims cleanup is atomic with the rest of the delete.","Inspect the wrapped inner error to see whether the claim delete or a lookup failed.","Keep host volume/CSI state healthy so claim deletion does not fail under load.","Avoid running manual writes to claim tables concurrently with job deregistration."],"tags":["nomad","state-store","volumes","job-deletion"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}