{"record":{"id":"6f85b5f95f879947","repo":"hashicorp/nomad","slug":"deleting-job-versions-failed-v","errorCode":null,"errorMessage":"deleting job versions failed: %v","messagePattern":"deleting job versions failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2223,"sourceCode":"\tfor {\n\t\traw := iter.Next()\n\t\tif raw == nil {\n\t\t\tbreak\n\t\t}\n\n\t\t// Ensure the ID is an exact match\n\t\tj := raw.(*structs.Job)\n\t\tif j.ID != job.ID {\n\t\t\tcontinue\n\t\t}\n\n\t\tjobs = append(jobs, j)\n\t}\n\n\t// Do the deletes\n\tfor _, j := range jobs {\n\t\tif err := txn.Delete(\"job_version\", j); err != nil {\n\t\t\treturn fmt.Errorf(\"deleting job versions failed: %v\", err)\n\t\t}\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{\"job_version\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// upsertJobVersion inserts a job into its historic version table and limits the\n// number of job versions that are tracked.\nfunc (s *StateStore) upsertJobVersion(index uint64, job *structs.Job, txn *txn) error {\n\t// JobTrackedVersions really must not be zero here\n\tif err := s.config.Validate(); err != nil {\n\t\treturn err\n\t}\n","sourceCodeStart":2205,"sourceCodeEnd":2241,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2205-L2241","documentation":"deleteJobByVersion (historic job version cleanup) deletes collected old job versions one by one with txn.Delete on the \"job_version\" table. This wrapped error means one of those deletions failed inside memdb, aborting the version-pruning transaction and preventing the new job version from being persisted cleanly.","triggerScenarios":"Job registration/update (upsertJobVersion pruning history) when txn.Delete(\"job_version\", j) errors for a collected old version — aborted transaction, memdb internal error, or a row deleted concurrently.","commonSituations":"Very frequent job updates keeping many historical versions (pruning triggers on every registration); concurrent registrations of the same job; state churn under load.","solutions":["Inspect the wrapped cause for the specific failing version row.","Retry the job registration; version collection and deletion are recomputed on a fresh txn.","Avoid concurrent submissions of the same job (Nomad handles this via Raft serialization).","Check job_version table integrity if errors repeat."],"exampleFix":"// before\nif err := txn.Delete(\"job_version\", j); err != nil {\n\treturn fmt.Errorf(\"deleting job versions failed: %v\", err)\n}\n// after\nif err := txn.Delete(\"job_version\", j); err != nil {\n\treturn fmt.Errorf(\"deleting job versions failed: %w\", err)\n}","handlingStrategy":"retry","validationCode":"// check existing versions before re-registering a job\nversions, err := state.jobVersionByID(nil, nil, ns, jobID, false)\nif err != nil {\n\treturn err\n}\nmaxVersion := uint64(0)\nfor _, v := range versions {\n\tif v.Version > maxVersion { maxVersion = v.Version }\n}\n_ = maxVersion // submit new job with maxVersion+1","typeGuard":null,"tryCatchPattern":"err := state.UpsertJob(msgType, index, job)\nif err != nil && strings.Contains(err.Error(), \"deleting job versions\") {\n\ttime.Sleep(backoff)\n\treturn state.UpsertJob(msgType, index, job)\n}","preventionTips":["Retry job submissions — version pruning is recomputed on a fresh txn.","Avoid rapid concurrent submissions of the same job ID (Nomad/Raft normally serializes these).","Keep job history pruning healthy so the job_version table does not grow unboundedly.","Use %w wrapping so callers can errors.Is on memdb causes."],"tags":["nomad","state-store","job-versions","pruning"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}