{"record":{"id":"66c4781e0b39c479","repo":"hashicorp/nomad","slug":"failed-to-delete-job-v-d-from-job-version","errorCode":null,"errorMessage":"failed to delete job %v (%d) from job_version","messagePattern":"failed to delete job (.+?) \\((.+?)\\) from job_version","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2283,"sourceCode":"\tstableIdx := -1\n\tfor i, j := range all {\n\t\tif j.Stable {\n\t\t\tstableIdx = i\n\t\t\tbreak\n\t\t}\n\t}\n\n\t// If the stable job is the oldest version, do a swap to bring it into the\n\t// keep set.\n\tmax := s.config.JobTrackedVersions\n\tif stableIdx == max {\n\t\tall[max-1], all[max] = all[max], all[max-1]\n\t}\n\n\t// Delete the oldest one\n\td := all[max]\n\tif err := txn.Delete(\"job_version\", d); err != nil {\n\t\treturn fmt.Errorf(\"failed to delete job %v (%d) from job_version\", d.ID, d.Version)\n\t}\n\n\treturn nil\n}\n\n// GetJobSubmissions returns an iterator that contains all job submissions\n// stored within state. This is not currently exposed via RPC and is only used\n// for snapshot persist and restore functionality.\nfunc (s *StateStore) GetJobSubmissions(ws memdb.WatchSet) (memdb.ResultIterator, error) {\n\ttxn := s.db.ReadTxn()\n\n\t// Walk the entire table to get all job submissions.\n\titer, err := txn.Get(TableJobSubmission, indexID)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"job submissions lookup failed: %v\", err)\n\t}\n\tws.Add(iter.WatchCh())\n","sourceCodeStart":2265,"sourceCodeEnd":2301,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2265-L2301","documentation":"This error occurs when the state store fails to delete the oldest tracked job version row from the job_version table during version trimming after a job update. The txn.Delete on the memdb table returned an error, so GC could not complete. It signals an internal persistence problem, since the row was just read in the same transaction.","triggerScenarios":"deleteJobVersion trimming the oldest version when len(job versions) > JobTrackedVersions and txn.Delete(\"job_version\", ...) errors.","commonSituations":"Very rare: memdb write failures, transaction conflicts, or corruption; typically seen with internal state store bugs during frequent job updates.","solutions":["Inspect the message's job ID and version to identify the affected row","Retry the job operation (txn is aborted atomically)","If reproducible, snapshot/restore Raft state and report to Nomad with the wrapped cause","Reduce job update churn so trimming happens less often"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := register(); err != nil && strings.Contains(err.Error(), \"from job_version\") {\n  time.Sleep(backoff)\n  err = register() // txn aborted atomically, safe to retry\n}","preventionTips":["Retry registration after transient failures","Reduce excessive job update churn","Report persistent occurrences to Nomad with server logs"],"tags":["nomad","state-store","job-versions","delete"],"backgroundTag":"state-store-write-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}