{"record":{"id":"5c5ea38e89698d21","repo":"hashicorp/nomad","slug":"failed-to-insert-job-into-job-version-table-v","errorCode":null,"errorMessage":"failed to insert job into job_version table: %v","messagePattern":"failed to insert job into job_version table: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":2244,"sourceCode":"\n\tif err := txn.Insert(\"index\", &IndexEntry{\"job_version\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// upsertJobVersion inserts a job into its historic version table and limits the\n// number of job versions that are tracked.\nfunc (s *StateStore) upsertJobVersion(index uint64, job *structs.Job, txn *txn) error {\n\t// JobTrackedVersions really must not be zero here\n\tif err := s.config.Validate(); err != nil {\n\t\treturn err\n\t}\n\n\t// Insert the job\n\tif err := txn.Insert(\"job_version\", job); err != nil {\n\t\treturn fmt.Errorf(\"failed to insert job into job_version table: %v\", err)\n\t}\n\n\tif err := txn.Insert(\"index\", &IndexEntry{\"job_version\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\t// Get all the historic jobs for this ID, except those with a VersionTag,\n\t// as they should always be kept. They are in Version order, high to low.\n\tall, err := s.jobVersionByID(txn, nil, job.Namespace, job.ID, false)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to look up job versions for %q: %v\", job.ID, err)\n\t}\n\n\t// If we are below the limit there is no GCing to be done\n\tif len(all) <= s.config.JobTrackedVersions {\n\t\treturn nil\n\t}\n","sourceCodeStart":2226,"sourceCodeEnd":2262,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L2226-L2262","documentation":"upsertJobVersion inserts the submitted job snapshot into the \"job_version\" table to record job history. This wrapped error means the row insert failed inside memdb, so the new job version is not recorded and the registration transaction aborts and rolls back.","triggerScenarios":"Job registration/update when txn.Insert(\"job_version\", job) errors — memdb allocation failure, already-aborted transaction, a duplicate primary key conflict (same Namespace/JobID/Version already present), or passing an invalid job struct.","commonSituations":"Job submissions with a Version colliding with an existing historical row (custom tooling or restores that mishandled version numbers); state store churn under load; snapshot/restore windows.","solutions":["Read the wrapped cause — a duplicate-key error indicates the Version already exists, so submit with a unique/incremented version.","Retry the registration on a fresh transaction if the cause is transient.","Ensure restored/migrated state does not reuse existing (Namespace, ID, Version) keys.","Verify job struct validity (non-empty namespace/ID/version) before upserting."],"exampleFix":"// before (caller re-submitting a job with an existing Version)\njob.Version = 3 // already stored\nerr := state.UpsertJob(msgType, index, job)\n// after\njob.Version = existingVersion + 1 // unique\nerr := state.UpsertJob(msgType, index, job)","handlingStrategy":"validation","validationCode":"// validate job identity and version uniqueness before upsert\nif job == nil || job.ID == \"\" || job.Namespace == \"\" {\n\treturn fmt.Errorf(\"job must have namespace and ID\")\n}\nexisting, _ := state.jobVersionByID(nil, nil, job.Namespace, job.ID, false)\nfor _, j := range existing {\n\tif j.Version == job.Version {\n\t\treturn fmt.Errorf(\"version %d already exists for %s/%s\", job.Version, job.Namespace, job.ID)\n\t}\n}","typeGuard":"func jobVersionIsUnique(existing []*structs.Job, version uint64) bool {\n\tfor _, j := range existing {\n\t\tif j.Version == version { return false }\n\t}\n\treturn true\n}","tryCatchPattern":"err := state.UpsertJob(msgType, index, job)\nif err != nil {\n\tif strings.Contains(err.Error(), \"job_version table\") {\n\t\t// possible duplicate key: resubmit with next version\n\t\tjob.Version++\n\t\treturn state.UpsertJob(msgType, index, job)\n\t}\n\treturn err\n}","preventionTips":["Never manually assign job Versions that may collide with stored history (restores/migrations).","Let Nomad assign versions via normal job submission flow.","Validate namespace/ID are non-empty before writing to the state store.","Use %w wrapping to distinguish duplicate-key vs transient memdb errors."],"tags":["nomad","state-store","job-versions","insert"],"backgroundTag":"duplicate-key-insert","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}