{"record":{"id":"3f0af95cc87da328","repo":"hashicorp/nomad","slug":"job-insert-failed-v","errorCode":null,"errorMessage":"job insert failed: %v","messagePattern":"job insert failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/state/state_store.go","lineNumber":1913,"sourceCode":"\tif err := s.updateJobRecommendations(index, txn, existingJob, job); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job recommendations: %v\", err)\n\t}\n\n\tif err := s.updateJobCSIPlugins(index, job, existingJob, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job csi plugins: %v\", err)\n\t}\n\n\tif err := s.updateJobSubmission(index, sub, job.Namespace, job.ID, job.Version, txn); err != nil {\n\t\treturn fmt.Errorf(\"unable to update job submission: %v\", err)\n\t}\n\n\tif err := s.updatePreservedValues(job, existingJob, req); err != nil {\n\t\treturn fmt.Errorf(\"unable to update preserved values: %v\", err)\n\t}\n\n\t// Insert the job\n\tif err := txn.Insert(\"jobs\", job); err != nil {\n\t\treturn fmt.Errorf(\"job insert failed: %v\", err)\n\t}\n\tif err := txn.Insert(\"index\", &IndexEntry{\"jobs\", index}); err != nil {\n\t\treturn fmt.Errorf(\"index update failed: %v\", err)\n\t}\n\n\treturn nil\n}\n\n// CheckIdempotencyToken finds all children of the parent job ID and checks to\n// make sure none of them were dispatched with the idempotency token passed as\n// an argument. Returns the child job found, if any.\nfunc (s *StateStore) CheckIdempotencyToken(ns, parentID, idempotencyToken string) (*structs.Job, error) {\n\titer, err := s.JobsByIDPrefix(nil, ns, parentID, SortDefault)\n\tif err != nil {\n\t\treturn nil, errors.New(\"failed to retrieve jobs for idempotency check\")\n\t}\n\n\tfor {","sourceCodeStart":1895,"sourceCodeEnd":1931,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/state/state_store.go#L1895-L1931","documentation":"The final txn.Insert(\"jobs\", job) inside UpsertJob failed, meaning the fully prepared job object could not be written to the jobs table in the memdb transaction. This wraps the low-level table insert error and aborts the whole registration.","triggerScenarios":"Job registration where the jobs-table insert itself fails: schema/table mismatch (wrong object type for the table), memdb write errors, or transaction state already invalid from prior steps.","commonSituations":"Most often an internal invariant bug or corrupted in-memory state; occasionally seen right after failed upgrades or during severe memory pressure on servers.","solutions":["Inspect the wrapped %v cause in server logs","Restart the Nomad server to rebuild state from raft and retry registration","Verify Nomad version consistency across servers (mixed versions can break schema invariants)","Upgrade to the latest Nomad patch release if reproducible"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Basic sanity: job must validate and be structurally complete\nif err := job.Validate(); len(err) > 0 {\n  return fmt.Errorf(\"invalid job before register: %v\", err)\n}\nif job.ID == \"\" || job.Namespace == \"\" {\n  return fmt.Errorf(\"job ID and namespace are required\")\n}","typeGuard":"func isJobInsertError(err error) bool {\n  return err != nil && strings.Contains(err.Error(), \"job insert failed\")\n}","tryCatchPattern":"_, err := client.Jobs().Register(job, nil, nil, nil)\nif isJobInsertError(err) {\n  // server-side insert failed; restart/recover then retry\n  _, err = client.Jobs().Register(job, nil, nil, nil)\n}\nreturn err","preventionTips":["Read the wrapped cause - it distinguishes memdb vs schema issues","Restart servers showing repeated insert failures to rebuild from raft","Keep all servers on the same Nomad version","Alert on state-store errors in server logs","Scale server memory to avoid OOM-driven memdb failures"],"tags":["nomad","state-store","memdb"],"backgroundTag":"state-store-insert-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}