{"record":{"id":"ecbc799703a5db37","repo":"hashicorp/nomad","slug":"upsertjob-failed-w","errorCode":null,"errorMessage":"UpsertJob failed: %w","messagePattern":"UpsertJob failed: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"nomad/fsm.go","lineNumber":933,"sourceCode":"\n\tif current == nil {\n\t\treturn fmt.Errorf(\"job %q in namespace %q doesn't exist to be deregistered\", jobID, namespace)\n\t}\n\n\tstopped := current.Copy()\n\tstopped.Stop = true\n\tif submitTime != 0 {\n\t\tstopped.SubmitTime = submitTime\n\t}\n\n\t// Disable scaling policies to avoid monitoring stopped jobs\n\tscalingPolicies := stopped.GetScalingPolicies()\n\tfor _, policy := range scalingPolicies {\n\t\tpolicy.Enabled = false\n\t}\n\n\tif err := n.state.UpsertJobTxn(index, nil, stopped, tx); err != nil {\n\t\treturn fmt.Errorf(\"UpsertJob failed: %w\", err)\n\t}\n\n\treturn nil\n}\n\nfunc (n *nomadFSM) applyUpdateEval(msgType structs.MessageType, buf []byte, index uint64) any {\n\tdefer metrics.MeasureSince([]string{\"nomad\", \"fsm\", \"update_eval\"}, time.Now())\n\n\tvar req structs.EvalUpdateRequest\n\tif err := structs.Decode(buf, &req); err != nil {\n\t\tpanic(fmt.Errorf(\"failed to decode request: %v\", err))\n\t}\n\n\treturn n.upsertEvals(msgType, index, req.Evals)\n}\n\nfunc (n *nomadFSM) upsertEvals(msgType structs.MessageType, index uint64, evals []*structs.Evaluation) error {\n\tif err := n.state.UpsertEvals(msgType, index, evals); err != nil {","sourceCodeStart":915,"sourceCodeEnd":951,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/nomad/fsm.go#L915-L951","documentation":"After copying the current job and marking it stopped, handleJobDeregister re-inserts it via state.UpsertJobTxn within the same transaction. Failure there (schema/validation/constraint issues in the state store) is wrapped as \"UpsertJob failed\" and aborts the deregistration apply.","triggerScenarios":"UpsertJobTxn rejects the stopped job copy — e.g. job spec validation within the state store fails, index conflict, or memdb write error during FSM apply.","commonSituations":"Jobs written by newer Nomad versions then applied by an older server (schema drift), corrupted state, or transaction-level errors after earlier steps.","solutions":["Read the wrapped underlying error for the exact state-store rejection reason.","Ensure all servers run a compatible Nomad version (`nomad server members` shows versions); upgrade stragglers.","Restart the server to rebuild in-memory state from Raft snapshots.","Restore a consistent snapshot if state corruption is suspected."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := client.Jobs().Deregister(jobID, false, nil)\nif err != nil && strings.Contains(err.Error(), \"UpsertJob failed\") {\n    // inspect wrapped cause; upgrade mismatched servers and retry\n}","preventionTips":["Keep all servers on identical Nomad versions","Rebuild state via restart if corruption is suspected","Maintain snapshot backups"],"tags":["nomad","raft","fsm","state-store"],"backgroundTag":"job-upsert-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}