{"record":{"id":"b8930c66d337f5b5","repo":"hashicorp/nomad","slug":"missing-state-refresh-after-partial-commit","errorCode":null,"errorMessage":"missing state refresh after partial commit","messagePattern":"missing state refresh after partial commit","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"scheduler/generic_sched.go","lineNumber":320,"sourceCode":"\t// If we got a state refresh, try again since we have stale data.\n\t//\n\t// Clear the in-memory deployment because the plan was rejected and nothing\n\t// was persisted; the next process() iteration will reload from state or\n\t// generate a new deployment if needed.\n\tif newState != nil {\n\t\ts.logger.Debug(\"refresh forced\")\n\t\ts.state = newState\n\t\ts.deployment = nil\n\t\treturn false, nil\n\t}\n\n\t// Try again if the plan was not fully committed, potential conflict. The\n\t// above conditional means that we are always missing a state refresh after\n\t// a partial commit.\n\tfullCommit, expected, actual := result.FullCommit(s.plan)\n\tif !fullCommit {\n\t\ts.logger.Debug(\"plan didn't fully commit\", \"attempted\", expected, \"placed\", actual)\n\t\treturn false, fmt.Errorf(\"missing state refresh after partial commit\")\n\t}\n\n\t// Success!\n\treturn true, nil\n}\n\n// computeJobAllocs is used to reconcile differences between the job,\n// existing allocations and node status to update the allocations.\nfunc (s *GenericScheduler) computeJobAllocs() error {\n\t// Lookup the allocations by JobID\n\tws := memdb.NewWatchSet()\n\tallocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to get allocs for job '%s': %v\",\n\t\t\ts.eval.JobID, err)\n\t}\n\n\t// Determine the tainted nodes containing job allocs","sourceCodeStart":302,"sourceCodeEnd":338,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/scheduler/generic_sched.go#L302-L338","documentation":"After submitting a plan, if it was only partially committed (FullCommit reports fewer placements than expected), the scheduler expects the eval state to have been refreshed so a retry sees current data. Hitting this error is an invariant violation — the scheduler is about to retry without a state refresh and refuses, indicating an internal bug or unsafe plan flow.","triggerScenarios":"Plan apply returns fullCommit=false AND the scheduler's internal state-refresh logic did not run — typically caused by scheduler bugs, race conditions between plan appliers, or unexpected plan outcomes on heavily contended clusters.","commonSituations":"High plan contention with many concurrent evals, plan applier conflicts at scale, or Nomad version bugs in plan submission/retry logic.","solutions":["Upgrade Nomad — this error often indicates a fixed scheduler bug","Reduce plan contention (fewer concurrent jobs/evals, spread deployments)","Inspect server logs around 'plan didn't fully commit' for the conflict details","Retry the eval — Nomad re-enqueues it and may succeed once contention drops"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"nomad eval list | grep -i failed // watch for repeated partial-commit failures","typeGuard":null,"tryCatchPattern":"// Mostly an internal invariant: on occurrence, capture logs around\n// \"plan didn't fully commit\", reduce concurrency, upgrade Nomad,\n// and let Nomad retry the eval","preventionTips":["Upgrade to the latest Nomad version","Avoid extremely high plan contention (stagger large job submissions)","Monitor plan apply conflicts in server metrics","Report recurring occurrences with logs to Nomad maintainers"],"tags":["nomad","scheduler","plan","invariant"],"backgroundTag":"plan-partial-commit","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}