hashicorp/nomad · error

missing state refresh after partial commit

Error message

missing state refresh after partial commit

What it means

After submitting a plan, if it was only partially committed (FullCommit reports fewer placements than expected), the scheduler expects the eval state to have been refreshed so a retry sees current data. Hitting this error is an invariant violation — the scheduler is about to retry without a state refresh and refuses, indicating an internal bug or unsafe plan flow.

Source

Thrown at scheduler/generic_sched.go:320

	// If we got a state refresh, try again since we have stale data.
	//
	// Clear the in-memory deployment because the plan was rejected and nothing
	// was persisted; the next process() iteration will reload from state or
	// generate a new deployment if needed.
	if newState != nil {
		s.logger.Debug("refresh forced")
		s.state = newState
		s.deployment = nil
		return false, nil
	}

	// Try again if the plan was not fully committed, potential conflict. The
	// above conditional means that we are always missing a state refresh after
	// a partial commit.
	fullCommit, expected, actual := result.FullCommit(s.plan)
	if !fullCommit {
		s.logger.Debug("plan didn't fully commit", "attempted", expected, "placed", actual)
		return false, fmt.Errorf("missing state refresh after partial commit")
	}

	// Success!
	return true, nil
}

// computeJobAllocs is used to reconcile differences between the job,
// existing allocations and node status to update the allocations.
func (s *GenericScheduler) computeJobAllocs() error {
	// Lookup the allocations by JobID
	ws := memdb.NewWatchSet()
	allocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)
	if err != nil {
		return fmt.Errorf("failed to get allocs for job '%s': %v",
			s.eval.JobID, err)
	}

	// Determine the tainted nodes containing job allocs

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Upgrade Nomad — this error often indicates a fixed scheduler bug
2. Reduce plan contention (fewer concurrent jobs/evals, spread deployments)
3. Inspect server logs around 'plan didn't fully commit' for the conflict details
4. Retry the eval — Nomad re-enqueues it and may succeed once contention drops

nomad eval list | grep -i failed // watch for repeated partial-commit failures

// Mostly an internal invariant: on occurrence, capture logs around
// "plan didn't fully commit", reduce concurrency, upgrade Nomad,
// and let Nomad retry the eval

When it happens

Trigger: Plan apply returns fullCommit=false AND the scheduler's internal state-refresh logic did not run — typically caused by scheduler bugs, race conditions between plan appliers, or unexpected plan outcomes on heavily contended clusters.

Common situations: High plan contention with many concurrent evals, plan applier conflicts at scale, or Nomad version bugs in plan submission/retry logic.

Related errors

• scheduler resulted in an unexpected number of plans: %v
• plan exceeds max allocation
• eval broker is enabled; eval broker must be paused to delete
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b8930c66d337f5b5. Report an issue: GitHub.