hashicorp/nomad · error

failed to get tainted nodes for job '%s': %v

Error message

failed to get tainted nodes for job '%s': %v

What it means

After fetching the job's allocations, computeJobAllocs calls taintedNodes to find nodes that are drained/down containing those allocs. If that state store query errors, processing aborts with this wrapped error.

Source

Thrown at scheduler/generic_sched.go:341

	// Success!
	return true, nil
}

// computeJobAllocs is used to reconcile differences between the job,
// existing allocations and node status to update the allocations.
func (s *GenericScheduler) computeJobAllocs() error {
	// Lookup the allocations by JobID
	ws := memdb.NewWatchSet()
	allocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)
	if err != nil {
		return fmt.Errorf("failed to get allocs for job '%s': %v",
			s.eval.JobID, err)
	}

	// Determine the tainted nodes containing job allocs
	tainted, err := taintedNodes(s.state, allocs)
	if err != nil {
		return fmt.Errorf("failed to get tainted nodes for job '%s': %v",
			s.eval.JobID, err)
	}

	// Update the allocations which are in pending/running state on tainted
	// nodes to lost, but only if the scheduler has already marked them
	updateNonTerminalAllocsToLost(s.plan, tainted, allocs)

	r := reconciler.NewAllocReconciler(s.logger,
		genericAllocUpdateFn(s.ctx, s.stack, s.eval.ID),
		reconciler.ReconcilerState{
			Job:               s.job,
			JobID:             s.eval.JobID,
			JobIsBatch:        s.batch,
			DeploymentCurrent: s.deployment,
			ExistingAllocs:    allocs,
			EvalID:            s.eval.ID,
			EvalPriority:      s.eval.Priority,
		},

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped underlying error for the root cause
2. Retry the eval; Nomad reprocesses failed evaluations
3. Verify node and raft health with 'nomad node status' and 'nomad operator raft list-peers'
4. Restore from a raft snapshot if state corruption is suspected

nomad node status // identify down/drained nodes and confirm cluster responsiveness

// Inspect wrapped error in server logs; retry via eval reprocessing
nomad node status -drained
nomad eval list | grep -i failed

When it happens

Trigger: taintedNodes(s.state, allocs) returns an error — internally it does StateStore.NodesByIDs or node status lookups that fail due to state store/memdb errors during computeJobAllocs.

Common situations: Server state store instability especially with many tainted/drained nodes, raft restore, resource pressure.

Related errors

• failed to get ready nodes: %v
• eval broker is enabled; eval broker must be paused to delete
• failed to get job's allocations: %v
• failed to lookup node ID %q: %v
• failed to get job %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/37318c3cac98dece. Report an issue: GitHub.