hashicorp/nomad · error

failed to get tainted nodes for job '%s': %v

Error message

failed to get tainted nodes for job '%s': %v

What it means

After fetching the job's allocations, SysBatchScheduler.computeJobAllocs calls taintedNodes to identify dead/drained nodes holding those allocs so non-terminal allocs can be marked lost; this error wraps a failure of the taintedNodes state-store lookups. Without the tainted-node set the scheduler cannot correctly update allocs on failed nodes.

Source

Thrown at scheduler/scheduler_sysbatch.go:223

	s.stack.SetJob(job)
	s.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))
	return nil
}

// computeJobAllocs is used to reconcile differences between the job,
// existing allocations and node status to update the allocations.
func (s *SysBatchScheduler) computeJobAllocs() error {
	// Lookup the allocations by JobID
	ws := memdb.NewWatchSet()
	allocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)
	if err != nil {
		return fmt.Errorf("failed to get allocs for job '%s': %v", s.eval.JobID, err)
	}

	// Determine the tainted nodes containing job allocs
	tainted, err := taintedNodes(s.state, allocs)
	if err != nil {
		return fmt.Errorf("failed to get tainted nodes for job '%s': %v", s.eval.JobID, err)
	}

	// Update the allocations which are in pending/running state on tainted
	// nodes to lost.
	updateNonTerminalAllocsToLost(s.plan, tainted, allocs)

	// Split out terminal allocations
	live, term := structs.SplitTerminalAllocs(allocs)

	// Diff the required and existing allocations
	nr := reconciler.NewNodeReconciler(nil)
	r := nr.Compute(s.job, s.nodes, s.notReadyNodes, tainted, live, term)
	if s.logger.IsDebug() {
		s.logger.Debug("reconciled current state with desired state", r.Fields()...)
	}

	// Add all the allocs to stop
	for _, e := range r.Stop {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause and server logs to find the failing node lookup / state-store error.
2. Check that node IDs referenced by the job's allocs exist and are consistent (nomad node status).
3. Repair or restart the Nomad server state store; restore from backup if corrupt.
4. Re-run the evaluation after recovery (nomad job eval).

// preflight: every node referenced by the job's allocs is resolvable
for _, a := range allocs {
    if _, err := state.NodeByID(nil, a.NodeID); err != nil {
        return fmt.Errorf("node %s unreadable: %w", a.NodeID, err)
    }
}

func taintedNodesResolvable(s structs.State, allocs []*structs.Allocation) bool {
    _, err := taintedNodes(s, allocs)
    return err == nil
}

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get tainted nodes") {
        return retryWithBackoff(func() error { return sched.Process(eval) })
    }
    return err
}

When it happens

Trigger: process() -> computeJobAllocs(): taintedNodes(s.state, allocs) returns err != nil — NodeByID state-store lookup failure for one of the job's allocation node IDs.

Common situations: State-store errors while resolving nodes referenced by allocations; nodes deregistered/purged while state store is degraded; raft/disk failures on the leader.

Related errors

• failed to get job '%s': %v
• failed to get ready nodes: %v
• failed to get job node pool %q: %v
• failed to get scheduler configuration: %v
• failed to get allocs for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/90fa92f5987c7a8d. Report an issue: GitHub.