What causes "failed to get ready nodes: %v" in hashicorp/nomad?

Inspect the wrapped cause and server logs for the underlying state-store error.

failed to get ready nodes: %v

Error message

failed to get ready nodes: %v

What it means

SysBatchScheduler.process calls readyNodesInDCsAndPool to list ready nodes in the job's datacenters and node pool; this error wraps any failure from that state-store query. It means the scheduler could not determine the eligible client nodes for placement.

Source

Thrown at scheduler/scheduler_sysbatch.go:127

	var err error
	ws := memdb.NewWatchSet()
	s.job, err = s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID)
	if err != nil {
		return false, fmt.Errorf("failed to get job '%s': %v", s.eval.JobID, err)
	}

	numTaskGroups := 0
	if !s.job.Stopped() {
		numTaskGroups = len(s.job.TaskGroups)
	}
	s.queuedAllocs = make(map[string]int, numTaskGroups)

	// Get the ready nodes in the required datacenters
	if !s.job.Stopped() {
		s.nodes, s.notReadyNodes, s.nodesByDC, err = readyNodesInDCsAndPool(
			s.state, s.job.Datacenters, s.job.NodePool)
		if err != nil {
			return false, fmt.Errorf("failed to get ready nodes: %v", err)
		}
	}

	// Create a plan
	s.plan = s.eval.MakePlan(s.job)

	// Reset the failed allocations
	s.failedTGAllocs = nil

	// Create an evaluation context
	s.ctx = feasible.NewEvalContext(s.eventsCh, s.state, s.plan, s.logger)

	// Construct the placement stack
	s.stack = feasible.NewSystemStack(true, s.ctx)
	if !s.job.Stopped() {
		s.setJob(s.job)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

Inspect the wrapped cause and server logs for the underlying state-store error.
Verify node pool and datacenter configuration on the job matches the cluster (nomad node status, nomad node pool list).
Check the health of the Nomad server state store and raft log; restart the server if corrupt.
Trigger a new evaluation once the state store is healthy.

Defensive patterns

Strategy: retry

Validate before calling

// preflight: node listing works and job's DCs/pool are sane
if _, _, _, err := readyNodesInDCsAndPool(state, job.Datacenters, job.NodePool); err != nil {
    return fmt.Errorf("node query preflight failed: %w", err)
}

Type guard

func nodesQueryable(s structs.State, dcs []string, pool string) bool {
    _, _, _, err := readyNodesInDCsAndPool(s, dcs, pool)
    return err == nil
}

Try / catch

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get ready nodes") {
        return retryWithBackoff(func() error { return sched.Process(eval) })
    }
    return err
}

Prevention

Validate job datacenters/node pool against actual cluster nodes before submit.
Watch state-store and raft health on Nomad servers.
Retry evaluations on transient state-store failures.
Keep node pool definitions consistent across the cluster.

When it happens

Trigger: readyNodesInDCsAndPool(s.state, job.Datacenters, job.NodePool) returns err != nil — failure iterating nodes from the state store (backend error), not merely zero ready nodes (which is handled without error).

Common situations: State-store read failure on the Nomad server; node pool / datacenter metadata inconsistency after cluster restore; underlying raft errors during evaluation.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/5d0c32a3eeb0f492. Report an issue: GitHub.