What causes "failed to get job node pool %q: %v" in hashicorp/nomad?

Check the wrapped cause and server logs for the underlying state-store failure.

failed to get job node pool %q: %v

Error message

failed to get job node pool %q: %v

What it means

SysBatchScheduler.setJob fetches the job's node pool via state.NodePoolByName before configuring the scheduler stack; this error wraps a failure of that lookup. Note that a nonexistent pool returns nil,nil here — this error indicates an actual state-store read failure, and the wrapped %v carries the cause.

Source

Thrown at scheduler/scheduler_sysbatch.go:197

	// Try again if the plan was not fully committed, potential conflict
	fullCommit, expected, actual := result.FullCommit(s.plan)
	if !fullCommit {
		s.logger.Debug("plan didn't fully commit", "attempted", expected, "placed", actual)
		return false, nil
	}

	// Success!
	return true, nil
}

// setJob updates the stack with the given job and job's node pool scheduler
// configuration.
func (s *SysBatchScheduler) setJob(job *structs.Job) error {
	// Fetch node pool and global scheduler configuration to determine how to
	// configure the scheduler.
	pool, err := s.state.NodePoolByName(nil, job.NodePool)
	if err != nil {
		return fmt.Errorf("failed to get job node pool %q: %v", job.NodePool, err)
	}

	_, schedConfig, err := s.state.SchedulerConfig()
	if err != nil {
		return fmt.Errorf("failed to get scheduler configuration: %v", err)
	}

	s.stack.SetJob(job)
	s.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))
	return nil
}

// computeJobAllocs is used to reconcile differences between the job,
// existing allocations and node status to update the allocations.
func (s *SysBatchScheduler) computeJobAllocs() error {
	// Lookup the allocations by JobID
	ws := memdb.NewWatchSet()
	allocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)

View on GitHub (pinned to 482b49bf1a)

Solutions

Check the wrapped cause and server logs for the underlying state-store failure.
Verify the node pool referenced by the job exists (nomad node pool list; create it with nomad node pool apply if missing).
Ensure the Nomad server state store/raft is healthy; restart or restore the server if needed.
Re-run the evaluation after the state store recovers.

Example fix

// before
job.NodePool = "gpu-pool" // pool never created
// after
// create the pool first: nomad node pool apply gpu-pool.hcl
job.NodePool = "gpu-pool" // ensure pool exists before submitting job

Defensive patterns

Strategy: validation

Validate before calling

if _, err := state.NodePoolByName(nil, job.NodePool); err != nil {
    return fmt.Errorf("node pool %q unreadable: %w", job.NodePool, err)
}

Type guard

func nodePoolExists(s structs.State, name string) bool {
    p, err := s.NodePoolByName(nil, name)
    return err == nil && p != nil
}

Try / catch

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get job node pool") {
        return fmt.Errorf("create/verify node pool before submitting job: %w", err)
    }
    return err
}

Prevention

Create node pools (nomad node pool apply) before submitting jobs that reference them.
Validate job.NodePool in CI against the cluster's pool list.
Avoid renaming/deleting pools while jobs reference them.
Monitor state-store health for read failures.

When it happens

Trigger: process() calls setJob(job); s.state.NodePoolByName(nil, job.NodePool) returns err != nil due to a state-store backend failure while reading the node pool table.

Common situations: Nomad server state-store errors; jobs referencing node pools during a degraded/restore state; raft I/O failures on the leader.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/768a56f41ca3e735. Report an issue: GitHub.