hashicorp/nomad · error

failed to get scheduler configuration: %v

Error message

failed to get scheduler configuration: %v

What it means

This error is returned by the scheduler's setJob helper (scheduler/generic_sched.go) when the state store's SchedulerConfig() call fails while configuring the scheduler for a job. It means the scheduler could not read the global scheduler configuration needed to set up the scheduling stack (after successfully fetching the job's node pool). The wrapped %v carries the underlying state-store error.

Source

Thrown at scheduler/generic_sched.go:804

			plan.NodeAllocation[original.NodeID][i] = updated
			return
		}
	}
}

// setJob updates the stack with the given job and job's node pool scheduler
// configuration.
func (s *GenericScheduler) setJob(job *structs.Job) error {
	// Fetch node pool and global scheduler configuration to determine how to
	// configure the scheduler.
	pool, err := s.state.NodePoolByName(nil, job.NodePool)
	if err != nil {
		return fmt.Errorf("failed to get job node pool %q: %v", job.NodePool, err)
	}

	_, schedConfig, err := s.state.SchedulerConfig()
	if err != nil {
		return fmt.Errorf("failed to get scheduler configuration: %v", err)
	}

	s.stack.SetJob(job)
	s.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))
	return nil
}

// setnodes updates the stack with the nodes that are ready for placement for
// the given job.
func (s *GenericScheduler) setNodes(job *structs.Job) ([]*structs.Node, map[string]int, error) {
	nodes, _, byDC, err := readyNodesInDCsAndPool(s.state, job.Datacenters, job.NodePool)
	if err != nil {
		return nil, nil, err
	}

	s.stack.SetNodes(nodes)
	return nodes, byDC, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause (%v) to identify the state-store error and check server logs for the underlying raft/state store failure.
2. Verify the leader is healthy and the state store is intact; run nomad server members and check for raft errors.
3. Restart the affected Nomad server agent; if state is corrupt, restore from a verified backup.
4. Re-trigger evaluation by retrying the job (nomad job eval <job>) once the state store is healthy.
5. Upgrade Nomad to the latest patch release if the state store bug persists.

Example fix

// Server-side fix is operational; client retry pattern
// before
err := scheduler.Process(eval)
if err != nil { return err }
// after
if err := scheduler.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get scheduler configuration") {
        time.Sleep(backoff)
        return scheduler.Process(eval) // retry after state store recovers
    }
    return err
}

// preflight: verify state store is reachable and config exists
if _, cfg, err := state.SchedulerConfig(); err != nil {
    return fmt.Errorf("state store not ready: %w", err)
}

func schedulerConfigReadable(s structs.State) bool {
    _, _, err := s.SchedulerConfig()
    return err == nil
}

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get scheduler configuration") {
        // retry with backoff; state store likely transiently unavailable
        return retryWithBackoff(func() error { return sched.Process(eval) })
    }
    return err
}

When it happens

Trigger: An evaluation is processed via process/computePlacements, setJob is invoked, NodePoolByName succeeds, but s.state.SchedulerConfig() returns a non-nil error (state store read failure, corrupted or missing scheduler config entry, or Raft/state backend I/O error).

Common situations: Nomad servers with a degraded or corrupt state store; scheduler config not yet initialized during early cluster bootstrap or restore from backup; disk/raft issues on the leader while a scheduler worker is evaluating a queued job.

Related errors

• failed to get scheduler configuration: %v
• eval broker is enabled; eval broker must be paused to delete
• Job registration, dispatch, and scale are disabled by the sc
• timeout cannot be negative
• failed to get job's allocations: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cd8c1f9611683d36. Report an issue: GitHub.