hashicorp/nomad · error

failed to get allocs for job '%s': %v

Error message

failed to get allocs for job '%s': %v

What it means

SysBatchScheduler.computeJobAllocs queries AllocsByJob (all allocations for the eval's namespace/JobID, including terminal ones) and wraps a failure of that state-store read in this error. The scheduler needs the existing allocs to compute placements and stop/lost updates for sysbatch jobs.

Source

Thrown at scheduler/scheduler_sysbatch.go:217

	_, schedConfig, err := s.state.SchedulerConfig()
	if err != nil {
		return fmt.Errorf("failed to get scheduler configuration: %v", err)
	}

	s.stack.SetJob(job)
	s.stack.SetSchedulerConfiguration(schedConfig.WithNodePool(pool))
	return nil
}

// computeJobAllocs is used to reconcile differences between the job,
// existing allocations and node status to update the allocations.
func (s *SysBatchScheduler) computeJobAllocs() error {
	// Lookup the allocations by JobID
	ws := memdb.NewWatchSet()
	allocs, err := s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true)
	if err != nil {
		return fmt.Errorf("failed to get allocs for job '%s': %v", s.eval.JobID, err)
	}

	// Determine the tainted nodes containing job allocs
	tainted, err := taintedNodes(s.state, allocs)
	if err != nil {
		return fmt.Errorf("failed to get tainted nodes for job '%s': %v", s.eval.JobID, err)
	}

	// Update the allocations which are in pending/running state on tainted
	// nodes to lost.
	updateNonTerminalAllocsToLost(s.plan, tainted, allocs)

	// Split out terminal allocations
	live, term := structs.SplitTerminalAllocs(allocs)

	// Diff the required and existing allocations
	nr := reconciler.NewNodeReconciler(nil)
	r := nr.Compute(s.job, s.nodes, s.notReadyNodes, tainted, live, term)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause and Nomad server logs for the underlying state-store error.
2. Verify raft/state store health; restart or replace the failing server.
3. Check alloc table integrity if corruption is suspected (nomad operator debug).
4. Re-trigger the evaluation once the state store is healthy.

ws := memdb.NewWatchSet()
if _, err := state.AllocsByJob(ws, ns, jobID, true); err != nil {
    return fmt.Errorf("allocs for %s/%s unreadable: %w", ns, jobID, err)
}

func allocsReadable(s structs.State, ns, jobID string) bool {
    _, err := s.AllocsByJob(memdb.NewWatchSet(), ns, jobID, true)
    return err == nil
}

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get allocs for job") {
        return retryWithBackoff(func() error { return sched.Process(eval) })
    }
    return err
}

When it happens

Trigger: process() -> computeJobAllocs(): s.state.AllocsByJob(ws, s.eval.Namespace, s.eval.JobID, true) returns err != nil due to a state-store backend failure.

Common situations: State-store read errors on a Nomad server under raft/disk failure; memory pressure in memdb; errors during leader failover while a sysbatch eval is being processed.

Related errors

• failed to get job '%s': %v
• failed to get ready nodes: %v
• failed to get job node pool %q: %v
• failed to get scheduler configuration: %v
• failed to get tainted nodes for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/d17e540194b07ac4. Report an issue: GitHub.