hashicorp/nomad · error

failed to get job '%s': %v

Error message

failed to get job '%s': %v

What it means

During SysBatchScheduler.process, the state store lookup JobByID for the evaluation's namespace/JobID failed, so the scheduler cannot load the job it is supposed to evaluate. The wrapped %v contains the underlying state-store error (not 'job not found' — a missing job returns nil, nil and is handled separately).

Source

Thrown at scheduler/scheduler_sysbatch.go:113

		}
		return err
	}

	// Update the status to complete
	return setStatus(s.logger, s.planner, s.eval, nil,
		s.failedTGAllocs, s.planAnnotations, structs.EvalStatusComplete, "",
		s.queuedAllocs, "")
}

// process is wrapped in retryMax to iteratively run the handler until we have no
// further work or we've made the maximum number of attempts.
func (s *SysBatchScheduler) process() (bool, error) {
	// Lookup the Job by ID
	var err error
	ws := memdb.NewWatchSet()
	s.job, err = s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID)
	if err != nil {
		return false, fmt.Errorf("failed to get job '%s': %v", s.eval.JobID, err)
	}

	numTaskGroups := 0
	if !s.job.Stopped() {
		numTaskGroups = len(s.job.TaskGroups)
	}
	s.queuedAllocs = make(map[string]int, numTaskGroups)

	// Get the ready nodes in the required datacenters
	if !s.job.Stopped() {
		s.nodes, s.notReadyNodes, s.nodesByDC, err = readyNodesInDCsAndPool(
			s.state, s.job.Datacenters, s.job.NodePool)
		if err != nil {
			return false, fmt.Errorf("failed to get ready nodes: %v", err)
		}
	}

	// Create a plan

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause in the error and check the Nomad server logs for the underlying state-store failure.
2. Verify the job still exists in the expected namespace (nomad job status -namespace <ns> <job>).
3. Confirm the server's raft/state store is healthy; restart the affected server if needed.
4. Re-submit or trigger a new evaluation (nomad job eval) once the state store recovers.

ws := memdb.NewWatchSet()
if _, err := state.JobByID(ws, ns, jobID); err != nil {
    return fmt.Errorf("job %s/%s not readable: %w", ns, jobID, err)
}

func jobReadable(s structs.State, ns, jobID string) bool {
    _, err := s.JobByID(memdb.NewWatchSet(), ns, jobID)
    return err == nil
}

if err := sched.Process(eval); err != nil {
    if strings.Contains(err.Error(), "failed to get job '") {
        return retryWithBackoff(func() error { return sched.Process(eval) })
    }
    return err
}

When it happens

Trigger: process() calls s.state.JobByID(ws, s.eval.Namespace, s.eval.JobID) and receives err != nil — state store backend failure, memdb/watch-set error, or I/O problem on the server.

Common situations: Degraded Nomad server state store; raft/IO errors during evaluation; memory pressure or corruption on the leader.

Related errors

• failed to get ready nodes: %v
• failed to get job node pool %q: %v
• failed to get scheduler configuration: %v
• failed to get allocs for job '%s': %v
• failed to get tainted nodes for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/414d96678c2d58b2. Report an issue: GitHub.