hashicorp/nomad · critical

failed to find system jobs for '%s': %v

Error message

failed to find system jobs for '%s': %v

What it means

createNodeEvals also reads all system jobs via snap.JobsByScheduler to decide which system-job evals must be re-created for the node. A failure in that state-store query is wrapped with the node ID. It is an internal server-side failure, not caused by the RPC payload.

Source

Thrown at nomad/node_endpoint.go:1824

// Each Eval is scoped to a job, so we need to potentially trigger many evals.
func (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {
	nodeID := node.ID

	// Snapshot the state
	snap, err := n.srv.fsm.State().Snapshot()
	if err != nil {
		return nil, 0, fmt.Errorf("failed to snapshot state: %v", err)
	}

	// Find all the allocations for this node
	allocs, err := snap.AllocsByNode(nil, nodeID)
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find allocs for '%s': %v", nodeID, err)
	}

	sysJobsIter, err := snap.JobsByScheduler(nil, "system")
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find system jobs for '%s': %v", nodeID, err)
	}

	var sysJobs []*structs.Job
	for jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {
		job := jobI.(*structs.Job)
		// Avoid creating evals for jobs that don't run in this datacenter or
		// node pool. We could perform an entire feasibility check here, but
		// datacenter/pool is a good optimization to start with as their
		// cardinality tends to be low so the check shouldn't add much work.
		// If the job is stopped, skip it as well, otherwise we will create an
		// eval with state and broker overhead that will be an immediate no-op.
		if node.IsInPool(job.NodePool) && node.IsInAnyDC(job.Datacenters) && !job.Stopped() {
			sysJobs = append(sysJobs, job)
		}
	}

	// Fast-path if nothing to do
	if len(allocs) == 0 && len(sysJobs) == 0 {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the affected Nomad server
2. Review server logs for the underlying state-store error message
3. Check for recently applied hot patches or version mismatch among servers
4. If persistent, restore the server's state from backup

resp, err := client.Nodes().UpdateStatus(req)
if err != nil && strings.Contains(err.Error(), "failed to find system jobs for") {
    // retry with backoff against another server
}

When it happens

Trigger: Any eval-creating RPC (Register, UpdateStatus, UpdateDrain, UpdateEligibility, Evaluate, deregister) when the JobsByScheduler(scheduler='system') query on the state snapshot errors.

Common situations: State store corruption or memdb index failures on an unhealthy server; rare, usually accompanied by other state-store errors in logs.

Related errors

• error getting plugin: %s, %v
• failed to determine last evaluation index for job %q: %v
• failed to retrieve jobs for idempotency check
• failed to snapshot state: %v
• failed to find allocs for '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cf9472aa71a99d33. Report an issue: GitHub.