hashicorp/nomad · critical

failed to find allocs for '%s': %v

Error message

failed to find allocs for '%s': %v

What it means

After snapshotting state, createNodeEvals looks up all allocations on the node via snap.AllocsByNode. If that state-store read fails, the error is wrapped with the node ID. Like the snapshot failure, it signals an internal state-store read error rather than invalid user input.

Source

Thrown at nomad/node_endpoint.go:1819

		}}
	return n.srv.blockingRPC(&opts)
}

// createNodeEvals is used to create evaluations for each alloc on a node.
// Each Eval is scoped to a job, so we need to potentially trigger many evals.
func (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {
	nodeID := node.ID

	// Snapshot the state
	snap, err := n.srv.fsm.State().Snapshot()
	if err != nil {
		return nil, 0, fmt.Errorf("failed to snapshot state: %v", err)
	}

	// Find all the allocations for this node
	allocs, err := snap.AllocsByNode(nil, nodeID)
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find allocs for '%s': %v", nodeID, err)
	}

	sysJobsIter, err := snap.JobsByScheduler(nil, "system")
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find system jobs for '%s': %v", nodeID, err)
	}

	var sysJobs []*structs.Job
	for jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {
		job := jobI.(*structs.Job)
		// Avoid creating evals for jobs that don't run in this datacenter or
		// node pool. We could perform an entire feasibility check here, but
		// datacenter/pool is a good optimization to start with as their
		// cardinality tends to be low so the check shouldn't add much work.
		// If the job is stopped, skip it as well, otherwise we will create an
		// eval with state and broker overhead that will be an immediate no-op.
		if node.IsInPool(job.NodePool) && node.IsInAnyDC(job.Datacenters) && !job.Stopped() {
			sysJobs = append(sysJobs, job)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the affected Nomad server to rebuild a clean in-memory state store
2. Inspect server logs around the error for the underlying memdb message
3. Check host memory/CPU; a very large cluster may need more server capacity
4. Restore from backup or re-add the node if the state store is corrupt

resp, err := client.Nodes().Register(req)
if err != nil && strings.Contains(err.Error(), "failed to find allocs for") {
    // retry with backoff; if persistent, restart the server
}

When it happens

Trigger: Register, deregister, UpdateStatus, UpdateDrain, UpdateEligibility, or Evaluate RPCs for a node whose allocation index lookup in the state store snapshot fails (internal memdb/iterator error).

Common situations: State store iterator failures under memory pressure, corrupted state store, or extremely large allocation tables causing resource exhaustion during iteration.

Related errors

• error getting plugin: %s, %v
• failed to determine last evaluation index for job %q: %v
• failed to retrieve jobs for idempotency check
• failed to snapshot state: %v
• failed to find system jobs for '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/2c3702d02a6ce51a. Report an issue: GitHub.