hashicorp/nomad · error

failed to lookup node ID %q: %v

Error message

failed to lookup node ID %q: %v

What it means

buildNodeMap resolves each candidate allocation's node via StateStore.NodeByID. If the state store lookup itself errors (not merely a missing node), it returns this wrapped error, aborting building the node map used for feasibility checking. A nil node (node absent) is tolerated; only an actual store error triggers this.

Source

Thrown at scheduler/feasible/propertyset.go:330

		}
	}
	return allocs[:n]
}

// buildNodeMap takes a list of allocations and returns a map of the nodes used
// by those allocations
func (p *propertySet) buildNodeMap(allocs []*structs.Allocation) (map[string]*structs.Node, error) {
	// Get all the nodes that have been used by the allocs
	nodes := make(map[string]*structs.Node)
	ws := memdb.NewWatchSet()
	for _, alloc := range allocs {
		if _, ok := nodes[alloc.NodeID]; ok {
			continue
		}

		node, err := p.ctx.State().NodeByID(ws, alloc.NodeID)
		if err != nil {
			return nil, fmt.Errorf("failed to lookup node ID %q: %v", alloc.NodeID, err)
		}

		nodes[alloc.NodeID] = node
	}

	return nodes, nil
}

// populateProperties goes through all allocations and builds up the used
// properties from the nodes storing the results in the passed properties map.
func (p *propertySet) populateProperties(allocs []*structs.Allocation, nodes map[string]*structs.Node,
	properties map[string]uint64) {

	for _, alloc := range allocs {
		nProperty, ok := getProperty(nodes[alloc.NodeID], p.targetAttribute)
		if !ok {
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped underlying error in the message to find the root cause
2. Retry the evaluation — Nomad re-enqueues failed evals; transient state issues often resolve
3. Check Nomad server logs and metrics for state store / raft health issues
4. Upgrade Nomad if on a version with known state store bugs

nomad node status // confirm nodes are queryable and the cluster is healthy

// State store errors are server-internal; users retry via Nomad's eval retry
// Check wrapped error in logs and cluster health
nomad eval list | grep -i failed

When it happens

Trigger: StateStore.NodeByID(ws, alloc.NodeID) returns a non-nil error while iterating existing/proposed allocations during populateExisting or PopulateProposed — state store failures, closed snapshot, memdb iteration error.

Common situations: Server under heavy load with state store timeouts, snapshot invalidated by a concurrent state refresh, or internal state store errors during eval processing.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• failed to get job's allocations: %v
• failed to get job %q: %v
• failed to get job deployment %q: %v
• failed to get allocs for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/79044b2e1ded06ee. Report an issue: GitHub.