hashicorp/nomad · error

failed to retrieve nodes: %w

Error message

failed to retrieve nodes: %w

What it means

Same upgrade check as the jobs pass, but for nodes: if state.StateStore.Nodes() fails to iterate nodes, the error is wrapped and returned. Indicates a failure reading the node table from the in-memory state store.

Source

Thrown at nomad/operator_endpoint.go:807

				for _, wid := range t.Identities {
					if wid.IsVault() {
						foundWID = true
						break
					}
				}
				if !foundWID {
					jobs = append(jobs, job.Stub(nil, nil))
					break TG_LOOP
				}
			}
		}
	}
	reply.JobsWithoutVaultIdentity = jobs

	// Find nodes that don't support workload identities for Vault.
	nodesIter, err := op.srv.State().Nodes(ws)
	if err != nil {
		return fmt.Errorf("failed to retrieve nodes: %w", err)
	}

	nodes := []*structs.NodeListStub{}
	for raw := nodesIter.Next(); raw != nil; raw = nodesIter.Next() {
		node := raw.(*structs.Node)

		v, err := version.NewVersion(node.Attributes["nomad.version"])
		if err != nil || v.LessThan(structs.MinNomadVersionVaultWID) {
			nodes = append(nodes, node.Stub(nil))
			continue
		}
	}
	reply.OutdatedNodes = nodes

	reply.QueryMeta.Index, _ = op.srv.State().LatestIndex()
	op.srv.setQueryMeta(&reply.QueryMeta)

	return nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the upgrade check
2. Inspect server logs for the wrapped underlying error
3. Validate raft/state store health; restore from a good snapshot if corrupt
4. Ensure the operation is hitting the current leader

resp, err := client.Operator().UpgradeCheck(nil)
if err != nil && strings.Contains(err.Error(), "failed to retrieve nodes") {
    // state store read failure: retry after checking server health
}

When it happens

Trigger: op.srv.State().Nodes(ws) returns an error from the memdb-backed state store — internal query failure or store inconsistency (e.g. mid-restore).

Common situations: Running the Vault WI upgrade check against a server with a corrupted or mid-restore state store.

Related errors

• failed to retrieve jobs: %w
• incompatible state version. expected %q but found %q
• error deleting unexpected key %q: %v
• unexpected bucket missing %q
• failed to get tainted nodes for job '%s': %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/25069a33363b241f. Report an issue: GitHub.