hashicorp/nomad · error

node %q not found

Error message

node %q not found

What it means

In allocwatcher.getNodeAddr, the watcher looks up a node (by nodeID) via the client's state/agent API to obtain the address of the previous allocation's node for migration. If the query response contains a nil Node for the requested nodeID, it returns "node %q not found". This means the node referenced by the earlier allocation is unknown to the cluster/state store.

Source

Thrown at client/allocwatcher/alloc_watcher.go:509

	resp := structs.SingleNodeResponse{}
	for {
		err := p.rpc.RPC("Node.GetNode", &req, &resp)
		if err != nil {
			p.logger.Error("failed to query node", "error", err, "node", nodeID)
			retry := getRemoteRetryIntv + helper.RandomStagger(getRemoteRetryIntv)
			select {
			case <-time.After(retry):
				continue
			case <-ctx.Done():
				return "", ctx.Err()
			}
		}
		break
	}

	if resp.Node == nil {
		return "", fmt.Errorf("node %q not found", nodeID)
	}

	scheme := "http://"
	if resp.Node.TLSEnabled {
		scheme = "https://"
	}
	return scheme + resp.Node.HTTPAddr, nil
}

// migrate a remote alloc dir to local node. Caller is responsible for calling
// Destroy on the returned allocdir if no error occurs.
func (p *remotePrevAlloc) migrateAllocDir(ctx context.Context, nodeAddr string) (*allocdir.AllocDir, error) {
	// Create the previous alloc dir
	prevAllocDir := allocdir.NewAllocDir(p.logger, p.config.AllocDir, p.config.AllocMountsDir, p.prevAllocID)
	if err := prevAllocDir.Build(); err != nil {
		return nil, fmt.Errorf("error building alloc dir for previous alloc %q: %w", p.prevAllocID, err)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Remove/ignore the stale allocation: purge it (nomad system gc or nomad alloc stop) so migration doesn't reference the dead node
2. Confirm the client is talking to the correct region/cluster that knows the node ID (check region/datacenter and servers list)
3. Re-register the node (bring the old client back or let a new client heartbeat) if migration data still lives there
4. If data locality isn't required, disable migration on the group (migrate { ... } / sticky volumes) so the alloc starts fresh without contacting the old node

Example fix

// before
new alloc tries to migrate from stopped alloc on deregistered node n1 -> error
// after
group "web" {
  migrate {
    # or disable sticky/previous-alloc reuse
  }
}
$ nomad system gc

// Before relying on migration, verify the previous alloc's node is still registered:
node, _, err := client.Nodes().Info(prevAlloc.NodeID, nil)
if err != nil || node == nil {
    // skip migration; schedule fresh
}

// Watcher already falls back; for API consumers:
addr, err := watcher.getNodeAddr(nodeID)
if err != nil && strings.Contains(err.Error(), "not found") {
    log.Warnf("previous node %q gone; starting alloc without migration", nodeID)
    return "", nil // proceed without remote migration
}
return addr, err

When it happens

Trigger: allocWatcher.Migrate (with migrate hooks) calls getNodeAddr for the node ID of the prior alloc; the node lookup (e.g. via the client's CSR/agent API or RPC Nomad.Node-specific query) returns resp.Node == nil - the node ID is not in the state store.

Common situations: The previous node was deregistered/garbage-collected (node down past heartbeat, client drained and removed) while an old stopped allocation is being migrated; restoring a cluster from backup/partial state; connecting to the wrong cluster/region so the node ID doesn't resolve; stale allocation references after cluster rebuild.

Related errors

• no servers
• missing NodeID
• missing AllocID
• missing allocation ID
• missing AllocID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8034be3d34cb87b3. Report an issue: GitHub.