hashicorp/nomad · error

Unknown node

Error message

Unknown node

What it means

Indicates the node referenced by an RPC (allocation restart, GC, signal, pause, stats, etc.) does not exist in Nomad's state store. The HTTP API layer maps it (along with no-node-conn and unknown-allocation) to a 404 response.

Source

Thrown at nomad/structs/errors.go:76

	ErrTokenInvalid               = errors.New(errTokenInvalid)
	ErrPermissionDenied           = errors.New(errPermissionDenied)
	ErrJobRegistrationDisabled    = errors.New(errJobRegistrationDisabled)
	ErrNoNodeConn                 = errors.New(errNoNodeConn)
	ErrUnknownMethod              = errors.New(errUnknownMethod)
	ErrUnknownNomadVersion        = errors.New(errUnknownNomadVersion)
	ErrNodeLacksRpc               = errors.New(errNodeLacksRpc)
	ErrMissingAllocID             = errors.New(errMissingAllocID)
	ErrIncompatibleFiltering      = errors.New(errIncompatibleFiltering)
	ErrMalformedChooseParameter   = errors.New(errMalformedChooseParameter)

	// ErrResultPaginatorCreation is returned by list RPC handlers when the
	// result paginator cannot be built, for example when the server cannot
	// evaluate a requested filter expression. api.ResultPaginatorErrorContent
	// duplicates its message so the CLI can match it without importing structs.
	// Keep the two in sync.
	ErrResultPaginatorCreation = errors.New(errResultPaginatorCreation)

	ErrUnknownNode = errors.New(ErrUnknownNodePrefix)

	ErrDeploymentTerminalNoCancel    = errors.New(errDeploymentTerminalNoCancel)
	ErrDeploymentTerminalNoFail      = errors.New(errDeploymentTerminalNoFail)
	ErrDeploymentTerminalNoPause     = errors.New(errDeploymentTerminalNoPause)
	ErrDeploymentTerminalNoPromote   = errors.New(errDeploymentTerminalNoPromote)
	ErrDeploymentTerminalNoResume    = errors.New(errDeploymentTerminalNoResume)
	ErrDeploymentTerminalNoUnblock   = errors.New(errDeploymentTerminalNoUnblock)
	ErrDeploymentTerminalNoRun       = errors.New(errDeploymentTerminalNoRun)
	ErrDeploymentTerminalNoSetHealth = errors.New(errDeploymentTerminalNoSetHealth)
	ErrDeploymentRunningNoUnblock    = errors.New(errDeploymentRunningNoUnblock)

	ErrCSIClientRPCIgnorable  = errors.New("CSI client error (ignorable)")
	ErrCSIClientRPCRetryable  = errors.New("CSI client error (retryable)")
	ErrCSIVolumeMaxClaims     = errors.New("volume max claims reached")
	ErrCSIVolumeUnschedulable = errors.New("volume is currently unschedulable")
	ErrCSIPluginInUse         = errors.New("plugin in use")
)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. List current nodes ('nomad node status') and use a valid Node ID.
2. Re-fetch the allocation to get its current NodeID before calling node-scoped RPCs.
3. Check you are targeting the correct cluster/region/agent.
4. If the node was decommissioned, recreate or re-register it before retrying.

Example fix

// before
client.Nodes().Info(staleNodeID)
// after
nodes, _ := client.Nodes().List(nil)
// verify staleNodeID exists in nodes before proceeding

// Pre-check the node exists:
n, err := client.Nodes().Info(nodeID, nil)
if err != nil { return fmt.Errorf("node %s not found", nodeID) }

func isUnknownNodeErr(err error) bool {
    return structs.IsErrUnknownNode(err) || structs.IsErrNoNodeConn(err)
}

rpcErr := doNodeRPC(...)
if structs.IsErrUnknownNode(rpcErr) {
    // treat as 404: refresh node list, do not retry blindly
    return refreshAndLookupNode(nodeID)
}

When it happens

Trigger: Calling allocRestart, allocGC, allocSignal, allocPauseGet, allocPauseSet, or allocStats with an allocation/node whose node ID is not registered; or after the node was reaped from state.

Common situations: Stale node IDs cached by automation, nodes removed after a cluster rebuild, typos in node IDs, or clients acting on data from a snapshot of a different cluster.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• %w %s
• Unknown node %q
• job not found
• variable not found
• no servers

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/84dd8bc034fb0a4b. Report an issue: GitHub.