What causes "Unknown node %q" in hashicorp/nomad?

Run `nomad node status` and confirm the node/alloc still exists; re-fetch a current alloc ID.

Unknown node %q

Error message

Unknown node %q

What it means

Thrown in the streaming filesystem RPC handler (stream) when the target nodeID does not exist in the server's node registry. The request cannot be forwarded to any client, so a 400 is returned to the streaming caller. It is the fs/stream flavor of "unknown node": the ID given is not a known Nomad client.

Source

Thrown at nomad/client_fs_endpoint.go:294

	if aclObj, err := f.srv.ResolveACL(&args); err != nil {
		handleStreamResultError(err, nil, encoder)
		return
	} else if !aclObj.AllowNsOp(alloc.Namespace, acl.NamespaceCapabilityReadFS) {
		handleStreamResultError(structs.ErrPermissionDenied, nil, encoder)
		return
	}

	nodeID := alloc.NodeID

	// Make sure Node is valid and new enough to support RPC
	node, err := snap.NodeByID(nil, nodeID)
	if err != nil {
		handleStreamResultError(err, new(int64(500)), encoder)
		return
	}

	if node == nil {
		err := fmt.Errorf("Unknown node %q", nodeID)
		handleStreamResultError(err, new(int64(400)), encoder)
		return
	}

	if err := nodeSupportsRpc(node); err != nil {
		handleStreamResultError(err, new(int64(400)), encoder)
		return
	}

	// Get the connection to the client either by forwarding to another server
	// or creating a direct stream
	var clientConn net.Conn
	state, ok := f.srv.getNodeConn(nodeID)
	if !ok {
		// Determine the Server that has a connection to the node.
		srv, err := f.srv.serverWithNodeConn(nodeID, f.srv.Region())
		if err != nil {
			var code *int64

View on GitHub (pinned to 482b49bf1a)

Solutions

Run `nomad node status` and confirm the node/alloc still exists; re-fetch a current alloc ID.
If the job's allocations were GC'd, re-run the job to get new allocations.
Check you are passing an alloc ID (not a job name without the -job flag) to the fs command.
If the node was removed intentionally, run the workload on an existing node.

Example fix

// before
nomad alloc fs <old-gc'd-alloc-id>
// after
nomad job status web | # get live alloc id
nomad alloc fs <current-alloc-id>

Defensive patterns

Strategy: try-catch

Validate before calling

alloc, _, err := client.Allocations().Info(allocID, nil)
if err != nil {
    return fmt.Errorf("alloc %s gone; refresh alloc list", allocID)
}
node, _, err := client.Nodes().Info(alloc.NodeID, nil)
if err != nil || node == nil {
    return fmt.Errorf("node %s unknown; re-run job", alloc.NodeID)
}

Try / catch

if strings.Contains(err.Error(), "Unknown node") {
    // refetch alloc/node IDs and retry once
    allocs, _, _ := client.Allocations().List(nil)
    _ = allocs
}

Prevention

Never cache alloc/node IDs across job restarts or GC windows.
Handle job GC by re-listing allocations before fs access.
Disable overly aggressive job GC for jobs you still need to inspect.

When it happens

Trigger: Calling the filesystem stream API (used by `nomad alloc fs`) with a node ID that is not registered, or where the alloc's node lookup returned nil before the stream was established.

Common situations: Stale alloc IDs from a GC'd/removed job; node deregistered between listing allocs and streaming files; typo in node/alloc identifier in scripts; state trimmed on servers after node GC.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/74d59e020ca1f2e1. Report an issue: GitHub.