hashicorp/nomad · error

state for allocation %s not found on client

Error message

state for allocation %s not found on client

What it means

Nomad's client filesystem streaming endpoint (HTTP GET /v1/client/fs/stream) rejects the request when the allocation's state has been destroyed on the client node. The AllocRunner's state exists but IsDestroyed() reports true, meaning the allocation has finished shutting down and its resources were reclaimed. The server returns HTTP 404 to the caller.

Source

Thrown at client/fs_endpoint.go:187

	if err := decoder.Decode(&req); err != nil {
		handleStreamResultError(err, new(int64(http.StatusInternalServerError)), encoder)
		return
	}

	if req.AllocID == "" {
		handleStreamResultError(allocIDNotPresentErr, new(int64(http.StatusBadRequest)), encoder)
		return
	}

	ar, err := f.c.getAllocRunner(req.AllocID)
	if err != nil {
		handleStreamResultError(structs.NewErrUnknownAllocation(req.AllocID), new(int64(http.StatusNotFound)), encoder)
		return
	}
	if ar.IsDestroyed() {
		handleStreamResultError(
			fmt.Errorf("state for allocation %s not found on client", req.AllocID),
			new(int64(http.StatusNotFound)),
			encoder,
		)
		return
	}
	alloc := ar.Alloc()

	// Check read permissions
	if aclObj, err := f.c.ResolveToken(req.QueryOptions.AuthToken); err != nil {
		handleStreamResultError(err, new(int64(http.StatusForbidden)), encoder)
		return
	} else if !aclObj.AllowNsOp(alloc.Namespace, acl.NamespaceCapabilityReadFS) {
		handleStreamResultError(structs.ErrPermissionDenied, new(int64(http.StatusForbidden)), encoder)
		return
	}

	// Validate the arguments
	if req.Path == "" {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Query the allocation's current status via the API and confirm it is running before streaming files.
2. Handle HTTP 404 by refreshing the alloc ID (e.g., look up the replacement allocation after a reschedule) and retrying against the new alloc.
3. Verify you are targeting the correct client node; a destroyed alloc may exist as a tombstone on one node while its replacement runs on another (use ?node_id= to pick the right node).
4. If this occurs during shutdown races, retry with backoff or treat it as a terminal 'alloc gone' condition rather than a transient failure.

Example fix

// before
stream, err := client.Allocs().Logs(alloc, trace, "web", "stdout", nil, nil)
// after
alloc, _, err := client.Allocations().Info(alloc.ID, nil)
if err != nil || alloc.ClientStatus != "running" {
    return fmt.Errorf("allocation %s is not running on client", allocID)
}
stream, err = client.Allocs().Logs(alloc, trace, "web", "stdout", nil, nil)

a, _, err := client.Allocations().Info(allocID, nil)
if err != nil || a.ClientStatus != "running" {
    return fmt.Errorf("alloc %s not running", allocID)
}

func isAllocGone(err error) bool {
    return strings.Contains(err.Error(), "not found on client") ||
        strings.Contains(err.Error(), "unknown allocation")
}

if err := stream(); err != nil {
    if isAllocGone(err) {
        alloc = reResolveAllocation(allocID) // handle reschedule
        return
    }
    return err
}

When it happens

Trigger: Calling the file stream API for an allocation whose AllocRunner on the target client node has been destroyed — typically after the alloc completed, was stopped, or was garbage collected on that node while a stream request was in flight or arrived just after destruction.

Common situations: Streaming logs/files from an allocation that just finished its task; a race where the job was stopped moments before the fs/stream request; stale alloc IDs cached in tooling pointing at a GC'd allocation; client node restarts where destroyed alloc state lingers as a tombstone.

Related errors

• nil allocation
• group name in allocation is not present in job
• task %q not started yet. No logs available
• failed to stream %q: %v
• unable to determine remaining read limit

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/82e2825fd26539a7. Report an issue: GitHub.