What causes "allocation not found" in hashicorp/nomad?

Send the fs request to the node actually running the allocation (nomad alloc status -> Node ID), or use `nomad alloc fs` which resolves it

hashicorp/nomad · error

allocation not found

Error message

allocation not found

What it means

allocSnapshot looks up the allocation's filesystem (GetAllocFS) on the local client to stream a snapshot of a file in the alloc dir. If GetAllocFS errors — the allocation is not hosted by this client node — it returns the 'allocation not found' sentinel (allocNotFoundErr).

Source

Thrown at command/agent/alloc_endpoint.go:513

	if rpcErr != nil {
		if structs.IsErrNoNodeConn(rpcErr) || structs.IsErrUnknownAllocation(rpcErr) || structs.IsErrUnknownNode(rpcErr) {
			rpcErr = CodedError(404, rpcErr.Error())
		}
	}

	return reply, rpcErr
}

func (s *HTTPServer) allocSnapshot(allocID string, resp http.ResponseWriter, req *http.Request) (any, error) {
	var secret string
	s.parseToken(req, &secret)
	if !s.agent.Client().ValidateMigrateToken(allocID, secret) {
		return nil, structs.ErrPermissionDenied
	}

	allocFS, err := s.agent.Client().GetAllocFS(allocID)
	if err != nil {
		return nil, fmt.Errorf(allocNotFoundErr)
	}
	if err := allocFS.Snapshot(resp); err != nil {
		return nil, fmt.Errorf("error making snapshot: %v", err)
	}
	return nil, nil
}

func (s *HTTPServer) allocStats(allocID string, resp http.ResponseWriter, req *http.Request) (any, error) {

	// Build the request and parse the ACL token
	task := req.URL.Query().Get("task")
	args := cstructs.AllocStatsRequest{
		AllocID: allocID,
		Task:    task,
	}
	s.parse(resp, req, &args.QueryOptions.Region, &args.QueryOptions)

	// Determine the handler to use

View on GitHub (pinned to 482b49bf1a)

Solutions

Send the fs request to the node actually running the allocation (nomad alloc status -> Node ID), or use `nomad alloc fs` which resolves it
Check `nomad alloc status <alloc_id>` to confirm the allocation still exists and was not GC'd
If using node_id/query routing, verify the target client hosts that alloc
Re-fetch logs/files promptly; GC'd allocations are unrecoverable

Example fix

// before
curl http://wrong-node:4646/v1/client/allocation/abc123/fs?path=/logs/x
// after
nomad alloc status abc123   # find hosting node
nomad alloc fs abc123 logs/app.log

Defensive patterns

Strategy: try-catch

Validate before calling

alloc, _, err := client.Allocations().Info(allocID, nil)
if err != nil || alloc == nil {
    return fmt.Errorf("allocation %s not found or GC'd", allocID)
}

Try / catch

snap, err := getAllocSnapshot(allocID)
if err != nil && strings.Contains(err.Error(), "allocation not found") {
    // resolve hosting node and retry there, or fail gracefully
    return redirectOr404(allocID)
}

Prevention

Resolve the hosting node via `nomad alloc status` before direct client fs calls
Prefer `nomad alloc fs`/API which forwards instead of direct client HTTP
Fetch alloc files before job stop/GC deadline
Handle rescheduled allocs: IDs change; re-resolve after rescheduling

When it happens

Trigger: GET /v1/client/allocation/<alloc_id>/fs?path=... routed to a client node that does not run the allocation, with a migrate token that validates but the alloc FS is absent locally; deleted/GC'd allocation; wrong alloc ID routed via node_id targeting.

Common situations: Load balancer or script hitting the wrong Nomad client node for alloc file access; allocation garbage-collected after job stop while logs are still being fetched; stale alloc ID cached in tooling after rescheduling.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1cd8a4564efa3c9a. Report an issue: GitHub.