What causes "Failed to retrieve allocation %q: %w" in hashicorp/nomad?

Verify the allocation still exists: nomad alloc status ; if it was GC'd, rerun nomad job restart on the job instead of a specific allocation.

hashicorp/nomad · error

Failed to retrieve allocation %q: %w

Error message

Failed to retrieve allocation %q: %w

What it means

This error is raised by the `nomad job restart` command when its background watcher (monitorReplacementAlloc) fails to fetch info about the allocation it is tracking via the Nomad HTTP API (Allocations().Info). The original API error (auth failure, allocation not found, connectivity problem) is wrapped with %w so the underlying cause is preserved. It is emitted on an error channel and surfaced to the user as the command's failure reason.

Source

Thrown at command/job_restart.go:1076

// Returns an error in errCh if anything goes wrong or nil when the new
// allocation is running.
func (c *JobRestartCommand) monitorReplacementAlloc(
	ctx context.Context,
	allocStub AllocationListStubWithJob,
	errCh chan<- error,
) {
	currentAllocID := allocStub.ID
	q := &api.QueryOptions{WaitIndex: 1}
	for {
		select {
		case <-ctx.Done():
			return
		default:
		}

		alloc, qm, err := c.client.Allocations().Info(currentAllocID, q)
		if err != nil {
			errCh <- fmt.Errorf("Failed to retrieve allocation %q: %w", limit(currentAllocID, c.length), err)
			return
		}

		// Follow replacement allocations. We expect the original allocation to
		// be replaced, but the replacements may be themselves replaced in
		// cases such as the allocation failing.
		if alloc.NextAllocation != "" {
			c.Ui.Output(fmt.Sprintf(
				"    %s: Allocation %q replaced by %[3]q, waiting for %[3]q to start running",
				formatTime(time.Now()),
				limit(alloc.ID, c.length),
				limit(alloc.NextAllocation, c.length),
			))
			currentAllocID = alloc.NextAllocation

			// Reset the blocking query so the Info() API call returns the new
			// allocation immediately.
			q.WaitIndex = 1

View on GitHub (pinned to 482b49bf1a)

Solutions

Verify the allocation still exists: nomad alloc status <alloc-id>; if it was GC'd, rerun nomad job restart on the job instead of a specific allocation.
Confirm the CLI targets the right cluster/namespace: check NOMAD_ADDR, NOMAD_NAMESPACE, NOMAD_REGION env vars and the -namespace/-region flags.
Check ACL permissions: the token needs allocation read access (alloc:lifecycle / namespace = read-alloc) in the target namespace; run nomad acl token self.
Test connectivity to the agent (curl $NOMAD_ADDR/v1/agent/health) and rerun the command if it was a transient network error.
If the wrapped cause is allocation not found after rescheduling, let the command follow the new allocation ID automatically or restart the whole job with -job.

Example fix

// before
client := api.NewClient(api.DefaultConfig()) // wrong address / namespace
alloc, _, err := client.Allocations().Info(id, nil)
// after
cfg := api.DefaultConfig()
cfg.Namespace = "default" // match the allocation's namespace
client := api.NewClient(cfg)
alloc, _, err := client.Allocations().Info(id, &api.QueryOptions{Namespace: "default"})
if err != nil && strings.Contains(err.Error(), "not found") {
    // allocation GC'd: fall back to restarting the job instead
}

Defensive patterns

Strategy: try-catch

Validate before calling

// check allocation exists and token works before running the restart
resp, err := http.Get(addr + "/v1/allocation/" + allocID)
if err != nil || resp.StatusCode == 403 {
    // fix NOMAD_ADDR / ACL token first
}

Type guard

func isNotFound(err error) bool {
    return err != nil && strings.Contains(strings.ToLower(err.Error()), "not found")
}

Try / catch

if err := cmd.Run(); err != nil {
    var apiErr *api.StatusError
    if errors.As(err, &apiErr) && apiErr.StatusCode == 404 {
        // allocation GC'd: fall back to `nomad job restart <job>`
    } else if isTransient(err) {
        // retry with backoff
    }
}

Prevention

Pin NOMAD_ADDR/NOMAD_NAMESPACE/NOMAD_REGION per environment in scripts or wrapper scripts.
Use ACL tokens with allocation read scope for restart automation.
Check nomad alloc status before targeting a specific allocation; avoid allocations pending GC.
Retry transient API failures with backoff instead of aborting the restart.

When it happens

Trigger: Allocations().Info(currentAllocID, q) returns an error while the command polls for the replacement allocation: the allocation was GC'd out of Nomad's state store, the agent/HTTP address is unreachable, the ACL token lacks alloc:lifecycle or namespace-scoped read on the allocation, or the query options (namespace/region) are wrong.

Common situations: Restarting a job whose stopped allocation has already been garbage collected; running the CLI against a different cluster/namespace after a context switch; expired or insufficient ACL token (token cannot read the target namespace's allocations); Nomad agent restarted or network partition mid-restart.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/af84754f13ea67c9. Report an issue: GitHub.