What causes "Failed to stop allocation: %w" in hashicorp/nomad?

Re-run nomad job restart; the command will pick up the current alloc IDs

Failed to stop allocation: %w

Error message

Failed to stop allocation: %w

What it means

Fires in stopAlloc during job restart: the Allocations().Stop API call to the server failed, so the allocation was not stopped and no replacement will be scheduled for it.

Source

Thrown at command/job_restart.go:955

		formatTime(time.Now()),
		shortAllocID,
		alloc.TaskGroup,
	))

	var q *api.QueryOptions

	if c.noShutdownDelay {
		q = &api.QueryOptions{
			Params: map[string]string{"no_shutdown_delay": "true"},
		}
	}

	// Stop allocation and wait for its replacement to be running or for a
	// blocked evaluation that prevents placements for this task group to
	// happen.
	resp, err := c.client.Allocations().Stop(&api.Allocation{ID: alloc.ID}, q)
	if err != nil {
		return fmt.Errorf("Failed to stop allocation: %w", err)
	}

	// Allocations for system jobs do not get replaced by the scheduler after
	// being stopped, so an eval is needed to trigger the reconciler.
	if alloc.isSystemJob() {
		opts := api.EvalOptions{
			ForceReschedule: true,
		}
		_, _, err := c.client.Jobs().EvaluateWithOpts(*alloc.Job.ID, opts, nil)
		if err != nil {
			return fmt.Errorf("Failed evaluate job: %w", err)
		}
	}

	// errCh receives an error if anything goes wrong or nil when the
	// replacement allocation is running.
	// Use a buffered channel to prevent both goroutine from blocking trying to
	// send a result back.

View on GitHub (pinned to 482b49bf1a)

Solutions

Re-run nomad job restart; the command will pick up the current alloc IDs
Check nomad alloc status and agent health/network connectivity
Extend the query timeout / wait options used by the command
Ensure the ACL token has permissions to stop allocations

Defensive patterns

Strategy: retry

Validate before calling

alloc, _, err := client.Allocations().Info(allocID, nil)
if err != nil {
    return err
}
if alloc.DesiredStatus == "stop" || isTerminal(alloc.ClientStatus) {
    return fmt.Errorf("alloc %s already stopping/terminal", allocID)
}

Type guard

func isStoppable(a *api.Allocation) bool {
    return a != nil && a.DesiredStatus != "stop" && a.ClientStatus == "running"
}

Try / catch

_, _, err := client.Allocations().Stop(&api.Allocation{ID: allocID}, q)
if err != nil {
    if errors.Is(err, context.DeadlineExceeded) || isTransientRPC(err) {
        return retryWithBackoff(func() error { _, _, err := client.Allocations().Stop(...); return err })
    }
    return err
}

Prevention

Re-fetch the alloc before stopping to avoid stale IDs
Use adequate query timeouts/waits
Ensure ACL token has alloc-lifecycle capability

When it happens

Trigger: The Stop RPC fails — agent unreachable, alloc already garbage-collected/terminal, context/query timeout expired, or ACL token lacks permission.

Common situations: Alloc was replaced by the scheduler between listing and stopping (stale ID); node down so the client RPC fails; too-short -detach or query timeout during a slow drain; missing alloc-lifecycle ACL capability.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b956d2b04cd62491. Report an issue: GitHub.