hashicorp/nomad · error

Failed evaluate job: %w

Error message

Failed evaluate job: %w

What it means

For system jobs, stopped allocs are not replaced automatically, so stopAlloc forces an evaluation via Jobs().EvaluateWithOpts with ForceReschedule. If that call errors, it is wrapped as 'Failed evaluate job: <cause>'.

Source

Thrown at command/job_restart.go:966

	}

	// Stop allocation and wait for its replacement to be running or for a
	// blocked evaluation that prevents placements for this task group to
	// happen.
	resp, err := c.client.Allocations().Stop(&api.Allocation{ID: alloc.ID}, q)
	if err != nil {
		return fmt.Errorf("Failed to stop allocation: %w", err)
	}

	// Allocations for system jobs do not get replaced by the scheduler after
	// being stopped, so an eval is needed to trigger the reconciler.
	if alloc.isSystemJob() {
		opts := api.EvalOptions{
			ForceReschedule: true,
		}
		_, _, err := c.client.Jobs().EvaluateWithOpts(*alloc.Job.ID, opts, nil)
		if err != nil {
			return fmt.Errorf("Failed evaluate job: %w", err)
		}
	}

	// errCh receives an error if anything goes wrong or nil when the
	// replacement allocation is running.
	// Use a buffered channel to prevent both goroutine from blocking trying to
	// send a result back.
	errCh := make(chan error, 1)
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Pass the LastIndex from the Stop() call to only monitor data that was
	// created after the Stop() call.
	go c.monitorPlacementFailures(ctx, alloc, resp.LastIndex, errCh)
	go c.monitorReplacementAlloc(ctx, alloc, errCh)

	// This process may take a while, so ping user from time to time to
	// indicate the command is still alive.

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the job still exists (nomad job status <job-id>)
2. Check connectivity to the Nomad agent / NOMAD_ADDR
3. Ensure the ACL token can submit evaluations for the job
4. Re-run the restart command after restoring API access

job, _, err := client.Jobs().Info(*alloc.Job.ID, nil)
if err != nil {
    return fmt.Errorf("job %s unavailable before evaluate: %w", *alloc.Job.ID, err)
}

_, _, err := client.Jobs().EvaluateWithOpts(jobID, opts, nil)
if err != nil {
    var apiErr error
    if errors.As(err, &apiErr) && strings.Contains(err.Error(), "403") {
        return fmt.Errorf("ACL token lacks submit-job permission: %w", err)
    }
    return fmt.Errorf("evaluate failed, check agent connectivity: %w", err)
}

When it happens

Trigger: EvaluateWithOpts fails while restarting a system-job alloc — agent unreachable, job ID stale (job deleted), or ACL token lacking job-submit/evaluate permission.

Common situations: Job was deleted or purged while a restart of its allocs was in progress; network interruption to the Nomad API; token without submit-job ACL capability.

Related errors

• Failed to retrieve evaluations for job %q: %w
• unknown task name %q
• error retrieving supported Envoy versions from Consul: %w
• node not found: %s
• allocation not found

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/95bf6078b3309973. Report an issue: GitHub.