hashicorp/nomad · error

failed to lookup eval allocs: %v

Error message

failed to lookup eval allocs: %v

What it means

StateStore.EvalIsUserDeleteSafe wraps failures from s.AllocsByEval when fetching allocations belonging to the eval being checked. This is a read-transaction failure in the allocs table, not simply 'no allocs exist' (an empty result is valid and safe). The eval delete safety check aborts and reports the eval as not safe to delete.

Source

Thrown at nomad/state/state_store.go:3669

// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its
// related allocation and job information. This follows similar, but different
// rules to the eval reap checking, to ensure evaluations for running allocs or
// allocs which need the evaluation detail are not deleted.
//
// Returns both a bool and an error so that error in querying the related
// objects can be differentiated from reporting that the eval isn't safe to
// delete.
func (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {

	job, err := s.JobByID(ws, eval.Namespace, eval.JobID)
	if err != nil {
		return false, fmt.Errorf("failed to lookup job for eval: %v", err)
	}

	allocs, err := s.AllocsByEval(ws, eval.ID)
	if err != nil {
		return false, fmt.Errorf("failed to lookup eval allocs: %v", err)
	}

	return isEvalDeleteSafe(allocs, job), nil
}

func isEvalDeleteSafe(allocs []*structs.Allocation, job *structs.Job) bool {

	// If the job is deleted, stopped, or dead, all allocs are terminal and
	// the eval can be deleted.
	if job == nil || job.Stop || job.Status == structs.JobStatusDead {
		return true
	}

	// Iterate the allocations associated to the eval, if any, and check
	// whether we can delete the eval.
	for _, alloc := range allocs {

		// If the allocation is still classed as running on the client, or

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause to identify the allocs table/index problem
2. Verify allocs index integrity; restore a known-good Raft snapshot if corruption is confirmed
3. Upgrade/align all servers to the same Nomad version to avoid schema mismatches
4. Retry after replacing or restarting the degraded server

// Pre-check allocs for the eval via the API before the safety check
allocs, _, err := client.Allocs().List(nil)
if err != nil { return err }
for _, a := range allocs {
	if a.EvalID == evalID { /* allocs exist; safety check will evaluate them */ }
}

func evalAllocLookupFailed(err error) bool {
	return err != nil && strings.Contains(err.Error(), "failed to lookup eval allocs")
}

if err != nil {
	if evalAllocLookupFailed(err) {
		return fmt.Errorf("allocs table read failed; verify server state store: %w", err)
	}
	return err
}

When it happens

Trigger: Calling the eval delete safety check when the allocs-by-eval memdb index is corrupted or the read transaction fails mid-lookup (schema mismatch, index rebuild failure).

Common situations: Corrupted allocs index after a bad snapshot restore; deleting evals for jobs whose allocations were written by an older Nomad schema version; disk corruption on the server's state store.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• alloc lookup failed: %v
• setting job status failed: %v
• job summary lookup failed: %v
• eval delete failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/673aec8e9e249b4f. Report an issue: GitHub.