hashicorp/nomad · error

failed to lookup job for eval: %v

Error message

failed to lookup job for eval: %v

What it means

StateStore.EvalIsUserDeleteSafe wraps any failure from s.JobByID when resolving the job referenced by the eval being checked for user-initiated deletion safety. It means the job lookup against the jobs table returned a transaction-level error (not merely 'not found' — a nil job is handled by isEvalDeleteSafe logic downstream). The caller gets a wrapped error and a false result, so the delete is treated as unsafe.

Source

Thrown at nomad/state/state_store.go:3664

	}

	err = txn.Commit()
	return err
}

// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its
// related allocation and job information. This follows similar, but different
// rules to the eval reap checking, to ensure evaluations for running allocs or
// allocs which need the evaluation detail are not deleted.
//
// Returns both a bool and an error so that error in querying the related
// objects can be differentiated from reporting that the eval isn't safe to
// delete.
func (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {

	job, err := s.JobByID(ws, eval.Namespace, eval.JobID)
	if err != nil {
		return false, fmt.Errorf("failed to lookup job for eval: %v", err)
	}

	allocs, err := s.AllocsByEval(ws, eval.ID)
	if err != nil {
		return false, fmt.Errorf("failed to lookup eval allocs: %v", err)
	}

	return isEvalDeleteSafe(allocs, job), nil
}

func isEvalDeleteSafe(allocs []*structs.Allocation, job *structs.Job) bool {

	// If the job is deleted, stopped, or dead, all allocs are terminal and
	// the eval can be deleted.
	if job == nil || job.Stop || job.Status == structs.JobStatusDead {
		return true
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped cause for the underlying memdb error and fix the store-level root cause
2. Confirm the eval's Namespace/JobID fields are valid; a corrupted eval record can make the lookup fail
3. Check for recent snapshot restores or upgrades; restore a clean snapshot or roll back the version skew
4. Retry the deletion check after the server's state store has been verified/replaced

// Verify the job exists and the eval is well-formed before the delete-safety check
eval, _, err := client.Evals().GetEval(evalID, nil)
if err != nil { return err }
job, _, err := client.Jobs().Info(eval.Namespace, eval.JobID, nil)
if err != nil { return fmt.Errorf("job %s/%s not found: %w", eval.Namespace, eval.JobID, err) }

func evalJobLookupFailed(err error) bool {
	return err != nil && strings.Contains(err.Error(), "failed to lookup job for eval")
}

if err != nil {
	if evalJobLookupFailed(err) {
		// store-level problem: verify server health before retrying
		return fmt.Errorf("state store job lookup failed; check server health: %w", err)
	}
	return err
}

When it happens

Trigger: Calling EvalIsUserDeleteSafe (via the eval delete user-safety check RPC) when the memdb jobs table transaction errors — schema/index corruption or a broken write transaction at read time.

Common situations: State store corruption after a failed snapshot restore; evaluating deletion of an eval whose job lives in a namespace being concurrently purged; version-skewed clusters where the jobs table schema changed.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• setting job status failed: %v
• job summary lookup failed: %v
• eval delete failed: %v
• failed to lookup eval allocs: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a554c7abf67af085. Report an issue: GitHub.