hashicorp/nomad · warning

failed to lookup evals: %v

Error message

failed to lookup evals: %v

What it means

Returned by DeleteEvalsByFilter (eval garbage collection) when iterating all evaluations via s.Evals(nil, SortDefault) fails. The wrapped memdb iterator error aborts the GC pass. It is a full-table scan failure inside the state store, not a filter-evaluation problem.

Source

Thrown at nomad/state/state_store.go:3614

	defer txn.Abort()

	// These are always user-initiated, so ensure the eval broker is paused.
	_, schedConfig, err := s.schedulerConfigTxn(txn)
	if err != nil {
		return err
	}
	if schedConfig == nil || !schedConfig.PauseEvalBroker {
		return errors.New("eval broker is enabled; eval broker must be paused to delete evals")
	}

	filter, err := bexpr.CreateEvaluator(filterExpr)
	if err != nil {
		return err
	}

	iter, err := s.Evals(nil, SortDefault)
	if err != nil {
		return fmt.Errorf("failed to lookup evals: %v", err)
	}

	// Note: Paginator imports this package for testing so we can't just use
	// Paginator
	pageCount := int32(0)

	for {
		if pageCount >= perPage {
			break
		}
		raw := iter.Next()
		if raw == nil {
			break
		}
		eval := raw.(*structs.Evaluation)
		if eval.ID < pageToken {
			continue
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Wait for the next GC cycle; the periodic GC will retry automatically
2. Reduce eval backlog manually (nomad system gc) and monitor server memory
3. Restart servers to rebuild state if the error is persistent
4. Upgrade Nomad; report with the wrapped inner error if reproducible

Example fix

// before
err := s.DeleteEvalsByFilter(...)
// after
if err != nil && strings.Contains(err.Error(), "failed to lookup evals") {
    // skip this GC round; retry on next tick
    return nil
}

// Go
if err := s.DeleteEvalsByFilter(...); err != nil {
    if strings.Contains(err.Error(), "failed to lookup evals") {
        time.Sleep(gcInterval); return s.DeleteEvalsByFilter(...)
    }
    return err
}

When it happens

Trigger: s.Evals(nil, SortDefault) errors during DeleteEvalsByFilter — typically an aborted transaction, memdb internal error, or resource exhaustion while building the eval iterator during GC.

Common situations: Eval GC on clusters with very large eval backlogs; memory pressure during compaction; server recovering after failover or snapshot restore.

Related errors

• deployment id not found: %q
• index update failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/8345bfbac1080770. Report an issue: GitHub.