hashicorp/nomad · error

eval delete failed: %v

Error message

eval delete failed: %v

What it means

Nomad's StateStore wraps every low-level memdb transaction failure with a contextual message, and this one occurs in DeleteEvalsByFilter when txn.Delete("evals", eval) fails inside the filtered batch delete. The wrapped error comes from the underlying memdb write transaction, which only fails on real store problems (table/index corruption, an object not matching the table schema, or a txn in an invalid state) — not because the eval is missing. It aborts the whole transaction before Commit, so no evals in the batch are removed.

Source

Thrown at nomad/state/state_store.go:3643

		raw := iter.Next()
		if raw == nil {
			break
		}
		eval := raw.(*structs.Evaluation)
		if eval.ID < pageToken {
			continue
		}

		deleteOk, err := s.EvalIsUserDeleteSafe(nil, eval)
		if !deleteOk || err != nil {
			continue
		}
		match, err := filter.Evaluate(eval)
		if !match || err != nil {
			continue
		}
		if err := txn.Delete("evals", eval); err != nil {
			return fmt.Errorf("eval delete failed: %v", err)
		}
		pageCount++
	}

	err = txn.Commit()
	return err
}

// EvalIsUserDeleteSafe ensures an evaluation is safe to delete based on its
// related allocation and job information. This follows similar, but different
// rules to the eval reap checking, to ensure evaluations for running allocs or
// allocs which need the evaluation detail are not deleted.
//
// Returns both a bool and an error so that error in querying the related
// objects can be differentiated from reporting that the eval isn't safe to
// delete.
func (s *StateStore) EvalIsUserDeleteSafe(ws memdb.WatchSet, eval *structs.Evaluation) (bool, error) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Inspect the wrapped %v cause — it identifies the actual memdb/table problem; fix that root cause first
2. Verify state store integrity: compare the failing server against peers (nomad server members / Raft state) and replace the degraded server
3. Restore a known-good Raft snapshot if corruption is confirmed
4. Restart the server and let Raft replay/resync; if persistent, rejoin the server as a fresh member

// Before issuing the eval delete-by-filter request, verify server health and store integrity:
health, _, _ := client.Agent().Health()
if health.Server.Ok == false { return fmt.Errorf("server not healthy; skip eval delete") }
// Optionally dry-run the filter:
// verify matching evals exist via client.Evals().List(nil) before deleting

func isMemdbCorruption(err error) bool {
	return err != nil && strings.Contains(err.Error(), "object is not valid")
}

// Nomad Go client surfaces RPC errors as error; retry transient failures
var lastErr error
for i := 0; i < 3; i++ {
	_, _, err := client.Evals().DeleteByFilter(nil, filter) // or equivalent RPC
	if err == nil { break }
	if isMemdbCorruption(err) { return err } // do not retry corruption
	lastErr = err; time.Sleep(backoff(i))
}

When it happens

Trigger: Calling the eval-delete-by-filter RPC/endpoint when the memdb evals table or its indexes are corrupted (e.g. snapshot restore of a bad snapshot), or an internal state store schema mismatch between the eval object being deleted and the table schema.

Common situations: Restored Raft snapshots on servers with version skew; disk-level corruption of the BoltDB state store; custom eval deletion tooling hitting servers mid-upgrade; operator-driven eval GC failing on a degraded server.

Related errors

• eval broker is enabled; eval broker must be paused to delete
• index update failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/0dbd3f316ccdb512. Report an issue: GitHub.