hashicorp/nomad · error

failed to get blocked evals for job %q in namespace %q: %v

Error message

failed to get blocked evals for job %q in namespace %q: %v

What it means

Returned by UpsertEvals in Nomad's state store when the memdb query to fetch blocked evaluations for a completed job (via txn.Get on the "evals" table with the "job" index) fails. It wraps the underlying memdb lookup error and aborts the eval upsert transaction. It signals an index/table lookup problem, not merely 'no evals found'.

Source

Thrown at nomad/state/state_store.go:3529

		// Insert the job summary
		if hasSummaryChanged {
			js.ModifyIndex = index
			if err := txn.Insert("job_summary", js); err != nil {
				return fmt.Errorf("job summary insert failed: %v", err)
			}
			if err := txn.Insert("index", &IndexEntry{"job_summary", index}); err != nil {
				return fmt.Errorf("index update failed: %v", err)
			}
		}
	}

	// Check if the job has any blocked evaluations and cancel them
	if eval.Status == structs.EvalStatusComplete && len(eval.FailedTGAllocs) == 0 {
		// Get the blocked evaluation for a job if it exists
		iter, err := txn.Get("evals", "job", eval.Namespace, eval.JobID, structs.EvalStatusBlocked)
		if err != nil {
			return fmt.Errorf("failed to get blocked evals for job %q in namespace %q: %v", eval.JobID, eval.Namespace, err)
		}

		var blocked []*structs.Evaluation
		for {
			raw := iter.Next()
			if raw == nil {
				break
			}
			blocked = append(blocked, raw.(*structs.Evaluation))
		}

		// Go through and update the evals
		for _, blockedEval := range blocked {
			newEval := blockedEval.Copy()
			newEval.Status = structs.EvalStatusCancelled
			newEval.StatusDescription = fmt.Sprintf("evaluation %q successful", eval.ID)
			newEval.ModifyIndex = index
			newEval.ModifyTime = eval.ModifyTime

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry the operation (the wrapping Raft apply will run in a new transaction)
2. Check server logs for a prior memdb panic; restart the Nomad server to rebuild state from Raft/snapshot
3. Upgrade Nomad if a matching memdb bug is fixed in a newer release
4. Capture the wrapped inner error (%v) for a HashiCorp bug report if reproducible

Example fix

// caller side: treat as retryable state-store failure
// before
err := stateStore.UpsertEvals(idx, evals)
// after
if err != nil && strings.HasPrefix(err.Error(), "failed to get blocked evals") {
    // schedule retry of the raft apply
}

// Go
if err := state.UpsertEvals(idx, evals); err != nil {
    if strings.Contains(err.Error(), "failed to get blocked evals") {
        return retry(fn) // fresh txn on retry
    }
    return err
}

When it happens

Trigger: txn.Get("evals", "job", eval.Namespace, eval.JobID, structs.EvalStatusBlocked) returns an error while cancelling blocked evals for a completed eval with no FailedTGAllocs — typically an aborted txn or memdb internal failure.

Common situations: Server state store under memory pressure; corrupted in-memory DB after failed restore; rare memdb bugs during high eval throughput.

Related errors

• index update failed: %v
• csi_plugin lookup error: %s %v
• csi_plugins insert error: %v
• csi_plugins lookup failed: %v
• csi_plugins lookup error %s: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/9ddd724ee99639d8. Report an issue: GitHub.