hashicorp/nomad · critical

failed to lookup state snapshot: %v

Error message

failed to lookup state snapshot: %v

What it means

Returned by Evaluations.Delete when the server cannot obtain a consistent state-store snapshot needed to verify the evals are safe to delete. The underlying state store error is embedded in the message.

Source

Thrown at nomad/eval_endpoint.go:502

		return errors.New("eval broker is enabled; eval broker must be paused to delete evals")
	}

	if args.Filter != "" {
		count, index, err := e.deleteEvalsByFilter(args)
		if err != nil {
			return err
		}

		// Update the index and return.
		reply.Index = index
		reply.Count = count
		return nil
	}

	// Grab the state snapshot, so we can look up relevant eval information.
	serverStateSnapshot, err := e.srv.State().Snapshot()
	if err != nil {
		return fmt.Errorf("failed to lookup state snapshot: %v", err)
	}
	ws := memdb.NewWatchSet()

	count := 0

	// Iterate the evaluations and ensure they are safe to delete. It is
	// possible passed evals are not safe to delete and would make Nomads state
	// a little wonky. The nature of the RPC return error, means a single
	// unsafe eval ID fails the whole call.
	for _, evalID := range args.EvalIDs {

		evalInfo, err := serverStateSnapshot.EvalByID(ws, evalID)
		if err != nil {
			return fmt.Errorf("failed to lookup eval: %v", err)
		}
		if evalInfo == nil {
			return errors.New("eval not found")
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the embedded cause in the error message and fix the underlying state store issue (disk space, permissions, corruption)
2. Retry the request against a healthy server/leader
3. Restore the server from backup or rejoin it to the cluster if the state store is corrupt

// check server health first
_, _, err := client.Agent().Health()
if err != nil { return fmt.Errorf("server unhealthy: %w", err) }

err := deleteEvals(...)
if err != nil && strings.Contains(err.Error(), "failed to lookup state snapshot") {
    // retry later or target another server
    return retryWithBackoff(err)
}

When it happens

Trigger: e.srv.State().Snapshot() fails during the Delete RPC, typically because the underlying BoltDB/raft state store is unavailable, corrupt, or the server is shutting down.

Common situations: Disk I/O errors or full disk on the server; state store closed during server shutdown/graceful leave; corruption after a crash.

Related errors

• failed to retrieve jobs for idempotency check
• error querying volume %q: %v
• failed to snapshot state: %v
• node lookup failed: %v
• node update failed: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c8297d685a0bae7a. Report an issue: GitHub.