hashicorp/nomad · critical

failed to snapshot state: %v

Error message

failed to snapshot state: %v

What it means

createNodeEvals snapshots the Raft-backed state store before scanning allocations. If the in-memory state store cannot produce a consistent snapshot (an internal memdb error), the endpoint returns this wrapped error. It indicates internal state-store trouble, not a user-input problem.

Source

Thrown at nomad/node_endpoint.go:1813

			}
			reply.Index = index

			// Set the query response
			n.srv.setQueryMeta(&reply.QueryMeta)
			return nil
		}}
	return n.srv.blockingRPC(&opts)
}

// createNodeEvals is used to create evaluations for each alloc on a node.
// Each Eval is scoped to a job, so we need to potentially trigger many evals.
func (n *Node) createNodeEvals(node *structs.Node, nodeIndex uint64) ([]string, uint64, error) {
	nodeID := node.ID

	// Snapshot the state
	snap, err := n.srv.fsm.State().Snapshot()
	if err != nil {
		return nil, 0, fmt.Errorf("failed to snapshot state: %v", err)
	}

	// Find all the allocations for this node
	allocs, err := snap.AllocsByNode(nil, nodeID)
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find allocs for '%s': %v", nodeID, err)
	}

	sysJobsIter, err := snap.JobsByScheduler(nil, "system")
	if err != nil {
		return nil, 0, fmt.Errorf("failed to find system jobs for '%s': %v", nodeID, err)
	}

	var sysJobs []*structs.Job
	for jobI := sysJobsIter.Next(); jobI != nil; jobI = sysJobsIter.Next() {
		job := jobI.(*structs.Job)
		// Avoid creating evals for jobs that don't run in this datacenter or
		// node pool. We could perform an entire feasibility check here, but

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Restart the affected Nomad server (followers first, then leader via a controlled leader election)
2. Check server logs for preceding state-store or memdb errors to identify the root cause
3. Verify memory resources on the server host; increase if OOM/thrashing is suspected
4. If corruption persists, restore the server from a backup or rebuild the node by re-adding it

// server-side: check state-store health before heavy RPCs
if err := fsm.State().Snapshot(); err != nil { /* backoff and retry RPC */ }

var evalErr *evals.CreateFailedEvalError // pseudo
resp, err := client.Nodes().UpdateStatus(req)
if err != nil && strings.Contains(err.Error(), "failed to snapshot state") {
    time.Sleep(backoff)
    resp, err = client.Nodes().UpdateStatus(req) // retry against another server
}

When it happens

Trigger: Any RPC that leads to eval creation — Register, deregister, UpdateStatus, UpdateDrain, UpdateEligibility, Evaluate — when n.srv.fsm.State().Snapshot() returns an error (state store corruption, memory pressure, internal memdb failure).

Common situations: Server under severe memory pressure, corrupted state store after a crash, or bugs in the FSM state store; often appears alongside other state-store errors on the same server.

Related errors

• error getting plugin: %s, %v
• error querying volume %q: %v
• failed to determine last evaluation index for job %q: %v
• failed to lookup state snapshot: %v
• failed to retrieve jobs for idempotency check

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/215b226d2ff268b3. Report an issue: GitHub.