hashicorp/nomad · critical

failed to restore from snapshot: %v

Error message

failed to restore from snapshot: %v

What it means

Wraps an error from snapshot.Restore when the server fails to restore state from a streamed snapshot in snapshotRestore. This runs after decodeStreamOutput, meaning the stream decoded fine but applying it to Raft failed. Status 500 is returned to the client.

Source

Thrown at nomad/operator_endpoint.go:698

	if aclObj, err := op.srv.ResolveACL(&args); err != nil {
		code := 500
		if err == structs.ErrTokenNotFound {
			code = 400
		}
		handleFailure(code, err)
		return
	} else if !aclObj.IsManagement() {
		handleFailure(403, structs.ErrPermissionDenied)
		return
	}

	op.srv.setQueryMeta(&reply.QueryMeta)

	reader, errCh := decodeStreamOutput(decoder)

	err = snapshot.Restore(op.logger.Named("snapshot"), reader, op.srv.raft)
	if err != nil {
		handleFailure(500, fmt.Errorf("failed to restore from snapshot: %v", err))
		return
	}

	err = <-errCh
	if err != nil {
		handleFailure(400, fmt.Errorf("failed to read stream: %v", err))
		return
	}

	// This'll be used for feedback from the leader loop.
	timeoutCh := time.After(time.Minute)

	lerrCh := make(chan error, 1)

	select {
	// Reassert leader actions and update all leader related state
	// with new state store content.
	case op.srv.reassertLeaderCh <- lerrCh:

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the snapshot was produced by a Nomad version compatible with this server
2. Re-download/re-transfer the snapshot and checksum it before restoring again
3. Check Raft peer health and quorum on the target cluster
4. Review the wrapped error in logs to distinguish corruption vs version mismatch

// inspect snapshot before restoring
out, err := exec.Command("nomad", "operator", "snapshot", "inspect", snapFile).CombinedOutput()
if err != nil { return fmt.Errorf("invalid snapshot: %v: %s", err, out) }

err := client.Operator().SnapshotRestore(snapReader, nil, nil)
if err != nil && strings.Contains(err.Error(), "failed to restore from snapshot") {
    // check version compatibility and raft health before retry
}

When it happens

Trigger: snapshot.Restore returns an error while ingesting the decoded snapshot reader into op.srv.raft — e.g. corrupt snapshot data, incompatible snapshot version, or Raft apply failure.

Common situations: Restoring a snapshot taken from a different Nomad version into an older cluster; corrupted snapshot file from a failed download; Raft cluster quorum issues during restore.

Related errors

• failed to restore state
• failed to read snapshot: %w
• failed to open snapshot dir: %v
• failed to restore from snapshot: %w
• Failed to load snapshot from archive: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f756462ecacd26c3. Report an issue: GitHub.