hashicorp/nomad · error

failed to restore from snapshot: %w

Error message

failed to restore from snapshot: %w

What it means

After piping the snapshot archive in, RestoreFromArchive calls fsm.RestoreWithFilter to stream the snapshot into the dummy FSM's state store. Any error from that restore (bad snapshot format, unsupported/unknown message types, decoding failures, filter errors) is wrapped as "failed to restore from snapshot: %w".

Source

Thrown at helper/raftutil/snapshot.go:44

	// r is closed by RestoreFiltered, w is closed by CopySnapshot
	r, w := io.Pipe()

	errCh := make(chan error)
	metaCh := make(chan *raft.SnapshotMeta)

	go func() {
		meta, err := snapshot.CopySnapshot(archive, w)
		if err != nil {
			errCh <- fmt.Errorf("failed to read snapshot: %w", err)
		} else {
			metaCh <- meta
		}
	}()

	err = fsm.RestoreWithFilter(r, filter)
	if err != nil {
		return nil, nil, nil, fmt.Errorf("failed to restore from snapshot: %w", err)
	}

	select {
	case err := <-errCh:
		return nil, nil, nil, err
	case meta := <-metaCh:
		return fsm, fsm.State(), meta, nil
	}
}

func RedactSnapshot(srcFile *os.File) error {
	srcFile.Seek(0, 0)
	fsm, store, meta, err := RestoreFromArchive(srcFile, nil)
	if err != nil {
		return fmt.Errorf("Failed to load snapshot from archive: %w", err)
	}

	iter, err := store.RootKeys(nil)

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped cause: decoding/unknown-type errors usually mean a Nomad version mismatch — upgrade the tool/binary to at least the version that wrote the snapshot.
2. Validate the archive with `nomad operator snapshot inspect` to rule out corruption.
3. If using a custom FSMFilter, check its logic — a filtering bug can make restore fail; try RestoreFromArchive with filter=nil to isolate.
4. Re-take the snapshot from a healthy leader if the archive is corrupt.

Example fix

// before: restoring a v1.5 snapshot with a v1.4 binary
fsm, _, _, err := raftutil.RestoreFromArchive(f, nil) // "failed to restore from snapshot: ... unknown message type"
// after: use a binary >= snapshot version
// (upgrade tooling) then:
fsm, _, _, err := raftutil.RestoreFromArchive(f, nil) // succeeds

// Check version compatibility before attempting restore
// snapshot meta (raft.SnapshotMeta) is only available after restore, so pre-validate with CLI:
if out, err := exec.Command("nomad", "operator", "snapshot", "inspect", path).CombinedOutput(); err != nil {
    return fmt.Errorf("snapshot not compatible with this binary: %s", out)
}

fsm, store, meta, err := raftutil.RestoreFromArchive(archive, filter)
if err != nil {
    if strings.Contains(err.Error(), "failed to restore from snapshot") {
        // decoding/unknown-type failures usually mean version skew: retry with a newer binary
        return fmt.Errorf("restore rejected archive (check Nomad version skew / FSMFilter): %w", err)
    }
    return err
}

When it happens

Trigger: RestoreWithFilter fails while consuming the piped snapshot: malformed archive contents, snapshot entries that fail to decode into the current FSM schema, an FSMFilter rejecting an entry, or the copy goroutine dying and closing the pipe mid-stream.

Common situations: Restoring a snapshot from a newer Nomad version into an older binary (unknown schemas/IMPL versions); a corrupted snapshot; a non-nil FSMFilter that rejects required entries; RedactSnapshot or NewHarnessFromSnapshot operating on an incompatible archive.

Related errors

• failed to create FSM: %w
• Failed to create redacted snapshot: %v
• failed to restore from snapshot: %v
• volume snapshot ID cannot be updated
• missing VolumeID

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/007187bb6c7290d4. Report an issue: GitHub.