hashicorp/nomad · error

failed to open snapshot: %v:

Error message

failed to open snapshot: %v:

What it means

After the Raft snapshot future completes, snapshot.New calls future.Open() to get the metadata and a reader for the snapshot store. This error wraps any failure opening the persisted snapshot from Raft's snapshot store, most commonly because the snapshot files were deleted or corrupted on disk between creation and open. Note the format string itself has a stray trailing colon after %v.

Source

Thrown at helper/snapshot/snapshot.go:44

	index    uint64
	checksum string
}

// New takes a state snapshot of the given Raft instance into a temporary file
// and returns an object that gives access to the file as an io.Reader. You must
// arrange to call Close() on the returned object or else you will leak a
// temporary file.
func New(logger hclog.Logger, r *raft.Raft) (*Snapshot, error) {
	// Take the snapshot.
	future := r.Snapshot()
	if err := future.Error(); err != nil {
		return nil, fmt.Errorf("Raft error when taking snapshot: %v", err)
	}

	// Open up the snapshot.
	metadata, snap, err := future.Open()
	if err != nil {
		return nil, fmt.Errorf("failed to open snapshot: %v:", err)
	}

	return writeSnapshot(logger, metadata, snap)
}

// NewFromFSM takes a state snapshot of the given FSM (for when we don't have a
// Raft instance setup) into a temporary file and returns an object that gives
// access to the file as an io.Reader. You must arrange to call Close() on the
// returned object or else you will leak a temporary file.
func NewFromFSM(logger hclog.Logger, fsm raft.FSM, meta *raft.SnapshotMeta) (*Snapshot, error) {
	_, trans := raft.NewInmemTransport("")
	snapshotStore := raft.NewInmemSnapshotStore()

	fsmSnap, err := fsm.Snapshot()
	if err != nil {
		return nil, err
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Retry snapshot.New; if the failure came from a transient raft retention race, a fresh r.Snapshot() will usually succeed.
2. Check the server's data dir (raft/ snapshots) for missing or unreadable files; stop any external cleanup scripts touching the Nomad data directory.
3. Check disk space and filesystem health (dmesg, mount ro state) on the volume holding the Nomad data dir.
4. If snapshots persistently fail, restart the Nomad server agent so Raft rebuilds its snapshot store state, then retake the snapshot.

Example fix

// before: single attempt, ambiguous error
snap, err := snapshot.New(logger, raftInst)
if err != nil {
    return err // "failed to open snapshot: ...:"
}

// after: retry transient open failures
var snap *snapshot.Snapshot
err = retry.Do(func() error {
    snap, err = snapshot.New(logger, raftInst)
    return err
}, retry.Attempts(3), retry.Delay(time.Second))

// Sanity-check the environment before opening a Raft snapshot
func snapshotEnvOK(dataDir string) error {
    info, err := os.Stat(filepath.Join(dataDir, "raft", "snapshots"))
    if err != nil || !info.IsDir() {
        return fmt.Errorf("raft snapshot store missing or unreadable: %w", err)
    }
    return checkDiskFree(dataDir, 256<<20)
}

snap, err := snapshot.New(logger, raftInst)
if err != nil && strings.Contains(err.Error(), "failed to open snapshot") {
    // snapshot vanished from the store between Snapshot() and Open():
    // retry once with a fresh snapshot request
    time.Sleep(time.Second)
    snap, err = snapshot.New(logger, raftInst)
}
if err != nil {
    return fmt.Errorf("snapshot open failed: %w", err)
}

When it happens

Trigger: Calling snapshot.New (via nomad operator snapshot save or automatic snapshotting) when future.Open() fails: the raft snapshot directory was pruned/cleaned concurrently, permissions on the snapshot file changed, disk I/O error reading the snapshot, or an internal raft race removed the snapshot sink before it was opened.

Common situations: External cleanup jobs deleting files under the Nomad data dir's raft/ subdirectory; full or failing disks; restoring a server from a partial backup missing raft snapshot files; running multiple snapshot operations that race with raft's own retention compaction.

Related errors

• Failed to copy snapshot to temporary file: %v
• failed to read snapshot: %w
• failed to open snapshot dir: %v
• Failed to load snapshot from archive: %w
• Failed to query for root keys: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/dfde10cc410a482d. Report an issue: GitHub.