hashicorp/nomad · error

Raft error when taking snapshot: %v

Error message

Raft error when taking snapshot: %v

What it means

snapshot.New asks the hashicorp/raft instance to take a snapshot via r.Snapshot() and waits on the returned future. When the Raft layer itself fails to produce a snapshot (leader lost leadership, snapshot in progress, FSM snapshot error, store error), the future's Error() is non-nil and this message wraps it. The root cause is always in the wrapped Raft error text.

Source

Thrown at helper/snapshot/snapshot.go:38

// Snapshot is a structure that holds state about a temporary file that is used
// to hold a snapshot. By using an intermediate file we avoid holding everything
// in memory.
type Snapshot struct {
	file     *os.File
	index    uint64
	checksum string
}

// New takes a state snapshot of the given Raft instance into a temporary file
// and returns an object that gives access to the file as an io.Reader. You must
// arrange to call Close() on the returned object or else you will leak a
// temporary file.
func New(logger hclog.Logger, r *raft.Raft) (*Snapshot, error) {
	// Take the snapshot.
	future := r.Snapshot()
	if err := future.Error(); err != nil {
		return nil, fmt.Errorf("Raft error when taking snapshot: %v", err)
	}

	// Open up the snapshot.
	metadata, snap, err := future.Open()
	if err != nil {
		return nil, fmt.Errorf("failed to open snapshot: %v:", err)
	}

	return writeSnapshot(logger, metadata, snap)
}

// NewFromFSM takes a state snapshot of the given FSM (for when we don't have a
// Raft instance setup) into a temporary file and returns an object that gives
// access to the file as an io.Reader. You must arrange to call Close() on the
// returned object or else you will leak a temporary file.
func NewFromFSM(logger hclog.Logger, fsm raft.FSM, meta *raft.SnapshotMeta) (*Snapshot, error) {
	_, trans := raft.NewInmemTransport("")
	snapshotStore := raft.NewInmemSnapshotStore()

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped Raft error: if it says 'node is not the leader', direct the snapshot request to the current leader (nomad operator snapshot save handles this; for direct raft use, wait for leadership or use raft.TakeSnapshot on the leader).
2. Check the server's disk for full volumes or permission problems on the data dir where Raft writes snapshots.
3. Retry after a short delay if a concurrent snapshot or a leadership change was in flight; add jitter to scheduled snapshot jobs.
4. Inspect Nomad server logs around the failure for FSM/persist errors and address the underlying state-store or storage issue.

Example fix

// before: saving a snapshot without checking leadership
snap, err := snapshot.New(logger, raftInst)

// after: ensure this node leads and retry transient failures
if raftInst.State() != raft.Leader {
    return fmt.Errorf("refusing to snapshot: node is not the leader")
}
var snap *snapshot.Snapshot
for i := 0; i < 3; i++ {
    snap, err = snapshot.New(logger, raftInst)
    if err == nil || !strings.Contains(fmt.Sprint(err), "leadership lost") {
        break
    }
    time.Sleep(2 * time.Second)
}

// Check preconditions before asking Raft for a snapshot
func canSnapshot(r *raft.Raft) error {
    if r.State() != raft.Leader {
        return fmt.Errorf("not the leader (state=%s)", r.State())
    }
    if err := checkDiskFree(defaultDataDir, 512<<20); err != nil {
        return fmt.Errorf("insufficient disk for snapshot: %w", err)
    }
    return nil
}

var snap *snapshot.Snapshot
var err error
for attempt := 0; attempt < 3; attempt++ {
    snap, err = snapshot.New(logger, raftInst)
    if err == nil {
        break
    }
    var transient = strings.Contains(err.Error(), "leadership lost") ||
        strings.Contains(err.Error(), "snapshot in progress")
    if !transient {
        return err
    }
    time.Sleep(time.Duration(1<<attempt) * time.Second)
}
if err != nil {
    return fmt.Errorf("Raft error when taking snapshot: %w", err)
}

When it happens

Trigger: Calling snapshot.New (e.g. via nomad operator snapshot save, automatic snapshot agents, or generateSnapshot/snapshotSave) when: the node is not the Raft leader, a snapshot is already being taken, the FSM's Snapshot() method errors, the snapshot store cannot create a sink (disk full/permissions), or leadership changes mid-snapshot (leadership lost).

Common situations: Running nomad operator snapshot save against a follower node; full disk on the server; Raft snapshot threshold/restore storms; elections occurring during a scheduled snapshot job; FSM (Nomad state store) returning an error under load.

Related errors

• failed to open snapshot dir: %v
• Failed to load snapshot from archive: %w
• Failed to query for root keys: %v
• Could not re-encode redacted key: %v
• Failed to create redacted snapshot: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/eae04e80dd255627. Report an issue: GitHub.