{"record":{"id":"eae04e80dd255627","repo":"hashicorp/nomad","slug":"raft-error-when-taking-snapshot-v","errorCode":null,"errorMessage":"Raft error when taking snapshot: %v","messagePattern":"Raft error when taking snapshot: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/snapshot/snapshot.go","lineNumber":38,"sourceCode":"\n// Snapshot is a structure that holds state about a temporary file that is used\n// to hold a snapshot. By using an intermediate file we avoid holding everything\n// in memory.\ntype Snapshot struct {\n\tfile     *os.File\n\tindex    uint64\n\tchecksum string\n}\n\n// New takes a state snapshot of the given Raft instance into a temporary file\n// and returns an object that gives access to the file as an io.Reader. You must\n// arrange to call Close() on the returned object or else you will leak a\n// temporary file.\nfunc New(logger hclog.Logger, r *raft.Raft) (*Snapshot, error) {\n\t// Take the snapshot.\n\tfuture := r.Snapshot()\n\tif err := future.Error(); err != nil {\n\t\treturn nil, fmt.Errorf(\"Raft error when taking snapshot: %v\", err)\n\t}\n\n\t// Open up the snapshot.\n\tmetadata, snap, err := future.Open()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to open snapshot: %v:\", err)\n\t}\n\n\treturn writeSnapshot(logger, metadata, snap)\n}\n\n// NewFromFSM takes a state snapshot of the given FSM (for when we don't have a\n// Raft instance setup) into a temporary file and returns an object that gives\n// access to the file as an io.Reader. You must arrange to call Close() on the\n// returned object or else you will leak a temporary file.\nfunc NewFromFSM(logger hclog.Logger, fsm raft.FSM, meta *raft.SnapshotMeta) (*Snapshot, error) {\n\t_, trans := raft.NewInmemTransport(\"\")\n\tsnapshotStore := raft.NewInmemSnapshotStore()","sourceCodeStart":20,"sourceCodeEnd":56,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/snapshot/snapshot.go#L20-L56","documentation":"snapshot.New asks the hashicorp/raft instance to take a snapshot via r.Snapshot() and waits on the returned future. When the Raft layer itself fails to produce a snapshot (leader lost leadership, snapshot in progress, FSM snapshot error, store error), the future's Error() is non-nil and this message wraps it. The root cause is always in the wrapped Raft error text.","triggerScenarios":"Calling snapshot.New (e.g. via nomad operator snapshot save, automatic snapshot agents, or generateSnapshot/snapshotSave) when: the node is not the Raft leader, a snapshot is already being taken, the FSM's Snapshot() method errors, the snapshot store cannot create a sink (disk full/permissions), or leadership changes mid-snapshot (leadership lost).","commonSituations":"Running nomad operator snapshot save against a follower node; full disk on the server; Raft snapshot threshold/restore storms; elections occurring during a scheduled snapshot job; FSM (Nomad state store) returning an error under load.","solutions":["Read the wrapped Raft error: if it says 'node is not the leader', direct the snapshot request to the current leader (nomad operator snapshot save handles this; for direct raft use, wait for leadership or use raft.TakeSnapshot on the leader).","Check the server's disk for full volumes or permission problems on the data dir where Raft writes snapshots.","Retry after a short delay if a concurrent snapshot or a leadership change was in flight; add jitter to scheduled snapshot jobs.","Inspect Nomad server logs around the failure for FSM/persist errors and address the underlying state-store or storage issue."],"exampleFix":"// before: saving a snapshot without checking leadership\nsnap, err := snapshot.New(logger, raftInst)\n\n// after: ensure this node leads and retry transient failures\nif raftInst.State() != raft.Leader {\n    return fmt.Errorf(\"refusing to snapshot: node is not the leader\")\n}\nvar snap *snapshot.Snapshot\nfor i := 0; i < 3; i++ {\n    snap, err = snapshot.New(logger, raftInst)\n    if err == nil || !strings.Contains(fmt.Sprint(err), \"leadership lost\") {\n        break\n    }\n    time.Sleep(2 * time.Second)\n}","handlingStrategy":"retry","validationCode":"// Check preconditions before asking Raft for a snapshot\nfunc canSnapshot(r *raft.Raft) error {\n    if r.State() != raft.Leader {\n        return fmt.Errorf(\"not the leader (state=%s)\", r.State())\n    }\n    if err := checkDiskFree(defaultDataDir, 512<<20); err != nil {\n        return fmt.Errorf(\"insufficient disk for snapshot: %w\", err)\n    }\n    return nil\n}","typeGuard":null,"tryCatchPattern":"var snap *snapshot.Snapshot\nvar err error\nfor attempt := 0; attempt < 3; attempt++ {\n    snap, err = snapshot.New(logger, raftInst)\n    if err == nil {\n        break\n    }\n    var transient = strings.Contains(err.Error(), \"leadership lost\") ||\n        strings.Contains(err.Error(), \"snapshot in progress\")\n    if !transient {\n        return err\n    }\n    time.Sleep(time.Duration(1<<attempt) * time.Second)\n}\nif err != nil {\n    return fmt.Errorf(\"Raft error when taking snapshot: %w\", err)\n}","preventionTips":["Only take snapshots on the Raft leader; route snapshot requests through the leader.","Monitor disk usage on Nomad servers and alert before volumes fill.","Add jitter to scheduled snapshot jobs to avoid election windows.","Log the wrapped Raft error text; it names the exact underlying cause."],"tags":["go","raft","consensus","snapshot"],"backgroundTag":"raft-snapshot-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}