{"record":{"id":"dfde10cc410a482d","repo":"hashicorp/nomad","slug":"failed-to-open-snapshot-v","errorCode":null,"errorMessage":"failed to open snapshot: %v:","messagePattern":"failed to open snapshot: (.+?):","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/snapshot/snapshot.go","lineNumber":44,"sourceCode":"\tindex    uint64\n\tchecksum string\n}\n\n// New takes a state snapshot of the given Raft instance into a temporary file\n// and returns an object that gives access to the file as an io.Reader. You must\n// arrange to call Close() on the returned object or else you will leak a\n// temporary file.\nfunc New(logger hclog.Logger, r *raft.Raft) (*Snapshot, error) {\n\t// Take the snapshot.\n\tfuture := r.Snapshot()\n\tif err := future.Error(); err != nil {\n\t\treturn nil, fmt.Errorf(\"Raft error when taking snapshot: %v\", err)\n\t}\n\n\t// Open up the snapshot.\n\tmetadata, snap, err := future.Open()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to open snapshot: %v:\", err)\n\t}\n\n\treturn writeSnapshot(logger, metadata, snap)\n}\n\n// NewFromFSM takes a state snapshot of the given FSM (for when we don't have a\n// Raft instance setup) into a temporary file and returns an object that gives\n// access to the file as an io.Reader. You must arrange to call Close() on the\n// returned object or else you will leak a temporary file.\nfunc NewFromFSM(logger hclog.Logger, fsm raft.FSM, meta *raft.SnapshotMeta) (*Snapshot, error) {\n\t_, trans := raft.NewInmemTransport(\"\")\n\tsnapshotStore := raft.NewInmemSnapshotStore()\n\n\tfsmSnap, err := fsm.Snapshot()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n","sourceCodeStart":26,"sourceCodeEnd":62,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/snapshot/snapshot.go#L26-L62","documentation":"After the Raft snapshot future completes, snapshot.New calls future.Open() to get the metadata and a reader for the snapshot store. This error wraps any failure opening the persisted snapshot from Raft's snapshot store, most commonly because the snapshot files were deleted or corrupted on disk between creation and open. Note the format string itself has a stray trailing colon after %v.","triggerScenarios":"Calling snapshot.New (via nomad operator snapshot save or automatic snapshotting) when future.Open() fails: the raft snapshot directory was pruned/cleaned concurrently, permissions on the snapshot file changed, disk I/O error reading the snapshot, or an internal raft race removed the snapshot sink before it was opened.","commonSituations":"External cleanup jobs deleting files under the Nomad data dir's raft/ subdirectory; full or failing disks; restoring a server from a partial backup missing raft snapshot files; running multiple snapshot operations that race with raft's own retention compaction.","solutions":["Retry snapshot.New; if the failure came from a transient raft retention race, a fresh r.Snapshot() will usually succeed.","Check the server's data dir (raft/ snapshots) for missing or unreadable files; stop any external cleanup scripts touching the Nomad data directory.","Check disk space and filesystem health (dmesg, mount ro state) on the volume holding the Nomad data dir.","If snapshots persistently fail, restart the Nomad server agent so Raft rebuilds its snapshot store state, then retake the snapshot."],"exampleFix":"// before: single attempt, ambiguous error\nsnap, err := snapshot.New(logger, raftInst)\nif err != nil {\n    return err // \"failed to open snapshot: ...:\"\n}\n\n// after: retry transient open failures\nvar snap *snapshot.Snapshot\nerr = retry.Do(func() error {\n    snap, err = snapshot.New(logger, raftInst)\n    return err\n}, retry.Attempts(3), retry.Delay(time.Second))","handlingStrategy":"retry","validationCode":"// Sanity-check the environment before opening a Raft snapshot\nfunc snapshotEnvOK(dataDir string) error {\n    info, err := os.Stat(filepath.Join(dataDir, \"raft\", \"snapshots\"))\n    if err != nil || !info.IsDir() {\n        return fmt.Errorf(\"raft snapshot store missing or unreadable: %w\", err)\n    }\n    return checkDiskFree(dataDir, 256<<20)\n}","typeGuard":null,"tryCatchPattern":"snap, err := snapshot.New(logger, raftInst)\nif err != nil && strings.Contains(err.Error(), \"failed to open snapshot\") {\n    // snapshot vanished from the store between Snapshot() and Open():\n    // retry once with a fresh snapshot request\n    time.Sleep(time.Second)\n    snap, err = snapshot.New(logger, raftInst)\n}\nif err != nil {\n    return fmt.Errorf(\"snapshot open failed: %w\", err)\n}","preventionTips":["Never run external cron/cleanup jobs that delete files inside the Nomad data directory.","Keep the data dir on healthy local storage; watch for read-only remounts and I/O errors.","Back up the whole raft/ directory, not a partial subset, when snapshotting server state manually.","Retry transient open failures; raft retention can race with user-initiated snapshots."],"tags":["go","raft","snapshot","io"],"backgroundTag":"snapshot-store-open-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}