{"record":{"id":"1b20aa9be66297e8","repo":"hashicorp/nomad","slug":"failed-to-stat-raft-store-s-v","errorCode":null,"errorMessage":"failed to stat raft store %s: %v","messagePattern":"failed to stat raft store (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"helper/raftutil/state.go","lineNumber":319,"sourceCode":"\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\treturn raftpath, nil\n}\n\n// FindRaftDir locates the raft data directory (the parent directory containing\n// either raft.db or wal/). Returns the directory path regardless of backend.\nfunc FindRaftDir(p string) (string, error) {\n\tstorePath, err := FindRaftStore(p)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\tinfo, statErr := os.Stat(storePath)\n\tif statErr != nil {\n\t\treturn \"\", fmt.Errorf(\"failed to stat raft store %s: %v\", storePath, statErr)\n\t}\n\n\t// For WAL the store path IS a directory (wal/); return its parent.\n\t// For BoltDB the store path is a file (raft.db); return its parent.\n\tif info.IsDir() {\n\t\treturn filepath.Dir(storePath), nil\n\t}\n\treturn filepath.Dir(storePath), nil\n}\n\n// FindFileInPath searches for file in path p\nfunc FindFileInPath(file string, p string) (path string, err error) {\n\t// Define walk function to find file\n\twalkFn := func(walkPath string, info os.FileInfo, err error) error {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n","sourceCodeStart":301,"sourceCodeEnd":337,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/helper/raftutil/state.go#L301-L337","documentation":"FindRaftDir resolves the raft data directory by first locating the store with FindRaftStore and then os.Stat-ing the resulting store path to determine whether it is a WAL directory or a BoltDB file, returning its parent directory. This error means the path that was just found could not be stat-ed — almost always a race where the file/directory was removed or renamed between discovery and the stat call.","triggerScenarios":"FindRaftDir → FindRaftStore finds a store path, then os.Stat(storePath) fails (file deleted, permissions changed, mount unmounted, or a stale/broken symlink) before the stat executes.","commonSituations":"Recovery scripts running while a server process is being cleaned up or the data dir is being wiped; running the tool as a user without permissions on the raft directory (e.g. not root and raft.db is 0600); network/ephemeral storage that unmounted between calls.","solutions":["Re-run FindRaftDir — if it's a race, the store may reappear or you need to run against a stable copy of the data dir.","Check permissions on the store path (ls -l); run as a user with read access (often root) since raft.db is typically 0600.","Copy the raft store to a stable location first and run FindRaftDir against the copy to avoid racing a live agent.","Verify the filesystem/mount backing the data dir is still mounted and healthy.","If the store is genuinely gone, restore it from backup/snapshot before running the tool."],"exampleFix":"// before: stat the live store, racing agent writes/cleanup\nstorePath, err := raftutil.FindRaftStore(dataDir)\ndir, err := raftutil.FindRaftDir(dataDir)\n// after: snapshot the store first, then operate on the stable copy\ncpCmd := exec.Command(\"cp\", \"-a\", dataDir, backupDir)\nif err := cpCmd.Run(); err != nil { log.Fatal(err) }\ndir, err := raftutil.FindRaftDir(backupDir)","handlingStrategy":"validation","validationCode":"// stat the expected store location yourself before calling FindRaftDir\nfor _, c := range []string{\"server/raft/raft.db\", \"server/raft/wal\"} {\n    if _, err := os.Stat(filepath.Join(dataDir, c)); err == nil {\n        break\n    }\n    return fmt.Errorf(\"raft store missing/unreadable under %s\", dataDir)\n}","typeGuard":null,"tryCatchPattern":"dir, err := raftutil.FindRaftDir(dataDir)\nif err != nil {\n    if strings.Contains(err.Error(), \"failed to stat raft store\") {\n        // retry once (possible race) or copy the data dir and retry\n        return retryOrCopyAndRetry(dataDir)\n    }\n    return err\n}","preventionTips":["Run inspection tooling against a stable copy of the data dir, not a live one.","Ensure the tooling user has read permission on raft.db/wal (often root-only).","Verify mounts backing the data dir are healthy before running.","Stop the Nomad agent before manipulating or copying raft data."],"tags":["raft","filesystem","permissions","race-condition"],"backgroundTag":"stat-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}