hashicorp/nomad · error

failed to stat raft store %s: %v

Error message

failed to stat raft store %s: %v

What it means

FindRaftDir resolves the raft data directory by first locating the store with FindRaftStore and then os.Stat-ing the resulting store path to determine whether it is a WAL directory or a BoltDB file, returning its parent directory. This error means the path that was just found could not be stat-ed — almost always a race where the file/directory was removed or renamed between discovery and the stat call.

Source

Thrown at helper/raftutil/state.go:319

	if err != nil {
		return "", err
	}

	return raftpath, nil
}

// FindRaftDir locates the raft data directory (the parent directory containing
// either raft.db or wal/). Returns the directory path regardless of backend.
func FindRaftDir(p string) (string, error) {
	storePath, err := FindRaftStore(p)
	if err != nil {
		return "", err
	}

	info, statErr := os.Stat(storePath)
	if statErr != nil {
		return "", fmt.Errorf("failed to stat raft store %s: %v", storePath, statErr)
	}

	// For WAL the store path IS a directory (wal/); return its parent.
	// For BoltDB the store path is a file (raft.db); return its parent.
	if info.IsDir() {
		return filepath.Dir(storePath), nil
	}
	return filepath.Dir(storePath), nil
}

// FindFileInPath searches for file in path p
func FindFileInPath(file string, p string) (path string, err error) {
	// Define walk function to find file
	walkFn := func(walkPath string, info os.FileInfo, err error) error {
		if err != nil {
			return err
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Re-run FindRaftDir — if it's a race, the store may reappear or you need to run against a stable copy of the data dir.
2. Check permissions on the store path (ls -l); run as a user with read access (often root) since raft.db is typically 0600.
3. Copy the raft store to a stable location first and run FindRaftDir against the copy to avoid racing a live agent.
4. Verify the filesystem/mount backing the data dir is still mounted and healthy.
5. If the store is genuinely gone, restore it from backup/snapshot before running the tool.

Example fix

// before: stat the live store, racing agent writes/cleanup
storePath, err := raftutil.FindRaftStore(dataDir)
dir, err := raftutil.FindRaftDir(dataDir)
// after: snapshot the store first, then operate on the stable copy
cpCmd := exec.Command("cp", "-a", dataDir, backupDir)
if err := cpCmd.Run(); err != nil { log.Fatal(err) }
dir, err := raftutil.FindRaftDir(backupDir)

// stat the expected store location yourself before calling FindRaftDir
for _, c := range []string{"server/raft/raft.db", "server/raft/wal"} {
    if _, err := os.Stat(filepath.Join(dataDir, c)); err == nil {
        break
    }
    return fmt.Errorf("raft store missing/unreadable under %s", dataDir)
}

dir, err := raftutil.FindRaftDir(dataDir)
if err != nil {
    if strings.Contains(err.Error(), "failed to stat raft store") {
        // retry once (possible race) or copy the data dir and retry
        return retryOrCopyAndRetry(dataDir)
    }
    return err
}

When it happens

Trigger: FindRaftDir → FindRaftStore finds a store path, then os.Stat(storePath) fails (file deleted, permissions changed, mount unmounted, or a stale/broken symlink) before the stat executes.

Common situations: Recovery scripts running while a server process is being cleaned up or the data dir is being wiped; running the tool as a user without permissions on the raft directory (e.g. not root and raft.db is 0600); network/ephemeral storage that unmounted between calls.

Related errors

• failed to open snapshot dir: %v
• failed to delete peers.json, please delete manually (see pee
• recovery failed to delete peers.json, please delete manually
• plugin not executable
• error reading symlink: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1b20aa9be66297e8. Report an issue: GitHub.