hashicorp/nomad · error

failed to open raft store %v: %v

Error message

failed to open raft store %v: %v

What it means

NewFSM wraps the error from RaftStateInfo when the located raft store (BoltDB or WAL) cannot be opened, e.g. corrupt file, lock contention, or unsupported/failed backend. The store path is included in the message.

Source

Thrown at helper/raftutil/fsm.go:54

	fsm   nomadFSM
	snaps *raft.FileSnapshotStore

	// raft
	logFirstIdx uint64
	logLastIdx  uint64
	nextIdx     uint64
}

func NewFSM(p string) (*FSMHelper, error) {
	// Auto-detect the backend: look for wal/ directory first, then raft.db.
	storePath, err := FindRaftStore(p)
	if err != nil {
		return nil, fmt.Errorf("failed to find raft store in %v: %v", p, err)
	}

	store, firstIdx, lastIdx, err := RaftStateInfo(storePath)
	if err != nil {
		return nil, fmt.Errorf("failed to open raft store %v: %v", storePath, err)
	}

	logger := hclog.L()

	snaps, err := raft.NewFileSnapshotStoreWithLogger(p, 1000, logger)
	if err != nil {
		store.Close()
		return nil, fmt.Errorf("failed to open snapshot dir: %v", err)
	}

	fsm, err := dummyFSM(logger)
	if err != nil {
		store.Close()
		return nil, err
	}

	return &FSMHelper{
		path:   p,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop the Nomad server process before opening the store (BoltDB allows only one writer/lock holder)
2. Check file permissions and disk health; look at the wrapped error for 'timeout' (lock) vs corruption
3. Restore the data dir from a snapshot or backup if the store is corrupt (nomad operator snapshot inspect/restore)

Example fix

// before
fsm, err := raftutil.NewFSM(dataDir) // fails if server is running
// after
if err := stopServerAndAcquireLock(dataDir); err != nil {
    return fmt.Errorf("ensure nomad server is stopped before opening %s: %w", dataDir, err)
}
fsm, err := raftutil.NewFSM(dataDir)

if err := probeBoltDBOpen(filepath.Join(dataDir, "raft.db")); err != nil {
    return fmt.Errorf("stop the nomad server or fix raft.db before proceeding: %w", err)
}

if err := raftutil.NewFSM(dataDir); err != nil {
    if strings.Contains(err.Error(), "failed to open raft store") && strings.Contains(err.Error(), "timeout") {
        return fmt.Errorf("raft.db locked — is the nomad server still running? %w", err)
    }
    return err
}

When it happens

Trigger: Calling NewFSM where raft.db is corrupt/truncated, another process holds the BoltDB file lock, disk I/O errors occur, or the WAL directory is unreadable/corrupt.

Common situations: Running migration/inspection tooling while the Nomad server is still running (BoltDB file lock held); corrupted data dir after crash or disk-full; insufficient file permissions on raft.db.

Related errors

• failed to open BoltDB store: %w
• failed to find raft store in %v: %v
• failed to read log entry at index %d: %v
• failed to open raft logs: %v
• failed to fetch first index: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/f0f45e70310b61eb. Report an issue: GitHub.