hashicorp/nomad · error

unable to open raft logs that are in use

Error message

unable to open raft logs that are in use

What it means

errAlreadyOpen is returned by RaftStateInfo (raftStateInfoBoltDB) when opening the raft BoltDB fails with a lock "timeout" — meaning another process (e.g. a running Nomad agent) holds the boltdb file lock. It tells the caller the logs exist but are actively in use.

Source

Thrown at helper/raftutil/state.go:24

import (
	"bytes"
	"errors"
	"fmt"
	"os"
	"path/filepath"
	"strings"
	"time"

	"github.com/hashicorp/go-msgpack/v2/codec"
	"github.com/hashicorp/nomad/nomad/structs"
	"github.com/hashicorp/raft"
	raftboltdb "github.com/hashicorp/raft-boltdb/v2"
	raftwal "github.com/hashicorp/raft-wal"
	"go.etcd.io/bbolt"
)

var (
	errAlreadyOpen = errors.New("unable to open raft logs that are in use")
)

// RaftStore is the interface returned by RaftStateInfo, satisfied by both
// *raftboltdb.BoltStore and *raftwal.WAL.
type RaftStore interface {
	raft.LogStore
	raft.StableStore
	Close() error
}

// RaftStateInfo returns info about the raft state found at path p. The path
// may point to a BoltDB file (raft.db) or a WAL directory. The returned
// RaftStore must be closed by the caller.
func RaftStateInfo(p string) (store RaftStore, firstIdx uint64, lastIdx uint64, err error) {
	info, statErr := os.Stat(p)
	if statErr != nil {
		return nil, 0, 0, fmt.Errorf("failed to stat %s: %v", p, statErr)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Stop the Nomad server process holding the raft.db lock before inspecting the data directory.
2. Point the tool at a copy/snapshot of the data dir instead of the live one.
3. Check for a second agent using the same data_dir (duplicate processes) and terminate it.
4. Compare with errors.Is(err, raftutil.ErrNoKeyID-style sentinel) — use errors.Is against errAlreadyOpen-equivalent to branch on 'in use' vs other failures.

Example fix

// before
store, _, _, err := raftutil.RaftStateInfo(dir)
if err != nil { return err }
// after
store, _, _, err := raftutil.RaftStateInfo(dir)
if err != nil {
    if strings.HasSuffix(err.Error(), "in use") {
        return fmt.Errorf("stop the nomad agent using %s before inspecting", dir)
    }
    return err
}

// verify no live agent holds the data dir before inspection
if pid := runningNomadPidFor(dataDir); pid != 0 {
    return fmt.Errorf("nomad agent pid %d is using %s; stop it first", pid, dataDir)
}

func isInUseError(err error) bool {
    return err != nil && strings.HasSuffix(err.Error(), "in use")
}

store, _, _, err := raftutil.RaftStateInfo(dir)
if err != nil {
    if strings.HasSuffix(err.Error(), "in use") {
        return fmt.Errorf("raft logs locked; stop the nomad agent or inspect a copy of %s", dir)
    }
    return fmt.Errorf("failed to open raft logs: %w", err)
}

When it happens

Trigger: Running RaftStateInfo or LogEntries against a data dir whose raft.db is locked by a live Nomad server; bbolt Open times out acquiring the flock and strings.HasSuffix(err.Error(), "timeout") maps it to this sentinel.

Common situations: Running inspection/ops tooling against a production data dir while the server is up; two agents pointed at the same data dir; leftover lock from a crashed process on filesystems without proper flock cleanup.

Related errors

• timed out while opening database, is another Nomad process a
• failed to open raft store %v: %v
• failed to open BoltDB store: %w
• failed to open raft logs: %v
• failed to fetch first index: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/cb0586063ec56d69. Report an issue: GitHub.