hashicorp/nomad · error

timed out while opening database, is another Nomad process a

Error message

timed out while opening database, is another Nomad process accessing data_dir %s?

What it means

NewBoltStateDB opens the client's bbolt-backed state database with a 5-second lock timeout. bbolt allows only one writer/process at a time; when Open fails specifically with bbolt.ErrTimeout (the file lock could not be acquired in time), Nomad converts it into this explanatory error naming the data_dir.

Source

Thrown at client/state/db_bolt.go:202

// NewBoltStateDB creates or opens an existing boltdb state file or returns an
// error.
func NewBoltStateDB(logger hclog.Logger, stateDir string) (StateDB, error) {
	fn := filepath.Join(stateDir, "state.db")

	// Check to see if the DB already exists
	fi, err := os.Stat(fn)
	if err != nil && !os.IsNotExist(err) {
		return nil, err
	}
	firstRun := fi == nil

	// Timeout to force failure when accessing a data dir that is already in use
	timeout := &bbolt.Options{Timeout: 5 * time.Second}

	// Create or open the boltdb state database
	db, err := boltdd.Open(fn, 0600, timeout)
	if err == bbolt.ErrTimeout {
		return nil, fmt.Errorf("timed out while opening database, is another Nomad process accessing data_dir %s?", stateDir)
	} else if err != nil {
		return nil, fmt.Errorf("failed to create state database: %v", err)
	}

	sdb := &BoltStateDB{
		stateDir: stateDir,
		db:       db,
		logger:   logger,
	}

	// If db did not already exist, initialize metadata fields
	if firstRun {
		if err := sdb.init(); err != nil {
			return nil, err
		}
	}

	return sdb, nil

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Find and stop the other Nomad process using the data_dir (pgrep nomad; check systemd/launchd units)
2. Verify only one Nomad client is configured to use this data_dir
3. Wait/retry if a previous process is still shutting down and holds the lock
4. Point the new agent at a different data_dir if concurrent instances are intentional

Example fix

# before
nomad agent -client -data-dir /var/nomad  # second instance, same dir
# after
systemctl stop nomad   # stop the existing holder first
nomad agent -client -data-dir /var/nomad

if _, err := os.Open(fname); err == nil {
    if isLockedByOtherProcess(fname) {
        return fmt.Errorf("data_dir %s state db is locked by another process; stop it first", stateDir)
    }
}

sdb, err := NewBoltStateDB(stateDir, logger)
if err != nil && strings.Contains(err.Error(), "timed out while opening database") {
    logger.Error("state db lock held; is another Nomad client running on this data_dir?", "dir", stateDir)
    // backoff and retry after ensuring the other process exited
    time.Sleep(5 * time.Second)
    return retry()
}

When it happens

Trigger: Starting a Nomad client whose data_dir/state database is already locked — typically a second Nomad agent process pointing at the same data_dir while the first still runs, or a dead process left an unclean lock in an environment where the lock lingers.

Common situations: Duplicate Nomad client services (systemd + manual run) on the same host; container/orchestrator restarts overlapping old instances; running nomad client commands against a live agent's data_dir in tests or debugging.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• unable to open raft logs that are in use
• failed to create state database: %v
• error deleting invalid task state for task %q: %v
• error deleting unexpected task bucket %q: %v
• error delting unexpected task key %q: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/b76a62be1a4156e2. Report an issue: GitHub.