What causes "failed to create state database: %v" in hashicorp/nomad?

Read the wrapped %v cause for the underlying OS/db error

failed to create state database: %v

Error message

failed to create state database: %v

What it means

NewBoltStateDB wraps any error from boltdd.Open (other than the lock timeout) with "failed to create state database". This covers permission problems, path issues, corruption, or any other failure to create/open the client's bolt state database file under the state directory.

Source

Thrown at client/state/db_bolt.go:204

func NewBoltStateDB(logger hclog.Logger, stateDir string) (StateDB, error) {
	fn := filepath.Join(stateDir, "state.db")

	// Check to see if the DB already exists
	fi, err := os.Stat(fn)
	if err != nil && !os.IsNotExist(err) {
		return nil, err
	}
	firstRun := fi == nil

	// Timeout to force failure when accessing a data dir that is already in use
	timeout := &bbolt.Options{Timeout: 5 * time.Second}

	// Create or open the boltdb state database
	db, err := boltdd.Open(fn, 0600, timeout)
	if err == bbolt.ErrTimeout {
		return nil, fmt.Errorf("timed out while opening database, is another Nomad process accessing data_dir %s?", stateDir)
	} else if err != nil {
		return nil, fmt.Errorf("failed to create state database: %v", err)
	}

	sdb := &BoltStateDB{
		stateDir: stateDir,
		db:       db,
		logger:   logger,
	}

	// If db did not already exist, initialize metadata fields
	if firstRun {
		if err := sdb.init(); err != nil {
			return nil, err
		}
	}

	return sdb, nil
}

View on GitHub (pinned to 482b49bf1a)

Solutions

Read the wrapped %v cause for the underlying OS/db error
Fix permissions on data_dir and the state db file so the Nomad agent user can write (chown/chmod)
Check disk space and that the filesystem is writable
If the bolt file is corrupt, restore from backup or move the corrupt state aside (losing local client state)

Example fix

# before
ls -l /var/nomad/client  # owned by root, agent runs as nomad
# after
chown -R nomad:nomad /var/nomad
systemctl restart nomad

Defensive patterns

Strategy: try-catch

Validate before calling

if err := canWriteDir(filepath.Join(stateDir, "client")); err != nil {
    return fmt.Errorf("data_dir not writable by nomad user: %w", err)
}

Try / catch

sdb, err := NewBoltStateDB(stateDir, logger)
if err != nil && strings.HasPrefix(err.Error(), "failed to create state database") {
    logger.Error("cannot open bolt state db; check permissions/disk/corruption", "dir", stateDir, "cause", err)
    return err
}

Prevention

Ensure data_dir is owned by and writable by the user Nomad runs as
Monitor disk free space and mount state on a writable, persistent volume
Back up the state db so corrupt files can be restored instead of discarded
Check LSM (SELinux/AppArmor) policies allow Nomad writes to data_dir

When it happens

Trigger: boltdd.Open(fn, 0600, timeout) returns a non-timeout error during client startup: the state db file path is unwritable (permissions/ownership), the parent state dir is missing or is not a directory, the db file is corrupt, or the filesystem is full/read-only.

Common situations: data_dir owned by a different user than the Nomad agent runs as; read-only container filesystem or full disk; corrupted bolt file after crash/power loss; SELinux/AppArmor denying write access to the data dir.

Related errors

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/1e32c96557f9c7e6. Report an issue: GitHub.