hashicorp/nomad · critical

failed to open state database

Error message

failed to open state database: %v

What it means

After determining the state directory, the client opens the persistent state database via conf.StateDBFactory (BoltDB-backed). If the factory returns an error, init fails with this message. The agent cannot start because all client state (allocs, task handles metadata) lives in this DB.

Solutions

1. Read the wrapped %v error to identify the cause (lock contention vs corruption vs permission).
2. Ensure no other Nomad client process is using the same data_dir/state_dir and that the directory is writable.
3. If the DB file is corrupt and unrecoverable, back it up and remove/rename it, accepting loss of local client state (the client will re-sync from servers).

Example fix

// before
sudo nomad agent -client   # fails: another agent holds state.db lock
// after
sudo systemctl stop nomad; pgrep nomad  # ensure single process
sudo chown -R nomad:nomad /var/lib/nomad/client
sudo systemctl start nomad

dbPath := filepath.Join(stateDir, "state.db")
if f, err := os.OpenFile(filepath.Dir(dbPath), os.O_WRONLY, 0o700); err != nil {
    return fmt.Errorf("state dir not writable: %v", err)
} else { f.Close() }
if fi, err := os.Stat(dbPath); err == nil && fi.Size() == 0 {
    return fmt.Errorf("state.db is 0 bytes — likely corrupt")
}

if err := clientInit(); err != nil {
    if strings.Contains(err.Error(), "failed to open state database") {
        // check for lock holder: lsof state.db; restore from backup or start fresh
    }
}

When it happens

Trigger: client init: StateDBFactory(logger, conf.StateDir) returns error, typically when the bolt file (state.db) is corrupt, locked, or the state dir is unwritable/uncreatable.

Common situations: Corrupt bolt.db after host crash or disk-full, another Nomad process holding the file lock, state_dir on a read-only or full filesystem, restored state files with wrong permissions.

Related errors

• failed to create state database
• failed creating alloc dir
• failed creating alloc mounts dir
• failed creating temporary directory for the AllocDir
• failed to change directory permissions for the AllocDir

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a6eb20e4d29d3342. Report an issue: GitHub.

Appendix: source

Thrown at client/client.go:711

		if err != nil {
			return fmt.Errorf("failed creating temporary directory for the StateDir: %v", err)
		}

		p, err = filepath.EvalSymlinks(p)
		if err != nil {
			return fmt.Errorf("failed to find temporary directory for the StateDir: %v", err)
		}

		conf = c.UpdateConfig(func(c *config.Config) {
			c.StateDir = p
		})
	}
	c.logger.Info("using state directory", "state_dir", conf.StateDir)

	// Open the state database
	db, err := conf.StateDBFactory(c.logger, conf.StateDir)
	if err != nil {
		return fmt.Errorf("failed to open state database: %v", err)
	}

	// Upgrade the state database
	if err := db.Upgrade(); err != nil {
		// Upgrade only returns an error on critical persistence
		// failures in which an operator should intervene before the
		// node is accessible. Upgrade drops and logs corrupt state it
		// encounters, so failing to start the agent should be extremely
		// rare.
		return fmt.Errorf("failed to upgrade state database: %v", err)
	}

	c.stateDB = db

	// Ensure host_volumes_dir config is not empty.
	if conf.HostVolumesDir == "" {
		conf = c.UpdateConfig(func(c *config.Config) {
			c.HostVolumesDir = filepath.Join(conf.StateDir, "host_volumes")

View on GitHub (pinned to 482b49bf1a)