hashicorp/nomad · critical

failed to open state database: %v

Error message

failed to open state database: %v

What it means

After determining the state directory, the client opens the persistent state database via conf.StateDBFactory (BoltDB-backed). If the factory returns an error, init fails with this message. The agent cannot start because all client state (allocs, task handles metadata) lives in this DB.

Source

Thrown at client/client.go:711

		if err != nil {
			return fmt.Errorf("failed creating temporary directory for the StateDir: %v", err)
		}

		p, err = filepath.EvalSymlinks(p)
		if err != nil {
			return fmt.Errorf("failed to find temporary directory for the StateDir: %v", err)
		}

		conf = c.UpdateConfig(func(c *config.Config) {
			c.StateDir = p
		})
	}
	c.logger.Info("using state directory", "state_dir", conf.StateDir)

	// Open the state database
	db, err := conf.StateDBFactory(c.logger, conf.StateDir)
	if err != nil {
		return fmt.Errorf("failed to open state database: %v", err)
	}

	// Upgrade the state database
	if err := db.Upgrade(); err != nil {
		// Upgrade only returns an error on critical persistence
		// failures in which an operator should intervene before the
		// node is accessible. Upgrade drops and logs corrupt state it
		// encounters, so failing to start the agent should be extremely
		// rare.
		return fmt.Errorf("failed to upgrade state database: %v", err)
	}

	c.stateDB = db

	// Ensure host_volumes_dir config is not empty.
	if conf.HostVolumesDir == "" {
		conf = c.UpdateConfig(func(c *config.Config) {
			c.HostVolumesDir = filepath.Join(conf.StateDir, "host_volumes")

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the wrapped %v error to identify the cause (lock contention vs corruption vs permission).
2. Ensure no other Nomad client process is using the same data_dir/state_dir and that the directory is writable.
3. If the DB file is corrupt and unrecoverable, back it up and remove/rename it, accepting loss of local client state (the client will re-sync from servers).

Example fix

// before
sudo nomad agent -client   # fails: another agent holds state.db lock
// after
sudo systemctl stop nomad; pgrep nomad  # ensure single process
sudo chown -R nomad:nomad /var/lib/nomad/client
sudo systemctl start nomad

dbPath := filepath.Join(stateDir, "state.db")
if f, err := os.OpenFile(filepath.Dir(dbPath), os.O_WRONLY, 0o700); err != nil {
    return fmt.Errorf("state dir not writable: %v", err)
} else { f.Close() }
if fi, err := os.Stat(dbPath); err == nil && fi.Size() == 0 {
    return fmt.Errorf("state.db is 0 bytes — likely corrupt")
}

if err := clientInit(); err != nil {
    if strings.Contains(err.Error(), "failed to open state database") {
        // check for lock holder: lsof state.db; restore from backup or start fresh
    }
}

When it happens

Trigger: client init: StateDBFactory(logger, conf.StateDir) returns error, typically when the bolt file (state.db) is corrupt, locked, or the state dir is unwritable/uncreatable.

Common situations: Corrupt bolt.db after host crash or disk-full, another Nomad process holding the file lock, state_dir on a read-only or full filesystem, restored state files with wrong permissions.

Related errors

• failed to create state database: %v
• failed to find temporary directory for the StateDir: %v
• failed to upgrade state database: %v
• failed creating alloc mounts dir: %w
• failed creating alloc dir: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a6eb20e4d29d3342. Report an issue: GitHub.