hashicorp/nomad · critical

failed to upgrade state database: %v

Error message

failed to upgrade state database: %v

What it means

After opening, the client calls db.Upgrade() to migrate persisted state. Per the source comment, Upgrade drops and logs corrupt state it encounters, so it returns an error only on critical persistence failures where an operator must intervene. Init then fails with this wrapper and the agent refuses to start.

Source

Thrown at client/client.go:721

			c.StateDir = p
		})
	}
	c.logger.Info("using state directory", "state_dir", conf.StateDir)

	// Open the state database
	db, err := conf.StateDBFactory(c.logger, conf.StateDir)
	if err != nil {
		return fmt.Errorf("failed to open state database: %v", err)
	}

	// Upgrade the state database
	if err := db.Upgrade(); err != nil {
		// Upgrade only returns an error on critical persistence
		// failures in which an operator should intervene before the
		// node is accessible. Upgrade drops and logs corrupt state it
		// encounters, so failing to start the agent should be extremely
		// rare.
		return fmt.Errorf("failed to upgrade state database: %v", err)
	}

	c.stateDB = db

	// Ensure host_volumes_dir config is not empty.
	if conf.HostVolumesDir == "" {
		conf = c.UpdateConfig(func(c *config.Config) {
			c.HostVolumesDir = filepath.Join(conf.StateDir, "host_volumes")
		})
	}

	// Ensure the alloc mounts dir exists if we are configured with a custom path.
	if conf.AllocMountsDir != "" {
		if err := os.MkdirAll(conf.AllocMountsDir, 0o711); err != nil {
			return fmt.Errorf("failed creating alloc mounts dir: %w", err)
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check agent logs for the upgrade-time errors preceding this failure to see which state was rejected.
2. Verify you are not downgrading Nomad across incompatible state schema versions.
3. Back up the state directory and remove/rename the state DB so the client starts fresh and re-syncs allocations from servers.

Example fix

// before: agent fails to start after downgrade
// after: start fresh
sudo systemctl stop nomad
sudo mv /var/lib/nomad/client/state.db /var/lib/nomad/client/state.db.bak
sudo systemctl start nomad

if err := clientInit(); err != nil {
    if strings.Contains(err.Error(), "failed to upgrade state database") {
        // operator intervention: stop agent, back up state dir, remove state.db,
        // restart so client re-syncs from servers
    }
}

When it happens

Trigger: client init: db.Upgrade() returns error — critical persistence failure during state migration (e.g. DB unusable at a structural level, not merely corrupt individual entries).

Common situations: Downgrading Nomad to an older version whose state schema cannot be handled, severe disk corruption, or a state DB stored on unreliable storage (network filesystems).

Related errors

• failed to open state database: %v
• failed to find temporary directory for the StateDir: %v
• failed creating alloc mounts dir: %w
• failed creating alloc dir: %w
• failed creating temporary directory for the AllocDir: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ac6d79f76fbc2142. Report an issue: GitHub.