hashicorp/nomad · error

error backing up state db: %v

Error message

error backing up state db: %v

What it means

Returned during state DB upgrade when the pre-upgrade backup of the BoltDB to state.db.backup fails. Upgrading aborts before any mutation so the live database is left untouched; this is a fail-safe step before destructive schema changes.

Source

Thrown at client/state/db_bolt.go:1230

}

// Upgrade bolt state db from 0.8 schema to 0.9 schema. Noop if already using
// 0.9 schema. Creates a backup before upgrading.
func (s *BoltStateDB) Upgrade() error {
	// Check to see if the underlying DB needs upgrading.
	upgrade09, upgrade13, err := NeedsUpgrade(s.db.BoltDB())
	if err != nil {
		return err
	}
	if !upgrade09 && !upgrade13 {
		// No upgrade needed!
		return nil
	}

	// Upgraded needed. Backup the boltdb first.
	backupFileName := filepath.Join(s.stateDir, "state.db.backup")
	if err := backupDB(s.db.BoltDB(), backupFileName); err != nil {
		return fmt.Errorf("error backing up state db: %v", err)
	}

	// Perform the upgrade
	if err := s.db.Update(func(tx *boltdd.Tx) error {

		if upgrade09 {
			if err := UpgradeAllocs(s.logger, tx); err != nil {
				return err
			}
		}
		if upgrade13 {
			if err := UpgradeDynamicPluginRegistry(s.logger, tx); err != nil {
				return err
			}
		}

		// Add standard metadata
		if err := addMeta(tx.BoltTx()); err != nil {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check disk space (df) and directory permissions on state_dir.
2. If a stale state.db.backup exists and you have another backup, remove or rename it, then restart the agent to retry the upgrade.
3. Verify the source state.db is readable by the Nomad process user.
4. If the backup keeps failing, copy state.db manually, delete the manual copy's lock files guidance aside, and restore it if the upgrade later corrupts.

Example fix

// before
upgrade fails: error backing up state db: backup file already exists
// after
systemctl stop nomad
mv /var/lib/nomad/state.db.backup /var/lib/nomad/state.db.backup.old
systemctl start nomad # upgrade retries and creates a fresh backup

backupPath := filepath.Join(stateDir, "state.db.backup")
if fi, err := os.Stat(backupPath); err == nil && !fi.IsDir() {
    return fmt.Errorf("stale backup %s exists; archive it before upgrading", backupPath)
}
if err := diskFreeAtLeast(stateDir, dbSize*2); err != nil { return err }

func backupPreconditionsOK(stateDir string, dbSize int64) bool {
    _, err := os.Stat(filepath.Join(stateDir, "state.db.backup"))
    return os.IsNotExist(err)
}

if err := upgradeState(); err != nil {
    if strings.Contains(err.Error(), "error backing up state db") {
        // remove stale backup and retry once
        os.Rename(backupPath, backupPath+".old")
        err = upgradeState()
    }
}

When it happens

Trigger: backupDB(s.db.BoltDB(), state.db.backup) fails — most commonly the destination file already exists (backup refuses to overwrite), or copy/read errors on the source DB (permissions, disk full).

Common situations: Leftover state.db.backup from a previously failed/aborted upgrade, read-only state directory, or insufficient disk space during a Nomad version upgrade on a client node.

Related errors

• error deleting unexpected key %q: %v
• error deleting invalid allocation state: %v
• unexpected bucket missing %q
• alloc entry not found
• Task can't ask for disk resources, they have to be specified

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/ba62294925fee1fc. Report an issue: GitHub.