hashicorp/nomad · error

migration succeeded but failed to rename %s to %s: %w

Error message

migration succeeded but failed to rename %s to %s: %w

What it means

This is the only post-success failure: the data was fully migrated and verified, but os.Rename of raft.db to raft.db.migrated.<timestamp> failed, so the new WAL directory exists alongside the original BoltDB file. Because raft.db is still present at its original path, the server may boot with the old BoltDB backend; the rename must be completed manually before starting on WAL.

Source

Thrown at helper/raftutil/migrate.go:148

	}

	// Close both stores before renaming files.
	if err := dst.Close(); err != nil {
		src.Close()
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close WAL store: %w", err)
	}

	if err := src.Close(); err != nil {
		cleanupWAL(walDir)
		return fmt.Errorf("failed to close BoltDB store: %w", err)
	}

	// Rename the old BoltDB file to preserve it as a backup with timestamp.
	timestamp := time.Now().Format("20060102-150405")
	backupPath := fmt.Sprintf("%s.migrated.%s", boltPath, timestamp)
	if err := os.Rename(boltPath, backupPath); err != nil {
		return fmt.Errorf("migration succeeded but failed to rename %s to %s: %w",
			boltPath, backupPath, err)
	}

	sendProgress(progress, fmt.Sprintf("migration complete; old BoltDB file preserved at %s", backupPath))
	return nil
}

func sendProgress(progress chan<- string, msg string) {
	if progress != nil {
		select {
		case progress <- msg:
		default:
			// Drop message if consumer is slow to avoid blocking migration.
		}
	}
}

func drainProgress(sub <-chan string, parent chan<- string, wg *sync.WaitGroup) {

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Manually move the file: mv <raftDir>/raft.db <raftDir>/raft.db.migrated.<timestamp>, then start the server (the wal directory is already complete and verified).
2. If a same-named backup already exists, use a distinct timestamp suffix for the manual rename.
3. Ensure raft dir and backup path are on the same filesystem; avoid bind-mount layouts that make rename cross-device.
4. On Windows, close processes (AV scanners, backup agents) holding raft.db open, then rename.

Example fix

// before (restart server right after failed rename)
systemctl start nomad
// after
mv /var/lib/nomad/raft.db /var/lib/nomad/raft.db.migrated.20260903-120000
systemctl start nomad

backupPath := fmt.Sprintf("%s.migrated.%s", filepath.Join(raftDir, "raft.db"), time.Now().Format("20060102-150405"))
if _, err := os.Stat(backupPath); err == nil {
    return fmt.Errorf("backup %s already exists; waiting avoids same-second collision", backupPath)
}
// also confirm raft.db and backup target are on the same filesystem
devSrc, _, _ := statDev(filepath.Join(raftDir, "raft.db"))
devDst, _, _ := statDev(backupPath)
if devSrc != devDst {
    return fmt.Errorf("rename would cross devices")
}

err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "migration succeeded but failed to rename") {
    // migration IS complete; finish manually, then start the server:
    // mv <raftDir>/raft.db <raftDir>/raft.db.migrated.manual
}

When it happens

Trigger: os.Rename(boltPath, backupPath) returns an error: a backup file with the same timestamped name already exists, cross-device rename not possible (backup target on a different filesystem via bind mounts), or permission/EPERM/EXDEV/EBUSY issues on the raft directory.

Common situations: Re-running migration within the same second after a prior success (same timestamp); raft dir bind-mounted such that rename crosses devices; antivirus/backup software holding raft.db open on Windows; read-only remount between migration and rename.

Related errors

• error moving data dir: %w
• error moving task %q local dir: %w
• failed to clean task directories after failed migration: %v
• cannot install new nomad binary - %s
• failed to find raft store in %v: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/091c815ae76136f5. Report an issue: GitHub.