hashicorp/nomad · error

WAL directory already exists at %s; remove it before retryin

Error message

WAL directory already exists at %s; remove it before retrying migration

What it means

preflightChecks refuses to run if <raftDir>/wal already exists, because a pre-existing WAL directory could contain partial data from an earlier run and MigrateToWAL does not merge into an existing store. This is a deliberate safety guard: migration only proceeds into a fresh wal directory. No state is modified.

Source

Thrown at helper/raftutil/migrate.go:191

	for msg := range sub {
		select {
		case parent <- msg:
		default:
			// Drop message if consumer is slow to avoid blocking migration.
		}
	}
}

func preflightChecks(boltPath, walDir, raftDir string) error {
	// Verify the BoltDB file exists.
	boltInfo, err := os.Stat(boltPath)
	if err != nil {
		return fmt.Errorf("BoltDB store not found at %s: %w", boltPath, err)
	}

	// Verify the WAL directory does not already exist.
	if _, err := os.Stat(walDir); err == nil {
		return fmt.Errorf(
			"WAL directory already exists at %s; remove it before retrying migration",
			walDir)
	}

	// Check write permissions on raft directory.
	testFile := filepath.Join(raftDir, ".permission-test")
	if err := os.WriteFile(testFile, []byte("test"), 0o600); err != nil {
		return fmt.Errorf("insufficient write permissions in %s: %w", raftDir, err)
	}
	os.Remove(testFile)

	// Check available disk space.
	usage, err := disk.Usage(raftDir)
	if err != nil {
		// Log warning but don't fail migration - disk space check is advisory.
		return fmt.Errorf("unable to check available disk space: %w", err)
	}
	availableSpace := usage.Free

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Confirm the previous migration did not complete (no raft.db.migrated.<timestamp> backup); if it did, do not re-migrate — just start the server.
2. If migration never completed, stop the server and remove the stale directory: rm -rf <raftDir>/wal, then re-run.
3. On Windows, close processes holding files in wal/ open before removing it.
4. Inspect wal/ contents before deleting to ensure it contains no live server data (server must be stopped).

Example fix

// before
MigrateToWAL(ctx, raftDir, progress) // fails: wal/ already exists
// after (server stopped, migration not previously completed)
// rm -rf <raftDir>/wal
MigrateToWAL(ctx, raftDir, progress)

walDir := filepath.Join(raftDir, "wal")
if _, err := os.Stat(walDir); err == nil {
    if _, err := os.Stat(filepath.Join(raftDir, "raft.db")); err != nil {
        return fmt.Errorf("wal exists but raft.db is gone: migration already completed; do not re-run")
    }
    return fmt.Errorf("stale %s present; server stopped? remove it before retrying", walDir)
}

err := raftutil.MigrateToWAL(ctx, raftDir, progress)
if err != nil && strings.Contains(err.Error(), "WAL directory already exists") {
    // inspect: if raft.db.migrated.* exists, migration done — start server.
    // else, remove the stale wal/ (server stopped) and retry once.
}

When it happens

Trigger: os.Stat(walDir) succeeds, meaning <raftDir>/wal exists: leftover from a previous cleanup failure (notably Windows handle-release delays), a previous successful migration (wal present but raft.db also present after a failed rename), or manually created directory.

Common situations: Retrying migration after a previous attempt whose cleanupWAL could not remove the directory; re-running migration after an error 1887 (rename failure) left both wal and raft.db in place; operator created the directory by hand or restored a backup into it.

Related errors

• BoltDB store not found at %s: %w
• failed to clean task directories after failed migration: %v
• failed to create migration marker: %w
• failed to create WAL directory: %w
• failed to open WAL store: %w

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/62912208662a1b1e. Report an issue: GitHub.