Yeachan-Heo/oh-my-codex · error · Error

canonical_scale_up_rollback_worker_verification_failed:${wor

Error message

canonical_scale_up_rollback_worker_verification_failed:${workerName}

What it means

During scale-up rollback, a worker directory that should have been deleted still exists under team state workers/. The message embeds the workerName whose directory survived cleanup.

Source

Thrown at src/team/scaling.ts:990

        ...(context.workerName && !unresolvedWorkerNames.has(context.workerName) ? [context.workerName] : []),
      ]);
      try {
        for (const taskId of createdTaskIds) {
          const task = await readTask(sanitized, taskId, leaderCwd);
          if (task && unresolvedWorkerNames.has(task.owner ?? '')) continue;
          await rm(join(teamStateRoot, 'team', sanitized, 'tasks', `task-${taskId}.json`), { force: true });
        }
        await Promise.all([...cleanupWorkerNames].map(async (workerName) => {
          await rm(join(teamStateRoot, 'team', sanitized, 'workers', workerName), { recursive: true, force: true });
        }));
        for (const taskId of createdTaskIds) {
          const task = await readTask(sanitized, taskId, leaderCwd);
          if (task && unresolvedWorkerNames.has(task.owner ?? '')) continue;
          if (task) throw new Error(`canonical_scale_up_rollback_task_verification_failed:${taskId}`);
        }
        for (const workerName of cleanupWorkerNames) {
          if (existsSync(join(teamStateRoot, 'team', sanitized, 'workers', workerName))) {
            throw new Error(`canonical_scale_up_rollback_worker_verification_failed:${workerName}`);
          }
        }
      } catch (rollbackError) {
        cleanupDebt.push(`canonical_cleanup_failed:${String(rollbackError)}`);
      }

      try {
        const contextWorkerName = context.worker?.name ?? context.workerName;
        const contextWorktreePath = context.worker?.worktree_path ?? context.worktreePath;
        if (contextWorkerName && contextWorktreePath && !unresolvedWorkerNames.has(contextWorkerName)) {
          await removeWorkerWorktreeRootAgentsFile(sanitized, contextWorkerName, teamStateRoot, contextWorktreePath);
        }
        const unresolvedWorktreePaths = new Set(rollbackWorkers
          .filter((worker) => unresolvedWorkerNames.has(worker.name) && typeof worker.worktree_path === 'string')
          .map((worker) => resolve(worker.worktree_path as string)));
        await rollbackProvisionedWorktrees(provisionedWorktrees.filter((worktree) => !unresolvedWorktreePaths.has(resolve(worktree.worktreePath))));
        await Promise.all([...preparedWorkerDirectoryOwner.entries()]
          .filter(([, workerName]) => !unresolvedWorkerNames.has(workerName))

View on GitHub (pinned to 3ad79a8a6f)

Solutions

  1. Manually remove the reported worker directory under team state workers/ and retry scaleUp
  2. Ensure the worker process/pane for that worker is terminated before scaling again
  3. Check filesystem permissions on the team state root
Defensive patterns

Strategy: retry

Validate before calling

import { existsSync, rmSync } from 'node:fs';
import { join } from 'node:path';
// before retrying, confirm worker dirs are gone
function workerDirsClean(stateRoot: string, team: string, workers: string[]): boolean {
  return workers.every((w) => !existsSync(join(stateRoot, 'team', team, 'workers', w)));
}

Try / catch

catch (e) {
  const m = /canonical_scale_up_rollback_worker_verification_failed:(.+)$/.exec(String((e as Error).message));
  if (m) { rmSync(join(stateRoot,'team',team,'workers',m[1]), { recursive: true, force: true }); return retryScaleUp(); }
  throw e;
}

Prevention

When it happens

Trigger: scaleUp rollback removes worker directories with rm recursive force, then existsSync still finds the worker directory path present.

Common situations: Filesystem race where a worker process recreates its directory; permission issues blocking deletion; Windows/macOS file locking holding the directory.

Related errors

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/b1d21bdd32dbd856. Report an issue: GitHub.