Yeachan-Heo/oh-my-codex · critical · Error

canonical_scale_up_rollback_resolved_membership_verification

Error message

canonical_scale_up_rollback_resolved_membership_verification_failed

What it means

Thrown during scale-up rollback verification after the team config was re-read from disk: a worker that was supposed to be rolled back (removed) still appears in the canonical config as resolved (not marked unresolved). It signals the rollback transaction left an inconsistent membership state where a rolled-back worker remains active.

Source

Thrown at src/team/scaling.ts:958

            baseGeneration: currentConfig.config_generation ?? 0,
            tasks: taskChanges,
            config: { oldBytes: currentConfigBytes, newBytes: JSON.stringify(desiredConfig, null, 2) },
            manifest: { oldBytes: currentManifestBytes, newBytes: desiredManifestBytes },
            recoverToNewOnFailure: true,
            retainJournalOnSuccess: true,
            failRollbackPersistence: hasScaleUpFailureInjection(env, 'rollback-membership-persistence'),
            failRollbackPersistenceAfter: hasScaleUpFailureInjection(env, 'rollback-membership-config-persistence')
              ? 'config'
              : hasScaleUpFailureInjection(env, 'rollback-membership-manifest-persistence')
                ? 'manifest'
                : undefined,
          });
          const verified = JSON.parse(await readFile(configPath, 'utf8')) as TeamConfig;
          if (!verified || retainedWorkers.some((worker) => !verified.workers.some((entry) => entry.name === worker.name && entry.pane_id === worker.pane_id))) {
            throw new Error('canonical_scale_up_rollback_membership_verification_failed');
          }
          if (verified.workers.some((worker) => rollbackWorkerNames.has(worker.name) && !unresolvedWorkerNames.has(worker.name))) {
            throw new Error('canonical_scale_up_rollback_resolved_membership_verification_failed');
          }
          await finalizeTeamMembershipTaskTransaction(sanitized, leaderCwd);
          Object.assign(config, verified);
        });
      } catch (rollbackError) {
        const journalPath = join(teamStateRoot, 'team', sanitized, '.membership-task-transaction.json');
        const suffix = existsSync(journalPath) ? String(rollbackError) : `no_recoverable_journal:${String(rollbackError)}`;
        return { ok: false, error: `scale_up_rollback_membership_persistence_failed:${suffix}` };
      }
      const cleanupWorkerNames = new Set([
        ...rollbackWorkers
          .filter((worker) => typeof worker.pane_id !== 'string' || resolvedPaneIds.has(worker.pane_id))
          .map((worker) => worker.name),
        ...(context.workerName && !unresolvedWorkerNames.has(context.workerName) ? [context.workerName] : []),
      ]);
      try {
        for (const taskId of createdTaskIds) {
          const task = await readTask(sanitized, taskId, leaderCwd);

View on GitHub (pinned to 3ad79a8a6f)

Solutions

Check for concurrent scale operations (other CLI/process mutating the same team state) and serialize them
Inspect team state .membership-task-transaction.json and recover or clear it, then retry scaleUp
Verify team config file integrity under the team state root and re-run scaleUp from a clean config

Defensive patterns

Strategy: try-catch

Try / catch

try {
  await scaleUp(team, count, opts);
} catch (e) {
  if (e instanceof Error && e.message.startsWith('canonical_scale_up_rollback')) {
    // membership state is suspect: re-read config, ensure no concurrent scalers, then retry once
    await inspectTeamState(team);
    return retryOnce(() => scaleUp(team, count, opts));
  }
  throw e;
}

Prevention

Run only one scale operation per team at a time
Monitor the membership transaction journal for stuck entries
Abort scale-up quickly on first failure to keep rollback windows small

When it happens

Trigger: Calling scaleUp, hitting a failure that triggers rollback, and the post-rollback config read finds a rollback worker name present that is not in the unresolved set.

Common situations: Concurrent mutation of the team config by another process during scale-up rollback; partially failed membership transaction recovery; stale .membership-task-transaction.json state.

Related errors

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/a8e9b092c27c04c6. Report an issue: GitHub.