Yeachan-Heo/oh-my-codex · critical · Error

canonical_scale_up_rollback_tracking_verification_failed

Error message

canonical_scale_up_rollback_tracking_verification_failed

What it means

Thrown during scaleUp rollback: after persisting the rolled-back (old) config/manifest, re-reading the team config fails to verify that all rollback workers are present with matching name and pane_id — the rollback may not have fully persisted.

Source

Thrown at src/team/scaling.ts:826

          };
          const trackedManifestBytes = currentManifestBytes === null
            ? null
            : JSON.stringify({
              ...(JSON.parse(currentManifestBytes) as Record<string, unknown>),
              workers: trackedConfig.workers,
              worker_count: trackedConfig.worker_count,
              next_worker_index: trackedConfig.next_worker_index,
            }, null, 2);
          await commitTeamMembershipTaskTransaction(sanitized, leaderCwd, {
            baseGeneration: currentConfig.config_generation ?? 0,
            tasks: [],
            config: { oldBytes: currentConfigBytes, newBytes: JSON.stringify(trackedConfig, null, 2) },
            manifest: { oldBytes: currentManifestBytes, newBytes: trackedManifestBytes },
            recoverToNewOnFailure: true,
          });
          const verified = await readTeamConfig(sanitized, leaderCwd);
          if (!verified || rollbackWorkers.some((worker) => !verified.workers.some((entry) => entry.name === worker.name && entry.pane_id === worker.pane_id))) {
            throw new Error('canonical_scale_up_rollback_tracking_verification_failed');
          }
          Object.assign(config, verified);
        });
      } catch (trackingError) {
        return { ok: false, error: `scale_up_rollback_membership_persistence_failed:${String(trackingError)}` };
      }

      const cleanupDebt: string[] = [];
      try {
        await removeDispatchRequestsForWorkers(sanitized, [...rollbackWorkerNames], leaderCwd);
      } catch (rollbackError) {
        cleanupDebt.push(`authoritative_dispatch_cleanup_failed:${String(rollbackError)}`);
      }
      const rollbackPaneIds = [...new Set([
        ...rollbackWorkers.map((worker) => worker.pane_id),
        context.paneId,
      ].filter((paneId): paneId is string => typeof paneId === 'string' && paneId.trim().startsWith('%')))];
      const resolvedPaneIds = new Set<string>();

View on GitHub (pinned to 3ad79a8a6f)

Solutions

Re-read the team config and compare worker entries against expected rollback membership
If inconsistent, restore the config from the oldBytes captured in the transaction record or recreate the team
Avoid concurrent scaling/shutdown operations on the same team

Defensive patterns

Strategy: fallback

Validate before calling

// after any scaleUp failure, verify membership consistency
const cfg = await readTeamConfig(name, cwd); const ok = expected.every(w => cfg?.workers.some(e => e.name === w.name && e.pane_id === w.pane_id));

Try / catch

try { await scaleUp(...) } catch (e) { if (e.message.includes('rollback_tracking_verification_failed')) await restoreConfigFromBackup(); else throw e; }

Prevention

Back up team config before scaling
Never run concurrent scaling or shutdown on one team
Disable failure-injection env vars outside tests

When it happens

Trigger: A scale-up failure triggers rollback; the atomic config rewrite succeeds but verification finds missing or mismatched worker entries (name/pane_id) in the persisted config.

Common situations: Concurrent writers to team config during rollback, crash mid-persist, or injected failure testing of rollback persistence.

Related errors

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/01bf04ec97a5bce5. Report an issue: GitHub.