Yeachan-Heo/oh-my-codex · critical · Error

canonical_scale_up_rollback_membership_verification_failed

Error message

canonical_scale_up_rollback_membership_verification_failed

What it means

Thrown during scaleUp rollback verification: after persisting retained workers, the on-disk config does not contain every retained worker with matching pane_id, or contains a rollback worker that should have been removed (see sibling error canonical_scale_up_rollback_resolved_membership_verification_failed).

Source

Thrown at src/team/scaling.ts:955

            });
          }
          await commitTeamMembershipTaskTransaction(sanitized, leaderCwd, {
            baseGeneration: currentConfig.config_generation ?? 0,
            tasks: taskChanges,
            config: { oldBytes: currentConfigBytes, newBytes: JSON.stringify(desiredConfig, null, 2) },
            manifest: { oldBytes: currentManifestBytes, newBytes: desiredManifestBytes },
            recoverToNewOnFailure: true,
            retainJournalOnSuccess: true,
            failRollbackPersistence: hasScaleUpFailureInjection(env, 'rollback-membership-persistence'),
            failRollbackPersistenceAfter: hasScaleUpFailureInjection(env, 'rollback-membership-config-persistence')
              ? 'config'
              : hasScaleUpFailureInjection(env, 'rollback-membership-manifest-persistence')
                ? 'manifest'
                : undefined,
          });
          const verified = JSON.parse(await readFile(configPath, 'utf8')) as TeamConfig;
          if (!verified || retainedWorkers.some((worker) => !verified.workers.some((entry) => entry.name === worker.name && entry.pane_id === worker.pane_id))) {
            throw new Error('canonical_scale_up_rollback_membership_verification_failed');
          }
          if (verified.workers.some((worker) => rollbackWorkerNames.has(worker.name) && !unresolvedWorkerNames.has(worker.name))) {
            throw new Error('canonical_scale_up_rollback_resolved_membership_verification_failed');
          }
          await finalizeTeamMembershipTaskTransaction(sanitized, leaderCwd);
          Object.assign(config, verified);
        });
      } catch (rollbackError) {
        const journalPath = join(teamStateRoot, 'team', sanitized, '.membership-task-transaction.json');
        const suffix = existsSync(journalPath) ? String(rollbackError) : `no_recoverable_journal:${String(rollbackError)}`;
        return { ok: false, error: `scale_up_rollback_membership_persistence_failed:${suffix}` };
      }
      const cleanupWorkerNames = new Set([
        ...rollbackWorkers
          .filter((worker) => typeof worker.pane_id !== 'string' || resolvedPaneIds.has(worker.pane_id))
          .map((worker) => worker.name),
        ...(context.workerName && !unresolvedWorkerNames.has(context.workerName) ? [context.workerName] : []),
      ]);

View on GitHub (pinned to 3ad79a8a6f)

Solutions

Inspect the team config file on disk for missing/extra worker entries
Restore from the transaction's recorded old bytes or recreate the team state
Re-run scaleUp once membership is consistent; disable failure-injection env vars if testing

Defensive patterns

Strategy: fallback

Validate before calling

const cfg: TeamConfig = JSON.parse(await readFile(configPath, 'utf8')); const ok = retained.every(w => cfg.workers.some(e => e.name === w.name && e.pane_id === w.pane_id)) && !cfg.workers.some(w => rollbackNames.has(w.name));

Try / catch

try { await scaleUp(...) } catch (e) { if (e.message.includes('rollback_membership_verification_failed')) { await restoreTeamConfigFromTransaction(); } else throw e; }

Prevention

Keep transactional backups of team config/manifest
Serialize scaling operations with a lock
Turn off scale-up failure injection flags in production

When it happens

Trigger: Scale-up failure path persists the retained membership, then verification via readTeamConfig/readFile finds missing/mismatched retained workers or resurrected rollback workers.

Common situations: Failure injection testing (rollback-membership-manifest-persistence), concurrent config writers, or partial writes during rollback.

Related errors

AI-assisted analysis of Yeachan-Heo/oh-my-codex@3ad79a8a6f (2026-08-27). Data as JSON: /api/errors/d782a78965a5c703. Report an issue: GitHub.