abBenchmark requires a content-aware executor. The provided

Error message

abBenchmark requires a content-aware executor. The provided IHeadlessExecutor lacks `setContext()`, so Config A and Config B will both read the same on-disk CLAUDE.md and the delta is guaranteed to be zero. Either use the DefaultHeadlessExecutor (content-aware as of @claude-flow/guidance@3.0.0-alpha.2) or implement IContentAwareExecutor on your custom executor.

What it means

abBenchmark() in @claude-flow/guidance compares guidance effectiveness by running Config A (empty context) and Config B (guidance content) through an executor. A valid comparison requires the executor to inject per-config context via `setContext()`; a bare IHeadlessExecutor reads the same on-disk CLAUDE.md for both configs, so the delta is structurally zero. The function duck-types the executor (isContentAwareExecutor) and aborts before burning roughly $23 of tokens on a meaningless run — it only fires when callers inject a custom executor, because the default one is content-aware.

Source

Thrown at v3/@claude-flow/guidance/src/analyzer.ts:3164

  } = {},
): Promise<ABReport> {
  const {
    executor = new DefaultHeadlessExecutor(),
    tasks = getABTasks(),
    proofKey,
    workDir = process.cwd(),
  } = options;

  const contentAware = isContentAwareExecutor(executor);

  // #1652: a non-content-aware executor reads CLAUDE.md from disk for both
  // configs, so the delta is architecturally guaranteed to be zero — yet
  // the verdict implies the user's CLAUDE.md is ineffective. Detect and
  // abort with a clear, actionable message before spending ~$23 in tokens
  // on a meaningless run. The default executor IS content-aware, so this
  // only triggers when callers inject a bare IHeadlessExecutor.
  if (!contentAware) {
    throw new Error(
      'abBenchmark requires a content-aware executor. The provided IHeadlessExecutor lacks `setContext()`, so Config A and Config B will both read the same on-disk CLAUDE.md and the delta is guaranteed to be zero. Either use the DefaultHeadlessExecutor (content-aware as of @claude-flow/guidance@3.0.0-alpha.2) or implement IContentAwareExecutor on your custom executor.',
    );
  }

  // ── Config A: No control plane ──────────────────────────────────────
  // For content-aware executors, set empty context (simulating no guidance)
  if (contentAware) executor.setContext('');
  const configAResults = await runABConfig(executor, tasks, workDir);
  const configAMetrics = computeABMetrics(configAResults);

  // ── Config B: With Phase 1 control plane ────────────────────────────
  // Hook wiring: setContext with guidance content
  // Retriever injection: the executor gets full guidance context
  // Persisted ledger: gate simulation logs violations
  // Deterministic tool gateway: assertions enforce compliance
  if (contentAware) executor.setContext(claudeMdContent);
  const configBResults = await runABConfig(executor, tasks, workDir);
  const configBMetrics = computeABMetrics(configBResults);

View on GitHub (pinned to fa13ee4ad6)

Solutions

Use the built-in DefaultHeadlessExecutor (content-aware since @claude-flow/guidance@3.0.0-alpha.2)
Add `setContext(content: string)` to your custom executor to satisfy IContentAwareExecutor
Update test mocks to include a no-op `setContext: (c: string) => {}`
Pin/upgrade @claude-flow/guidance so the default executor you import has setContext

Example fix

// before
class MyExecutor implements IHeadlessExecutor {
  async run(task: string) { /* ... */ }
}
await abBenchmark({ executor: new MyExecutor(), tasks, workDir }); // throws

// after
class MyExecutor implements IContentAwareExecutor {
  private ctx = '';
  setContext(content: string): void { this.ctx = content; }
  async run(task: string) { /* use this.ctx */ }
}
await abBenchmark({ executor: new MyExecutor(), tasks, workDir });

Defensive patterns

Strategy: type-guard

Validate before calling

const hasSetContext =
  typeof (executor as { setContext?: unknown }).setContext === 'function';
if (!hasSetContext) {
  throw new Error('executor must implement setContext() before abBenchmark');
}

Type guard

type ContentAware = IHeadlessExecutor & { setContext(content: string): void };
const isContentAware = (e: IHeadlessExecutor): e is ContentAware =>
  typeof (e as ContentAware).setContext === 'function';

Prevention

Import DefaultHeadlessExecutor from @claude-flow/guidance >= 3.0.0-alpha.2
Keep custom executors aligned with the IContentAwareExecutor interface
Include setContext in every executor mock/stub

When it happens

Trigger: Passing a custom executor that implements run() but not setContext(); passing a test mock/stub of IHeadlessExecutor; using a DefaultHeadlessExecutor from a package older than 3.0.0-alpha.2 where setContext did not exist.

Common situations: Upgrading @claude-flow/guidance to >= 3.0.0-alpha.2 which made this check mandatory; DI containers or test harnesses injecting minimal executor stubs; copy-pasting an old custom executor class that predates IContentAwareExecutor.

Related errors

AI-assisted analysis of ruvnet/ruflo@fa13ee4ad6 (2026-08-18). Data as JSON: /api/errors/9017334b18f5ae7f. Report an issue: GitHub.