{"record":{"id":"9017334b18f5ae7f","repo":"ruvnet/ruflo","slug":"abbenchmark-requires-a-content-aware-executor-the","errorCode":null,"errorMessage":"abBenchmark requires a content-aware executor. The provided IHeadlessExecutor lacks `setContext()`, so Config A and Config B will both read the same on-disk CLAUDE.md and the delta is guaranteed to be zero. Either use the DefaultHeadlessExecutor (content-aware as of @claude-flow/guidance@3.0.0-alpha.2) or implement IContentAwareExecutor on your custom executor.","messagePattern":"abBenchmark requires a content-aware executor\\. The provided IHeadlessExecutor lacks `setContext\\(\\)`, so Config A and Config B will both read the same on-disk CLAUDE\\.md and the delta is guaranteed to be zero\\. Either use the DefaultHeadlessExecutor \\(content-aware as of @claude-flow/guidance@3\\.0\\.0-alpha\\.2\\) or implement IContentAwareExecutor on your custom executor\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"v3/@claude-flow/guidance/src/analyzer.ts","lineNumber":3164,"sourceCode":"  } = {},\n): Promise<ABReport> {\n  const {\n    executor = new DefaultHeadlessExecutor(),\n    tasks = getABTasks(),\n    proofKey,\n    workDir = process.cwd(),\n  } = options;\n\n  const contentAware = isContentAwareExecutor(executor);\n\n  // #1652: a non-content-aware executor reads CLAUDE.md from disk for both\n  // configs, so the delta is architecturally guaranteed to be zero — yet\n  // the verdict implies the user's CLAUDE.md is ineffective. Detect and\n  // abort with a clear, actionable message before spending ~$23 in tokens\n  // on a meaningless run. The default executor IS content-aware, so this\n  // only triggers when callers inject a bare IHeadlessExecutor.\n  if (!contentAware) {\n    throw new Error(\n      'abBenchmark requires a content-aware executor. The provided IHeadlessExecutor lacks `setContext()`, so Config A and Config B will both read the same on-disk CLAUDE.md and the delta is guaranteed to be zero. Either use the DefaultHeadlessExecutor (content-aware as of @claude-flow/guidance@3.0.0-alpha.2) or implement IContentAwareExecutor on your custom executor.',\n    );\n  }\n\n  // ── Config A: No control plane ──────────────────────────────────────\n  // For content-aware executors, set empty context (simulating no guidance)\n  if (contentAware) executor.setContext('');\n  const configAResults = await runABConfig(executor, tasks, workDir);\n  const configAMetrics = computeABMetrics(configAResults);\n\n  // ── Config B: With Phase 1 control plane ────────────────────────────\n  // Hook wiring: setContext with guidance content\n  // Retriever injection: the executor gets full guidance context\n  // Persisted ledger: gate simulation logs violations\n  // Deterministic tool gateway: assertions enforce compliance\n  if (contentAware) executor.setContext(claudeMdContent);\n  const configBResults = await runABConfig(executor, tasks, workDir);\n  const configBMetrics = computeABMetrics(configBResults);","sourceCodeStart":3146,"sourceCodeEnd":3182,"githubUrl":"https://github.com/ruvnet/ruflo/blob/fa13ee4ad60ac2090b1480656eb233521790d640/v3/@claude-flow/guidance/src/analyzer.ts#L3146-L3182","documentation":"abBenchmark() in @claude-flow/guidance compares guidance effectiveness by running Config A (empty context) and Config B (guidance content) through an executor. A valid comparison requires the executor to inject per-config context via `setContext()`; a bare IHeadlessExecutor reads the same on-disk CLAUDE.md for both configs, so the delta is structurally zero. The function duck-types the executor (isContentAwareExecutor) and aborts before burning roughly $23 of tokens on a meaningless run — it only fires when callers inject a custom executor, because the default one is content-aware.","triggerScenarios":"Passing a custom executor that implements run() but not setContext(); passing a test mock/stub of IHeadlessExecutor; using a DefaultHeadlessExecutor from a package older than 3.0.0-alpha.2 where setContext did not exist.","commonSituations":"Upgrading @claude-flow/guidance to >= 3.0.0-alpha.2 which made this check mandatory; DI containers or test harnesses injecting minimal executor stubs; copy-pasting an old custom executor class that predates IContentAwareExecutor.","solutions":["Use the built-in DefaultHeadlessExecutor (content-aware since @claude-flow/guidance@3.0.0-alpha.2)","Add `setContext(content: string)` to your custom executor to satisfy IContentAwareExecutor","Update test mocks to include a no-op `setContext: (c: string) => {}`","Pin/upgrade @claude-flow/guidance so the default executor you import has setContext"],"exampleFix":"// before\nclass MyExecutor implements IHeadlessExecutor {\n  async run(task: string) { /* ... */ }\n}\nawait abBenchmark({ executor: new MyExecutor(), tasks, workDir }); // throws\n\n// after\nclass MyExecutor implements IContentAwareExecutor {\n  private ctx = '';\n  setContext(content: string): void { this.ctx = content; }\n  async run(task: string) { /* use this.ctx */ }\n}\nawait abBenchmark({ executor: new MyExecutor(), tasks, workDir });","handlingStrategy":"type-guard","validationCode":"const hasSetContext =\n  typeof (executor as { setContext?: unknown }).setContext === 'function';\nif (!hasSetContext) {\n  throw new Error('executor must implement setContext() before abBenchmark');\n}","typeGuard":"type ContentAware = IHeadlessExecutor & { setContext(content: string): void };\nconst isContentAware = (e: IHeadlessExecutor): e is ContentAware =>\n  typeof (e as ContentAware).setContext === 'function';","tryCatchPattern":null,"preventionTips":["Import DefaultHeadlessExecutor from @claude-flow/guidance >= 3.0.0-alpha.2","Keep custom executors aligned with the IContentAwareExecutor interface","Include setContext in every executor mock/stub"],"tags":["ab-testing","executor","guidance","duck-typing","version-mismatch","cost-protection"],"backgroundTag":"interface-implementation-missing","analyzedSha":"fa13ee4ad60ac2090b1480656eb233521790d640","analyzedAt":"2026-08-18T21:34:22.708Z","schemaVersion":2},"datasetVersion":"2026-08-22T09:17:25.309Z"}