{"record":{"id":"d897cc39365912d6","repo":"JuliusBrussee/caveman","slug":"cave-harness-transform-not-evaluated","errorCode":"cave_harness_transform_not_evaluated","errorMessage":"cave_harness_transform_not_evaluated","messagePattern":"cave_harness_transform_not_evaluated","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/agent/src/adapters.ts","lineNumber":676,"sourceCode":"  });\n}\n\nfunction validateTransformEvidence(\n  evidence: Pick<HarnessRequest, \"evaluatedTransformIDs\" | \"appliedTransformIDs\" | \"recoveryResolved\">,\n  plan: CavePlan,\n): void {\n  if (!Array.isArray(evidence.evaluatedTransformIDs) ||\n      !Array.isArray(evidence.appliedTransformIDs) ||\n      evidence.evaluatedTransformIDs.some((value) => typeof value !== \"string\" || value.length === 0) ||\n      evidence.appliedTransformIDs.some((value) => typeof value !== \"string\" || value.length === 0) ||\n      evidence.recoveryResolved !== true) {\n    throw new Error(\"cave_harness_incomplete_evidence\");\n  }\n  const selected = [...new Set(plan.segment_routes.map((route) => route.transform_id))].sort();\n  const evaluated = [...new Set(evidence.evaluatedTransformIDs)].sort();\n  const applied = [...new Set(evidence.appliedTransformIDs)].sort();\n  if (selected.some((transformID) => !evaluated.includes(transformID))) {\n    throw new Error(\"cave_harness_transform_not_evaluated\");\n  }\n  if (evaluated.some((transformID) => !selected.includes(transformID)) ||\n      applied.some((transformID) => !selected.includes(transformID))) {\n    throw new Error(\"cave_harness_unknown_transform\");\n  }\n}\n\nfunction validateExecution(execution: HarnessExecution, plan: CavePlan): void {\n  const [provider, ...modelParts] = plan.model.split(\"/\");\n  const model = modelParts.join(\"/\");\n  try {\n    validateProviderUsage(execution, {\n      expected: { provider: provider ?? \"\", model },\n      reportedCostUsd: execution.costUsd,\n      requirePriced: true,\n    });\n  } catch {\n    throw new Error(\"cave_harness_incomplete_evidence\");","sourceCodeStart":658,"sourceCodeEnd":694,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/agent/src/adapters.ts#L658-L694","documentation":"The plan's `segment_routes` selected a set of transform IDs, and `validateTransformEvidence` requires every selected ID to appear in the execution's `evaluatedTransformIDs` (duplicates ignored, order-insensitive). If any planned transform was never evaluated during execution, the adapter throws `cave_harness_transform_not_evaluated`: the plan promised transforms that the run did not honor, breaking plan-to-execution parity.","triggerScenarios":"A plan with `segment_routes: [{ transform_id: \"terminal-ccr\" }, { transform_id: \"history-text\" }]` but an execution whose `evaluatedTransformIDs` contains only one of them; a harness that skips evaluation for a route kind (e.g. no tool_result segments, so the terminal transform never evaluated).","commonSituations":"Reusing a plan compiled for a different workload shape (routes for tool results on a run with no tools); a harness implementation that lazily evaluates only encountered segment kinds; editing the plan after compilation without re-running the compiler.","solutions":["Use a plan compiled for the actual workload — if the run has no tool-result segments, compile a plan without the tool_result transform route.","Fix the harness so it evaluates every selected transform, even when that kind's segment count is zero.","Never hand-edit `segment_routes` on a compiled plan; recompile instead."],"exampleFix":"// before: plan routes tool_result->terminal, but run had no tools\nconst plan = compiledWithRoutes([\"terminal-ccr\", \"history-text\"]);\n// execution.evaluatedTransformIDs = [\"history-text\"] -> throws\n\n// after: compile the plan for this workload\nconst plan = compileFor({ segments: [\"history\"] });\n// execution.evaluatedTransformIDs = [\"history-text\"] -> passes","handlingStrategy":"validation","validationCode":"function allPlanTransformsEvaluated(plan: CavePlan, evidence: { evaluatedTransformIDs: string[] }): boolean {\n  const selected = new Set(plan.segment_routes.map((r) => r.transform_id));\n  return [...selected].every((id) => evidence.evaluatedTransformIDs.includes(id));\n}\nif (!allPlanTransformsEvaluated(plan, execution)) throw new Error(\"recompile plan for this workload\");","typeGuard":null,"tryCatchPattern":"try {\n  validateTransformEvidence(execution, plan);\n} catch (err) {\n  if (err instanceof Error && err.message === \"cave_harness_transform_not_evaluated\") {\n    // plan/workload mismatch: recompile the plan, do not patch evidence arrays\n  }\n}","preventionTips":["Compile plans per workload shape so routes match the segments actually present.","Evaluate every selected transform even when its segment kind has zero instances.","Never hand-edit segment_routes on a compiled plan."],"tags":["adapter","evidence","transforms","plan-parity"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}