JuliusBrussee/caveman · error · Error

cave_harness_transform_not_evaluated

cave_harness_transform_not_evaluated

Error message

cave_harness_transform_not_evaluated

What it means

The plan's `segment_routes` selected a set of transform IDs, and `validateTransformEvidence` requires every selected ID to appear in the execution's `evaluatedTransformIDs` (duplicates ignored, order-insensitive). If any planned transform was never evaluated during execution, the adapter throws `cave_harness_transform_not_evaluated`: the plan promised transforms that the run did not honor, breaking plan-to-execution parity.

Source

Thrown at packages/agent/src/adapters.ts:676

  });
}

function validateTransformEvidence(
  evidence: Pick<HarnessRequest, "evaluatedTransformIDs" | "appliedTransformIDs" | "recoveryResolved">,
  plan: CavePlan,
): void {
  if (!Array.isArray(evidence.evaluatedTransformIDs) ||
      !Array.isArray(evidence.appliedTransformIDs) ||
      evidence.evaluatedTransformIDs.some((value) => typeof value !== "string" || value.length === 0) ||
      evidence.appliedTransformIDs.some((value) => typeof value !== "string" || value.length === 0) ||
      evidence.recoveryResolved !== true) {
    throw new Error("cave_harness_incomplete_evidence");
  }
  const selected = [...new Set(plan.segment_routes.map((route) => route.transform_id))].sort();
  const evaluated = [...new Set(evidence.evaluatedTransformIDs)].sort();
  const applied = [...new Set(evidence.appliedTransformIDs)].sort();
  if (selected.some((transformID) => !evaluated.includes(transformID))) {
    throw new Error("cave_harness_transform_not_evaluated");
  }
  if (evaluated.some((transformID) => !selected.includes(transformID)) ||
      applied.some((transformID) => !selected.includes(transformID))) {
    throw new Error("cave_harness_unknown_transform");
  }
}

function validateExecution(execution: HarnessExecution, plan: CavePlan): void {
  const [provider, ...modelParts] = plan.model.split("/");
  const model = modelParts.join("/");
  try {
    validateProviderUsage(execution, {
      expected: { provider: provider ?? "", model },
      reportedCostUsd: execution.costUsd,
      requirePriced: true,
    });
  } catch {
    throw new Error("cave_harness_incomplete_evidence");

View on GitHub (pinned to 27d5a3981a)

Solutions

  1. Use a plan compiled for the actual workload — if the run has no tool-result segments, compile a plan without the tool_result transform route.
  2. Fix the harness so it evaluates every selected transform, even when that kind's segment count is zero.
  3. Never hand-edit `segment_routes` on a compiled plan; recompile instead.

Example fix

// before: plan routes tool_result->terminal, but run had no tools
const plan = compiledWithRoutes(["terminal-ccr", "history-text"]);
// execution.evaluatedTransformIDs = ["history-text"] -> throws

// after: compile the plan for this workload
const plan = compileFor({ segments: ["history"] });
// execution.evaluatedTransformIDs = ["history-text"] -> passes
Defensive patterns

Strategy: validation

Validate before calling

function allPlanTransformsEvaluated(plan: CavePlan, evidence: { evaluatedTransformIDs: string[] }): boolean {
  const selected = new Set(plan.segment_routes.map((r) => r.transform_id));
  return [...selected].every((id) => evidence.evaluatedTransformIDs.includes(id));
}
if (!allPlanTransformsEvaluated(plan, execution)) throw new Error("recompile plan for this workload");

Try / catch

try {
  validateTransformEvidence(execution, plan);
} catch (err) {
  if (err instanceof Error && err.message === "cave_harness_transform_not_evaluated") {
    // plan/workload mismatch: recompile the plan, do not patch evidence arrays
  }
}

Prevention

When it happens

Trigger: A plan with `segment_routes: [{ transform_id: "terminal-ccr" }, { transform_id: "history-text" }]` but an execution whose `evaluatedTransformIDs` contains only one of them; a harness that skips evaluation for a route kind (e.g. no tool_result segments, so the terminal transform never evaluated).

Common situations: Reusing a plan compiled for a different workload shape (routes for tool results on a run with no tools); a harness implementation that lazily evaluates only encountered segment kinds; editing the plan after compilation without re-running the compiler.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/d897cc39365912d6. Report an issue: GitHub.