JuliusBrussee/caveman · error · Error

cave_harness_incomplete_evidence

cave_harness_incomplete_evidence

Error message

cave_harness_incomplete_evidence

What it means

`harnessExecution` is the shared post-generation evidence builder: it derives the expected `{provider, model}` by splitting `plan.model` on its first `/` and compares both with what the harness actually reported. If either differs, `cave_harness_incomplete_evidence` throws before pricing — the run's provenance is broken because the answer may have come from a different model than the locked plan selected. For Vercel/Mastra the provider comes from the plan while the model comes from `response.response.modelId` (after stripping the provider prefix); for Eve both come from `session.started`.

Solutions

1. Use exact, concrete model IDs in `plan.model` on both sides of the comparison
2. Disable gateway fallback / model aliasing for locked runs
3. Rebuild the plan against the exact `modelId` the provider reports (log it once from a raw generate)
4. Add a contract test asserting the configured model equals the reported modelId

Example fix

// before — plan pins an alias, provider reports the concrete ID
// plan.model: 'anthropic/claude-sonnet-4'  (response.modelId: 'claude-sonnet-4-20250514')
// after
// plan.model: 'anthropic/claude-sonnet-4-20250514'

// before locked runs, assert the plan's model is the exact concrete ID the
// harness/gateway will report (aliases and fallback routing cause the mismatch)
const planModel = plan.model; // e.g. 'anthropic/claude-sonnet-4-20250514'
const reported = await resolveConcreteModelId(gateway, planModel); // one probe call
if (reported !== planModel) {
  throw new Error(`plan.model ${planModel} != reported ${reported}; pin the concrete ID`);
}

try {
  return await adapter.run(req);
} catch (e) {
  if (e instanceof Error && e.message === 'cave_harness_incomplete_evidence') {
    // provider/model provenance broke: log plan.model vs the harness-reported
    // modelId, then pin the concrete ID and disable fallback routing before retry
  }
  throw e;
}

When it happens

Trigger: A gateway/proxy rerouting `anthropic/claude-…` to a fallback model so the reported modelId differs; using a model alias in `plan.model` while the provider reports the concrete ID (`claude-sonnet-4` vs `claude-sonnet-4-20250514`); an Eve session started with a different model than the plan's.

Common situations: Provider failover or load-balancer model rewriting; aliases in plans while providers report dated concrete IDs; stale plans after a model rename; custom bindings returning a normalized modelId.

Related errors

• cave_eve_runtime_identity_missing
• cave_eve_terminal_${result.status}
• cave_eve_usage_missing
• cave_mastra_terminal_failure
• cave_mastra_usage_missing

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/3b2a617ba4f0ec7d. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/adapters.ts:354

  inputTokens: number;
  outputTokens: number;
  cacheReadTokens: number;
  cacheWriteTokens: number;
  reasoningTokens: number;
  totalTokens: number;
}

function harnessExecution(input: {
  request: Readonly<HarnessRequest>;
  text: string;
  provider: string;
  model: string;
  usage: NormalizedUsage;
  latencyMs: number;
}): HarnessExecution {
  const expected = expectedProviderModel(input.request.plan);
  if (input.provider !== expected.provider || input.model !== expected.model) {
    throw new Error("cave_harness_incomplete_evidence");
  }
  const priced = validateProviderUsage({
    provider: input.provider,
    model: input.model,
    ...input.usage,
  }, { requirePriced: true });
  return {
    terminal: true,
    text: input.text,
    provider: input.provider,
    model: input.model,
    ...input.usage,
    costUsd: priced.catalogCostUsd,
    usageBasis: "provider_reported",
    priceBasis: "public_catalog",
    evaluatedTransformIDs: [...input.request.evaluatedTransformIDs],
    appliedTransformIDs: [...input.request.appliedTransformIDs],
    recoveryResolved: input.request.recoveryResolved,

View on GitHub (pinned to 766dce6b13)