{"record":{"id":"3b2a617ba4f0ec7d","repo":"JuliusBrussee/caveman","slug":"cave-harness-incomplete-evidence","errorCode":"cave_harness_incomplete_evidence","errorMessage":"cave_harness_incomplete_evidence","messagePattern":"cave_harness_incomplete_evidence","errorType":"error_code","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/agent/src/adapters.ts","lineNumber":354,"sourceCode":"  inputTokens: number;\n  outputTokens: number;\n  cacheReadTokens: number;\n  cacheWriteTokens: number;\n  reasoningTokens: number;\n  totalTokens: number;\n}\n\nfunction harnessExecution(input: {\n  request: Readonly<HarnessRequest>;\n  text: string;\n  provider: string;\n  model: string;\n  usage: NormalizedUsage;\n  latencyMs: number;\n}): HarnessExecution {\n  const expected = expectedProviderModel(input.request.plan);\n  if (input.provider !== expected.provider || input.model !== expected.model) {\n    throw new Error(\"cave_harness_incomplete_evidence\");\n  }\n  const priced = validateProviderUsage({\n    provider: input.provider,\n    model: input.model,\n    ...input.usage,\n  }, { requirePriced: true });\n  return {\n    terminal: true,\n    text: input.text,\n    provider: input.provider,\n    model: input.model,\n    ...input.usage,\n    costUsd: priced.catalogCostUsd,\n    usageBasis: \"provider_reported\",\n    priceBasis: \"public_catalog\",\n    evaluatedTransformIDs: [...input.request.evaluatedTransformIDs],\n    appliedTransformIDs: [...input.request.appliedTransformIDs],\n    recoveryResolved: input.request.recoveryResolved,","sourceCodeStart":336,"sourceCodeEnd":372,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/packages/agent/src/adapters.ts#L336-L372","documentation":"`harnessExecution` is the shared post-generation evidence builder: it derives the expected `{provider, model}` by splitting `plan.model` on its first `/` and compares both with what the harness actually reported. If either differs, `cave_harness_incomplete_evidence` throws before pricing — the run's provenance is broken because the answer may have come from a different model than the locked plan selected. For Vercel/Mastra the provider comes from the plan while the model comes from `response.response.modelId` (after stripping the provider prefix); for Eve both come from `session.started`.","triggerScenarios":"A gateway/proxy rerouting `anthropic/claude-…` to a fallback model so the reported modelId differs; using a model alias in `plan.model` while the provider reports the concrete ID (`claude-sonnet-4` vs `claude-sonnet-4-20250514`); an Eve session started with a different model than the plan's.","commonSituations":"Provider failover or load-balancer model rewriting; aliases in plans while providers report dated concrete IDs; stale plans after a model rename; custom bindings returning a normalized modelId.","solutions":["Use exact, concrete model IDs in `plan.model` on both sides of the comparison","Disable gateway fallback / model aliasing for locked runs","Rebuild the plan against the exact `modelId` the provider reports (log it once from a raw generate)","Add a contract test asserting the configured model equals the reported modelId"],"exampleFix":"// before — plan pins an alias, provider reports the concrete ID\n// plan.model: 'anthropic/claude-sonnet-4'  (response.modelId: 'claude-sonnet-4-20250514')\n// after\n// plan.model: 'anthropic/claude-sonnet-4-20250514'","handlingStrategy":"validation","validationCode":"// before locked runs, assert the plan's model is the exact concrete ID the\n// harness/gateway will report (aliases and fallback routing cause the mismatch)\nconst planModel = plan.model; // e.g. 'anthropic/claude-sonnet-4-20250514'\nconst reported = await resolveConcreteModelId(gateway, planModel); // one probe call\nif (reported !== planModel) {\n  throw new Error(`plan.model ${planModel} != reported ${reported}; pin the concrete ID`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  return await adapter.run(req);\n} catch (e) {\n  if (e instanceof Error && e.message === 'cave_harness_incomplete_evidence') {\n    // provider/model provenance broke: log plan.model vs the harness-reported\n    // modelId, then pin the concrete ID and disable fallback routing before retry\n  }\n  throw e;\n}","preventionTips":["Use concrete dated model IDs in plans, never aliases","Disable gateway failover/model rewriting for locked executions","Add a contract test asserting configured model === reported modelId"],"tags":["typescript","llm","model-identity","evidence","integrity"],"backgroundTag":"model-identity-mismatch","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}