{"record":{"id":"13cd139aac8fbc77","repo":"JuliusBrussee/caveman","slug":"cave-fixture-terminal-evidence-missing","errorCode":"cave_fixture_terminal_evidence_missing","errorMessage":"cave_fixture_terminal_evidence_missing","messagePattern":"cave_fixture_terminal_evidence_missing","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/agent/src/cli.ts","lineNumber":1222,"sourceCode":"      graders,\n      latency_ms: result.latencyMs,\n      provider_visible_tokens: result.inputTokens + result.cacheReadTokens + result.cacheWriteTokens,\n      cache_prefix_sha256: result.cachePrefixSHA256,\n      cache_boundary_known: result.cacheBoundaryKnown,\n      cache_read_tokens: result.cacheReadTokens,\n      cache_write_tokens: result.cacheWriteTokens,\n      cache_bust: result.cacheBust,\n      error: false,\n      recovery_resolved: result.recoveryResolved && result.transformFailures.length === 0,\n      privacy_passed: privacyConformance && contentBlindRunEvidence(result, fixture),\n      sandbox_passed: sandboxConformance && (definition.tools.length === 0 ||\n        (definition.sandbox === \"required\" && result.toolCalls.length > 0)),\n      unknown_transform: plan.segment_routes.some((route) =>\n        !result.evaluatedTransformIDs.includes(route.transform_id)),\n      output_digest: sha256(result.text),\n    };\n  } catch (error) {\n    throw new Error(\"cave_fixture_terminal_evidence_missing\", { cause: error });\n  }\n}\n\nasync function loadEvalSandboxProfile(\n  root: string,\n  fixture: EvalDefinition,\n): Promise<{\n  network: boolean;\n  childProcess: boolean;\n  credentialEnv: readonly string[];\n}> {\n  const path = fixture.tools.sandbox;\n  if (!path) throw new Error(\"cave_live_eval_sandbox_profile_missing\");\n  const fullPath = resolve(root, path);\n  const relativePath = relative(root, fullPath);\n  if (relativePath === \"..\" || relativePath.startsWith(\"../\") ||\n      relativePath.startsWith(\"..\\\\\") || isAbsolute(relativePath)) {\n    throw new Error(\"cave_live_eval_sandbox_profile_escapes_root\");","sourceCodeStart":1204,"sourceCodeEnd":1240,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/agent/src/cli.ts#L1204-L1240","documentation":"runFixture() executes an eval fixture through the agent runtime and produces terminal RunEvidence. If anything inside the try block throws (runtime failure, pricing, grading, sandbox evidence assembly, etc.), the error is wrapped as 'cave_fixture_terminal_evidence_missing' with the original error as cause. The evidence contract requires every fixture run to yield terminal evidence; a run without it cannot contribute to eval-complete build selection.","triggerScenarios":"Any throw while running a fixture in runFixture (packages/agent/src/cli.ts:1160): runAgentInternal failing (provider error, abort, sandbox violation), loadEvalSandboxProfile throwing for live fixtures, catalogCost failing, grader errors — during CLI compile/dev eval runs.","commonSituations":"Provider API errors or invalid API keys during eval runs; live-mode fixtures with a missing/invalid sandbox profile; abort signal firing mid-run; a model/tool failure leaving the run without complete usage evidence.","solutions":["Inspect error.cause to find the underlying failure (provider error, sandbox profile, abort, etc.)","Fix the root cause (e.g. valid API key/credentials, reachable provider, correct fixture tools config)","Re-run the eval/compile; if a specific fixture is flaky, isolate it and retry or fix its input/tools","For live fixtures, confirm the sandbox profile file exists and passes loadEvalSandboxProfile validation"],"exampleFix":"// debugging\ntry {\n  await runFixture(/* ... */);\n} catch (error) {\n  console.error(\"fixture failed:\", (error as Error & { cause?: unknown }).cause); // real reason\n}","handlingStrategy":"try-catch","validationCode":"// pre-flight a fixture before the eval run\nif (fixture.tools.mode === \"live\") {\n  if (!fixture.tools.sandbox) throw new Error(\"live fixture needs tools.sandbox\");\n  await loadEvalSandboxProfile(root, fixture); // validates path + schema early\n}\nif (!process.env.ANTHROPIC_API_KEY && !process.env.CAVE_API_KEY) {\n  throw new Error(\"eval run needs provider credentials\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  await runFixture(/* ... */);\n} catch (error) {\n  const cause = (error as Error & { cause?: unknown }).cause;\n  console.error(\"fixture lost terminal evidence; root cause:\", cause);\n  // retry transient provider errors, abort on real config problems\n  if (isTransient(cause)) return await runFixture(/* ... */);\n  throw cause ?? error;\n}","preventionTips":["Always read error.cause — the wrapper message alone hides the real failure","Pre-validate fixture definitions (mode, sandbox path, inputs) before runs","Use valid provider credentials and reachable network for eval runs","Keep fixtures small and deterministic so abort/timeouts are rare"],"tags":["eval","fixture","evidence","wrapped-error"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}