What causes "cave_fixture_terminal_evidence_missing" in JuliusBrussee/caveman?

Inspect error.cause to find the underlying failure (provider error, sandbox profile, abort, etc.)

cave_fixture_terminal_evidence_missing

cave_fixture_terminal_evidence_missing

Error message

cave_fixture_terminal_evidence_missing

What it means

runFixture() executes an eval fixture through the agent runtime and produces terminal RunEvidence. If anything inside the try block throws (runtime failure, pricing, grading, sandbox evidence assembly, etc.), the error is wrapped as 'cave_fixture_terminal_evidence_missing' with the original error as cause. The evidence contract requires every fixture run to yield terminal evidence; a run without it cannot contribute to eval-complete build selection.

Solutions

Inspect error.cause to find the underlying failure (provider error, sandbox profile, abort, etc.)
Fix the root cause (e.g. valid API key/credentials, reachable provider, correct fixture tools config)
Re-run the eval/compile; if a specific fixture is flaky, isolate it and retry or fix its input/tools
For live fixtures, confirm the sandbox profile file exists and passes loadEvalSandboxProfile validation

Example fix

// debugging
try {
  await runFixture(/* ... */);
} catch (error) {
  console.error("fixture failed:", (error as Error & { cause?: unknown }).cause); // real reason
}

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-flight a fixture before the eval run
if (fixture.tools.mode === "live") {
  if (!fixture.tools.sandbox) throw new Error("live fixture needs tools.sandbox");
  await loadEvalSandboxProfile(root, fixture); // validates path + schema early
}
if (!process.env.ANTHROPIC_API_KEY && !process.env.CAVE_API_KEY) {
  throw new Error("eval run needs provider credentials");
}

Try / catch

try {
  await runFixture(/* ... */);
} catch (error) {
  const cause = (error as Error & { cause?: unknown }).cause;
  console.error("fixture lost terminal evidence; root cause:", cause);
  // retry transient provider errors, abort on real config problems
  if (isTransient(cause)) return await runFixture(/* ... */);
  throw cause ?? error;
}

Prevention

Always read error.cause — the wrapper message alone hides the real failure
Pre-validate fixture definitions (mode, sandbox path, inputs) before runs
Use valid provider credentials and reachable network for eval runs
Keep fixtures small and deterministic so abort/timeouts are rare

When it happens

Trigger: Any throw while running a fixture in runFixture (packages/agent/src/cli.ts:1160): runAgentInternal failing (provider error, abort, sandbox violation), loadEvalSandboxProfile throwing for live fixtures, catalogCost failing, grader errors — during CLI compile/dev eval runs.

Common situations: Provider API errors or invalid API keys during eval runs; live-mode fixtures with a missing/invalid sandbox profile; abort signal firing mid-run; a model/tool failure leaving the run without complete usage evidence.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/13cd139aac8fbc77. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/cli.ts:1222

      graders,
      latency_ms: result.latencyMs,
      provider_visible_tokens: result.inputTokens + result.cacheReadTokens + result.cacheWriteTokens,
      cache_prefix_sha256: result.cachePrefixSHA256,
      cache_boundary_known: result.cacheBoundaryKnown,
      cache_read_tokens: result.cacheReadTokens,
      cache_write_tokens: result.cacheWriteTokens,
      cache_bust: result.cacheBust,
      error: false,
      recovery_resolved: result.recoveryResolved && result.transformFailures.length === 0,
      privacy_passed: privacyConformance && contentBlindRunEvidence(result, fixture),
      sandbox_passed: sandboxConformance && (definition.tools.length === 0 ||
        (definition.sandbox === "required" && result.toolCalls.length > 0)),
      unknown_transform: plan.segment_routes.some((route) =>
        !result.evaluatedTransformIDs.includes(route.transform_id)),
      output_digest: sha256(result.text),
    };
  } catch (error) {
    throw new Error("cave_fixture_terminal_evidence_missing", { cause: error });
  }
}

async function loadEvalSandboxProfile(
  root: string,
  fixture: EvalDefinition,
): Promise<{
  network: boolean;
  childProcess: boolean;
  credentialEnv: readonly string[];
}> {
  const path = fixture.tools.sandbox;
  if (!path) throw new Error("cave_live_eval_sandbox_profile_missing");
  const fullPath = resolve(root, path);
  const relativePath = relative(root, fullPath);
  if (relativePath === ".." || relativePath.startsWith("../") ||
      relativePath.startsWith("..\\") || isAbsolute(relativePath)) {
    throw new Error("cave_live_eval_sandbox_profile_escapes_root");

View on GitHub (pinned to 3ee70a1026)