{"record":{"id":"87e3a962019a8845","repo":"JuliusBrussee/caveman","slug":"cave-live-eval-sandbox-profile-missing","errorCode":"cave_live_eval_sandbox_profile_missing","errorMessage":"cave_live_eval_sandbox_profile_missing","messagePattern":"cave_live_eval_sandbox_profile_missing","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/agent/src/cli.ts","lineNumber":1235,"sourceCode":"      unknown_transform: plan.segment_routes.some((route) =>\n        !result.evaluatedTransformIDs.includes(route.transform_id)),\n      output_digest: sha256(result.text),\n    };\n  } catch (error) {\n    throw new Error(\"cave_fixture_terminal_evidence_missing\", { cause: error });\n  }\n}\n\nasync function loadEvalSandboxProfile(\n  root: string,\n  fixture: EvalDefinition,\n): Promise<{\n  network: boolean;\n  childProcess: boolean;\n  credentialEnv: readonly string[];\n}> {\n  const path = fixture.tools.sandbox;\n  if (!path) throw new Error(\"cave_live_eval_sandbox_profile_missing\");\n  const fullPath = resolve(root, path);\n  const relativePath = relative(root, fullPath);\n  if (relativePath === \"..\" || relativePath.startsWith(\"../\") ||\n      relativePath.startsWith(\"..\\\\\") || isAbsolute(relativePath)) {\n    throw new Error(\"cave_live_eval_sandbox_profile_escapes_root\");\n  }\n  const parsed = JSON.parse(await readFile(fullPath, \"utf8\")) as Record<string, unknown>;\n  const keys = Object.keys(parsed).sort();\n  const expected = [\"child_process\", \"credential_env\", \"network\", \"schema_version\"];\n  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||\n      parsed.schema_version !== 1 || typeof parsed.network !== \"boolean\" ||\n      typeof parsed.child_process !== \"boolean\" || !Array.isArray(parsed.credential_env) ||\n      parsed.credential_env.some((name) => typeof name !== \"string\" ||\n        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {\n    throw new Error(\"cave_live_eval_sandbox_profile_invalid\");\n  }\n  validateSandboxCredentialEnv(parsed.credential_env as string[]);\n  return {","sourceCodeStart":1217,"sourceCodeEnd":1253,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/agent/src/cli.ts#L1217-L1253","documentation":"loadEvalSandboxProfile() resolves the sandbox profile path declared by a live-mode eval fixture (fixture.tools.sandbox). If the fixture declares no sandbox path, it throws 'cave_live_eval_sandbox_profile_missing'. Live tool mode runs execute real sandboxed tools, so a declared profile is mandatory before any provider traffic.","triggerScenarios":"Running a fixture whose tools.mode is 'live' (runFixture passes sandboxProfile: await loadEvalSandboxProfile(...)) while fixture.tools.sandbox is undefined/null/empty in the eval definition.","commonSituations":"Authoring a new eval fixture with mode 'live' but forgetting to set tools.sandbox; copying a fixture-tool config from a 'fixture'-mode eval where sandbox is optional; a schema/migration dropping the sandbox field.","solutions":["Add a sandbox profile path to the fixture: set fixture.tools.sandbox to a project-relative JSON profile file","Switch the fixture to tools.mode 'fixture' if it does not actually need live sandboxed tools","Write the missing sandbox profile file (with network, child_process, credential_env, schema_version keys) and reference it"],"exampleFix":"// before\n{ id: \"live-eval\", tools: { mode: \"live\" } }\n// after\n{ id: \"live-eval\", tools: { mode: \"live\", sandbox: \"eval/sandbox.profile.json\" } }","handlingStrategy":"validation","validationCode":"function liveFixtureHasSandbox(fixture: EvalDefinition): boolean {\n  return fixture.tools.mode !== \"live\" || typeof fixture.tools.sandbox === \"string\" && fixture.tools.sandbox.length > 0;\n}\nconst invalid = evals.filter((f) => !liveFixtureHasSandbox(f));\nif (invalid.length) throw new Error(`live fixtures missing sandbox: ${invalid.map((f) => f.id).join(\", \")}`);","typeGuard":"function hasSandboxProfile(f: EvalDefinition): f is EvalDefinition & { tools: { mode: \"live\"; sandbox: string } } {\n  return f.tools.mode !== \"live\" || (typeof (f.tools as { sandbox?: unknown }).sandbox === \"string\"\n    && (f.tools.sandbox as string).length > 0);\n}","tryCatchPattern":"try {\n  await loadEvalSandboxProfile(root, fixture);\n} catch (error) {\n  if ((error as Error).message === \"cave_live_eval_sandbox_profile_missing\") {\n    throw new Error(`fixture '${fixture.id}' uses tools.mode 'live' but declares no tools.sandbox; add a project-relative profile path`);\n  }\n  throw error;\n}","preventionTips":["Add a lint/schema check requiring tools.sandbox on every live-mode fixture","Reuse a shared typed fixture schema so sandbox is mandatory for mode 'live'","Default new fixtures to 'fixture' mode until a profile exists"],"tags":["eval","sandbox","config","missing-field"],"backgroundTag":"missing-required-config-field","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}