{"record":{"id":"87e3a962019a8845","repo":"JuliusBrussee/caveman","slug":"cave-live-eval-sandbox-profile-missing","errorCode":null,"errorMessage":"cave_live_eval_sandbox_profile_missing","messagePattern":"cave_live_eval_sandbox_profile_missing","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/agent/src/cli.ts","lineNumber":1235,"sourceCode":"      unknown_transform: plan.segment_routes.some((route) =>\n        !result.evaluatedTransformIDs.includes(route.transform_id)),\n      output_digest: sha256(result.text),\n    };\n  } catch (error) {\n    throw new Error(\"cave_fixture_terminal_evidence_missing\", { cause: error });\n  }\n}\n\nasync function loadEvalSandboxProfile(\n  root: string,\n  fixture: EvalDefinition,\n): Promise<{\n  network: boolean;\n  childProcess: boolean;\n  credentialEnv: readonly string[];\n}> {\n  const path = fixture.tools.sandbox;\n  if (!path) throw new Error(\"cave_live_eval_sandbox_profile_missing\");\n  const fullPath = resolve(root, path);\n  const relativePath = relative(root, fullPath);\n  if (relativePath === \"..\" || relativePath.startsWith(\"../\") ||\n      relativePath.startsWith(\"..\\\\\") || isAbsolute(relativePath)) {\n    throw new Error(\"cave_live_eval_sandbox_profile_escapes_root\");\n  }\n  const parsed = JSON.parse(await readFile(fullPath, \"utf8\")) as Record<string, unknown>;\n  const keys = Object.keys(parsed).sort();\n  const expected = [\"child_process\", \"credential_env\", \"network\", \"schema_version\"];\n  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||\n      parsed.schema_version !== 1 || typeof parsed.network !== \"boolean\" ||\n      typeof parsed.child_process !== \"boolean\" || !Array.isArray(parsed.credential_env) ||\n      parsed.credential_env.some((name) => typeof name !== \"string\" ||\n        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {\n    throw new Error(\"cave_live_eval_sandbox_profile_invalid\");\n  }\n  validateSandboxCredentialEnv(parsed.credential_env as string[]);\n  return {","sourceCodeStart":1217,"sourceCodeEnd":1253,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/agent/src/cli.ts#L1217-L1253","documentation":"Thrown by `loadEvalSandboxProfile` when a live eval fixture has no `fixture.tools.sandbox` path. Live (non-fixture-only) eval runs must declare a sandbox profile JSON — the file that grants/denies network, child processes, and credential env vars for the eval — and the build fails closed when it is absent rather than defaulting to an unsafe sandbox.","triggerScenarios":"An approved required eval definition passed to the live run path whose `tools` object lacks `sandbox`. Typically a fixture authored for static-only checking, then promoted to a live eval without adding the sandbox profile reference.","commonSituations":"Writing a new eval via the fixture helper that doesn't set `tools.sandbox`; copying an eval from a sample project and dropping the profile path; refactoring eval definitions and losing the `tools` field.","solutions":["Add `tools: { sandbox: \"<project-relative path>.json\" }` to the eval definition, pointing at a sandbox profile.","Create the profile file with the required schema (schema_version 1, network, child_process, credential_env).","If the eval should not run live, mark it so it stays out of the live-run set rather than leaving sandbox undefined."],"exampleFix":"// before\nexport const myEval = evalDef({ input: \"...\", tools: {} });\n\n// after\nexport const myEval = evalDef({ input: \"...\", tools: { sandbox: \"evals/sandbox.json\" } });","handlingStrategy":"validation","validationCode":"function evalHasSandboxProfile(fixture: { tools?: { sandbox?: unknown } }): boolean {\n  return typeof fixture.tools?.sandbox === \"string\" && fixture.tools.sandbox.length > 0;\n}\n// assert for every eval before marking it approved+required","typeGuard":"function hasSandboxPath(fixture: unknown): fixture is { tools: { sandbox: string } } {\n  return typeof fixture === \"object\" && fixture !== null &&\n    typeof (fixture as { tools?: { sandbox?: unknown } }).tools?.sandbox === \"string\";\n}","tryCatchPattern":"try {\n  await build(args);\n} catch (error) {\n  if (error instanceof Error && error.message === \"cave_live_eval_sandbox_profile_missing\") {\n    // add tools.sandbox path to the eval definition and create the profile file\n  } else throw error;\n}","preventionTips":["Use the eval helper that requires a sandbox profile at authoring time.","Lint eval definitions in CI: approved+required implies tools.sandbox set."],"tags":["eval","sandbox","build","validation"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}