{"record":{"id":"f288f3d14f534d58","repo":"JuliusBrussee/caveman","slug":"caveman-agent-live-eval-tools-require-an-explicit","errorCode":null,"errorMessage":"caveman agent: live eval tools require an explicit sandbox","messagePattern":"caveman agent: live eval tools require an explicit sandbox","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/agent/src/primitives.ts","lineNumber":463,"sourceCode":"  const known = new Set([\"contains\", \"tool_called\", \"exact_match\", \"json_schema\"]);\n  for (const grader of options.quality) {\n    if (!known.has(grader.type)) {\n      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);\n    }\n  }\n  for (const guardrail of options.guardrails ?? []) {\n    if (guardrail.type === \"latency_threshold\" &&\n        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {\n      throw new Error(\"caveman agent: latency guardrail requires positive integer p95_ms\");\n    }\n    if (guardrail.type === \"error_rate\" &&\n        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {\n      throw new Error(\"caveman agent: error-rate guardrail max must be in [0,1]\");\n    }\n  }\n  const tools = options.tools ?? { mode: \"fixture\" as const };\n  if (tools.mode === \"live\" && !tools.sandbox) {\n    throw new Error(\"caveman agent: live eval tools require an explicit sandbox\");\n  }\n  return Object.freeze({\n    kind: \"eval\",\n    id: options.id,\n    approved: options.approved ?? false,\n    required: options.required ?? true,\n    input: options.input,\n    tools,\n    quality: Object.freeze([...options.quality]),\n    guardrails: Object.freeze([...(options.guardrails ?? [])]),\n  });\n}\n\nexport { evalFixture as eval };\n\n// The real `subagent` builder + its SubagentDefinition live in index.ts and\n// shadow this module's `export *`. A stale duplicate here (different shape:\n// contextBudget/modelCallBudget) was dead and, worse, still re-exported a","sourceCodeStart":445,"sourceCodeEnd":481,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/agent/src/primitives.ts#L445-L481","documentation":"evalFixture() defaults tools to { mode: \"fixture\" }, but if tools.mode is \"live\" an explicit sandbox string is required. Live tool execution is untrusted, so the library refuses to run it without a named sandbox isolation boundary and throws this error instead.","triggerScenarios":"evalFixture({ tools: { mode: \"live\" } }) with sandbox omitted or set to an empty/undefined value. Only the combination mode:\"live\" plus a missing sandbox triggers it; mode:\"fixture\" needs no sandbox.","commonSituations":"Switching a fixture eval to live mode for realistic testing and forgetting the sandbox; a shared config template that only sets mode; an empty sandbox string from an unset environment variable.","solutions":["Provide the sandbox: tools: { mode: \"live\", sandbox: \"my-sandbox\" }.","Keep mode: \"fixture\" if the eval does not actually need live tool execution.","Read the sandbox name from a validated config/env value and fail fast if it is empty before building the fixture."],"exampleFix":"// before\ntools: { mode: \"live\" }\n// after\ntools: { mode: \"live\", sandbox: \"eval-sandbox-1\" }","handlingStrategy":"validation","validationCode":"if (tools?.mode === \"live\" && !tools.sandbox) {\n  throw new Error(`live eval tools require a sandbox; set tools.sandbox for eval ${id}`);\n}","typeGuard":"function liveToolsValid(t?: { mode: string; sandbox?: string }): boolean { return t?.mode !== \"live\" || typeof t.sandbox === \"string\" && t.sandbox.length > 0; }","tryCatchPattern":"try { ev = evalFixture(opts); }\ncatch (e) { if (String(e.message).includes(\"live eval tools require\")) ev = evalFixture({ ...opts, tools: { ...opts.tools, mode: \"fixture\" } }); else throw e; }","preventionTips":["Whenever you set mode: \"live\", add the sandbox in the same edit.","Resolve sandbox names from env/config at startup and fail fast on empties.","Default eval templates to mode: \"fixture\" unless live execution is explicitly needed."],"tags":["validation","eval","sandbox","missing-option"],"backgroundTag":"missing-required-option","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}