{"record":{"id":"f288f3d14f534d58","repo":"JuliusBrussee/caveman","slug":"caveman-agent-live-eval-tools-require-an-explicit","errorCode":null,"errorMessage":"caveman agent: live eval tools require an explicit sandbox","messagePattern":"caveman agent: live eval tools require an explicit sandbox","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/agent/src/primitives.ts","lineNumber":463,"sourceCode":"  const known = new Set([\"contains\", \"tool_called\", \"exact_match\", \"json_schema\"]);\n  for (const grader of options.quality) {\n    if (!known.has(grader.type)) {\n      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);\n    }\n  }\n  for (const guardrail of options.guardrails ?? []) {\n    if (guardrail.type === \"latency_threshold\" &&\n        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {\n      throw new Error(\"caveman agent: latency guardrail requires positive integer p95_ms\");\n    }\n    if (guardrail.type === \"error_rate\" &&\n        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {\n      throw new Error(\"caveman agent: error-rate guardrail max must be in [0,1]\");\n    }\n  }\n  const tools = options.tools ?? { mode: \"fixture\" as const };\n  if (tools.mode === \"live\" && !tools.sandbox) {\n    throw new Error(\"caveman agent: live eval tools require an explicit sandbox\");\n  }\n  return Object.freeze({\n    kind: \"eval\",\n    id: options.id,\n    approved: options.approved ?? false,\n    required: options.required ?? true,\n    input: options.input,\n    tools,\n    quality: Object.freeze([...options.quality]),\n    guardrails: Object.freeze([...(options.guardrails ?? [])]),\n  });\n}\n\nexport { evalFixture as eval };\n\n// The real `subagent` builder + its SubagentDefinition live in index.ts and\n// shadow this module's `export *`. A stale duplicate here (different shape:\n// contextBudget/modelCallBudget) was dead and, worse, still re-exported a","sourceCodeStart":445,"sourceCodeEnd":481,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/agent/src/primitives.ts#L445-L481","documentation":"Thrown by evalFixture() when tools.mode is 'live' but no sandbox is named. Live-tool evals execute real tools, so the framework requires an explicit sandbox string to bound what a live run may touch; fixture mode (the default when tools is omitted) needs no sandbox.","triggerScenarios":"Calling evalFixture({ ..., tools: { mode: 'live' } }) with no sandbox property, or with sandbox: '' (falsy). tools: { mode: 'fixture' } or omitting tools never triggers it.","commonSituations":"Upgrading a fixture-based eval to live tools and forgetting the sandbox field; passing a sandbox name from config where the key is missing so it resolves to undefined; assuming the CLI's default sandbox applies to programmatic fixtures (it does not).","solutions":["Name an explicit sandbox for live tools: tools: { mode: 'live', sandbox: 'eval-tools' }","If the eval does not truly need real tool execution, keep the default fixture mode by omitting tools","Validate that a configured sandbox id is a non-empty string before building live fixtures"],"exampleFix":"// before\nevalFixture({ id: 'e1', input: q, quality: q1, tools: { mode: 'live' } });\n\n// after\nevalFixture({ id: 'e1', input: q, quality: q1, tools: { mode: 'live', sandbox: 'eval-tools' } });","handlingStrategy":"validation","validationCode":"function liveToolsConfig(sandbox: unknown): { mode: 'live'; sandbox: string } | { mode: 'fixture' } {\n  if (typeof sandbox !== 'string' || sandbox.trim() === '') return { mode: 'fixture' };\n  return { mode: 'live', sandbox };\n}","typeGuard":"function isLiveTools(value: unknown): value is { mode: 'live'; sandbox: string } { return typeof value === 'object' && value !== null && (value as { mode?: unknown }).mode === 'live' && typeof (value as { sandbox?: unknown }).sandbox === 'string' && (value as { sandbox: string }).sandbox !== ''; }","tryCatchPattern":null,"preventionTips":["Default to fixture mode; opt into live only when a test genuinely needs real tools","Require a non-empty sandbox string in the config schema when mode is 'live'","Keep sandbox ids in a named registry so typos fail loudly"],"tags":["eval","sandbox","live-tools","fail-closed"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}