JuliusBrussee/caveman · error

caveman agent: live eval tools require an explicit sandbox

Error message

caveman agent: live eval tools require an explicit sandbox

What it means

Thrown by evalFixture() when tools.mode is 'live' but no sandbox is named. Live-tool evals execute real tools, so the framework requires an explicit sandbox string to bound what a live run may touch; fixture mode (the default when tools is omitted) needs no sandbox.

Source

Thrown at packages/agent/src/primitives.ts:463

  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",
    id: options.id,
    approved: options.approved ?? false,
    required: options.required ?? true,
    input: options.input,
    tools,
    quality: Object.freeze([...options.quality]),
    guardrails: Object.freeze([...(options.guardrails ?? [])]),
  });
}

export { evalFixture as eval };

// The real `subagent` builder + its SubagentDefinition live in index.ts and
// shadow this module's `export *`. A stale duplicate here (different shape:
// contextBudget/modelCallBudget) was dead and, worse, still re-exported a

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Name an explicit sandbox for live tools: tools: { mode: 'live', sandbox: 'eval-tools' }
2. If the eval does not truly need real tool execution, keep the default fixture mode by omitting tools
3. Validate that a configured sandbox id is a non-empty string before building live fixtures

Example fix

// before
evalFixture({ id: 'e1', input: q, quality: q1, tools: { mode: 'live' } });

// after
evalFixture({ id: 'e1', input: q, quality: q1, tools: { mode: 'live', sandbox: 'eval-tools' } });

function liveToolsConfig(sandbox: unknown): { mode: 'live'; sandbox: string } | { mode: 'fixture' } {
  if (typeof sandbox !== 'string' || sandbox.trim() === '') return { mode: 'fixture' };
  return { mode: 'live', sandbox };
}

function isLiveTools(value: unknown): value is { mode: 'live'; sandbox: string } { return typeof value === 'object' && value !== null && (value as { mode?: unknown }).mode === 'live' && typeof (value as { sandbox?: unknown }).sandbox === 'string' && (value as { sandbox: string }).sandbox !== ''; }

When it happens

Trigger: Calling evalFixture({ ..., tools: { mode: 'live' } }) with no sandbox property, or with sandbox: '' (falsy). tools: { mode: 'fixture' } or omitting tools never triggers it.

Common situations: Upgrading a fixture-based eval to live tools and forgetting the sandbox field; passing a sandbox name from config where the key is missing so it resolves to undefined; assuming the CLI's default sandbox applies to programmatic fixtures (it does not).

Related errors

• cave_sandbox_conformance_failed
• cave_live_eval_sandbox_profile_missing
• cave_live_eval_sandbox_profile_escapes_root
• cave_live_eval_sandbox_profile_invalid
• cave_sandbox_credential_env_not_allowlisted

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/f288f3d14f534d58. Report an issue: GitHub.