JuliusBrussee/caveman · error · Error

caveman agent: live eval tools require an explicit sandbox

Error message

caveman agent: live eval tools require an explicit sandbox

What it means

evalFixture() defaults tools to { mode: "fixture" }, but if tools.mode is "live" an explicit sandbox string is required. Live tool execution is untrusted, so the library refuses to run it without a named sandbox isolation boundary and throws this error instead.

Solutions

1. Provide the sandbox: tools: { mode: "live", sandbox: "my-sandbox" }.
2. Keep mode: "fixture" if the eval does not actually need live tool execution.
3. Read the sandbox name from a validated config/env value and fail fast if it is empty before building the fixture.

Example fix

// before
tools: { mode: "live" }
// after
tools: { mode: "live", sandbox: "eval-sandbox-1" }

if (tools?.mode === "live" && !tools.sandbox) {
  throw new Error(`live eval tools require a sandbox; set tools.sandbox for eval ${id}`);
}

function liveToolsValid(t?: { mode: string; sandbox?: string }): boolean { return t?.mode !== "live" || typeof t.sandbox === "string" && t.sandbox.length > 0; }

try { ev = evalFixture(opts); }
catch (e) { if (String(e.message).includes("live eval tools require")) ev = evalFixture({ ...opts, tools: { ...opts.tools, mode: "fixture" } }); else throw e; }

When it happens

Trigger: evalFixture({ tools: { mode: "live" } }) with sandbox omitted or set to an empty/undefined value. Only the combination mode:"live" plus a missing sandbox triggers it; mode:"fixture" needs no sandbox.

Common situations: Switching a fixture eval to live mode for realistic testing and forgetting the sandbox; a shared config template that only sets mode; an empty sandbox string from an unset environment variable.

Understand the failure class

Background: "Must pass :limit option" / "Missing required option" — required option errors explained — this error's family across 41 libraries.

Related errors

• cave_live_eval_sandbox_profile_escapes_root
• cave_live_eval_sandbox_profile_missing
• cave_sandbox_agent_export_missing
• cave_sandbox_request_invalid
• cave_sandbox_subagent_definition_invalid

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/f288f3d14f534d58. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/primitives.ts:463

  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",
    id: options.id,
    approved: options.approved ?? false,
    required: options.required ?? true,
    input: options.input,
    tools,
    quality: Object.freeze([...options.quality]),
    guardrails: Object.freeze([...(options.guardrails ?? [])]),
  });
}

export { evalFixture as eval };

// The real `subagent` builder + its SubagentDefinition live in index.ts and
// shadow this module's `export *`. A stale duplicate here (different shape:
// contextBudget/modelCallBudget) was dead and, worse, still re-exported a

View on GitHub (pinned to 3ee70a1026)