JuliusBrussee/caveman · error

cave_live_eval_sandbox_profile_missing

cave_live_eval_sandbox_profile_missing

Error message

cave_live_eval_sandbox_profile_missing

What it means

loadEvalSandboxProfile() resolves the sandbox profile path declared by a live-mode eval fixture (fixture.tools.sandbox). If the fixture declares no sandbox path, it throws 'cave_live_eval_sandbox_profile_missing'. Live tool mode runs execute real sandboxed tools, so a declared profile is mandatory before any provider traffic.

Solutions

1. Add a sandbox profile path to the fixture: set fixture.tools.sandbox to a project-relative JSON profile file
2. Switch the fixture to tools.mode 'fixture' if it does not actually need live sandboxed tools
3. Write the missing sandbox profile file (with network, child_process, credential_env, schema_version keys) and reference it

Example fix

// before
{ id: "live-eval", tools: { mode: "live" } }
// after
{ id: "live-eval", tools: { mode: "live", sandbox: "eval/sandbox.profile.json" } }

function liveFixtureHasSandbox(fixture: EvalDefinition): boolean {
  return fixture.tools.mode !== "live" || typeof fixture.tools.sandbox === "string" && fixture.tools.sandbox.length > 0;
}
const invalid = evals.filter((f) => !liveFixtureHasSandbox(f));
if (invalid.length) throw new Error(`live fixtures missing sandbox: ${invalid.map((f) => f.id).join(", ")}`);

function hasSandboxProfile(f: EvalDefinition): f is EvalDefinition & { tools: { mode: "live"; sandbox: string } } {
  return f.tools.mode !== "live" || (typeof (f.tools as { sandbox?: unknown }).sandbox === "string"
    && (f.tools.sandbox as string).length > 0);
}

try {
  await loadEvalSandboxProfile(root, fixture);
} catch (error) {
  if ((error as Error).message === "cave_live_eval_sandbox_profile_missing") {
    throw new Error(`fixture '${fixture.id}' uses tools.mode 'live' but declares no tools.sandbox; add a project-relative profile path`);
  }
  throw error;
}

When it happens

Trigger: Running a fixture whose tools.mode is 'live' (runFixture passes sandboxProfile: await loadEvalSandboxProfile(...)) while fixture.tools.sandbox is undefined/null/empty in the eval definition.

Common situations: Authoring a new eval fixture with mode 'live' but forgetting to set tools.sandbox; copying a fixture-tool config from a 'fixture'-mode eval where sandbox is optional; a schema/migration dropping the sandbox field.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• cave_live_eval_sandbox_profile_escapes_root
• caveman agent: live eval tools require an explicit sandbox
• compat upstream : base_url is required
• : MCP ownership journal failed; native config may already…
• MCP transaction failed and rolled back

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/87e3a962019a8845. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/cli.ts:1235

      unknown_transform: plan.segment_routes.some((route) =>
        !result.evaluatedTransformIDs.includes(route.transform_id)),
      output_digest: sha256(result.text),
    };
  } catch (error) {
    throw new Error("cave_fixture_terminal_evidence_missing", { cause: error });
  }
}

async function loadEvalSandboxProfile(
  root: string,
  fixture: EvalDefinition,
): Promise<{
  network: boolean;
  childProcess: boolean;
  credentialEnv: readonly string[];
}> {
  const path = fixture.tools.sandbox;
  if (!path) throw new Error("cave_live_eval_sandbox_profile_missing");
  const fullPath = resolve(root, path);
  const relativePath = relative(root, fullPath);
  if (relativePath === ".." || relativePath.startsWith("../") ||
      relativePath.startsWith("..\\") || isAbsolute(relativePath)) {
    throw new Error("cave_live_eval_sandbox_profile_escapes_root");
  }
  const parsed = JSON.parse(await readFile(fullPath, "utf8")) as Record<string, unknown>;
  const keys = Object.keys(parsed).sort();
  const expected = ["child_process", "credential_env", "network", "schema_version"];
  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||
      parsed.schema_version !== 1 || typeof parsed.network !== "boolean" ||
      typeof parsed.child_process !== "boolean" || !Array.isArray(parsed.credential_env) ||
      parsed.credential_env.some((name) => typeof name !== "string" ||
        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {
    throw new Error("cave_live_eval_sandbox_profile_invalid");
  }
  validateSandboxCredentialEnv(parsed.credential_env as string[]);
  return {

View on GitHub (pinned to 3ee70a1026)