JuliusBrussee/caveman · error · Error

cave_live_eval_sandbox_profile_missing

Error message

cave_live_eval_sandbox_profile_missing

What it means

Thrown by `loadEvalSandboxProfile` when a live eval fixture has no `fixture.tools.sandbox` path. Live (non-fixture-only) eval runs must declare a sandbox profile JSON — the file that grants/denies network, child processes, and credential env vars for the eval — and the build fails closed when it is absent rather than defaulting to an unsafe sandbox.

Source

Thrown at packages/agent/src/cli.ts:1235

      unknown_transform: plan.segment_routes.some((route) =>
        !result.evaluatedTransformIDs.includes(route.transform_id)),
      output_digest: sha256(result.text),
    };
  } catch (error) {
    throw new Error("cave_fixture_terminal_evidence_missing", { cause: error });
  }
}

async function loadEvalSandboxProfile(
  root: string,
  fixture: EvalDefinition,
): Promise<{
  network: boolean;
  childProcess: boolean;
  credentialEnv: readonly string[];
}> {
  const path = fixture.tools.sandbox;
  if (!path) throw new Error("cave_live_eval_sandbox_profile_missing");
  const fullPath = resolve(root, path);
  const relativePath = relative(root, fullPath);
  if (relativePath === ".." || relativePath.startsWith("../") ||
      relativePath.startsWith("..\\") || isAbsolute(relativePath)) {
    throw new Error("cave_live_eval_sandbox_profile_escapes_root");
  }
  const parsed = JSON.parse(await readFile(fullPath, "utf8")) as Record<string, unknown>;
  const keys = Object.keys(parsed).sort();
  const expected = ["child_process", "credential_env", "network", "schema_version"];
  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||
      parsed.schema_version !== 1 || typeof parsed.network !== "boolean" ||
      typeof parsed.child_process !== "boolean" || !Array.isArray(parsed.credential_env) ||
      parsed.credential_env.some((name) => typeof name !== "string" ||
        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {
    throw new Error("cave_live_eval_sandbox_profile_invalid");
  }
  validateSandboxCredentialEnv(parsed.credential_env as string[]);
  return {

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Add `tools: { sandbox: "<project-relative path>.json" }` to the eval definition, pointing at a sandbox profile.
2. Create the profile file with the required schema (schema_version 1, network, child_process, credential_env).
3. If the eval should not run live, mark it so it stays out of the live-run set rather than leaving sandbox undefined.

Example fix

// before
export const myEval = evalDef({ input: "...", tools: {} });

// after
export const myEval = evalDef({ input: "...", tools: { sandbox: "evals/sandbox.json" } });

function evalHasSandboxProfile(fixture: { tools?: { sandbox?: unknown } }): boolean {
  return typeof fixture.tools?.sandbox === "string" && fixture.tools.sandbox.length > 0;
}
// assert for every eval before marking it approved+required

function hasSandboxPath(fixture: unknown): fixture is { tools: { sandbox: string } } {
  return typeof fixture === "object" && fixture !== null &&
    typeof (fixture as { tools?: { sandbox?: unknown } }).tools?.sandbox === "string";
}

try {
  await build(args);
} catch (error) {
  if (error instanceof Error && error.message === "cave_live_eval_sandbox_profile_missing") {
    // add tools.sandbox path to the eval definition and create the profile file
  } else throw error;
}

When it happens

Trigger: An approved required eval definition passed to the live run path whose `tools` object lacks `sandbox`. Typically a fixture authored for static-only checking, then promoted to a live eval without adding the sandbox profile reference.

Common situations: Writing a new eval via the fixture helper that doesn't set `tools.sandbox`; copying an eval from a sample project and dropping the profile path; refactoring eval definitions and losing the `tools` field.

Related errors

• cave_live_eval_sandbox_profile_invalid
• caveman build: requiredFixturePassRate must be in (0,1]
• caveman build: qualityRetention must be in (0,1]
• cave_sandbox_conformance_failed
• cave_privacy_conformance_failed

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/87e3a962019a8845. Report an issue: GitHub.