JuliusBrussee/caveman · error

cave_live_eval_sandbox_profile_invalid

cave_live_eval_sandbox_profile_invalid

Error message

cave_live_eval_sandbox_profile_invalid

What it means

The live-eval sandbox profile parser in packages/agent/src/cli.ts validates a strict JSON schema: keys must be exactly [child_process, credential_env, network, schema_version] in order, schema_version must be 1, network/child_process must be booleans, and every credential_env entry must match /^[A-Z][A-Z0-9_]{0,127}$/. Any deviation throws 'cave_live_eval_sandbox_profile_invalid' before validateSandboxCredentialEnv runs further checks. It is a deliberate fail-fast guard so a malformed sandbox profile never reaches evaluation runs.

Solutions

1. Rewrite the sandbox profile so its top-level keys are exactly, in order: child_process, credential_env, network, schema_version.
2. Set schema_version to 1 and make network/child_process strict JSON booleans (true/false, not strings).
3. Ensure every credential_env entry is an uppercase identifier matching /^[A-Z][A-Z0-9_]{0,127}$/ (e.g. ANTHROPIC_API_KEY).
4. Regenerate the profile from a template or the tool's own emitter instead of hand-editing.
5. Fix any remaining issues flagged by validateSandboxCredentialEnv (e.g. duplicated or reserved credential names).

Example fix

// before
{ "network": true, "schema_version": 1, "child_process": false, "credential_env": ["api_key"] }
// after
{ "child_process": false, "credential_env": ["API_KEY"], "network": true, "schema_version": 1 }

function isValidSandboxProfile(p) {
  const keys = Object.keys(p);
  const expected = ["child_process", "credential_env", "network", "schema_version"];
  return keys.length === expected.length &&
    keys.every((k, i) => k === expected[i]) &&
    p.schema_version === 1 &&
    typeof p.network === "boolean" &&
    typeof p.child_process === "boolean" &&
    Array.isArray(p.credential_env) &&
    p.credential_env.every((n) => typeof n === "string" && /^[A-Z][A-Z0-9_]{0,127}$/.test(n));
}

const isSandboxProfile = (p: unknown): p is SandboxProfile =>
  typeof p === "object" && p !== null && isValidSandboxProfile(p as Record<string, unknown>);

try {
  runLiveEval(profile);
} catch (e) {
  if (e.message === "cave_live_eval_sandbox_profile_invalid") {
    console.error("Sandbox profile failed schema check; regenerate it from the template.");
  } else throw e;
}

When it happens

Trigger: Calling the cave live-eval flow with a sandbox profile JSON that has missing/extra/reordered keys, a schema_version other than 1, network or child_process that is not a boolean, or credential_env entries that are non-strings or contain characters outside A-Z, 0-9, and underscore (or exceed 128 chars or do not start with a letter).

Common situations: Hand-editing .caveman sandbox profile JSON and reordering keys or adding a comment-like field; writing credential env names like 'api_key' or 'API-KEY' that violate the uppercase pattern; bumping or omitting schema_version after a tool upgrade; generating the profile with a script that emits lowercase key names.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• at least one report path and the canonical span fixture…
• at least one sink id is required
• base and head must be full Git object IDs
• cave_breaker_retry_backoff_invalid
• cave_breaker_retry_requires_budget

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/83bdbdbdaa4c6617. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/cli.ts:1250

  credentialEnv: readonly string[];
}> {
  const path = fixture.tools.sandbox;
  if (!path) throw new Error("cave_live_eval_sandbox_profile_missing");
  const fullPath = resolve(root, path);
  const relativePath = relative(root, fullPath);
  if (relativePath === ".." || relativePath.startsWith("../") ||
      relativePath.startsWith("..\\") || isAbsolute(relativePath)) {
    throw new Error("cave_live_eval_sandbox_profile_escapes_root");
  }
  const parsed = JSON.parse(await readFile(fullPath, "utf8")) as Record<string, unknown>;
  const keys = Object.keys(parsed).sort();
  const expected = ["child_process", "credential_env", "network", "schema_version"];
  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||
      parsed.schema_version !== 1 || typeof parsed.network !== "boolean" ||
      typeof parsed.child_process !== "boolean" || !Array.isArray(parsed.credential_env) ||
      parsed.credential_env.some((name) => typeof name !== "string" ||
        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {
    throw new Error("cave_live_eval_sandbox_profile_invalid");
  }
  validateSandboxCredentialEnv(parsed.credential_env as string[]);
  return {
    network: parsed.network,
    childProcess: parsed.child_process,
    credentialEnv: parsed.credential_env as string[],
  };
}

function contextIRIsContentBlind(ir: Awaited<ReturnType<typeof lowerContext>>["ir"]): boolean {
  const allowed = new Set([
    "id", "kind", "stability", "safety", "priority", "recovery", "cacheRegion",
    "privacy", "opaque", "ttlTurns", "provenanceDigest", "tokenCount", "bodyHandle",
  ]);
  return ir.segments.every((segment) =>
    Object.keys(segment).every((key) => allowed.has(key)) &&
    /^cave_local_sha256:[0-9a-f]{64}$/.test(segment.bodyHandle) &&
    /^[0-9a-f]{64}$/.test(segment.provenanceDigest));

View on GitHub (pinned to 3ee70a1026)