{"record":{"id":"83bdbdbdaa4c6617","repo":"JuliusBrussee/caveman","slug":"cave-live-eval-sandbox-profile-invalid","errorCode":"cave_live_eval_sandbox_profile_invalid","errorMessage":"cave_live_eval_sandbox_profile_invalid","messagePattern":"cave_live_eval_sandbox_profile_invalid","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/agent/src/cli.ts","lineNumber":1250,"sourceCode":"  credentialEnv: readonly string[];\n}> {\n  const path = fixture.tools.sandbox;\n  if (!path) throw new Error(\"cave_live_eval_sandbox_profile_missing\");\n  const fullPath = resolve(root, path);\n  const relativePath = relative(root, fullPath);\n  if (relativePath === \"..\" || relativePath.startsWith(\"../\") ||\n      relativePath.startsWith(\"..\\\\\") || isAbsolute(relativePath)) {\n    throw new Error(\"cave_live_eval_sandbox_profile_escapes_root\");\n  }\n  const parsed = JSON.parse(await readFile(fullPath, \"utf8\")) as Record<string, unknown>;\n  const keys = Object.keys(parsed).sort();\n  const expected = [\"child_process\", \"credential_env\", \"network\", \"schema_version\"];\n  if (keys.length !== expected.length || keys.some((key, index) => key !== expected[index]) ||\n      parsed.schema_version !== 1 || typeof parsed.network !== \"boolean\" ||\n      typeof parsed.child_process !== \"boolean\" || !Array.isArray(parsed.credential_env) ||\n      parsed.credential_env.some((name) => typeof name !== \"string\" ||\n        !/^[A-Z][A-Z0-9_]{0,127}$/.test(name))) {\n    throw new Error(\"cave_live_eval_sandbox_profile_invalid\");\n  }\n  validateSandboxCredentialEnv(parsed.credential_env as string[]);\n  return {\n    network: parsed.network,\n    childProcess: parsed.child_process,\n    credentialEnv: parsed.credential_env as string[],\n  };\n}\n\nfunction contextIRIsContentBlind(ir: Awaited<ReturnType<typeof lowerContext>>[\"ir\"]): boolean {\n  const allowed = new Set([\n    \"id\", \"kind\", \"stability\", \"safety\", \"priority\", \"recovery\", \"cacheRegion\",\n    \"privacy\", \"opaque\", \"ttlTurns\", \"provenanceDigest\", \"tokenCount\", \"bodyHandle\",\n  ]);\n  return ir.segments.every((segment) =>\n    Object.keys(segment).every((key) => allowed.has(key)) &&\n    /^cave_local_sha256:[0-9a-f]{64}$/.test(segment.bodyHandle) &&\n    /^[0-9a-f]{64}$/.test(segment.provenanceDigest));","sourceCodeStart":1232,"sourceCodeEnd":1268,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/agent/src/cli.ts#L1232-L1268","documentation":"The live-eval sandbox profile parser in packages/agent/src/cli.ts validates a strict JSON schema: keys must be exactly [child_process, credential_env, network, schema_version] in order, schema_version must be 1, network/child_process must be booleans, and every credential_env entry must match /^[A-Z][A-Z0-9_]{0,127}$/. Any deviation throws 'cave_live_eval_sandbox_profile_invalid' before validateSandboxCredentialEnv runs further checks. It is a deliberate fail-fast guard so a malformed sandbox profile never reaches evaluation runs.","triggerScenarios":"Calling the cave live-eval flow with a sandbox profile JSON that has missing/extra/reordered keys, a schema_version other than 1, network or child_process that is not a boolean, or credential_env entries that are non-strings or contain characters outside A-Z, 0-9, and underscore (or exceed 128 chars or do not start with a letter).","commonSituations":"Hand-editing .caveman sandbox profile JSON and reordering keys or adding a comment-like field; writing credential env names like 'api_key' or 'API-KEY' that violate the uppercase pattern; bumping or omitting schema_version after a tool upgrade; generating the profile with a script that emits lowercase key names.","solutions":["Rewrite the sandbox profile so its top-level keys are exactly, in order: child_process, credential_env, network, schema_version.","Set schema_version to 1 and make network/child_process strict JSON booleans (true/false, not strings).","Ensure every credential_env entry is an uppercase identifier matching /^[A-Z][A-Z0-9_]{0,127}$/ (e.g. ANTHROPIC_API_KEY).","Regenerate the profile from a template or the tool's own emitter instead of hand-editing.","Fix any remaining issues flagged by validateSandboxCredentialEnv (e.g. duplicated or reserved credential names)."],"exampleFix":"// before\n{ \"network\": true, \"schema_version\": 1, \"child_process\": false, \"credential_env\": [\"api_key\"] }\n// after\n{ \"child_process\": false, \"credential_env\": [\"API_KEY\"], \"network\": true, \"schema_version\": 1 }","handlingStrategy":"validation","validationCode":"function isValidSandboxProfile(p) {\n  const keys = Object.keys(p);\n  const expected = [\"child_process\", \"credential_env\", \"network\", \"schema_version\"];\n  return keys.length === expected.length &&\n    keys.every((k, i) => k === expected[i]) &&\n    p.schema_version === 1 &&\n    typeof p.network === \"boolean\" &&\n    typeof p.child_process === \"boolean\" &&\n    Array.isArray(p.credential_env) &&\n    p.credential_env.every((n) => typeof n === \"string\" && /^[A-Z][A-Z0-9_]{0,127}$/.test(n));\n}","typeGuard":"const isSandboxProfile = (p: unknown): p is SandboxProfile =>\n  typeof p === \"object\" && p !== null && isValidSandboxProfile(p as Record<string, unknown>);","tryCatchPattern":"try {\n  runLiveEval(profile);\n} catch (e) {\n  if (e.message === \"cave_live_eval_sandbox_profile_invalid\") {\n    console.error(\"Sandbox profile failed schema check; regenerate it from the template.\");\n  } else throw e;\n}","preventionTips":["Generate sandbox profiles from the tool's emitter, never hand-edit the JSON.","Keep credential env names uppercase snake_case matching /^[A-Z][A-Z0-9_]{0,127}$/.","Pin schema_version: 1 and validate the profile in CI before running live evals."],"tags":["config","validation","cli"],"backgroundTag":"schema-validation-failed","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}