{"record":{"id":"11700db8e68fe79d","repo":"JuliusBrussee/caveman","slug":"eval-evidence-batch-exceeds-1000-records","errorCode":null,"errorMessage":"eval evidence batch exceeds 1000 records","messagePattern":"eval evidence batch exceeds 1000 records","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/cli/src/index.ts","lineNumber":17161,"sourceCode":"  if (!file) throw new Error(`usage: ${invokedCommand(\"audit\")} eval-import <evidence.jsonl> [--project <uuid>] [--dry-run]`);\n  const data = await readFile(file);\n  if (data.byteLength > 4 * 1024 * 1024) throw new Error(\"eval evidence file exceeds 4 MiB batch limit\");\n\n  const items: Record<string, unknown>[] = [];\n  for (const [index, rawLine] of data.toString(\"utf8\").split(/\\r?\\n/).entries()) {\n    const line = rawLine.trim();\n    if (!line) continue;\n    let parsed: unknown;\n    try {\n      parsed = JSON.parse(line);\n    } catch {\n      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);\n    }\n    if (!parsed || typeof parsed !== \"object\" || Array.isArray(parsed)) {\n      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);\n    }\n    items.push(parsed as Record<string, unknown>);\n    if (items.length > 1000) throw new Error(\"eval evidence batch exceeds 1000 records\");\n  }\n  if (items.length === 0) throw new Error(\"eval evidence file contains no records\");\n\n  const cfg = await config();\n  const project = flagFrom(argv, \"--project\", cfg.projectId ?? \"\");\n  const response = await fetch(`${cfg.baseURL}/api/v1/eval-evidence/batches`, {\n    method: \"POST\",\n    headers: {\n      authorization: `Bearer ${cfg.token}`,\n      \"content-type\": \"application/json\",\n      \"x-cave-csrf\": \"cli\",\n    },\n    body: JSON.stringify({\n      ...(project ? { project_id: project } : {}),\n      dry_run: argv.includes(\"--dry-run\"),\n      items,\n    }),\n  });","sourceCodeStart":17143,"sourceCodeEnd":17179,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/5184b3d11ac6a1acb7d44b9bfaa31698157cff97/packages/cli/src/index.ts#L17143-L17179","documentation":"Cardinality guard: an eval-import batch may carry at most 1000 records. The CLI counts object lines while parsing and throws as soon as the 1001st record is seen, keeping the single POST bounded alongside the 4 MiB byte cap — whichever limit trips first.","triggerScenarios":"Any evidence file with more than 1000 non-blank lines; dense files that pass the size check on early lines but keep accumulating records.","commonSituations":"Bulk historical imports; concatenated multi-day eval logs; generated stress-test evidence.","solutions":["Split the file: `split -l 1000 evidence.jsonl part-` and import each part","Filter to the records you actually need before exporting","Automate a sequential loop over the parts in CI"],"exampleFix":"# before\ncaveman audit eval-import all-evidence.jsonl   # 5300 records\n# after\nsplit -l 1000 all-evidence.jsonl ev-\nfor f in ev-*; do caveman audit eval-import \"$f\"; done","handlingStrategy":"validation","validationCode":"const count = (await readFile(file, 'utf8')).split(/\\r?\\n/).filter((l) => l.trim()).length;\nif (count > 1000) throw new Error(`${count} records — split into batches of at most 1000`);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Count non-blank lines before importing","Standardize on 1000-record chunks in pipelines","Watch both caps together: 1000 records AND 4 MiB"],"tags":["cli","limits","batching","eval","import"],"backgroundTag":"batch-size-limit-exceeded","analyzedSha":"5184b3d11ac6a1acb7d44b9bfaa31698157cff97","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}