JuliusBrussee/caveman · error · Error

eval evidence batch exceeds 1000 records

Error message

eval evidence batch exceeds 1000 records

What it means

Cardinality guard: an eval-import batch may carry at most 1000 records. The CLI counts object lines while parsing and throws as soon as the 1001st record is seen, keeping the single POST bounded alongside the 4 MiB byte cap — whichever limit trips first.

Source

Thrown at packages/cli/src/index.ts:17161

  if (!file) throw new Error(`usage: ${invokedCommand("audit")} eval-import <evidence.jsonl> [--project <uuid>] [--dry-run]`);
  const data = await readFile(file);
  if (data.byteLength > 4 * 1024 * 1024) throw new Error("eval evidence file exceeds 4 MiB batch limit");

  const items: Record<string, unknown>[] = [];
  for (const [index, rawLine] of data.toString("utf8").split(/\r?\n/).entries()) {
    const line = rawLine.trim();
    if (!line) continue;
    let parsed: unknown;
    try {
      parsed = JSON.parse(line);
    } catch {
      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);
    }
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);
    }
    items.push(parsed as Record<string, unknown>);
    if (items.length > 1000) throw new Error("eval evidence batch exceeds 1000 records");
  }
  if (items.length === 0) throw new Error("eval evidence file contains no records");

  const cfg = await config();
  const project = flagFrom(argv, "--project", cfg.projectId ?? "");
  const response = await fetch(`${cfg.baseURL}/api/v1/eval-evidence/batches`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${cfg.token}`,
      "content-type": "application/json",
      "x-cave-csrf": "cli",
    },
    body: JSON.stringify({
      ...(project ? { project_id: project } : {}),
      dry_run: argv.includes("--dry-run"),
      items,
    }),
  });

View on GitHub (pinned to 5184b3d11a)

Solutions

1. Split the file: `split -l 1000 evidence.jsonl part-` and import each part
2. Filter to the records you actually need before exporting
3. Automate a sequential loop over the parts in CI

Example fix

# before
caveman audit eval-import all-evidence.jsonl   # 5300 records
# after
split -l 1000 all-evidence.jsonl ev-
for f in ev-*; do caveman audit eval-import "$f"; done

const count = (await readFile(file, 'utf8')).split(/\r?\n/).filter((l) => l.trim()).length;
if (count > 1000) throw new Error(`${count} records — split into batches of at most 1000`);

When it happens

Trigger: Any evidence file with more than 1000 non-blank lines; dense files that pass the size check on early lines but keep accumulating records.

Common situations: Bulk historical imports; concatenated multi-day eval logs; generated stress-test evidence.

Related errors

• eval evidence file exceeds 4 MiB batch limit
• usage: ${invokedCommand("audit")} eval-import <evidence.json
• eval evidence file contains no records
• usage: ${invokedCommand("audit")} import --format <fmt> <fil
• eval evidence line ${index + 1} is not valid JSON

AI-assisted analysis of JuliusBrussee/caveman@5184b3d11a (2026-08-18). Data as JSON: /api/errors/11700db8e68fe79d. Report an issue: GitHub.