JuliusBrussee/caveman · error · Error

eval evidence file exceeds 4 MiB batch limit

Error message

eval evidence file exceeds 4 MiB batch limit

What it means

Hard batch-size guard inside `caveman audit eval-import`: the evidence file must be at most 4 MiB (4 * 1024 * 1024 bytes). The limit keeps a single import request bounded; larger files are rejected client-side before any line is parsed or any upload starts.

Source

Thrown at packages/cli/src/index.ts:17145

  };
  const fieldMap = flagFrom(argv, "--field-map", "");
  if (fieldMap) headers["x-cave-field-map"] = fieldMap;
  const response = await fetch(`${cfg.baseURL}/api/v1/imports?format=${encodeURIComponent(format)}`, {
    method: "POST",
    headers,
    body: data
  });
  print(await response.json());
}

// auditEvalImport turns newline-delimited, source-neutral eval records into one
// bounded batch. Server stamps tenant scope, observed basis, and external-only
// authority; CI cannot promote its own results into rollout authority.
async function auditEvalImport(argv: string[]) {
  const file = positionalAfterOptions(argv.slice(1), new Set(["--project"]));
  if (!file) throw new Error(`usage: ${invokedCommand("audit")} eval-import <evidence.jsonl> [--project <uuid>] [--dry-run]`);
  const data = await readFile(file);
  if (data.byteLength > 4 * 1024 * 1024) throw new Error("eval evidence file exceeds 4 MiB batch limit");

  const items: Record<string, unknown>[] = [];
  for (const [index, rawLine] of data.toString("utf8").split(/\r?\n/).entries()) {
    const line = rawLine.trim();
    if (!line) continue;
    let parsed: unknown;
    try {
      parsed = JSON.parse(line);
    } catch {
      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);
    }
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);
    }
    items.push(parsed as Record<string, unknown>);
    if (items.length > 1000) throw new Error("eval evidence batch exceeds 1000 records");
  }
  if (items.length === 0) throw new Error("eval evidence file contains no records");

View on GitHub (pinned to 5184b3d11a)

Solutions

1. Split the JSONL into files under 4 MiB and import each (also respect the 1000-record cap)
2. Trim bloated fields from records before export
3. Store references to large content instead of inlining it in evidence records

Example fix

# before
caveman audit eval-import big-evidence.jsonl
# after
split -l 500 big-evidence.jsonl chunk-
for f in chunk-*; do caveman audit eval-import "$f"; done

import { stat } from 'node:fs/promises';
const { size } = await stat(file);
if (size > 4 * 1024 * 1024) throw new Error(`evidence too large (${size} bytes) — split into <4 MiB batches`);

When it happens

Trigger: A JSONL export larger than 4 MiB — long eval runs with many records, or records that inline large payloads such as full prompts and completions.

Common situations: CI artifacts that grow over months until they cross the limit; verbose eval records embedding whole transcripts; concatenating several runs into one file before importing.

Related errors

• eval evidence batch exceeds 1000 records
• usage: ${invokedCommand("audit")} eval-import <evidence.json
• eval evidence file contains no records
• SKILL.md must be 1..262144 bytes
• usage: ${invokedCommand("audit")} import --format <fmt> <fil

AI-assisted analysis of JuliusBrussee/caveman@5184b3d11a (2026-08-18). Data as JSON: /api/errors/389dc6cb36a7a2a1. Report an issue: GitHub.