JuliusBrussee/caveman · error · Error

eval evidence line ${index + 1} is not valid JSON

Error message

eval evidence line ${index + 1} is not valid JSON

What it means

Thrown while parsing the eval-import JSONL: every non-empty line must be a complete, valid JSON document. Line numbers are 1-based over the raw file (blank lines are skipped), so the message pinpoints the exact offending line. Parsing happens client-side before the batch POST.

Source

Thrown at packages/cli/src/index.ts:17155

// auditEvalImport turns newline-delimited, source-neutral eval records into one
// bounded batch. Server stamps tenant scope, observed basis, and external-only
// authority; CI cannot promote its own results into rollout authority.
async function auditEvalImport(argv: string[]) {
  const file = positionalAfterOptions(argv.slice(1), new Set(["--project"]));
  if (!file) throw new Error(`usage: ${invokedCommand("audit")} eval-import <evidence.jsonl> [--project <uuid>] [--dry-run]`);
  const data = await readFile(file);
  if (data.byteLength > 4 * 1024 * 1024) throw new Error("eval evidence file exceeds 4 MiB batch limit");

  const items: Record<string, unknown>[] = [];
  for (const [index, rawLine] of data.toString("utf8").split(/\r?\n/).entries()) {
    const line = rawLine.trim();
    if (!line) continue;
    let parsed: unknown;
    try {
      parsed = JSON.parse(line);
    } catch {
      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);
    }
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);
    }
    items.push(parsed as Record<string, unknown>);
    if (items.length > 1000) throw new Error("eval evidence batch exceeds 1000 records");
  }
  if (items.length === 0) throw new Error("eval evidence file contains no records");

  const cfg = await config();
  const project = flagFrom(argv, "--project", cfg.projectId ?? "");
  const response = await fetch(`${cfg.baseURL}/api/v1/eval-evidence/batches`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${cfg.token}`,
      "content-type": "application/json",
      "x-cave-csrf": "cli",
    },

View on GitHub (pinned to 5184b3d11a)

Solutions

1. Open the file at the reported line and fix or delete the malformed record
2. Re-emit the file programmatically: exactly one JSON.stringify(record) per line
3. Validate locally first: `jq -c . file.jsonl > /dev/null` reports the first bad line
4. If only the last line is truncated, regenerate the export

Example fix

// before (pretty-printed object inside JSONL)
{
  "suite": "eval"
}
// after
{"suite":"eval"}

// Validate every line before handing the file to the CLI.
const bad: number[] = [];
for (const [i, line] of (await readFile(file, 'utf8')).split(/\r?\n/).entries()) {
  const t = line.trim();
  if (!t) continue;
  try { JSON.parse(t); } catch { bad.push(i + 1); }
}
if (bad.length) throw new Error(`invalid JSON on lines: ${bad.join(',')}`);

When it happens

Trigger: A truncated final line from an interrupted write; a pretty-printed object spanning multiple lines; stray commas or BOM characters; artifacts cut mid-record by head/tail; corrupted strings from line-ending conversions.

Common situations: Manually edited evidence files; concatenating outputs mid-record; producers using JSON.stringify with indentation instead of one compact object per line; files transferred through editors that re-encode them.

Related errors

• eval evidence line ${index + 1} must be a JSON object
• usage: ${invokedCommand("audit")} eval-import <evidence.json
• eval evidence file exceeds 4 MiB batch limit
• eval evidence batch exceeds 1000 records
• eval evidence file contains no records

AI-assisted analysis of JuliusBrussee/caveman@5184b3d11a (2026-08-18). Data as JSON: /api/errors/915bfbd0b91960d9. Report an issue: GitHub.