JuliusBrussee/caveman · error · Error

eval evidence file contains no records

Error message

eval evidence file contains no records

What it means

Emptiness guard: after skipping blank lines, the evidence file yielded zero records, so there is nothing to POST. Distinct from the parse and shape errors — the file parsed fine, it simply contains no records. Usually a symptom that the producing eval run failed or the path points at a placeholder.

Source

Thrown at packages/cli/src/index.ts:17163

  if (data.byteLength > 4 * 1024 * 1024) throw new Error("eval evidence file exceeds 4 MiB batch limit");

  const items: Record<string, unknown>[] = [];
  for (const [index, rawLine] of data.toString("utf8").split(/\r?\n/).entries()) {
    const line = rawLine.trim();
    if (!line) continue;
    let parsed: unknown;
    try {
      parsed = JSON.parse(line);
    } catch {
      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);
    }
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);
    }
    items.push(parsed as Record<string, unknown>);
    if (items.length > 1000) throw new Error("eval evidence batch exceeds 1000 records");
  }
  if (items.length === 0) throw new Error("eval evidence file contains no records");

  const cfg = await config();
  const project = flagFrom(argv, "--project", cfg.projectId ?? "");
  const response = await fetch(`${cfg.baseURL}/api/v1/eval-evidence/batches`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${cfg.token}`,
      "content-type": "application/json",
      "x-cave-csrf": "cli",
    },
    body: JSON.stringify({
      ...(project ? { project_id: project } : {}),
      dry_run: argv.includes("--dry-run"),
      items,
    }),
  });
  const body = await response.json();
  if (!response.ok) throw new Error(`eval evidence import failed (${response.status}): ${JSON.stringify(body)}`);

View on GitHub (pinned to 5184b3d11a)

Solutions

1. Check the pipeline that produced the file — an empty export usually means the eval run itself failed
2. Point the command at the real artifact path
3. Skip the import step when the file has no records instead of failing the job

Example fix

# before
caveman audit eval-import empty.jsonl
# after
[ -s evidence.jsonl ] && caveman audit eval-import evidence.jsonl || echo 'no records — skipping'

import { stat } from 'node:fs/promises';
const { size } = await stat(file);
if (size === 0) throw new Error('evidence file is empty — the producing step likely failed');

When it happens

Trigger: A zero-byte file; a file containing only newlines/whitespace; a wrong path pointing at an accidentally created empty file; a generator whose filter removed every row.

Common situations: CI creating the artifact with touch before a failed produce step; glob patterns matching an empty placeholder; upstream eval run writing no rows.

Related errors

• usage: ${invokedCommand("audit")} eval-import <evidence.json
• eval evidence file exceeds 4 MiB batch limit
• eval evidence batch exceeds 1000 records
• usage: ${invokedCommand("audit")} import --format <fmt> <fil
• eval evidence line ${index + 1} is not valid JSON

AI-assisted analysis of JuliusBrussee/caveman@5184b3d11a (2026-08-18). Data as JSON: /api/errors/20c7068c29d8a96d. Report an issue: GitHub.