JuliusBrussee/caveman · error · Error

eval evidence line ${index + 1} must be a JSON object

Error message

eval evidence line ${index + 1} must be a JSON object

What it means

Shape guard for eval-import records: each JSONL line must parse to a JSON object at the top level. Arrays, strings, numbers, booleans, and null are rejected even though they are valid JSON, because the batch endpoint consumes only object-shaped records.

Source

Thrown at packages/cli/src/index.ts:17158

// authority; CI cannot promote its own results into rollout authority.
async function auditEvalImport(argv: string[]) {
  const file = positionalAfterOptions(argv.slice(1), new Set(["--project"]));
  if (!file) throw new Error(`usage: ${invokedCommand("audit")} eval-import <evidence.jsonl> [--project <uuid>] [--dry-run]`);
  const data = await readFile(file);
  if (data.byteLength > 4 * 1024 * 1024) throw new Error("eval evidence file exceeds 4 MiB batch limit");

  const items: Record<string, unknown>[] = [];
  for (const [index, rawLine] of data.toString("utf8").split(/\r?\n/).entries()) {
    const line = rawLine.trim();
    if (!line) continue;
    let parsed: unknown;
    try {
      parsed = JSON.parse(line);
    } catch {
      throw new Error(`eval evidence line ${index + 1} is not valid JSON`);
    }
    if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
      throw new Error(`eval evidence line ${index + 1} must be a JSON object`);
    }
    items.push(parsed as Record<string, unknown>);
    if (items.length > 1000) throw new Error("eval evidence batch exceeds 1000 records");
  }
  if (items.length === 0) throw new Error("eval evidence file contains no records");

  const cfg = await config();
  const project = flagFrom(argv, "--project", cfg.projectId ?? "");
  const response = await fetch(`${cfg.baseURL}/api/v1/eval-evidence/batches`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${cfg.token}`,
      "content-type": "application/json",
      "x-cave-csrf": "cli",
    },
    body: JSON.stringify({
      ...(project ? { project_id: project } : {}),
      dry_run: argv.includes("--dry-run"),

View on GitHub (pinned to 5184b3d11a)

Solutions

1. Fix the reported line by wrapping the value in an object or dropping it
2. Regenerate the export as exactly one object per line
3. Validate shape locally: `jq -c 'select(type != "object")' file.jsonl` lists every non-object line

Example fix

// before
["suite-a", "suite-b"]
// after
{"suite":"suite-a"}
{"suite":"suite-b"}

const allObjects = lines
  .filter((l) => l.trim())
  .every((l) => {
    const v = JSON.parse(l);
    return v !== null && typeof v === 'object' && !Array.isArray(v);
  });
if (!allObjects) throw new Error('evidence contains non-object lines — expected one JSON object per line');

const isRecord = (v: unknown): v is Record<string, unknown> =>
  typeof v === 'object' && v !== null && !Array.isArray(v);

When it happens

Trigger: A line holding a bare array (e.g. a JSON-array export stripped of brackets but leaving array tokens), a quoted string line, a bare number/boolean, or a null serialized by sparse pipelines.

Common situations: Converting a JSON array export to JSONL incorrectly; logging frameworks emitting string lines into the same file; empty rows serialized as null.

Related errors

• eval evidence line ${index + 1} is not valid JSON
• cave_live_eval_sandbox_profile_invalid
• usage: ${invokedCommand("audit")} eval-import <evidence.json
• eval evidence file exceeds 4 MiB batch limit
• eval evidence batch exceeds 1000 records

AI-assisted analysis of JuliusBrussee/caveman@5184b3d11a (2026-08-18). Data as JSON: /api/errors/67efc2220f7b2134. Report an issue: GitHub.