What causes "caveman agent: eval id is required" in JuliusBrussee/caveman?

Pass a non-empty, non-whitespace id string.

JuliusBrussee/caveman · error · Error

caveman agent: eval id is required

Error message

caveman agent: eval id is required

What it means

evalFixture() requires a non-empty id: it throws when options.id.trim() === "" (empty string or whitespace-only) or when id is missing/undefined. The id identifies the eval fixture in results and approvals, so the library refuses to build an EvalDefinition without one.

Solutions

Pass a non-empty, non-whitespace id string.
Trim and validate the id at config load time before constructing the fixture.
If ids are generated, add a fallback like a slugified name or a counter to guarantee uniqueness and non-emptiness.

Example fix

// before
evalFixture({ id: "", input: "q", quality: [exactMatch()] });
// after
evalFixture({ id: "qa-basic-1", input: "q", quality: [exactMatch()] });

Defensive patterns

Strategy: validation

Validate before calling

function hasId(id) { return typeof id === "string" && id.trim().length > 0; }
if (!hasId(fixture.id)) throw new Error(`eval fixture id required, got ${JSON.stringify(fixture.id)}`);

Type guard

function hasNonEmptyId(o: { id?: string }): o is { id: string } { return typeof o.id === "string" && o.id.trim().length > 0; }

Try / catch

let ev;
try { ev = evalFixture(opts); }
catch (e) { if (String(e.message).includes("eval id is required")) ev = evalFixture({ ...opts, id: opts.id?.trim() || generatedId() }); else throw e; }

Prevention

Validate fixture configs (id non-empty) when loading from YAML/JSON.
Trim ids sourced from user input or spreadsheets.
Auto-generate deterministic ids when a source row lacks one.

When it happens

Trigger: evalFixture({ id: "", ... }), evalFixture({ id: " ", ... }), or omitting id altogether while supplying the required input and quality fields.

Common situations: Generating ids from a variable that is empty or whitespace-padded; building fixtures programmatically from table rows where the id column is blank; YAML/JSON config with id: "".

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/7404a237132c231e. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/primitives.ts:443

  readonly id: string;
  readonly approved: boolean;
  readonly required: boolean;
  readonly input: unknown;
  readonly tools: { mode: "fixture" | "live"; sandbox?: string };
  readonly quality: readonly QualityGrader[];
  readonly guardrails: readonly EvalGuardrail[];
}

export function evalFixture(options: {
  id: string;
  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };

View on GitHub (pinned to 3ee70a1026)