JuliusBrussee/caveman · error · Error

caveman agent: eval needs at least one quality grader

Error message

caveman agent: eval needs at least one quality grader

What it means

evalFixture requires at least one quality grader in options.quality; an empty grader list fails this guard right after the id check. Evals must declare how output quality will be judged.

Source

Thrown at packages/agent/src/primitives.ts:444

  readonly approved: boolean;
  readonly required: boolean;
  readonly input: unknown;
  readonly tools: { mode: "fixture" | "live"; sandbox?: string };
  readonly quality: readonly QualityGrader[];
  readonly guardrails: readonly EvalGuardrail[];
}

export function evalFixture(options: {
  id: string;
  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {

View on GitHub (pinned to 766dce6b13)

Solutions

1. Add at least one grader (contains, tool_called, exact_match, or json_schema) to quality
2. Seed a trivial 'contains' grader while authoring, then refine
3. Skip empty eval entries when generating fixtures from data

When it happens

Trigger: Thrown at packages/agent/src/primitives.ts:444 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/8d8de547520595b8. Report an issue: GitHub.