JuliusBrussee/caveman · error · Error

caveman agent: unknown grader

Error message

caveman agent: unknown grader ${type}

What it means

evalFixture() validates each quality grader's type against the known set: contains, tool_called, exact_match, and json_schema. It throws this templated error (with the offending type interpolated) when a grader's type is not one of those four strings. This catches typos and graders from unsupported versions at definition time.

Solutions

1. Use one of the supported types exactly: contains, tool_called, exact_match, or json_schema.
2. Fix typos and casing — e.g. exact_match not exactMatch or exact-match.
3. Check the installed library version's supported grader list if a type used to work.

Example fix

// before
quality: [{ type: "semantic", value: "x" }]
// after
quality: [{ type: "contains", value: "x" }]

const KNOWN = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
const bad = fixture.quality.filter(g => !KNOWN.has(g.type));
if (bad.length) throw new Error(`unknown grader types: ${bad.map(g => g.type).join(", ")}`);

function isKnownGrader(g: { type: string }): boolean { return ["contains","tool_called","exact_match","json_schema"].includes(g.type); }

try { ev = evalFixture(opts); }
catch (e) { const m = /unknown grader (\S+)/.exec(String(e.message)); if (m) throw new Error(`Fix grader type "${m[1]}" — allowed: contains, tool_called, exact_match, json_schema`); throw e; }

When it happens

Trigger: Passing quality: [{ type: "semantic_similarity", ... }] or any type string outside the known set, including typos like "contians" or "exact-match" (hyphen instead of underscore).

Common situations: Misspelling a grader type; copying a grader name from another testing library; upgrading/downgrading the library so a previously valid (or future) grader type is not in the current known set.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• assembly slot has unknown stability
• assembly slot has unknown stability
• bedrock endpoint must be runtime or mantle
• caveman agent: error-rate guardrail max must be in [0,1]
• caveman agent: eval id is required

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/c544fb45d63b5e2b. Report an issue: GitHub.

Appendix: source

Thrown at packages/agent/src/primitives.ts:448

  readonly quality: readonly QualityGrader[];
  readonly guardrails: readonly EvalGuardrail[];
}

export function evalFixture(options: {
  id: string;
  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",

View on GitHub (pinned to 3ee70a1026)