JuliusBrussee/caveman · error

caveman agent: unknown grader ${(grader as { type: string })

Error message

caveman agent: unknown grader ${(grader as { type: string }).type}

What it means

Thrown by evalFixture() when a QualityGrader's type is not one of the four known kinds: 'contains', 'tool_called', 'exact_match', 'json_schema'. The quality array is checked entry-by-entry at construction so a typo'd or future grader type fails before the eval runs and is attributed to the exact definition.

Source

Thrown at packages/agent/src/primitives.ts:448

  readonly quality: readonly QualityGrader[];
  readonly guardrails: readonly EvalGuardrail[];
}

export function evalFixture(options: {
  id: string;
  approved?: boolean;
  required?: boolean;
  input: unknown;
  tools?: { mode: "fixture" | "live"; sandbox?: string };
  quality: QualityGrader[];
  guardrails?: EvalGuardrail[];
}): EvalDefinition {
  if (options.id.trim() === "") throw new Error("caveman agent: eval id is required");
  if (options.quality.length === 0) throw new Error("caveman agent: eval needs at least one quality grader");
  const known = new Set(["contains", "tool_called", "exact_match", "json_schema"]);
  for (const grader of options.quality) {
    if (!known.has(grader.type)) {
      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);
    }
  }
  for (const guardrail of options.guardrails ?? []) {
    if (guardrail.type === "latency_threshold" &&
        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {
      throw new Error("caveman agent: latency guardrail requires positive integer p95_ms");
    }
    if (guardrail.type === "error_rate" &&
        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {
      throw new Error("caveman agent: error-rate guardrail max must be in [0,1]");
    }
  }
  const tools = options.tools ?? { mode: "fixture" as const };
  if (tools.mode === "live" && !tools.sandbox) {
    throw new Error("caveman agent: live eval tools require an explicit sandbox");
  }
  return Object.freeze({
    kind: "eval",

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Use exactly one of: contains (with fragments), tool_called (with tools), exact_match (with expected), json_schema
2. Constrain grader config to a parsed union at load time (e.g. a zod/TypeBox enum) so bad types fail in the loader with context
3. Check the error message — it interpolates the offending type, which pinpoints the typo

Example fix

// before
evalFixture({ id: 'e1', input: q, quality: [{ type: 'exactMatch', expected: '42' }] });

// after
evalFixture({ id: 'e1', input: q, quality: [{ type: 'exact_match', expected: '42' }] });

const GRADER_TYPES = new Set(['contains', 'tool_called', 'exact_match', 'json_schema']);
function assertGraders(quality: unknown[]): void {
  for (const g of quality) if (!GRADER_TYPES.has((g as { type?: string }).type ?? '')) throw new Error(`unknown grader type ${(g as { type?: string }).type}`);
}

function isQualityGrader(value: unknown): value is { type: 'contains' | 'tool_called' | 'exact_match' | 'json_schema' } { return typeof value === 'object' && value !== null && ['contains','tool_called','exact_match','json_schema'].includes((value as { type?: unknown }).type as string); }

try { evalFixture(fixture); } catch (e) { if (e instanceof Error && e.message.startsWith('caveman agent: unknown grader')) throw new ConfigError(`fixture '${fixture.id}': ${e.message}`, { cause: e }); throw e; }

When it happens

Trigger: Passing quality: [{ type: 'contain' }], { type: 'regex' }, { type: 'jsonschema' }, or any grader object whose type string is misspelled or comes from an unvalidated config union.

Common situations: Typos in hand-written fixture arrays ('toolCall', 'exactMatch' camelCase from other frameworks); loading graders from YAML/JSON where the type field is free text; version drift where a grader kind was renamed between releases.

Related errors

• cave_budget_on_exhausted_invalid
• cave_live_eval_sandbox_profile_missing
• cave_live_eval_sandbox_profile_invalid
• caveman agent: eval id is required
• caveman agent: latency guardrail requires positive integer p

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/610a2cb8444470a1. Report an issue: GitHub.