{"record":{"id":"610a2cb8444470a1","repo":"JuliusBrussee/caveman","slug":"caveman-agent-unknown-grader-grader-as-type","errorCode":null,"errorMessage":"caveman agent: unknown grader ${(grader as { type: string }).type}","messagePattern":"caveman agent: unknown grader (.+?)\\)\\.type\\}","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/agent/src/primitives.ts","lineNumber":448,"sourceCode":"  readonly quality: readonly QualityGrader[];\n  readonly guardrails: readonly EvalGuardrail[];\n}\n\nexport function evalFixture(options: {\n  id: string;\n  approved?: boolean;\n  required?: boolean;\n  input: unknown;\n  tools?: { mode: \"fixture\" | \"live\"; sandbox?: string };\n  quality: QualityGrader[];\n  guardrails?: EvalGuardrail[];\n}): EvalDefinition {\n  if (options.id.trim() === \"\") throw new Error(\"caveman agent: eval id is required\");\n  if (options.quality.length === 0) throw new Error(\"caveman agent: eval needs at least one quality grader\");\n  const known = new Set([\"contains\", \"tool_called\", \"exact_match\", \"json_schema\"]);\n  for (const grader of options.quality) {\n    if (!known.has(grader.type)) {\n      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);\n    }\n  }\n  for (const guardrail of options.guardrails ?? []) {\n    if (guardrail.type === \"latency_threshold\" &&\n        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {\n      throw new Error(\"caveman agent: latency guardrail requires positive integer p95_ms\");\n    }\n    if (guardrail.type === \"error_rate\" &&\n        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {\n      throw new Error(\"caveman agent: error-rate guardrail max must be in [0,1]\");\n    }\n  }\n  const tools = options.tools ?? { mode: \"fixture\" as const };\n  if (tools.mode === \"live\" && !tools.sandbox) {\n    throw new Error(\"caveman agent: live eval tools require an explicit sandbox\");\n  }\n  return Object.freeze({\n    kind: \"eval\",","sourceCodeStart":430,"sourceCodeEnd":466,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/agent/src/primitives.ts#L430-L466","documentation":"Thrown by evalFixture() when a QualityGrader's type is not one of the four known kinds: 'contains', 'tool_called', 'exact_match', 'json_schema'. The quality array is checked entry-by-entry at construction so a typo'd or future grader type fails before the eval runs and is attributed to the exact definition.","triggerScenarios":"Passing quality: [{ type: 'contain' }], { type: 'regex' }, { type: 'jsonschema' }, or any grader object whose type string is misspelled or comes from an unvalidated config union.","commonSituations":"Typos in hand-written fixture arrays ('toolCall', 'exactMatch' camelCase from other frameworks); loading graders from YAML/JSON where the type field is free text; version drift where a grader kind was renamed between releases.","solutions":["Use exactly one of: contains (with fragments), tool_called (with tools), exact_match (with expected), json_schema","Constrain grader config to a parsed union at load time (e.g. a zod/TypeBox enum) so bad types fail in the loader with context","Check the error message — it interpolates the offending type, which pinpoints the typo"],"exampleFix":"// before\nevalFixture({ id: 'e1', input: q, quality: [{ type: 'exactMatch', expected: '42' }] });\n\n// after\nevalFixture({ id: 'e1', input: q, quality: [{ type: 'exact_match', expected: '42' }] });","handlingStrategy":"type-guard","validationCode":"const GRADER_TYPES = new Set(['contains', 'tool_called', 'exact_match', 'json_schema']);\nfunction assertGraders(quality: unknown[]): void {\n  for (const g of quality) if (!GRADER_TYPES.has((g as { type?: string }).type ?? '')) throw new Error(`unknown grader type ${(g as { type?: string }).type}`);\n}","typeGuard":"function isQualityGrader(value: unknown): value is { type: 'contains' | 'tool_called' | 'exact_match' | 'json_schema' } { return typeof value === 'object' && value !== null && ['contains','tool_called','exact_match','json_schema'].includes((value as { type?: unknown }).type as string); }","tryCatchPattern":"try { evalFixture(fixture); } catch (e) { if (e instanceof Error && e.message.startsWith('caveman agent: unknown grader')) throw new ConfigError(`fixture '${fixture.id}': ${e.message}`, { cause: e }); throw e; }","preventionTips":["Parse grader config through an enum schema (zod/TypeBox) before building fixtures","Use snake_case grader type literals exactly as documented","Add a compile-time satisfies QualityGrader[] on hand-written arrays"],"tags":["validation","eval","grader","enum"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}