{"record":{"id":"c544fb45d63b5e2b","repo":"JuliusBrussee/caveman","slug":"caveman-agent-unknown-grader-type","errorCode":null,"errorMessage":"caveman agent: unknown grader ${type}","messagePattern":"caveman agent: unknown grader (.+?)","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/agent/src/primitives.ts","lineNumber":448,"sourceCode":"  readonly quality: readonly QualityGrader[];\n  readonly guardrails: readonly EvalGuardrail[];\n}\n\nexport function evalFixture(options: {\n  id: string;\n  approved?: boolean;\n  required?: boolean;\n  input: unknown;\n  tools?: { mode: \"fixture\" | \"live\"; sandbox?: string };\n  quality: QualityGrader[];\n  guardrails?: EvalGuardrail[];\n}): EvalDefinition {\n  if (options.id.trim() === \"\") throw new Error(\"caveman agent: eval id is required\");\n  if (options.quality.length === 0) throw new Error(\"caveman agent: eval needs at least one quality grader\");\n  const known = new Set([\"contains\", \"tool_called\", \"exact_match\", \"json_schema\"]);\n  for (const grader of options.quality) {\n    if (!known.has(grader.type)) {\n      throw new Error(`caveman agent: unknown grader ${(grader as { type: string }).type}`);\n    }\n  }\n  for (const guardrail of options.guardrails ?? []) {\n    if (guardrail.type === \"latency_threshold\" &&\n        (!Number.isSafeInteger(guardrail.p95_ms) || guardrail.p95_ms <= 0)) {\n      throw new Error(\"caveman agent: latency guardrail requires positive integer p95_ms\");\n    }\n    if (guardrail.type === \"error_rate\" &&\n        (!Number.isFinite(guardrail.max) || guardrail.max < 0 || guardrail.max > 1)) {\n      throw new Error(\"caveman agent: error-rate guardrail max must be in [0,1]\");\n    }\n  }\n  const tools = options.tools ?? { mode: \"fixture\" as const };\n  if (tools.mode === \"live\" && !tools.sandbox) {\n    throw new Error(\"caveman agent: live eval tools require an explicit sandbox\");\n  }\n  return Object.freeze({\n    kind: \"eval\",","sourceCodeStart":430,"sourceCodeEnd":466,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/agent/src/primitives.ts#L430-L466","documentation":"evalFixture() validates each quality grader's type against the known set: contains, tool_called, exact_match, and json_schema. It throws this templated error (with the offending type interpolated) when a grader's type is not one of those four strings. This catches typos and graders from unsupported versions at definition time.","triggerScenarios":"Passing quality: [{ type: \"semantic_similarity\", ... }] or any type string outside the known set, including typos like \"contians\" or \"exact-match\" (hyphen instead of underscore).","commonSituations":"Misspelling a grader type; copying a grader name from another testing library; upgrading/downgrading the library so a previously valid (or future) grader type is not in the current known set.","solutions":["Use one of the supported types exactly: contains, tool_called, exact_match, or json_schema.","Fix typos and casing — e.g. exact_match not exactMatch or exact-match.","Check the installed library version's supported grader list if a type used to work."],"exampleFix":"// before\nquality: [{ type: \"semantic\", value: \"x\" }]\n// after\nquality: [{ type: \"contains\", value: \"x\" }]","handlingStrategy":"validation","validationCode":"const KNOWN = new Set([\"contains\", \"tool_called\", \"exact_match\", \"json_schema\"]);\nconst bad = fixture.quality.filter(g => !KNOWN.has(g.type));\nif (bad.length) throw new Error(`unknown grader types: ${bad.map(g => g.type).join(\", \")}`);","typeGuard":"function isKnownGrader(g: { type: string }): boolean { return [\"contains\",\"tool_called\",\"exact_match\",\"json_schema\"].includes(g.type); }","tryCatchPattern":"try { ev = evalFixture(opts); }\ncatch (e) { const m = /unknown grader (\\S+)/.exec(String(e.message)); if (m) throw new Error(`Fix grader type \"${m[1]}\" — allowed: contains, tool_called, exact_match, json_schema`); throw e; }","preventionTips":["Copy grader type strings from the library's exported constants, not from memory.","Use underscore forms: exact_match, tool_called.","Re-check grader names after upgrading the library version."],"tags":["validation","eval","enum","unknown-value"],"backgroundTag":"invalid-enum-value","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}