{"record":{"id":"4394fe4d6a3c6598","repo":"JuliusBrussee/caveman","slug":"where-requires-grader-grader-which-the-reco","errorCode":null,"errorMessage":"${where} requires grader ${grader}, which the recorded interpreter does not compute","messagePattern":"(.+?) requires grader (.+?), which the recorded interpreter does not compute","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":240,"sourceCode":"      throw new Error(`${where} change set or eval pack is bound to another case`);\n    }\n\n    // The case must investigate the pair its own opportunity named. A cohort\n    // built from a differently-derived pair would describe one comparison and\n    // measure another.\n    const source = report.opportunities.find((opportunity) => opportunity.id === item.opportunity_id);\n    if (!source) throw new Error(`${where} names an opportunity that is not in this report`);\n    if (armVariants.get(\"baseline\").id !== source.current_variant_id || armVariants.get(\"alternative\").id !== source.alternative_variant_id) {\n      throw new Error(`${where} cohort arms (${armVariants.get(\"baseline\").id}/${armVariants.get(\"alternative\").id}) are not the variants opportunity ${source.id} named`);\n    }\n\n    // Every required grader is one the recorded interpreter actually computes.\n    // `json_schema` is deliberately absent: over a typed interpreter result it\n    // could only assert that a hash is non-empty, which is a grader that cannot\n    // fail and therefore inflates how much a replay was checked.\n    const knownGraders = new Set([\"exact_match\", \"tool_order\", \"tool_count\", \"guard_respected\"]);\n    for (const grader of item.eval_pack.graders) {\n      if (!knownGraders.has(grader)) throw new Error(`${where} requires grader ${grader}, which the recorded interpreter does not compute`);\n    }\n\n    // Every dataset case — recorded or generated — must name a real analysis\n    // unit of this case's own task family, and its generator and perturbation\n    // must agree. A generated case whose source unit cannot be resolved would be\n    // invented evidence wearing an adversarial label.\n    const dataset = item.eval_pack.dataset;\n    const generatorPerturbation = new Map([\n      [\"recorded_case.v1\", \"none\"],\n      [\"boundary_guard_threshold.v1\", \"guard_threshold_boundary\"],\n      [\"adversarial_misleading_tool_output.v1\", \"misleading_tool_output\"],\n      [\"adversarial_stale_input.v1\", \"stale_input\"],\n    ]);\n    const roleCounts = { target_failure: 0, prior_success: 0, boundary: 0, adversarial: 0 };\n    const recordedSources = new Set();\n    const datasetCaseIDs = new Set();\n    for (const entry of dataset.cases) {\n      const at = `${where} dataset case ${entry.id}`;","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L222-L258","documentation":"Thrown when eval_pack.graders names a grader outside the set the recorded interpreter actually computes: {\"exact_match\", \"tool_order\", \"tool_count\", \"guard_respected\"}. Requiring a grader that is never computed would silently count as coverage that does not exist (json_schema is deliberately excluded for this reason).","triggerScenarios":"Adding a grader name like \"json_schema\", \"bleu\" or a typo (\"exact-match\") to eval_pack.graders; upgrading the interpreter to drop a grader without updating the packs that require it.","commonSituations":"See trigger scenarios.","solutions":["Remove the unknown grader from item.eval_pack.graders, or replace it with one of exact_match, tool_order, tool_count, guard_respected.","Check the exact spelling and separator (underscores, not dashes).","If a new grader is genuinely needed, extend knownGraders in the validator only after the interpreter actually computes it."],"exampleFix":"// before\ngraders: [\"exact_match\", \"json_schema\"]\n\n// after\ngraders: [\"exact_match\", \"guard_respected\"]","handlingStrategy":"type-guard","validationCode":"const KNOWN_GRADERS = new Set([\"exact_match\", \"tool_order\", \"tool_count\", \"guard_respected\"]);\nconst gradersOk = item.eval_pack.graders.every((g) => KNOWN_GRADERS.has(g));","typeGuard":"const GRADER_NAMES = [\"exact_match\", \"tool_order\", \"tool_count\", \"guard_respected\"];\nfunction isGraderName(g) { return GRADER_NAMES.includes(g); }\n// typed composer: graders: GraderName[]","tryCatchPattern":null,"preventionTips":["Type eval_pack.graders as a closed union of the four grader names in your authoring code so invalid ones cannot be typed.","Keep the interpreter's computed-grader list and the validator's set in one shared constant."],"tags":["validation","eval-pack","graders"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}