JuliusBrussee/caveman · error · Error

${where} requires grader ${grader}, which the recorded inter

Error message

${where} requires grader ${grader}, which the recorded interpreter does not compute

What it means

Thrown when eval_pack.graders names a grader outside the set the recorded interpreter actually computes: {"exact_match", "tool_order", "tool_count", "guard_respected"}. Requiring a grader that is never computed would silently count as coverage that does not exist (json_schema is deliberately excluded for this reason).

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:240

      throw new Error(`${where} change set or eval pack is bound to another case`);
    }

    // The case must investigate the pair its own opportunity named. A cohort
    // built from a differently-derived pair would describe one comparison and
    // measure another.
    const source = report.opportunities.find((opportunity) => opportunity.id === item.opportunity_id);
    if (!source) throw new Error(`${where} names an opportunity that is not in this report`);
    if (armVariants.get("baseline").id !== source.current_variant_id || armVariants.get("alternative").id !== source.alternative_variant_id) {
      throw new Error(`${where} cohort arms (${armVariants.get("baseline").id}/${armVariants.get("alternative").id}) are not the variants opportunity ${source.id} named`);
    }

    // Every required grader is one the recorded interpreter actually computes.
    // `json_schema` is deliberately absent: over a typed interpreter result it
    // could only assert that a hash is non-empty, which is a grader that cannot
    // fail and therefore inflates how much a replay was checked.
    const knownGraders = new Set(["exact_match", "tool_order", "tool_count", "guard_respected"]);
    for (const grader of item.eval_pack.graders) {
      if (!knownGraders.has(grader)) throw new Error(`${where} requires grader ${grader}, which the recorded interpreter does not compute`);
    }

    // Every dataset case — recorded or generated — must name a real analysis
    // unit of this case's own task family, and its generator and perturbation
    // must agree. A generated case whose source unit cannot be resolved would be
    // invented evidence wearing an adversarial label.
    const dataset = item.eval_pack.dataset;
    const generatorPerturbation = new Map([
      ["recorded_case.v1", "none"],
      ["boundary_guard_threshold.v1", "guard_threshold_boundary"],
      ["adversarial_misleading_tool_output.v1", "misleading_tool_output"],
      ["adversarial_stale_input.v1", "stale_input"],
    ]);
    const roleCounts = { target_failure: 0, prior_success: 0, boundary: 0, adversarial: 0 };
    const recordedSources = new Set();
    const datasetCaseIDs = new Set();
    for (const entry of dataset.cases) {
      const at = `${where} dataset case ${entry.id}`;

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Remove the unknown grader from item.eval_pack.graders, or replace it with one of exact_match, tool_order, tool_count, guard_respected.
2. Check the exact spelling and separator (underscores, not dashes).
3. If a new grader is genuinely needed, extend knownGraders in the validator only after the interpreter actually computes it.

Example fix

// before
graders: ["exact_match", "json_schema"]

// after
graders: ["exact_match", "guard_respected"]

const KNOWN_GRADERS = new Set(["exact_match", "tool_order", "tool_count", "guard_respected"]);
const gradersOk = item.eval_pack.graders.every((g) => KNOWN_GRADERS.has(g));

const GRADER_NAMES = ["exact_match", "tool_order", "tool_count", "guard_respected"];
function isGraderName(g) { return GRADER_NAMES.includes(g); }
// typed composer: graders: GraderName[]

When it happens

Trigger: Adding a grader name like "json_schema", "bleu" or a typo ("exact-match") to eval_pack.graders; upgrading the interpreter to drop a grader without updating the packs that require it.

Common situations: See trigger scenarios.

Related errors

• option not found
• cave_harness_adapter_version_invalid
• cave_harness_model_invalid
• cave_harness_model_identity_missing
• cave_harness_wire_contract_invalid

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/4394fe4d6a3c6598. Report an issue: GitHub.