JuliusBrussee/caveman · error · Error

${where} cohort arms (${armVariants.get("baseline").id}/${ar

Error message

${where} cohort arms (${armVariants.get("baseline").id}/${armVariants.get("alternative").id}) are not the variants opportunity ${source.id} named

What it means

Thrown when the case's cohort arms (baseline/alternative variant ids) do not equal the current_variant_id / alternative_variant_id recorded on the opportunity the case cites. The case must measure exactly the pair its opportunity named; otherwise the report describes one comparison and measures another.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:231

    // A case that proposes nothing must carry nothing: no operator, no eval
    // pack, no replay. Anything else would present an abstention as a proposal.
    if (item.status === "diagnosis_only") {
      if (item.change_set.operator_id !== "" || item.eval_pack.id !== "" || item.replay.status !== "not_run") {
        throw new Error(`${where} is diagnosis-only but carries a change set, eval pack or replay`);
      }
      continue;
    }
    if (item.change_set.case_id !== item.id || item.eval_pack.case_id !== item.id) {
      throw new Error(`${where} change set or eval pack is bound to another case`);
    }

    // The case must investigate the pair its own opportunity named. A cohort
    // built from a differently-derived pair would describe one comparison and
    // measure another.
    const source = report.opportunities.find((opportunity) => opportunity.id === item.opportunity_id);
    if (!source) throw new Error(`${where} names an opportunity that is not in this report`);
    if (armVariants.get("baseline").id !== source.current_variant_id || armVariants.get("alternative").id !== source.alternative_variant_id) {
      throw new Error(`${where} cohort arms (${armVariants.get("baseline").id}/${armVariants.get("alternative").id}) are not the variants opportunity ${source.id} named`);
    }

    // Every required grader is one the recorded interpreter actually computes.
    // `json_schema` is deliberately absent: over a typed interpreter result it
    // could only assert that a hash is non-empty, which is a grader that cannot
    // fail and therefore inflates how much a replay was checked.
    const knownGraders = new Set(["exact_match", "tool_order", "tool_count", "guard_respected"]);
    for (const grader of item.eval_pack.graders) {
      if (!knownGraders.has(grader)) throw new Error(`${where} requires grader ${grader}, which the recorded interpreter does not compute`);
    }

    // Every dataset case — recorded or generated — must name a real analysis
    // unit of this case's own task family, and its generator and perturbation
    // must agree. A generated case whose source unit cannot be resolved would be
    // invented evidence wearing an adversarial label.
    const dataset = item.eval_pack.dataset;
    const generatorPerturbation = new Map([
      ["recorded_case.v1", "none"],

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Align the cohort arms: arms[0].variant_id must equal source.current_variant_id and arms[1].variant_id must equal source.alternative_variant_id (with roles baseline, alternative in that order).
2. If the cohort is correct, fix the opportunity's current_variant_id/alternative_variant_id instead — but update every case citing that opportunity.
3. Regenerate cohorts from the opportunity list so arms always derive from the named pair.

Example fix

// before
opportunity: {current_variant_id: "wf-1.2", alternative_variant_id: "wf-1.3"}
cohort.arms: [{role:"baseline",variant_id:"wf-1.0"},{role:"alternative",variant_id:"wf-1.3"}]

// after
cohort.arms: [{role:"baseline",variant_id:"wf-1.2"},{role:"alternative",variant_id:"wf-1.3"}]

const opp = report.opportunities.find((o) => o.id === item.opportunity_id);
const aligned = opp !== undefined &&
  item.cohort.arms[0].variant_id === opp.current_variant_id &&
  item.cohort.arms[1].variant_id === opp.alternative_variant_id;

When it happens

Trigger: Editing item.cohort.arms to point at different variants after the case was created; updating the opportunity's variant pair without rebuilding the case's cohort; assembling a cohort from variants of the wrong derivation.

Common situations: Re-baselining an opportunity to a new variant; copy-pasting a cohort between cases that investigate different variant pairs.

Related errors

• ${where} names an opportunity that is not in this report
• option not found
• cave_harness_adapter_version_invalid
• cave_harness_model_invalid
• cave_harness_model_identity_missing

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/cfde9f22cfafa705. Report an issue: GitHub.