{"record":{"id":"cfde9f22cfafa705","repo":"JuliusBrussee/caveman","slug":"where-cohort-arms-armvariants-get-baseline","errorCode":null,"errorMessage":"${where} cohort arms (${armVariants.get(\"baseline\").id}/${armVariants.get(\"alternative\").id}) are not the variants opportunity ${source.id} named","messagePattern":"(.+?) cohort arms \\((.+?)/(.+?)\\) are not the variants opportunity (.+?) named","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":231,"sourceCode":"    // A case that proposes nothing must carry nothing: no operator, no eval\n    // pack, no replay. Anything else would present an abstention as a proposal.\n    if (item.status === \"diagnosis_only\") {\n      if (item.change_set.operator_id !== \"\" || item.eval_pack.id !== \"\" || item.replay.status !== \"not_run\") {\n        throw new Error(`${where} is diagnosis-only but carries a change set, eval pack or replay`);\n      }\n      continue;\n    }\n    if (item.change_set.case_id !== item.id || item.eval_pack.case_id !== item.id) {\n      throw new Error(`${where} change set or eval pack is bound to another case`);\n    }\n\n    // The case must investigate the pair its own opportunity named. A cohort\n    // built from a differently-derived pair would describe one comparison and\n    // measure another.\n    const source = report.opportunities.find((opportunity) => opportunity.id === item.opportunity_id);\n    if (!source) throw new Error(`${where} names an opportunity that is not in this report`);\n    if (armVariants.get(\"baseline\").id !== source.current_variant_id || armVariants.get(\"alternative\").id !== source.alternative_variant_id) {\n      throw new Error(`${where} cohort arms (${armVariants.get(\"baseline\").id}/${armVariants.get(\"alternative\").id}) are not the variants opportunity ${source.id} named`);\n    }\n\n    // Every required grader is one the recorded interpreter actually computes.\n    // `json_schema` is deliberately absent: over a typed interpreter result it\n    // could only assert that a hash is non-empty, which is a grader that cannot\n    // fail and therefore inflates how much a replay was checked.\n    const knownGraders = new Set([\"exact_match\", \"tool_order\", \"tool_count\", \"guard_respected\"]);\n    for (const grader of item.eval_pack.graders) {\n      if (!knownGraders.has(grader)) throw new Error(`${where} requires grader ${grader}, which the recorded interpreter does not compute`);\n    }\n\n    // Every dataset case — recorded or generated — must name a real analysis\n    // unit of this case's own task family, and its generator and perturbation\n    // must agree. A generated case whose source unit cannot be resolved would be\n    // invented evidence wearing an adversarial label.\n    const dataset = item.eval_pack.dataset;\n    const generatorPerturbation = new Map([\n      [\"recorded_case.v1\", \"none\"],","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L213-L249","documentation":"Thrown when the case's cohort arms (baseline/alternative variant ids) do not equal the current_variant_id / alternative_variant_id recorded on the opportunity the case cites. The case must measure exactly the pair its opportunity named; otherwise the report describes one comparison and measures another.","triggerScenarios":"Editing item.cohort.arms to point at different variants after the case was created; updating the opportunity's variant pair without rebuilding the case's cohort; assembling a cohort from variants of the wrong derivation.","commonSituations":"Re-baselining an opportunity to a new variant; copy-pasting a cohort between cases that investigate different variant pairs.","solutions":["Align the cohort arms: arms[0].variant_id must equal source.current_variant_id and arms[1].variant_id must equal source.alternative_variant_id (with roles baseline, alternative in that order).","If the cohort is correct, fix the opportunity's current_variant_id/alternative_variant_id instead — but update every case citing that opportunity.","Regenerate cohorts from the opportunity list so arms always derive from the named pair."],"exampleFix":"// before\nopportunity: {current_variant_id: \"wf-1.2\", alternative_variant_id: \"wf-1.3\"}\ncohort.arms: [{role:\"baseline\",variant_id:\"wf-1.0\"},{role:\"alternative\",variant_id:\"wf-1.3\"}]\n\n// after\ncohort.arms: [{role:\"baseline\",variant_id:\"wf-1.2\"},{role:\"alternative\",variant_id:\"wf-1.3\"}]","handlingStrategy":"validation","validationCode":"const opp = report.opportunities.find((o) => o.id === item.opportunity_id);\nconst aligned = opp !== undefined &&\n  item.cohort.arms[0].variant_id === opp.current_variant_id &&\n  item.cohort.arms[1].variant_id === opp.alternative_variant_id;","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Build each case's cohort from its opportunity's variant pair at creation time, not by hand.","When re-baselining an opportunity, regenerate every case that cites it in the same change."],"tags":["validation","cohort","report-consistency"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}