{"record":{"id":"b710720d51fcfe74","repo":"JuliusBrussee/caveman","slug":"where-cohort-arm-arm-role-uses-a-variant-of","errorCode":null,"errorMessage":"${where} cohort arm ${arm.role} uses a variant of another task family","messagePattern":"(.+?) cohort arm (.+?) uses a variant of another task family","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":149,"sourceCode":"\n  // The causal investigation of every case: the cohort's arms, the traces it\n  // selected from them, and the backward hard-dependency slice.\n  const unitsByID = new Map(report.analysis_units.map((unit) => [unit.id, unit]));\n  const familiesByID = new Map(report.task_families.map((family) => [family.id, family]));\n  for (const item of report.cases) {\n    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;\n    const family = familiesByID.get(item.cohort.task_family_id);\n    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);\n    const familyUnits = new Set(family.analysis_unit_ids);\n    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);\n    const roles = item.cohort.arms.map((arm) => arm.role);\n    if (roles.join(\",\") !== \"baseline,alternative\") throw new Error(`${where} cohort arms are not baseline then alternative`);\n    const armVariants = new Map();\n    let comparedUnits = 0;\n    for (const arm of item.cohort.arms) {\n      const variant = variantsByID.get(arm.variant_id);\n      if (!variant) throw new Error(`${where} cohort arm ${arm.role} references a workflow variant that is not in this report`);\n      if (variant.task_family_id !== family.id) throw new Error(`${where} cohort arm ${arm.role} uses a variant of another task family`);\n      armVariants.set(arm.role, variant);\n      comparedUnits += arm.unit_count;\n    }\n    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);\n    if (comparedUnits + excluded !== item.cohort.family_unit_count) {\n      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);\n    }\n\n    const selectedPerArm = new Map();\n    for (const trace of item.representative_traces) {\n      const variant = armVariants.get(trace.arm);\n      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);\n      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);\n      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);\n      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);\n      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);\n    }\n    for (const [arm, count] of selectedPerArm) {","sourceCodeStart":131,"sourceCodeEnd":167,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L131-L167","documentation":"Beyond merely existing, both arms' variants must belong to the cohort's own task family: variant.task_family_id must equal the referenced family's id (validate-continuous-improvement.mjs:149). A cohort that compares variants from two different families would describe one experiment and measure another, so the validator rejects cross-family arms even though both ids resolve.","triggerScenarios":"The alternative arm is pointed at a variant whose task_family_id is another family (wf-review-fast while the cohort is family-codegen); a variant's task_family_id was edited in a refactor; fixtures with parallel families and a copy-paste slip between them.","commonSituations":"Fixtures that model several task families side by side; re-assigning a variant to a new family without updating the cohorts that used it; hand-picking a 'similar' variant from another family to fill an arm.","solutions":["Point the arm at a variant whose task_family_id equals the cohort's task family id.","If the variant was mislabeled, correct its task_family_id — then re-check every other cohort using that variant.","Grep the fixture for the family id to confirm which variants legitimately belong to it, then re-run the validator."],"exampleFix":"// before — cohort family is family-codegen, but the variant belongs to family-review\n{ \"role\": \"alternative\", \"variant_id\": \"wf-review-fast\", \"unit_count\": 25 }\n// after\n{ \"role\": \"alternative\", \"variant_id\": \"wf-build-02\", \"unit_count\": 25 }","handlingStrategy":"validation","validationCode":"const crossFamilyArms = (report) => {\n  const byID = new Map(report.workflow_variants.map((v) => [v.id, v]));\n  return report.cases.flatMap((c) => {\n    const family = report.task_families.find((f) => f.id === c.cohort.task_family_id);\n    return c.cohort.arms\n      .filter((a) => byID.get(a.variant_id)?.task_family_id !== family?.id)\n      .map((a) => `${c.id}:${a.role}`);\n  });\n};","typeGuard":null,"tryCatchPattern":"try {\n  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);\n} catch (err) {\n  if (/variant of another task family/.test(err.message)) {\n    failCI(`cross-family comparison — repoint the arm: ${err.message}`);\n  } else throw err;\n}","preventionTips":["Pick arm variants from the cohort family's own variant list, never by similarity.","When re-assigning a variant to a new family, grep cohorts that reference it.","Name variants with their family (wf-build-*) so cross-family slips are visible on review."],"tags":["fixtures","cross-validation","cohort","task-family","referential-integrity"],"backgroundTag":"wrong-owner-reference","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}