{"record":{"id":"9c3e5b786a4b965e","repo":"JuliusBrussee/caveman","slug":"where-cohort-arms-comparedunits-plus-exclu","errorCode":null,"errorMessage":"${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units","messagePattern":"(.+?) cohort arms \\((.+?)\\) plus exclusions \\((.+?)\\) do not account for its (.+?) family units","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":155,"sourceCode":"    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;\n    const family = familiesByID.get(item.cohort.task_family_id);\n    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);\n    const familyUnits = new Set(family.analysis_unit_ids);\n    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);\n    const roles = item.cohort.arms.map((arm) => arm.role);\n    if (roles.join(\",\") !== \"baseline,alternative\") throw new Error(`${where} cohort arms are not baseline then alternative`);\n    const armVariants = new Map();\n    let comparedUnits = 0;\n    for (const arm of item.cohort.arms) {\n      const variant = variantsByID.get(arm.variant_id);\n      if (!variant) throw new Error(`${where} cohort arm ${arm.role} references a workflow variant that is not in this report`);\n      if (variant.task_family_id !== family.id) throw new Error(`${where} cohort arm ${arm.role} uses a variant of another task family`);\n      armVariants.set(arm.role, variant);\n      comparedUnits += arm.unit_count;\n    }\n    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);\n    if (comparedUnits + excluded !== item.cohort.family_unit_count) {\n      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);\n    }\n\n    const selectedPerArm = new Map();\n    for (const trace of item.representative_traces) {\n      const variant = armVariants.get(trace.arm);\n      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);\n      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);\n      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);\n      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);\n      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);\n    }\n    for (const [arm, count] of selectedPerArm) {\n      if (count > 3) throw new Error(`${where} arm ${arm} carries ${count} representative traces, more than three`);\n    }\n\n    const slice = item.causal_slice;\n    const sliceVariant = variantsByID.get(slice.variant_id);\n    if (!sliceVariant) throw new Error(`${where} causal slice references a workflow variant that is not in this report`);","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L137-L173","documentation":"The cohort must be a complete partition of its family's units: the sum of both arms' unit_count plus the sum of excluded_units[].unit_count must equal cohort.family_unit_count (validate-continuous-improvement.mjs:151-156). Exclusions are how the cohort accounts for units that ran neither arm's variant; any slack or double counting fails here.","triggerScenarios":"An exclusion entry was added after the family grew, without adjusting either side; arm.unit_count is stale after units moved between arms; a unit is counted both in an arm and in excluded_units. Note family_unit_count itself is checked first (error 603), so it agrees with the family's membership list.","commonSituations":"Editing a fixture to exclude noisy units by hand; generators that compute arm counts and exclusion counts in separate passes; appending new units to the family but only bumping one side of the arithmetic.","solutions":["Verify family_unit_count against the family's unique unit ids first (see error 603), then adjust the arm unit_counts or the excluded_units entries so comparedUnits + excluded equals it.","Make sure no unit is counted twice — an excluded unit must not appear in an arm's unit_count.","Prefer regenerating the fixture from the report generator so the partition is recomputed atomically."],"exampleFix":"// before — arms 24 + 25, exclusions 3, but the family has 52 units\n\"family_unit_count\": 50\n// after — 24 + 25 + 3 = 52\n\"family_unit_count\": 52","handlingStrategy":"validation","validationCode":"const partitionDrift = (report) =>\n  report.cases.filter((c) => {\n    const compared = c.cohort.arms.reduce((n, a) => n + a.unit_count, 0);\n    const excluded = c.cohort.excluded_units.reduce((n, e) => n + e.unit_count, 0);\n    return compared + excluded !== c.cohort.family_unit_count;\n  }).map((c) => c.id);","typeGuard":null,"tryCatchPattern":"try {\n  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);\n} catch (err) {\n  if (/do not account for/.test(err.message)) {\n    failCI(`cohort partition broken — arms + exclusions must equal family units: ${err.message}`);\n  } else throw err;\n}","preventionTips":["Fix family_unit_count (error 603) first, then balance arms and exclusions against it.","Never count a unit in both an arm and the exclusion list.","Regenerate the fixture instead of patching one side of the arithmetic by hand."],"tags":["fixtures","cross-validation","cohort","accounting","arithmetic"],"backgroundTag":"conservation-check-failed","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}