JuliusBrussee/caveman · error · Error

${where} cohort arm ${arm.role} uses a variant of another ta

Error message

${where} cohort arm ${arm.role} uses a variant of another task family

What it means

Beyond merely existing, both arms' variants must belong to the cohort's own task family: variant.task_family_id must equal the referenced family's id (validate-continuous-improvement.mjs:149). A cohort that compares variants from two different families would describe one experiment and measure another, so the validator rejects cross-family arms even though both ids resolve.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:149

  // The causal investigation of every case: the cohort's arms, the traces it
  // selected from them, and the backward hard-dependency slice.
  const unitsByID = new Map(report.analysis_units.map((unit) => [unit.id, unit]));
  const familiesByID = new Map(report.task_families.map((family) => [family.id, family]));
  for (const item of report.cases) {
    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;
    const family = familiesByID.get(item.cohort.task_family_id);
    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);
    const familyUnits = new Set(family.analysis_unit_ids);
    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);
    const roles = item.cohort.arms.map((arm) => arm.role);
    if (roles.join(",") !== "baseline,alternative") throw new Error(`${where} cohort arms are not baseline then alternative`);
    const armVariants = new Map();
    let comparedUnits = 0;
    for (const arm of item.cohort.arms) {
      const variant = variantsByID.get(arm.variant_id);
      if (!variant) throw new Error(`${where} cohort arm ${arm.role} references a workflow variant that is not in this report`);
      if (variant.task_family_id !== family.id) throw new Error(`${where} cohort arm ${arm.role} uses a variant of another task family`);
      armVariants.set(arm.role, variant);
      comparedUnits += arm.unit_count;
    }
    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);
    if (comparedUnits + excluded !== item.cohort.family_unit_count) {
      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);
    }

    const selectedPerArm = new Map();
    for (const trace of item.representative_traces) {
      const variant = armVariants.get(trace.arm);
      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);
      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);
      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);
      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);
      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);
    }
    for (const [arm, count] of selectedPerArm) {

View on GitHub (pinned to 766dce6b13)

Solutions

Point the arm at a variant whose task_family_id equals the cohort's task family id.
If the variant was mislabeled, correct its task_family_id — then re-check every other cohort using that variant.
Grep the fixture for the family id to confirm which variants legitimately belong to it, then re-run the validator.

Example fix

// before — cohort family is family-codegen, but the variant belongs to family-review
{ "role": "alternative", "variant_id": "wf-review-fast", "unit_count": 25 }
// after
{ "role": "alternative", "variant_id": "wf-build-02", "unit_count": 25 }

Defensive patterns

Strategy: validation

Validate before calling

const crossFamilyArms = (report) => {
  const byID = new Map(report.workflow_variants.map((v) => [v.id, v]));
  return report.cases.flatMap((c) => {
    const family = report.task_families.find((f) => f.id === c.cohort.task_family_id);
    return c.cohort.arms
      .filter((a) => byID.get(a.variant_id)?.task_family_id !== family?.id)
      .map((a) => `${c.id}:${a.role}`);
  });
};

Try / catch

try {
  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);
} catch (err) {
  if (/variant of another task family/.test(err.message)) {
    failCI(`cross-family comparison — repoint the arm: ${err.message}`);
  } else throw err;
}

Prevention

Pick arm variants from the cohort family's own variant list, never by similarity.
When re-assigning a variant to a new family, grep cohorts that reference it.
Name variants with their family (wf-build-*) so cross-family slips are visible on review.

When it happens

Trigger: The alternative arm is pointed at a variant whose task_family_id is another family (wf-review-fast while the cohort is family-codegen); a variant's task_family_id was edited in a refactor; fixtures with parallel families and a copy-paste slip between them.

Common situations: Fixtures that model several task families side by side; re-assigning a variant to a new family without updating the cohorts that used it; hand-picking a 'similar' variant from another family to fill an arm.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/b710720d51fcfe74. Report an issue: GitHub.