JuliusBrussee/caveman · error · Error

${where} cohort arm ${arm.role} references a workflow varian

Error message

${where} cohort arm ${arm.role} references a workflow variant that is not in this report

What it means

Each cohort arm must name a workflow variant defined in the same report: the validator resolves arm.variant_id against variantsByID (report.workflow_variants) and throws when the lookup misses (validate-continuous-improvement.mjs:146-148). Like the task-family check this enforces report self-containment — the baseline/alternative comparison may only use variants the report itself carries.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:148

  motifCount += report.motifs.length;

  // The causal investigation of every case: the cohort's arms, the traces it
  // selected from them, and the backward hard-dependency slice.
  const unitsByID = new Map(report.analysis_units.map((unit) => [unit.id, unit]));
  const familiesByID = new Map(report.task_families.map((family) => [family.id, family]));
  for (const item of report.cases) {
    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;
    const family = familiesByID.get(item.cohort.task_family_id);
    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);
    const familyUnits = new Set(family.analysis_unit_ids);
    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);
    const roles = item.cohort.arms.map((arm) => arm.role);
    if (roles.join(",") !== "baseline,alternative") throw new Error(`${where} cohort arms are not baseline then alternative`);
    const armVariants = new Map();
    let comparedUnits = 0;
    for (const arm of item.cohort.arms) {
      const variant = variantsByID.get(arm.variant_id);
      if (!variant) throw new Error(`${where} cohort arm ${arm.role} references a workflow variant that is not in this report`);
      if (variant.task_family_id !== family.id) throw new Error(`${where} cohort arm ${arm.role} uses a variant of another task family`);
      armVariants.set(arm.role, variant);
      comparedUnits += arm.unit_count;
    }
    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);
    if (comparedUnits + excluded !== item.cohort.family_unit_count) {
      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);
    }

    const selectedPerArm = new Map();
    for (const trace of item.representative_traces) {
      const variant = armVariants.get(trace.arm);
      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);
      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);
      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);
      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);
      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);
    }

View on GitHub (pinned to 766dce6b13)

Solutions

Set arm.variant_id to an id that exists in this report's workflow_variants.
If the variant object was accidentally removed, restore it (including signature, eligible_runs and analysis_unit_ids).
Keep the case and both its arm variants in the same report file, then re-run the validator.

Example fix

// before — report carries variant "wf-build-01" and "wf-build-02"
{ "role": "baseline", "variant_id": "wf-build-2", "unit_count": 24 }
// after
{ "role": "baseline", "variant_id": "wf-build-01", "unit_count": 24 }

Defensive patterns

Strategy: validation

Validate before calling

const danglingArmVariants = (report) => {
  const ids = new Set(report.workflow_variants.map((v) => v.id));
  return report.cases.flatMap((c) =>
    c.cohort.arms.filter((a) => !ids.has(a.variant_id)).map((a) => `${c.id}:${a.role}`));
};

Try / catch

try {
  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);
} catch (err) {
  if (/cohort arm .* workflow variant that is not/.test(err.message)) {
    failCI(`arm points at a variant outside this report: ${err.message}`);
  } else throw err;
}

Prevention

Keep each case and both its arm variants in one self-contained report file.
On variant renames, sweep cohort arms, motifs, and causal slices in the same commit.
Run the validator as a pre-commit hook so dangling ids never land on main.

When it happens

Trigger: arm.variant_id is a typo or stale id after a rename (wf-build-2 vs wf-build-02); the workflow_variants entry was deleted while the case was kept; the variant exists only in a sibling fixture file.

Common situations: Renaming variant ids in a refactor without sweeping arm references; trimming unused variants from a fixture that still referenced them; merging fixtures where ids collided and were renumbered.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/175ad5de37b4ca17. Report an issue: GitHub.