What causes "${where} cohort arms are not baseline then alternative" in JuliusBrussee/caveman?

Reorder the arms so the baseline arm is first and the alternative arm is second.

JuliusBrussee/caveman · error · Error

${where} cohort arms are not baseline then alternative

Error message

${where} cohort arms are not baseline then alternative

What it means

A case cohort must carry exactly two arms ordered baseline first, alternative second — the comparison direction that every downstream check (causal slice on baseline, opportunity pairing) depends on. The validator joins the arm roles and compares the result against the literal 'baseline,alternative' (validate-continuous-improvement.mjs:142-143), so a swapped pair, a missing arm, an extra arm, or a role typo all fail with this one message.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:143

    const expectedShare = runs > 0 ? weighted / runs : 0;
    if (Math.abs(motif.structural_cost_share - expectedShare) > 1e-6) {
      throw new Error(`${where} structural_cost_share ${motif.structural_cost_share} != ${expectedShare}`);
    }
  }
  motifCount += report.motifs.length;

  // The causal investigation of every case: the cohort's arms, the traces it
  // selected from them, and the backward hard-dependency slice.
  const unitsByID = new Map(report.analysis_units.map((unit) => [unit.id, unit]));
  const familiesByID = new Map(report.task_families.map((family) => [family.id, family]));
  for (const item of report.cases) {
    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;
    const family = familiesByID.get(item.cohort.task_family_id);
    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);
    const familyUnits = new Set(family.analysis_unit_ids);
    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);
    const roles = item.cohort.arms.map((arm) => arm.role);
    if (roles.join(",") !== "baseline,alternative") throw new Error(`${where} cohort arms are not baseline then alternative`);
    const armVariants = new Map();
    let comparedUnits = 0;
    for (const arm of item.cohort.arms) {
      const variant = variantsByID.get(arm.variant_id);
      if (!variant) throw new Error(`${where} cohort arm ${arm.role} references a workflow variant that is not in this report`);
      if (variant.task_family_id !== family.id) throw new Error(`${where} cohort arm ${arm.role} uses a variant of another task family`);
      armVariants.set(arm.role, variant);
      comparedUnits += arm.unit_count;
    }
    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);
    if (comparedUnits + excluded !== item.cohort.family_unit_count) {
      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);
    }

    const selectedPerArm = new Map();
    for (const trace of item.representative_traces) {
      const variant = armVariants.get(trace.arm);
      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);

View on GitHub (pinned to 766dce6b13)

Solutions

Reorder the arms so the baseline arm is first and the alternative arm is second.
Ensure there are exactly two arms and their role fields are exactly "baseline" and "alternative".
Re-run the validator — arm-order errors also cascade into error 614, so fix this first.

Example fix

// before
"arms": [
  { "role": "alternative", "variant_id": "wf-build-02", "unit_count": 25 },
  { "role": "baseline", "variant_id": "wf-build-01", "unit_count": 24 }
]
// after
"arms": [
  { "role": "baseline", "variant_id": "wf-build-01", "unit_count": 24 },
  { "role": "alternative", "variant_id": "wf-build-02", "unit_count": 25 }
]

Defensive patterns

Strategy: validation

Validate before calling

const armSequenceOK = (report) =>
  report.cases.every((c) => c.cohort.arms.map((a) => a.role).join(',') === 'baseline,alternative');

Try / catch

try {
  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);
} catch (err) {
  if (/not baseline then alternative/.test(err.message)) {
    failCI(`arm roles/order wrong — exactly [baseline, alternative]: ${err.message}`);
  } else throw err;
}

Prevention

Treat arm order as part of the fixture's contract, not presentation.
When cloning a case, re-check the arms array before anything else — many later checks depend on it.
Avoid inventing role names; only baseline and alternative are accepted.

When it happens

Trigger: arms array ordered [alternative, baseline]; a third arm appended; a role spelled 'control'/'treatment'/'base'; a single-arm cohort. The JSON schema does not pin the sequence, so these shapes reach the validator and are rejected here.

Common situations: Hand-authoring cohorts with intuitive role names; reordering arms while polishing a fixture; porting a fixture from an older format that allowed more arms.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/91d5fab3e7d79be1. Report an issue: GitHub.