JuliusBrussee/caveman · error · Error

${where} arm ${arm} carries ${count} representative traces,

Error message

${where} arm ${arm} carries ${count} representative traces, more than three

What it means

At most three representative traces per arm: the validator counts traces per arm and throws when an arm carries more than three (validate-continuous-improvement.mjs:165-168). The cap keeps the 'representative' claim honest — four or more cherry-picked traces start to look like a distribution, which the report does not assert.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:168

      armVariants.set(arm.role, variant);
      comparedUnits += arm.unit_count;
    }
    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);
    if (comparedUnits + excluded !== item.cohort.family_unit_count) {
      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);
    }

    const selectedPerArm = new Map();
    for (const trace of item.representative_traces) {
      const variant = armVariants.get(trace.arm);
      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);
      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);
      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);
      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);
      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);
    }
    for (const [arm, count] of selectedPerArm) {
      if (count > 3) throw new Error(`${where} arm ${arm} carries ${count} representative traces, more than three`);
    }

    const slice = item.causal_slice;
    const sliceVariant = variantsByID.get(slice.variant_id);
    if (!sliceVariant) throw new Error(`${where} causal slice references a workflow variant that is not in this report`);
    if (sliceVariant.id !== armVariants.get("baseline").id) throw new Error(`${where} causal slice is not taken from the baseline arm's variant`);
    const sliceNodes = new Set(sliceVariant.nodes.map((node) => node.id));
    if (slice.manifestation_node_id !== "" && !sliceNodes.has(slice.manifestation_node_id)) {
      throw new Error(`${where} manifestation node ${slice.manifestation_node_id} is not a node of variant ${sliceVariant.id}`);
    }
    if (item.diagnosis.manifestation.node_id !== slice.manifestation_node_id) {
      throw new Error(`${where} diagnosis manifestation node disagrees with its causal slice`);
    }
    for (const nodeID of slice.node_chain) {
      if (!sliceNodes.has(nodeID)) throw new Error(`${where} causal slice node ${nodeID} is not a node of variant ${sliceVariant.id}`);
    }
    if (slice.manifestation_node_id !== "" && !slice.node_chain.includes(slice.manifestation_node_id)) {
      throw new Error(`${where} causal slice omits its own manifestation node`);

View on GitHub (pinned to 766dce6b13)

Solutions

  1. Trim each arm to at most three traces (at most six total across baseline and alternative), keeping the most informative ones.
  2. Fix the generator's per-arm selection constant to 3.
  3. When merging cases, re-select traces instead of concatenating, then re-run the validator.

Example fix

// before — five traces with "arm": "baseline"
"representative_traces": [t1, t2, t3, t4, t5]
// after — keep the three most informative baseline traces
"representative_traces": [t1, t2, t3]
Defensive patterns

Strategy: validation

Validate before calling

const overRepresentedArms = (report) =>
  report.cases.flatMap((c) => {
    const per = new Map();
    for (const t of c.representative_traces) per.set(t.arm, (per.get(t.arm) ?? 0) + 1);
    return [...per].filter(([, n]) => n > 3).map(([arm, n]) => `${c.id}:${arm}=${n}`);
  });

Try / catch

try {
  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);
} catch (err) {
  if (/more than three/.test(err.message)) {
    failCI(`too many representative traces for one arm: ${err.message}`);
  } else throw err;
}

Prevention

When it happens

Trigger: A fourth baseline trace is added while hand-enriching a fixture; a generator's selection cap is off by one and samples up to 4 traces per arm; traces duplicated to make a case look better evidenced.

Common situations: Padding fixtures for coverage; tweaking the generator's k-constant without updating the validator's expectation; merging two cases and concatenating their trace lists.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/905f2edbaf240caa. Report an issue: GitHub.