JuliusBrussee/caveman · error · Error

${where} support run count ${motif.support_run_count} != ${r

Error message

${where} support run count ${motif.support_run_count} != ${runs}

What it means

Thrown by the continuous-improvement report conformance validator (packages/shared/contracts/scripts/validate-continuous-improvement.mjs:124) when a motif's declared support_run_count does not equal the sum of eligible_runs over the workflow variants listed in that motif's variant_ids. A motif is a structural count over the report's own variants, so every number it carries must be re-derivable from them; the validator recomputes the sum and rejects the fixture on any mismatch. The message prints both the stored value and the recomputed sum so the drift is immediately visible.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:124

  const variantsByID = new Map(report.workflow_variants.map((variant) => [variant.id, variant]));
  const familyIDs = new Set(report.task_families.map((family) => family.id));
  for (const motif of report.motifs) {
    const where = `report fixture ${reportPaths[index]}: motif ${motif.id}`;
    if (!familyIDs.has(motif.task_family_id)) throw new Error(`${where} references a task family that is not in this report`);
    if (motif.support_variant_count !== motif.variant_ids.length) throw new Error(`${where} support variant count disagrees with its variant ids`);
    let runs = 0;
    let weighted = 0;
    for (const variantID of motif.variant_ids) {
      const variant = variantsByID.get(variantID);
      if (!variant) throw new Error(`${where} references workflow variant ${variantID} that is not in this report`);
      if (variant.task_family_id !== motif.task_family_id) throw new Error(`${where} supporting variant ${variantID} belongs to another task family`);
      if (!contiguouslyContains(variant.signature, motif.operations)) {
        throw new Error(`${where} operations are not a contiguous subsequence of variant ${variantID}'s signature`);
      }
      runs += variant.eligible_runs;
      weighted += (motif.operations.length / variant.signature.length) * variant.eligible_runs;
    }
    if (motif.support_run_count !== runs) throw new Error(`${where} support run count ${motif.support_run_count} != ${runs}`);
    const expectedShare = runs > 0 ? weighted / runs : 0;
    if (Math.abs(motif.structural_cost_share - expectedShare) > 1e-6) {
      throw new Error(`${where} structural_cost_share ${motif.structural_cost_share} != ${expectedShare}`);
    }
  }
  motifCount += report.motifs.length;

  // The causal investigation of every case: the cohort's arms, the traces it
  // selected from them, and the backward hard-dependency slice.
  const unitsByID = new Map(report.analysis_units.map((unit) => [unit.id, unit]));
  const familiesByID = new Map(report.task_families.map((family) => [family.id, family]));
  for (const item of report.cases) {
    const where = `report fixture ${reportPaths[index]}: case ${item.id}`;
    const family = familiesByID.get(item.cohort.task_family_id);
    if (!family) throw new Error(`${where} cohort references a task family that is not in this report`);
    const familyUnits = new Set(family.analysis_unit_ids);
    if (item.cohort.family_unit_count !== familyUnits.size) throw new Error(`${where} cohort family unit count disagrees with the task family`);
    const roles = item.cohort.arms.map((arm) => arm.role);

View on GitHub (pinned to 766dce6b13)

Solutions

  1. Recompute support_run_count as the exact sum of eligible_runs over every variant id in motif.variant_ids and write that value into the fixture.
  2. If the recomputed sum looks wrong, verify variant_ids lists exactly the supporting variants (each must contain the motif operations as a contiguous subsequence of its signature) — a stale entry silently changes the sum.
  3. If a generator produced the fixture, fix it to derive the count at emit time (after eligible_runs is final) and add a unit test that recomputes it.
  4. Re-run the validator: node packages/shared/contracts/scripts/validate-continuous-improvement.mjs <reportPath> <spansPath>

Example fix

// before — variant_ids: [wf-a (eligible_runs 24), wf-b (eligible_runs 31)]
{
  "id": "motif-retry-loop",
  "variant_ids": ["wf-a", "wf-b"],
  "support_run_count": 40
}
// after — 24 + 31 = 55
{
  "id": "motif-retry-loop",
  "variant_ids": ["wf-a", "wf-b"],
  "support_run_count": 55
}
Defensive patterns

Strategy: validation

Validate before calling

const motifRunDrift = (report) => {
  const byID = new Map(report.workflow_variants.map((v) => [v.id, v]));
  return report.motifs.filter((m) =>
    m.support_run_count !== m.variant_ids.reduce((n, id) => n + (byID.get(id)?.eligible_runs ?? 0), 0));
};

Try / catch

try {
  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);
} catch (err) {
  if (/support run count/.test(err.message)) {
    // err.message carries fixture path, motif id, stored and expected sums
    failCI(`derived count drift — regenerate fixture: ${err.message}`);
  } else throw err;
}

Prevention

When it happens

Trigger: A report fixture passes the ajv schema check but motif.variant_ids names variants whose eligible_runs sum to something other than motif.support_run_count — e.g. one variant's eligible_runs was edited from 24 to 31 without touching the motif, or a variant id was added to or removed from variant_ids without re-summing. Produced by running: node packages/shared/contracts/scripts/validate-continuous-improvement.mjs <report.json> <spans.json>.

Common situations: Hand-editing generated fixtures to tweak numbers; a report generator that computes support_run_count from a snapshot taken before eligible_runs was finalized; rebasing fixtures after a schema or generator change; copying a motif block from another fixture that has different variant ids.

Related errors

AI-assisted analysis of JuliusBrussee/caveman@766dce6b13 (2026-08-18). Data as JSON: /api/errors/681906fa727565ef. Report an issue: GitHub.