{"record":{"id":"905f2edbaf240caa","repo":"JuliusBrussee/caveman","slug":"where-arm-arm-carries-count-representativ","errorCode":null,"errorMessage":"${where} arm ${arm} carries ${count} representative traces, more than three","messagePattern":"(.+?) arm (.+?) carries (.+?) representative traces, more than three","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":168,"sourceCode":"      armVariants.set(arm.role, variant);\n      comparedUnits += arm.unit_count;\n    }\n    const excluded = item.cohort.excluded_units.reduce((total, exclusion) => total + exclusion.unit_count, 0);\n    if (comparedUnits + excluded !== item.cohort.family_unit_count) {\n      throw new Error(`${where} cohort arms (${comparedUnits}) plus exclusions (${excluded}) do not account for its ${item.cohort.family_unit_count} family units`);\n    }\n\n    const selectedPerArm = new Map();\n    for (const trace of item.representative_traces) {\n      const variant = armVariants.get(trace.arm);\n      if (!variant || variant.id !== trace.variant_id) throw new Error(`${where} representative trace names a variant that is not that arm's variant`);\n      if (!unitsByID.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not an analysis unit of this report`);\n      if (!familyUnits.has(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} is not a member of the cohort's task family`);\n      if (!variant.analysis_unit_ids.includes(trace.analysis_unit_id)) throw new Error(`${where} representative trace ${trace.analysis_unit_id} did not run that arm's workflow variant`);\n      selectedPerArm.set(trace.arm, (selectedPerArm.get(trace.arm) ?? 0) + 1);\n    }\n    for (const [arm, count] of selectedPerArm) {\n      if (count > 3) throw new Error(`${where} arm ${arm} carries ${count} representative traces, more than three`);\n    }\n\n    const slice = item.causal_slice;\n    const sliceVariant = variantsByID.get(slice.variant_id);\n    if (!sliceVariant) throw new Error(`${where} causal slice references a workflow variant that is not in this report`);\n    if (sliceVariant.id !== armVariants.get(\"baseline\").id) throw new Error(`${where} causal slice is not taken from the baseline arm's variant`);\n    const sliceNodes = new Set(sliceVariant.nodes.map((node) => node.id));\n    if (slice.manifestation_node_id !== \"\" && !sliceNodes.has(slice.manifestation_node_id)) {\n      throw new Error(`${where} manifestation node ${slice.manifestation_node_id} is not a node of variant ${sliceVariant.id}`);\n    }\n    if (item.diagnosis.manifestation.node_id !== slice.manifestation_node_id) {\n      throw new Error(`${where} diagnosis manifestation node disagrees with its causal slice`);\n    }\n    for (const nodeID of slice.node_chain) {\n      if (!sliceNodes.has(nodeID)) throw new Error(`${where} causal slice node ${nodeID} is not a node of variant ${sliceVariant.id}`);\n    }\n    if (slice.manifestation_node_id !== \"\" && !slice.node_chain.includes(slice.manifestation_node_id)) {\n      throw new Error(`${where} causal slice omits its own manifestation node`);","sourceCodeStart":150,"sourceCodeEnd":186,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/766dce6b1394ebb56a3090748d5a0240a5aefb36/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L150-L186","documentation":"At most three representative traces per arm: the validator counts traces per arm and throws when an arm carries more than three (validate-continuous-improvement.mjs:165-168). The cap keeps the 'representative' claim honest — four or more cherry-picked traces start to look like a distribution, which the report does not assert.","triggerScenarios":"A fourth baseline trace is added while hand-enriching a fixture; a generator's selection cap is off by one and samples up to 4 traces per arm; traces duplicated to make a case look better evidenced.","commonSituations":"Padding fixtures for coverage; tweaking the generator's k-constant without updating the validator's expectation; merging two cases and concatenating their trace lists.","solutions":["Trim each arm to at most three traces (at most six total across baseline and alternative), keeping the most informative ones.","Fix the generator's per-arm selection constant to 3.","When merging cases, re-select traces instead of concatenating, then re-run the validator."],"exampleFix":"// before — five traces with \"arm\": \"baseline\"\n\"representative_traces\": [t1, t2, t3, t4, t5]\n// after — keep the three most informative baseline traces\n\"representative_traces\": [t1, t2, t3]","handlingStrategy":"validation","validationCode":"const overRepresentedArms = (report) =>\n  report.cases.flatMap((c) => {\n    const per = new Map();\n    for (const t of c.representative_traces) per.set(t.arm, (per.get(t.arm) ?? 0) + 1);\n    return [...per].filter(([, n]) => n > 3).map(([arm, n]) => `${c.id}:${arm}=${n}`);\n  });","typeGuard":null,"tryCatchPattern":"try {\n  execFileSync(process.execPath, [VALIDATOR, reportPath, spansPath]);\n} catch (err) {\n  if (/more than three/.test(err.message)) {\n    failCI(`too many representative traces for one arm: ${err.message}`);\n  } else throw err;\n}","preventionTips":["Cap per-arm trace selection at three in the generator, matching the validator.","When merging cases, re-select traces rather than concatenating lists.","Treat a fourth trace as a signal the case should be split, not padded."],"tags":["fixtures","cross-validation","representative-trace","limits"],"backgroundTag":"count-limit-exceeded","analyzedSha":"766dce6b1394ebb56a3090748d5a0240a5aefb36","analyzedAt":"2026-08-18T03:14:35.516Z","contentChangedAt":"2026-08-18T03:14:35.516Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}