{"record":{"id":"f09ebe8ea59cd8b9","repo":"JuliusBrussee/caveman","slug":"where-compares-two-workflows-without-naming-both-variants","errorCode":null,"errorMessage":"${where} compares two workflows without naming both variants","messagePattern":"(.+?) compares two workflows without naming both variants","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":64,"sourceCode":"      throw new Error(`report fixture ${reportPaths[index]}: theme ${theme.id} inherited a registry id that is not one of its predecessors`);\n    }\n  }\n\n  // An opportunity names the exact pair of workflow variants it was emitted\n  // from, and a safety finding carries no borrowed dollar figure: copying the\n  // efficiency finding's alternative metrics and expected value would let the\n  // same money be counted twice under two detectors.\n  const variantIds = new Set(report.workflow_variants.map((variant) => variant.id));\n  for (const opportunity of report.opportunities) {\n    const where = `report fixture ${reportPaths[index]}: opportunity ${opportunity.id}`;\n    for (const key of [\"current_variant_id\", \"alternative_variant_id\"]) {\n      if (opportunity[key] && !variantIds.has(opportunity[key])) {\n        throw new Error(`${where} ${key} ${opportunity[key]} is not a workflow variant of this report`);\n      }\n    }\n    if (opportunity.detector_id === \"dominated-workflow\") {\n      if (!opportunity.current_variant_id || !opportunity.alternative_variant_id) {\n        throw new Error(`${where} compares two workflows without naming both variants`);\n      }\n    }\n    if (opportunity.type === \"safety\") {\n      if (opportunity.alternative_variant_id) throw new Error(`${where} is a safety finding with an alternative variant`);\n      if (opportunity.expected_value !== 0) throw new Error(`${where} is a safety finding carrying expected value ${opportunity.expected_value}`);\n      if (opportunity.alternative_metrics.cost_per_outcome_usd !== null || opportunity.alternative_metrics.eligible_runs !== 0) {\n        throw new Error(`${where} is a safety finding carrying another workflow's metrics`);\n      }\n    }\n  }\n\n  // A relationship is a count, so it must be recomputable from the counts it\n  // carries. Anything a reader cannot re-derive is a claim, not evidence.\n  const themeIds = new Set(report.themes.map((theme) => theme.id));\n  for (const relationship of report.relationships) {\n    const where = `report fixture ${reportPaths[index]}: relationship ${relationship.id}`;\n    if (!themeIds.has(relationship.theme_a_id) || !themeIds.has(relationship.theme_b_id)) {\n      throw new Error(`${where} references a theme that is not in this report`);","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/3ee70a102609e550bd2e68004bf5990a9341c851/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L46-L82","documentation":"A dominated-workflow opportunity is by definition a comparison between two workflow variants, so the validator requires it to name both current_variant_id and alternative_variant_id. It throws when detector_id is \"dominated-workflow\" but either id is empty/missing. The JSON schema cannot express this per-detector requirement, so the script enforces it after schema validation.","triggerScenarios":"Running the validator on a report containing an opportunity with detector_id === \"dominated-workflow\" where current_variant_id or alternative_variant_id is null, empty string, or absent.","commonSituations":"A detector emitting a dominance finding without recording which variant lost; report generators copying a safety-finding template (which omits the alternative variant) for an efficiency opportunity; partial backfill of a new field on old fixtures.","solutions":["Set both current_variant_id and alternative_variant_id on the opportunity to the two compared variant ids from report.workflow_variants.","If the opportunity genuinely compares no second workflow, change its detector_id to one that matches its shape (e.g. a non-comparison detector).","Regenerate the report from the canonical span fixture if the detector output was truncated.","Re-run the validator to confirm the detector/variant pairing holds."],"exampleFix":"// before\n{ \"id\": \"opp-7\", \"detector_id\": \"dominated-workflow\", \"current_variant_id\": \"variant-a\", \"alternative_variant_id\": \"\" }\n// after\n{ \"id\": \"opp-7\", \"detector_id\": \"dominated-workflow\", \"current_variant_id\": \"variant-a\", \"alternative_variant_id\": \"variant-b\" }","handlingStrategy":"validation","validationCode":"if (report.opportunities.some((o) => o.detector_id === \"dominated-workflow\" && (!o.current_variant_id || !o.alternative_variant_id))) {\n  throw new Error(\"dominated-workflow opportunity missing a compared variant\");\n}","typeGuard":"const isCompleteComparison = (o) => o.detector_id !== \"dominated-workflow\" || (Boolean(o.current_variant_id) && Boolean(o.alternative_variant_id));","tryCatchPattern":"try {\n  await runValidator([reportPath, spansPath]);\n} catch (err) {\n  if (String(err.message).includes(\"compares two workflows without naming both variants\")) {\n    console.error(\"Fill both current_variant_id and alternative_variant_id on the dominated-workflow opportunity.\");\n  }\n  throw err;\n}","preventionTips":["Have the dominated-workflow detector take both variants as required inputs so empties are impossible at the source.","Assert the pair exists when the opportunity object is constructed, not only at validation time.","Do not reuse the safety-finding template for efficiency opportunities."],"tags":["validation","fixtures","opportunities","required-field"],"backgroundTag":"missing-required-argument","analyzedSha":"3ee70a102609e550bd2e68004bf5990a9341c851","analyzedAt":"2026-09-20T15:53:39.229Z","contentChangedAt":"2026-09-20T15:53:39.229Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}