JuliusBrussee/caveman · error · Error

compares two workflows without naming both variants

Error message

${where} compares two workflows without naming both variants

What it means

A dominated-workflow opportunity is by definition a comparison between two workflow variants, so the validator requires it to name both current_variant_id and alternative_variant_id. It throws when detector_id is "dominated-workflow" but either id is empty/missing. The JSON schema cannot express this per-detector requirement, so the script enforces it after schema validation.

Solutions

1. Set both current_variant_id and alternative_variant_id on the opportunity to the two compared variant ids from report.workflow_variants.
2. If the opportunity genuinely compares no second workflow, change its detector_id to one that matches its shape (e.g. a non-comparison detector).
3. Regenerate the report from the canonical span fixture if the detector output was truncated.
4. Re-run the validator to confirm the detector/variant pairing holds.

Example fix

// before
{ "id": "opp-7", "detector_id": "dominated-workflow", "current_variant_id": "variant-a", "alternative_variant_id": "" }
// after
{ "id": "opp-7", "detector_id": "dominated-workflow", "current_variant_id": "variant-a", "alternative_variant_id": "variant-b" }

if (report.opportunities.some((o) => o.detector_id === "dominated-workflow" && (!o.current_variant_id || !o.alternative_variant_id))) {
  throw new Error("dominated-workflow opportunity missing a compared variant");
}

const isCompleteComparison = (o) => o.detector_id !== "dominated-workflow" || (Boolean(o.current_variant_id) && Boolean(o.alternative_variant_id));

try {
  await runValidator([reportPath, spansPath]);
} catch (err) {
  if (String(err.message).includes("compares two workflows without naming both variants")) {
    console.error("Fill both current_variant_id and alternative_variant_id on the dominated-workflow opportunity.");
  }
  throw err;
}

When it happens

Trigger: Running the validator on a report containing an opportunity with detector_id === "dominated-workflow" where current_variant_id or alternative_variant_id is null, empty string, or absent.

Common situations: A detector emitting a dominance finding without recording which variant lost; report generators copying a safety-finding template (which omits the alternative variant) for an efficiency opportunity; partial backfill of a new field on old fixtures.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• is a safety finding carrying another workflow's metrics
• is a safety finding carrying expected value
• is a safety finding with an alternative variant
• is not a workflow variant of this report
• canonical span

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/f09ebe8ea59cd8b9. Report an issue: GitHub.

Appendix: source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:64

      throw new Error(`report fixture ${reportPaths[index]}: theme ${theme.id} inherited a registry id that is not one of its predecessors`);
    }
  }

  // An opportunity names the exact pair of workflow variants it was emitted
  // from, and a safety finding carries no borrowed dollar figure: copying the
  // efficiency finding's alternative metrics and expected value would let the
  // same money be counted twice under two detectors.
  const variantIds = new Set(report.workflow_variants.map((variant) => variant.id));
  for (const opportunity of report.opportunities) {
    const where = `report fixture ${reportPaths[index]}: opportunity ${opportunity.id}`;
    for (const key of ["current_variant_id", "alternative_variant_id"]) {
      if (opportunity[key] && !variantIds.has(opportunity[key])) {
        throw new Error(`${where} ${key} ${opportunity[key]} is not a workflow variant of this report`);
      }
    }
    if (opportunity.detector_id === "dominated-workflow") {
      if (!opportunity.current_variant_id || !opportunity.alternative_variant_id) {
        throw new Error(`${where} compares two workflows without naming both variants`);
      }
    }
    if (opportunity.type === "safety") {
      if (opportunity.alternative_variant_id) throw new Error(`${where} is a safety finding with an alternative variant`);
      if (opportunity.expected_value !== 0) throw new Error(`${where} is a safety finding carrying expected value ${opportunity.expected_value}`);
      if (opportunity.alternative_metrics.cost_per_outcome_usd !== null || opportunity.alternative_metrics.eligible_runs !== 0) {
        throw new Error(`${where} is a safety finding carrying another workflow's metrics`);
      }
    }
  }

  // A relationship is a count, so it must be recomputable from the counts it
  // carries. Anything a reader cannot re-derive is a claim, not evidence.
  const themeIds = new Set(report.themes.map((theme) => theme.id));
  for (const relationship of report.relationships) {
    const where = `report fixture ${reportPaths[index]}: relationship ${relationship.id}`;
    if (!themeIds.has(relationship.theme_a_id) || !themeIds.has(relationship.theme_b_id)) {
      throw new Error(`${where} references a theme that is not in this report`);

View on GitHub (pinned to 3ee70a1026)