JuliusBrussee/caveman · error · Error

is a safety finding carrying another workflow's metrics

Error message

${where} is a safety finding carrying another workflow's metrics

What it means

A safety finding must not carry another workflow's alternative metrics: when type === "safety", the validator requires alternative_metrics.cost_per_outcome_usd === null and alternative_metrics.eligible_runs === 0, and throws if either is populated. Borrowed cost-per-outcome figures would double-count money already claimed by an efficiency detector, so hazard reports must leave the alternative-metrics block empty.

Solutions

1. Set alternative_metrics.cost_per_outcome_usd to null and alternative_metrics.eligible_runs to 0 on the safety opportunity.
2. Fix the report generator/detector so alternative_metrics is zeroed for safety findings.
3. If the metrics are real, re-emit the finding as an efficiency opportunity rather than safety.
4. Re-run the validator to confirm safety findings carry no borrowed metrics.

Example fix

// before
{ "id": "opp-3", "type": "safety", "alternative_metrics": { "cost_per_outcome_usd": 0.42, "eligible_runs": 17 } }
// after
{ "id": "opp-3", "type": "safety", "alternative_metrics": { "cost_per_outcome_usd": null, "eligible_runs": 0 } }

if (report.opportunities.some((o) => o.type === "safety" && (o.alternative_metrics.cost_per_outcome_usd !== null || o.alternative_metrics.eligible_runs !== 0))) {
  throw new Error("safety finding carries alternative metrics");
}

const hasEmptyAlternativeMetrics = (o) => o.type !== "safety" || (o.alternative_metrics.cost_per_outcome_usd === null && o.alternative_metrics.eligible_runs === 0);

try {
  await runValidator([reportPath, spansPath]);
} catch (err) {
  if (String(err.message).includes("carrying another workflow's metrics")) {
    console.error("Zero/null alternative_metrics on safety findings to avoid double-counted savings.");
  }
  throw err;
}

When it happens

Trigger: Running the validator on a report where a type === "safety" opportunity has alternative_metrics.cost_per_outcome_usd set to a number or alternative_metrics.eligible_runs set to a non-zero count.

Common situations: Generator code default-initializing alternative_metrics with real numbers for every opportunity; a detector converted from efficiency to safety while reusing its metric block; copy-pasted fixtures between opportunity types.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• compares two workflows without naming both variants
• is a safety finding carrying expected value
• is a safety finding with an alternative variant
• is not a workflow variant of this report
• canonical span

AI-assisted analysis of JuliusBrussee/caveman@3ee70a1026 (2026-09-20). Data as JSON: /api/errors/869edb57dfee55b0. Report an issue: GitHub.

Appendix: source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:71

  // same money be counted twice under two detectors.
  const variantIds = new Set(report.workflow_variants.map((variant) => variant.id));
  for (const opportunity of report.opportunities) {
    const where = `report fixture ${reportPaths[index]}: opportunity ${opportunity.id}`;
    for (const key of ["current_variant_id", "alternative_variant_id"]) {
      if (opportunity[key] && !variantIds.has(opportunity[key])) {
        throw new Error(`${where} ${key} ${opportunity[key]} is not a workflow variant of this report`);
      }
    }
    if (opportunity.detector_id === "dominated-workflow") {
      if (!opportunity.current_variant_id || !opportunity.alternative_variant_id) {
        throw new Error(`${where} compares two workflows without naming both variants`);
      }
    }
    if (opportunity.type === "safety") {
      if (opportunity.alternative_variant_id) throw new Error(`${where} is a safety finding with an alternative variant`);
      if (opportunity.expected_value !== 0) throw new Error(`${where} is a safety finding carrying expected value ${opportunity.expected_value}`);
      if (opportunity.alternative_metrics.cost_per_outcome_usd !== null || opportunity.alternative_metrics.eligible_runs !== 0) {
        throw new Error(`${where} is a safety finding carrying another workflow's metrics`);
      }
    }
  }

  // A relationship is a count, so it must be recomputable from the counts it
  // carries. Anything a reader cannot re-derive is a claim, not evidence.
  const themeIds = new Set(report.themes.map((theme) => theme.id));
  for (const relationship of report.relationships) {
    const where = `report fixture ${reportPaths[index]}: relationship ${relationship.id}`;
    if (!themeIds.has(relationship.theme_a_id) || !themeIds.has(relationship.theme_b_id)) {
      throw new Error(`${where} references a theme that is not in this report`);
    }
    if (!(relationship.theme_a_id < relationship.theme_b_id)) {
      throw new Error(`${where} theme ids are not in canonical order`);
    }
    if (relationship.shared_unit_count > relationship.theme_a_unit_count || relationship.shared_unit_count > relationship.theme_b_unit_count) {
      throw new Error(`${where} shares more units than either theme has`);
    }

View on GitHub (pinned to 3ee70a1026)