JuliusBrussee/caveman · error · Error

${where} names an opportunity that is not in this report

Error message

${where} names an opportunity that is not in this report

What it means

Thrown when a case's opportunity_id does not match any opportunity in report.opportunities. Cases are derived from opportunities recorded in the same report; a case citing a foreign or missing opportunity breaks the report's internal derivation chain.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:229

    }

    // A case that proposes nothing must carry nothing: no operator, no eval
    // pack, no replay. Anything else would present an abstention as a proposal.
    if (item.status === "diagnosis_only") {
      if (item.change_set.operator_id !== "" || item.eval_pack.id !== "" || item.replay.status !== "not_run") {
        throw new Error(`${where} is diagnosis-only but carries a change set, eval pack or replay`);
      }
      continue;
    }
    if (item.change_set.case_id !== item.id || item.eval_pack.case_id !== item.id) {
      throw new Error(`${where} change set or eval pack is bound to another case`);
    }

    // The case must investigate the pair its own opportunity named. A cohort
    // built from a differently-derived pair would describe one comparison and
    // measure another.
    const source = report.opportunities.find((opportunity) => opportunity.id === item.opportunity_id);
    if (!source) throw new Error(`${where} names an opportunity that is not in this report`);
    if (armVariants.get("baseline").id !== source.current_variant_id || armVariants.get("alternative").id !== source.alternative_variant_id) {
      throw new Error(`${where} cohort arms (${armVariants.get("baseline").id}/${armVariants.get("alternative").id}) are not the variants opportunity ${source.id} named`);
    }

    // Every required grader is one the recorded interpreter actually computes.
    // `json_schema` is deliberately absent: over a typed interpreter result it
    // could only assert that a hash is non-empty, which is a grader that cannot
    // fail and therefore inflates how much a replay was checked.
    const knownGraders = new Set(["exact_match", "tool_order", "tool_count", "guard_respected"]);
    for (const grader of item.eval_pack.graders) {
      if (!knownGraders.has(grader)) throw new Error(`${where} requires grader ${grader}, which the recorded interpreter does not compute`);
    }

    // Every dataset case — recorded or generated — must name a real analysis
    // unit of this case's own task family, and its generator and perturbation
    // must agree. A generated case whose source unit cannot be resolved would be
    // invented evidence wearing an adversarial label.
    const dataset = item.eval_pack.dataset;

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Check report.opportunities for the id the case should reference and correct opportunity_id.
2. If the opportunity was intentionally removed, remove the case that cites it too.
3. If cases and opportunities are generated by different steps, regenerate them together from one source list.

Example fix

// before
item.opportunity_id = "opp-041"
report.opportunities = [{id: "opp-41"}, ...]

// after
item.opportunity_id = "opp-41"

const known = new Set(report.opportunities.map((o) => o.id));
const resolvable = known.has(item.opportunity_id);

When it happens

Trigger: Typo in opportunity_id; deleting an opportunity from the report while its case remains; copying a case between two different reports.

Common situations: Splitting a report into two files; renaming opportunity ids; generating cases from a different (older) opportunity list than the one embedded in the report.

Related errors

• ${where} change set or eval pack is bound to another case
• ${where} cohort arms (${armVariants.get("baseline").id}/${ar
• ${at} names source unit ${entry.source_unit_id} that is not
• option not found
• cave_harness_adapter_version_invalid

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/80a151b9ca78b00f. Report an issue: GitHub.