JuliusBrussee/caveman · error · Error

${at} names source unit ${entry.source_unit_id} that is not

Error message

${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report

What it means

Thrown when a dataset case's source_unit_id is not in unitsByID, the map of all analysis units of this report. Every dataset case — recorded or generated — must be derived from a real analysis unit of the report; an unresolvable source unit would be invented evidence wearing an adversarial label.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:261

    // Every dataset case — recorded or generated — must name a real analysis
    // unit of this case's own task family, and its generator and perturbation
    // must agree. A generated case whose source unit cannot be resolved would be
    // invented evidence wearing an adversarial label.
    const dataset = item.eval_pack.dataset;
    const generatorPerturbation = new Map([
      ["recorded_case.v1", "none"],
      ["boundary_guard_threshold.v1", "guard_threshold_boundary"],
      ["adversarial_misleading_tool_output.v1", "misleading_tool_output"],
      ["adversarial_stale_input.v1", "stale_input"],
    ]);
    const roleCounts = { target_failure: 0, prior_success: 0, boundary: 0, adversarial: 0 };
    const recordedSources = new Set();
    const datasetCaseIDs = new Set();
    for (const entry of dataset.cases) {
      const at = `${where} dataset case ${entry.id}`;
      if (datasetCaseIDs.has(entry.id)) throw new Error(`${at} appears twice in the dataset`);
      datasetCaseIDs.add(entry.id);
      if (!unitsByID.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report`);
      if (!familyUnits.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not a member of task family ${family.id}`);
      if (generatorPerturbation.get(entry.generator) !== entry.perturbation) {
        throw new Error(`${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}`);
      }
      if (entry.perturbation === "none" && entry.id !== entry.source_unit_id) {
        throw new Error(`${at} replays a recorded flow but is not identified by its source unit`);
      }
      if (entry.perturbation !== "none" && entry.id !== `${entry.generator}:${entry.source_unit_id}`) {
        throw new Error(`${at} is a generated case whose id does not name its generator and source unit`);
      }
      roleCounts[entry.role] += 1;
      if (entry.role === "target_failure" || entry.role === "prior_success") recordedSources.add(entry.source_unit_id);
    }
    for (const entry of dataset.cases) {
      // Leakage: a generated case must not perturb a unit the manifest already
      // replays unperturbed.
      if (entry.perturbation !== "none" && recordedSources.has(entry.source_unit_id)) {
        throw new Error(`${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed`);

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Correct source_unit_id to an id present in the report's analysis units (and in the case's own task family — the next check enforces family membership).
2. If the unit no longer exists, delete the dataset case that cites it.
3. Regenerate the dataset from the same unit snapshot the report embeds.

Example fix

// before
entry.source_unit_id = "unit-77"   // not in report.analysis_units

// after
entry.source_unit_id = "unit-07"

const units = new Set(report.analysis_units.map((u) => u.id));
const resolvable = dataset.cases.every((c) => units.has(c.source_unit_id));

When it happens

Trigger: Typing a wrong unit id; generating dataset cases from a unit list that is out of sync with report.analysis_units; removing an analysis unit from the report while dataset cases still cite it.

Common situations: Regenerating analysis units under new ids (e.g., hashing or renumbering) without regenerating eval packs; cross-report case reuse.

Related errors

• ${where} change set or eval pack is bound to another case
• ${where} names an opportunity that is not in this report
• ${at} appears twice in the dataset
• ${at} generator ${entry.generator} disagrees with perturbati
• ${at} replays a recorded flow but is not identified by its s

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/e7403df084ef5711. Report an issue: GitHub.