JuliusBrussee/caveman · error · Error

${where} generated case ${entry.id} perturbs a unit the data

Error message

${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed

What it means

Thrown when a generated (perturbed) case and a recorded (perturbation "none") case share the same source_unit_id. This is leakage: the dataset would both replay a unit unperturbed and perturb it, so the perturbed run's behavior is contaminated by the clean replay of the same flow.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:279

      if (!unitsByID.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report`);
      if (!familyUnits.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not a member of task family ${family.id}`);
      if (generatorPerturbation.get(entry.generator) !== entry.perturbation) {
        throw new Error(`${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}`);
      }
      if (entry.perturbation === "none" && entry.id !== entry.source_unit_id) {
        throw new Error(`${at} replays a recorded flow but is not identified by its source unit`);
      }
      if (entry.perturbation !== "none" && entry.id !== `${entry.generator}:${entry.source_unit_id}`) {
        throw new Error(`${at} is a generated case whose id does not name its generator and source unit`);
      }
      roleCounts[entry.role] += 1;
      if (entry.role === "target_failure" || entry.role === "prior_success") recordedSources.add(entry.source_unit_id);
    }
    for (const entry of dataset.cases) {
      // Leakage: a generated case must not perturb a unit the manifest already
      // replays unperturbed.
      if (entry.perturbation !== "none" && recordedSources.has(entry.source_unit_id)) {
        throw new Error(`${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed`);
      }
    }

    // Manifest composition arithmetic (spec 18.2): target-failure + prior-success
    // + one boundary + up to two adversarial, each capped and each recomputable.
    //
    // The boundary and stale-input generators perturb the localization evidence
    // a confidence guard reads. A ChangeSet whose applicability never reads that
    // evidence has no threshold to sit beside, so those two cases must NOT be
    // generated for it — a "boundary" case against a guard that does not exist
    // tests nothing while counting as adversarial coverage.
    const confidenceGuard = item.change_set.applicability.all.some((condition) =>
      condition === "failure_location_confidence >= 0.90" ||
      condition === "symbol_resolution == unique" ||
      condition === "targeted_test_reproduces == true");
    const perturbations = new Set(dataset.cases.map((entry) => entry.perturbation));
    for (const [perturbation, label] of [["guard_threshold_boundary", "boundary"], ["stale_input", "stale-input"]]) {
      if (perturbations.has(perturbation) !== confidenceGuard) {

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Pick a different source unit for the generated case — one not already replayed unperturbed in this dataset.
2. Or drop the recorded case for that unit if the perturbed version is the one that matters.
3. Enforce the exclusion in the dataset composer: generated-case source units = family units minus recordedSources.

Example fix

// before
[{id:"unit-07", source_unit_id:"unit-07", perturbation:"none"},
 {id:"adversarial_stale_input.v1:unit-07", source_unit_id:"unit-07", perturbation:"stale_input"}]

// after
[{id:"unit-07", source_unit_id:"unit-07", perturbation:"none"},
 {id:"adversarial_stale_input.v1:unit-12", source_unit_id:"unit-12", perturbation:"stale_input"}]

const recordedSources = new Set(
  dataset.cases.filter((c) => c.role === "target_failure" || c.role === "prior_success").map((c) => c.source_unit_id)
);
const noLeak = dataset.cases.every((c) => c.perturbation === "none" || !recordedSources.has(c.source_unit_id));

When it happens

Trigger: Composing a dataset that includes a unit as a target_failure/prior_success recorded case AND derives an adversarial or boundary case from that same unit.

Common situations: Auto-selecting adversarial sources from the same pool as recorded cases without exclusion; growing the dataset by adding perturbations of already-included units.

Related errors

• ${at} appears twice in the dataset
• ${at} names source unit ${entry.source_unit_id} that is not
• ${at} generator ${entry.generator} disagrees with perturbati
• ${at} replays a recorded flow but is not identified by its s
• ${at} is a generated case whose id does not name its generat

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/0b631d5011673519. Report an issue: GitHub.