JuliusBrussee/caveman · error · Error

${at} generator ${entry.generator} disagrees with perturbati

Error message

${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}

What it means

Thrown when a dataset case's generator and perturbation disagree per the fixed mapping: recorded_case.v1→none, boundary_guard_threshold.v1→guard_threshold_boundary, adversarial_misleading_tool_output.v1→misleading_tool_output, adversarial_stale_input.v1→stale_input. Note that generatorPerturbation.get on an UNKNOWN generator returns undefined, so an unrecognized generator string also triggers this error.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:264

    // invented evidence wearing an adversarial label.
    const dataset = item.eval_pack.dataset;
    const generatorPerturbation = new Map([
      ["recorded_case.v1", "none"],
      ["boundary_guard_threshold.v1", "guard_threshold_boundary"],
      ["adversarial_misleading_tool_output.v1", "misleading_tool_output"],
      ["adversarial_stale_input.v1", "stale_input"],
    ]);
    const roleCounts = { target_failure: 0, prior_success: 0, boundary: 0, adversarial: 0 };
    const recordedSources = new Set();
    const datasetCaseIDs = new Set();
    for (const entry of dataset.cases) {
      const at = `${where} dataset case ${entry.id}`;
      if (datasetCaseIDs.has(entry.id)) throw new Error(`${at} appears twice in the dataset`);
      datasetCaseIDs.add(entry.id);
      if (!unitsByID.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report`);
      if (!familyUnits.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not a member of task family ${family.id}`);
      if (generatorPerturbation.get(entry.generator) !== entry.perturbation) {
        throw new Error(`${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}`);
      }
      if (entry.perturbation === "none" && entry.id !== entry.source_unit_id) {
        throw new Error(`${at} replays a recorded flow but is not identified by its source unit`);
      }
      if (entry.perturbation !== "none" && entry.id !== `${entry.generator}:${entry.source_unit_id}`) {
        throw new Error(`${at} is a generated case whose id does not name its generator and source unit`);
      }
      roleCounts[entry.role] += 1;
      if (entry.role === "target_failure" || entry.role === "prior_success") recordedSources.add(entry.source_unit_id);
    }
    for (const entry of dataset.cases) {
      // Leakage: a generated case must not perturb a unit the manifest already
      // replays unperturbed.
      if (entry.perturbation !== "none" && recordedSources.has(entry.source_unit_id)) {
        throw new Error(`${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed`);
      }
    }

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Set generator and perturbation to a matched pair from the map (e.g. boundary_guard_threshold.v1 with guard_threshold_boundary).
2. If the generator name is unrecognized, correct it to one of the four known generators or extend generatorPerturbation in lockstep with the composer.
3. Re-run the validator; a mismatch here usually means the dataset composer and validator drifted, so check both.

Example fix

// before
{generator: "boundary_guard_threshold.v1", perturbation: "none", ...}

// after
{generator: "boundary_guard_threshold.v1", perturbation: "guard_threshold_boundary", ...}

const GEN_PERT = new Map([
  ["recorded_case.v1", "none"],
  ["boundary_guard_threshold.v1", "guard_threshold_boundary"],
  ["adversarial_misleading_tool_output.v1", "misleading_tool_output"],
  ["adversarial_stale_input.v1", "stale_input"],
]);
const pairsOk = dataset.cases.every((c) => GEN_PERT.get(c.generator) === c.perturbation);

const GENERATORS = ["recorded_case.v1", "boundary_guard_threshold.v1", "adversarial_misleading_tool_output.v1", "adversarial_stale_input.v1"];
function isKnownGenerator(g) { return GENERATORS.includes(g); }

When it happens

Trigger: Pairing e.g. generator "recorded_case.v1" with perturbation "stale_input"; introducing a new generator id not in the map; misspelling either field.

Common situations: Adding a new perturbation type to the pipeline without extending the validator's map; hand-editing entries; version skew between the composer that builds datasets and the validator.

Related errors

• ${at} appears twice in the dataset
• ${at} names source unit ${entry.source_unit_id} that is not
• ${at} replays a recorded flow but is not identified by its s
• ${at} is a generated case whose id does not name its generat
• ${where} generated case ${entry.id} perturbs a unit the data

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/6cb90ba3ac1aada2. Report an issue: GitHub.