JuliusBrussee/caveman · error · Error

${at} replays a recorded flow but is not identified by its s

Error message

${at} replays a recorded flow but is not identified by its source unit

What it means

Thrown when a dataset case has perturbation "none" (a recorded replay) but its id is not exactly its source_unit_id. Recorded cases are identified by the unit they replay; any other id breaks the 1:1 correspondence between a replayed flow and its provenance.

Source

Thrown at packages/shared/contracts/scripts/validate-continuous-improvement.mjs:267

      ["recorded_case.v1", "none"],
      ["boundary_guard_threshold.v1", "guard_threshold_boundary"],
      ["adversarial_misleading_tool_output.v1", "misleading_tool_output"],
      ["adversarial_stale_input.v1", "stale_input"],
    ]);
    const roleCounts = { target_failure: 0, prior_success: 0, boundary: 0, adversarial: 0 };
    const recordedSources = new Set();
    const datasetCaseIDs = new Set();
    for (const entry of dataset.cases) {
      const at = `${where} dataset case ${entry.id}`;
      if (datasetCaseIDs.has(entry.id)) throw new Error(`${at} appears twice in the dataset`);
      datasetCaseIDs.add(entry.id);
      if (!unitsByID.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report`);
      if (!familyUnits.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not a member of task family ${family.id}`);
      if (generatorPerturbation.get(entry.generator) !== entry.perturbation) {
        throw new Error(`${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}`);
      }
      if (entry.perturbation === "none" && entry.id !== entry.source_unit_id) {
        throw new Error(`${at} replays a recorded flow but is not identified by its source unit`);
      }
      if (entry.perturbation !== "none" && entry.id !== `${entry.generator}:${entry.source_unit_id}`) {
        throw new Error(`${at} is a generated case whose id does not name its generator and source unit`);
      }
      roleCounts[entry.role] += 1;
      if (entry.role === "target_failure" || entry.role === "prior_success") recordedSources.add(entry.source_unit_id);
    }
    for (const entry of dataset.cases) {
      // Leakage: a generated case must not perturb a unit the manifest already
      // replays unperturbed.
      if (entry.perturbation !== "none" && recordedSources.has(entry.source_unit_id)) {
        throw new Error(`${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed`);
      }
    }

    // Manifest composition arithmetic (spec 18.2): target-failure + prior-success
    // + one boundary + up to two adversarial, each capped and each recomputable.
    //

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Set the entry's id equal to its source_unit_id.
2. If a distinct id is genuinely needed, the case is not a plain replay — model it as a generated case with an explicit generator and matching perturbation.

Example fix

// before
{id: "replay-unit-07", source_unit_id: "unit-07", perturbation: "none"}

// after
{id: "unit-07", source_unit_id: "unit-07", perturbation: "none"}

const recordedIdsOk = dataset.cases
  .filter((c) => c.perturbation === "none")
  .every((c) => c.id === c.source_unit_id);

When it happens

Trigger: Renaming a recorded case's id (e.g., adding a prefix or suffix) while keeping perturbation "none"; reusing a generated-case id format for a recorded case.

Common situations: Hand-editing ids for readability; merging datasets where ids were namespaced.

Related errors

• ${at} is a generated case whose id does not name its generat
• ${at} appears twice in the dataset
• ${at} names source unit ${entry.source_unit_id} that is not
• ${at} generator ${entry.generator} disagrees with perturbati
• ${where} generated case ${entry.id} perturbs a unit the data

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/7b5e501a0e51f328. Report an issue: GitHub.