{"record":{"id":"0b631d5011673519","repo":"JuliusBrussee/caveman","slug":"where-generated-case-entry-id-perturbs-a-uni","errorCode":null,"errorMessage":"${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed","messagePattern":"(.+?) generated case (.+?) perturbs a unit the dataset already replays unperturbed","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":279,"sourceCode":"      if (!unitsByID.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not an analysis unit of this report`);\n      if (!familyUnits.has(entry.source_unit_id)) throw new Error(`${at} names source unit ${entry.source_unit_id} that is not a member of task family ${family.id}`);\n      if (generatorPerturbation.get(entry.generator) !== entry.perturbation) {\n        throw new Error(`${at} generator ${entry.generator} disagrees with perturbation ${entry.perturbation}`);\n      }\n      if (entry.perturbation === \"none\" && entry.id !== entry.source_unit_id) {\n        throw new Error(`${at} replays a recorded flow but is not identified by its source unit`);\n      }\n      if (entry.perturbation !== \"none\" && entry.id !== `${entry.generator}:${entry.source_unit_id}`) {\n        throw new Error(`${at} is a generated case whose id does not name its generator and source unit`);\n      }\n      roleCounts[entry.role] += 1;\n      if (entry.role === \"target_failure\" || entry.role === \"prior_success\") recordedSources.add(entry.source_unit_id);\n    }\n    for (const entry of dataset.cases) {\n      // Leakage: a generated case must not perturb a unit the manifest already\n      // replays unperturbed.\n      if (entry.perturbation !== \"none\" && recordedSources.has(entry.source_unit_id)) {\n        throw new Error(`${where} generated case ${entry.id} perturbs a unit the dataset already replays unperturbed`);\n      }\n    }\n\n    // Manifest composition arithmetic (spec 18.2): target-failure + prior-success\n    // + one boundary + up to two adversarial, each capped and each recomputable.\n    //\n    // The boundary and stale-input generators perturb the localization evidence\n    // a confidence guard reads. A ChangeSet whose applicability never reads that\n    // evidence has no threshold to sit beside, so those two cases must NOT be\n    // generated for it — a \"boundary\" case against a guard that does not exist\n    // tests nothing while counting as adversarial coverage.\n    const confidenceGuard = item.change_set.applicability.all.some((condition) =>\n      condition === \"failure_location_confidence >= 0.90\" ||\n      condition === \"symbol_resolution == unique\" ||\n      condition === \"targeted_test_reproduces == true\");\n    const perturbations = new Set(dataset.cases.map((entry) => entry.perturbation));\n    for (const [perturbation, label] of [[\"guard_threshold_boundary\", \"boundary\"], [\"stale_input\", \"stale-input\"]]) {\n      if (perturbations.has(perturbation) !== confidenceGuard) {","sourceCodeStart":261,"sourceCodeEnd":297,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L261-L297","documentation":"Thrown when a generated (perturbed) case and a recorded (perturbation \"none\") case share the same source_unit_id. This is leakage: the dataset would both replay a unit unperturbed and perturb it, so the perturbed run's behavior is contaminated by the clean replay of the same flow.","triggerScenarios":"Composing a dataset that includes a unit as a target_failure/prior_success recorded case AND derives an adversarial or boundary case from that same unit.","commonSituations":"Auto-selecting adversarial sources from the same pool as recorded cases without exclusion; growing the dataset by adding perturbations of already-included units.","solutions":["Pick a different source unit for the generated case — one not already replayed unperturbed in this dataset.","Or drop the recorded case for that unit if the perturbed version is the one that matters.","Enforce the exclusion in the dataset composer: generated-case source units = family units minus recordedSources."],"exampleFix":"// before\n[{id:\"unit-07\", source_unit_id:\"unit-07\", perturbation:\"none\"},\n {id:\"adversarial_stale_input.v1:unit-07\", source_unit_id:\"unit-07\", perturbation:\"stale_input\"}]\n\n// after\n[{id:\"unit-07\", source_unit_id:\"unit-07\", perturbation:\"none\"},\n {id:\"adversarial_stale_input.v1:unit-12\", source_unit_id:\"unit-12\", perturbation:\"stale_input\"}]","handlingStrategy":"validation","validationCode":"const recordedSources = new Set(\n  dataset.cases.filter((c) => c.role === \"target_failure\" || c.role === \"prior_success\").map((c) => c.source_unit_id)\n);\nconst noLeak = dataset.cases.every((c) => c.perturbation === \"none\" || !recordedSources.has(c.source_unit_id));","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Select generated-case source units from the complement of the recorded pool (family units minus recorded sources).","Add a leakage assertion to the dataset composer's own tests, not just the report validator."],"tags":["validation","dataset","leakage"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}