{"record":{"id":"0daab5805573fac0","repo":"JuliusBrussee/caveman","slug":"where-confidenceguard-omits-generated","errorCode":null,"errorMessage":"${where} ${confidenceGuard ? \"omits\" : \"generated\"} a ${label} case ${confidenceGuard ? \"for\" : \"against\"} a change set that ${confidenceGuard ? \"declares\" : \"declares no\"} confidence guard","messagePattern":"(.+?) (.+?) a (.+?) case (.+?) a change set that (.+?) confidence guard","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/shared/contracts/scripts/validate-continuous-improvement.mjs","lineNumber":298,"sourceCode":"      }\n    }\n\n    // Manifest composition arithmetic (spec 18.2): target-failure + prior-success\n    // + one boundary + up to two adversarial, each capped and each recomputable.\n    //\n    // The boundary and stale-input generators perturb the localization evidence\n    // a confidence guard reads. A ChangeSet whose applicability never reads that\n    // evidence has no threshold to sit beside, so those two cases must NOT be\n    // generated for it — a \"boundary\" case against a guard that does not exist\n    // tests nothing while counting as adversarial coverage.\n    const confidenceGuard = item.change_set.applicability.all.some((condition) =>\n      condition === \"failure_location_confidence >= 0.90\" ||\n      condition === \"symbol_resolution == unique\" ||\n      condition === \"targeted_test_reproduces == true\");\n    const perturbations = new Set(dataset.cases.map((entry) => entry.perturbation));\n    for (const [perturbation, label] of [[\"guard_threshold_boundary\", \"boundary\"], [\"stale_input\", \"stale-input\"]]) {\n      if (perturbations.has(perturbation) !== confidenceGuard) {\n        throw new Error(`${where} ${confidenceGuard ? \"omits\" : \"generated\"} a ${label} case ${confidenceGuard ? \"for\" : \"against\"} a change set that ${confidenceGuard ? \"declares\" : \"declares no\"} confidence guard`);\n      }\n    }\n    const expectedTargets = Math.min(4, dataset.target_failure_cases.length);\n    const expectedPriors = Math.min(2, dataset.prior_success_cases.length);\n    if (roleCounts.target_failure !== expectedTargets) throw new Error(`${where} dataset carries ${roleCounts.target_failure} target-failure cases, want ${expectedTargets}`);\n    if (roleCounts.prior_success !== expectedPriors) throw new Error(`${where} dataset carries ${roleCounts.prior_success} prior-success cases, want ${expectedPriors}`);\n    if (roleCounts.boundary > 1) throw new Error(`${where} dataset carries ${roleCounts.boundary} boundary cases, want at most one`);\n    if (roleCounts.adversarial > (confidenceGuard ? 2 : 1)) throw new Error(`${where} dataset carries ${roleCounts.adversarial} adversarial cases`);\n    if (roleCounts.boundary !== dataset.boundary_cases.length) throw new Error(`${where} boundary case list disagrees with the composed dataset`);\n    if (roleCounts.adversarial !== dataset.generated_cases.length) throw new Error(`${where} generated case list disagrees with the composed dataset`);\n    const composed = roleCounts.target_failure + roleCounts.prior_success + roleCounts.boundary + roleCounts.adversarial;\n    if (composed !== dataset.cases.length) throw new Error(`${where} dataset roles (${composed}) do not account for its ${dataset.cases.length} cases`);\n    if (dataset.replay_case_ids.length !== datasetCaseIDs.size) throw new Error(`${where} replay manifest carries ${dataset.replay_case_ids.length} ids for ${datasetCaseIDs.size} dataset cases`);\n    for (const datasetCaseID of dataset.replay_case_ids) {\n      if (!datasetCaseIDs.has(datasetCaseID)) throw new Error(`${where} replay manifest case ${datasetCaseID} is not a composed dataset case`);\n    }\n\n    // The guard grader exists exactly when there is a perturbed case to catch a","sourceCodeStart":280,"sourceCodeEnd":316,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/packages/shared/contracts/scripts/validate-continuous-improvement.mjs#L280-L316","documentation":"Thrown when the presence of a boundary (guard_threshold_boundary) or stale-input case in the dataset does not match whether the change set's applicability declares a confidence guard (one of the exact conditions \"failure_location_confidence >= 0.90\", \"symbol_resolution == unique\", \"targeted_test_reproduces == true\"). Those two generators perturb localization evidence a guard reads; with no guard there is no threshold to test, and with a guard present omitting them loses the coverage that matters most.","triggerScenarios":"A change set whose applicability.all contains none of the three guard conditions but whose dataset includes a boundary or stale-input case; conversely, a guarded change set whose dataset lacks them.","commonSituations":"Editing applicability conditions (rewording or removing a guard) without recomposing the dataset; reusing one dataset template for all cases regardless of applicability.","solutions":["If the change set declares a confidence guard (exact string match), ensure the dataset contains both a guard_threshold_boundary case and a stale_input case.","If it declares no guard, remove boundary and stale-input cases from the dataset.","If you believe a condition IS a guard but is worded differently, align the string to one of the three recognized conditions (the check is exact-match, not semantic)."],"exampleFix":"// before\napplicability.all: []            // no guard\ndataset includes {generator:\"boundary_guard_threshold.v1\", perturbation:\"guard_threshold_boundary\"}\n\n// after — either drop the case, or declare the guard\napplicability.all: [\"failure_location_confidence >= 0.90\"]","handlingStrategy":"validation","validationCode":"const GUARD_CONDITIONS = new Set([\n  \"failure_location_confidence >= 0.90\",\n  \"symbol_resolution == unique\",\n  \"targeted_test_reproduces == true\",\n]);\nconst hasGuard = item.change_set.applicability.all.some((c) => GUARD_CONDITIONS.has(c));\nconst perts = new Set(dataset.cases.map((c) => c.perturbation));\nconst aligned = perts.has(\"guard_threshold_boundary\") === hasGuard && perts.has(\"stale_input\") === hasGuard;","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Make dataset composition a function of applicability: emit boundary/stale-input cases iff a recognized guard condition is present.","Use the exact guard-condition strings verbatim — the check is exact-match; paraphrases silently disable the coverage."],"tags":["validation","dataset","change-set","guard-coverage"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}