{"record":{"id":"862b5290e7712ea0","repo":"Hmbown/CodeWhale","slug":"baseline-provenance-build-profile-sample-count-cha","errorCode":null,"errorMessage":"baseline provenance build profile/sample count changed","messagePattern":"baseline provenance build profile/sample count changed","errorType":"exception","errorClass":"PersistenceBacklogError","httpStatus":null,"severity":"error","filePath":"scripts/check-persistence-backlog-budget.py","lineNumber":316,"sourceCode":"    provenance = baseline.get(\"provenance\")\n    if not isinstance(provenance, dict):\n        raise PersistenceBacklogError(\"baseline_observation needs provenance\")\n    if provenance.get(\"platform\") != \"macos\":\n        raise PersistenceBacklogError(\"baseline provenance platform must be macos\")\n    if not isinstance(provenance.get(\"source_sha\"), str) or not SOURCE_SHA_PATTERN.fullmatch(\n        provenance[\"source_sha\"]\n    ):\n        raise PersistenceBacklogError(\"baseline provenance needs an exact source SHA\")\n    if provenance.get(\"source_dirty\") is not False:\n        raise PersistenceBacklogError(\"baseline provenance must identify a clean source tree\")\n    for field, prefix in ((\"rustc_version\", \"rustc \"), (\"cargo_version\", \"cargo \")):\n        if not isinstance(provenance.get(field), str) or not provenance[field].startswith(prefix):\n            raise PersistenceBacklogError(f\"baseline provenance needs {field}\")\n    if provenance.get(\"build_profile\") != \"test\" or not (\n        type(provenance.get(\"sample_count\")) is int\n        and provenance[\"sample_count\"] == 1\n    ):\n        raise PersistenceBacklogError(\"baseline provenance build profile/sample count changed\")\n\n\ndef validate_baseline_receipt(\n    budget: dict[str, Any], baseline_receipt: dict[str, Any]\n) -> None:\n    validate_receipt(baseline_receipt, require_clean_source=True)\n    baseline = budget[\"baseline_observation\"]\n    for field in (\"accepted_requests\", \"applied_version\", *CEILING_FIELDS):\n        if baseline_receipt[field] != baseline[field]:\n            raise PersistenceBacklogError(\n                f\"baseline receipt {field} does not match baseline_observation\"\n            )\n    provenance = baseline[\"provenance\"]\n    for field in (\n        \"platform\",\n        \"source_sha\",\n        \"source_dirty\",\n        \"rustc_version\",","sourceCodeStart":298,"sourceCodeEnd":334,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/8880682c63083a91624de936797efa3ce9e498fd/scripts/check-persistence-backlog-budget.py#L298-L334","documentation":"Raised by scripts/check-persistence-backlog-budget.py while validating scripts/persistence-backlog-budget.json: the baseline_observation.provenance block no longer records the frozen measurement methodology — build_profile must be exactly \"test\" and sample_count must be the integer 1. The gate pins how the macOS baseline was produced so ceiling comparisons stay meaningful across runs. The sample_count check uses exact type equality, so a bool or float also fails.","triggerScenarios":"Running `python3 scripts/check-persistence-backlog-budget.py` (or the CI job wrapping it) after baseline_observation.provenance.build_profile was changed from \"test\" (e.g. to \"release\" or \"bench\"), or sample_count from 1 (e.g. to 3 for averaging, to 1.0, or to true).","commonSituations":"A developer tries to make the baseline \"more realistic\" with a release-profile or multi-sample measurement; hand-editing the budget JSON in an editor that rewrites 1 as 1.0; pasting a provenance block from a receipt measured under a different configuration.","solutions":["Restore \"build_profile\": \"test\" and \"sample_count\": 1 in baseline_observation.provenance of scripts/persistence-backlog-budget.json","If the methodology genuinely changed, re-measure the baseline on a clean macOS checkout with scripts/measure-persistence-backlog.py and update both the budget's baseline_observation and scripts/persistence-backlog-baseline-receipt.json in the same change","Make sure sample_count is a JSON integer (not 1.0 or true), because the checker compares types strictly"],"exampleFix":"// before (scripts/persistence-backlog-budget.json)\n\"provenance\": { \"platform\": \"macos\", \"source_sha\": \"<40-hex>\", \"source_dirty\": false, \"rustc_version\": \"rustc 1.79.0\", \"cargo_version\": \"cargo 1.79.0\", \"build_profile\": \"release\", \"sample_count\": 3 }\n// after\n\"provenance\": { \"platform\": \"macos\", \"source_sha\": \"<40-hex>\", \"source_dirty\": false, \"rustc_version\": \"rustc 1.79.0\", \"cargo_version\": \"cargo 1.79.0\", \"build_profile\": \"test\", \"sample_count\": 1 }","handlingStrategy":"validation","validationCode":"prov = budget['baseline_observation']['provenance']\nassert prov.get('build_profile') == 'test', 'build_profile drifted'\nassert type(prov.get('sample_count')) is int and prov['sample_count'] == 1, 'sample_count drifted'","typeGuard":"def is_frozen_provenance(prov):\n    return (isinstance(prov, dict)\n            and prov.get('build_profile') == 'test'\n            and type(prov.get('sample_count')) is int\n            and prov['sample_count'] == 1)","tryCatchPattern":null,"preventionTips":["Never hand-edit baseline_observation.provenance; regenerate it from a real measurement","Treat build_profile='test' and sample_count=1 as frozen constants of the gate, not tunables","Run the checker in CI on every change that touches the budget files"],"tags":["python","json","validation","benchmarking","ci"],"backgroundTag":null,"analyzedSha":"8880682c63083a91624de936797efa3ce9e498fd","analyzedAt":"2026-08-16T11:31:27.956Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}