{"record":{"id":"49c727ca2cb1e05d","repo":"nextlevelbuilder/ui-ux-pro-max-skill","slug":"invalid-threshold-manifest-n-manifest-errors","errorCode":null,"errorMessage":"Invalid threshold manifest:\\n- {manifest_errors}","messagePattern":"Invalid threshold manifest:\\\\n- (.+?)","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"error","filePath":"scripts/evaluate-relevance.py","lineNumber":239,"sourceCode":"    fixture = load_json(args.cases)\n    fingerprint = runtime_fingerprint()\n    oracle = oracle_fingerprint(args.cases)\n    errors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)\n    if errors:\n        raise SystemExit(\"Invalid relevance fixture:\\n- \" + \"\\n- \".join(errors))\n    selected_split = args.split\n    if selected_split != \"all\":\n        fixture = {**fixture, \"cases\": [case for case in fixture[\"cases\"]\n                                        if case[\"split\"] == selected_split]}\n    report = evaluate(fixture, oracle)\n    if args.write_baseline:\n        args.write_baseline.write_text(json.dumps(report, indent=2, sort_keys=True) + \"\\n\", encoding=\"utf-8\")\n    print(json.dumps({key: report[key] for key in (\"metrics\", \"samples\", \"splits\")},\n                     indent=2, sort_keys=True))\n    manifest = load_json(args.thresholds)\n    manifest_errors = validate_manifest(manifest, fingerprint, oracle)\n    if manifest_errors and not args.no_thresholds:\n        raise SystemExit(\"Invalid threshold manifest:\\n- \" + \"\\n- \".join(manifest_errors))\n    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)\n    if failures:\n        baseline = load_json(args.baseline)\n        old = {record[\"id\"]: record for record in baseline[\"cases\"]}\n        suspect = [record for record in report[\"cases\"] if old.get(record[\"id\"]) != record]\n        if not suspect:\n            suspect = [record for record in report[\"cases\"]\n                       if (record.get(\"grades\") and record[\"grades\"][0] < 2)\n                       or record.get(\"falsePositives\", 0)\n                       or (\"coherence\" in record and not all(record[\"coherence\"]))]\n        detail = \"\\n\".join(f\"  {item['id']}: {item.get('actual', item)}\"\n                           for item in suspect)\n        raise SystemExit(\"Relevance gate failed:\\n- \" + \"\\n- \".join(failures)\n                         + (\"\\nProblem cases:\\n\" + detail if detail else \"\"))\n\n\nif __name__ == \"__main__\":\n    main()","sourceCodeStart":221,"sourceCodeEnd":257,"githubUrl":"https://github.com/nextlevelbuilder/ui-ux-pro-max-skill/blob/a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5/scripts/evaluate-relevance.py#L221-L257","documentation":"After evaluation, the thresholds manifest (relevance-thresholds.json) is validated against the runtime fingerprint (hash of core.py, design_system.py, reasoning_contract.py, and all CSVs) and the oracle fingerprint of the cases file. Errors mean the manifest was generated for different code/data than what's running — the thresholds can't be trusted for this tree — so the gate refuses rather than compare apples to oranges.","triggerScenarios":"Editing any CSV, core.py, design_system.py, or reasoning_contract.py after the manifest was written; editing relevance-cases.json (changes oracle fingerprint) without regenerating thresholds; checking out a branch whose data differs from the branch that produced the manifest.","commonSituations":"Data updates merged without re-running threshold generation; a stale manifest committed on one branch and evaluated on another; local CSV tweaks during search-quality experiments.","solutions":["Regenerate the manifest with the harness's own generation flow so it records the current fingerprints.","If the change is intentional, commit the regenerated relevance-thresholds.json together with the code/CSV changes.","If you only want metrics without gating, pass --no-thresholds explicitly — but don't ship that result as a pass.","Verify both fingerprints (runtime + oracle) match after regeneration; the error lines tell you which one mismatched."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# detect drift before evaluating\nmanifest = load_json(args.thresholds)\nif manifest.get('runtimeFingerprint') != fingerprint or manifest.get('oracleFingerprint') != oracle:\n    raise SystemExit('Threshold manifest out of date - regenerate it for this tree')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Regenerate the thresholds manifest in the same commit as code/CSV changes that affect search behavior.","Never edit thresholds by hand - let the tooling write fingerprints.","Treat a fingerprint mismatch as 'results not comparable', not a nuisance error to bypass with --no-thresholds."],"tags":["python","testing","fingerprint","thresholds","ci"],"backgroundTag":null,"analyzedSha":"a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5","analyzedAt":"2026-08-14T18:51:02.321Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}