{"record":{"id":"f929ca5e0bafeaa5","repo":"nextlevelbuilder/ui-ux-pro-max-skill","slug":"relevance-gate-failed-n-failures","errorCode":null,"errorMessage":"Relevance gate failed:\\n- {failures}","messagePattern":"Relevance gate failed:\\\\n- (.+?)","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"error","filePath":"scripts/evaluate-relevance.py","lineNumber":252,"sourceCode":"    print(json.dumps({key: report[key] for key in (\"metrics\", \"samples\", \"splits\")},\n                     indent=2, sort_keys=True))\n    manifest = load_json(args.thresholds)\n    manifest_errors = validate_manifest(manifest, fingerprint, oracle)\n    if manifest_errors and not args.no_thresholds:\n        raise SystemExit(\"Invalid threshold manifest:\\n- \" + \"\\n- \".join(manifest_errors))\n    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)\n    if failures:\n        baseline = load_json(args.baseline)\n        old = {record[\"id\"]: record for record in baseline[\"cases\"]}\n        suspect = [record for record in report[\"cases\"] if old.get(record[\"id\"]) != record]\n        if not suspect:\n            suspect = [record for record in report[\"cases\"]\n                       if (record.get(\"grades\") and record[\"grades\"][0] < 2)\n                       or record.get(\"falsePositives\", 0)\n                       or (\"coherence\" in record and not all(record[\"coherence\"]))]\n        detail = \"\\n\".join(f\"  {item['id']}: {item.get('actual', item)}\"\n                           for item in suspect)\n        raise SystemExit(\"Relevance gate failed:\\n- \" + \"\\n- \".join(failures)\n                         + (\"\\nProblem cases:\\n\" + detail if detail else \"\"))\n\n\nif __name__ == \"__main__\":\n    main()\n","sourceCodeStart":234,"sourceCodeEnd":258,"githubUrl":"https://github.com/nextlevelbuilder/ui-ux-pro-max-skill/blob/a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5/scripts/evaluate-relevance.py#L234-L258","documentation":"The final gate: check_thresholds() compares evaluation metrics (NDCG@k, precision@k, reciprocal rank, grades, false positives, coherence) against the manifest's floors for the selected split; any failure becomes a line in the message. It also prints 'problem cases' — cases whose results differ from the saved baseline, or failing heuristics like top-grade < 2, false positives, or incoherent results — to point at regressions.","triggerScenarios":"Changing search logic in core.py, CSV content, or reasoning weights so ranking quality drops below the recorded floors; a case newly graded poorly (grades[0] < 2), accumulating falsePositives, or failing coherence checks; running on a split whose thresholds are tighter than the whole-set results.","commonSituations":"Refactoring BM25 scoring or tokenization and accidentally degrading ranking; new CSV rows diluting results for existing queries; thresholds regenerated on a lucky run so normal runs fail.","solutions":["Start from the printed problem-case ids: re-run those queries via search.py and inspect why expected results no longer rank.","If a legitimate improvement changed ordering, update the baseline (--write-baseline) and regenerate thresholds so gates track the new behavior.","If a regression, bisect recent changes to core.py / CSVs and fix the ranking regression.","Use --split calibration vs held_out to narrow whether calibration or generalization is failing."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# preview metrics against thresholds before gating the CI job\nreport = evaluate(fixture, oracle)\nfor failure in check_thresholds(report, manifest, selected_split):\n    print('will fail:', failure)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run the relevance gate locally before pushing changes to core.py or CSVs.","When rankings legitimately improve, regenerate baseline AND thresholds together.","Watch the printed problem-case ids during development, not just pass/fail."],"tags":["python","testing","search-quality","thresholds","ci"],"backgroundTag":null,"analyzedSha":"a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5","analyzedAt":"2026-08-14T18:51:02.321Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}