nextlevelbuilder/ui-ux-pro-max-skill · error · SystemExit

Relevance gate failed:\n- {failures}

Error message

Relevance gate failed:\n- {failures}

What it means

The final gate: check_thresholds() compares evaluation metrics (NDCG@k, precision@k, reciprocal rank, grades, false positives, coherence) against the manifest's floors for the selected split; any failure becomes a line in the message. It also prints 'problem cases' — cases whose results differ from the saved baseline, or failing heuristics like top-grade < 2, false positives, or incoherent results — to point at regressions.

Source

Thrown at scripts/evaluate-relevance.py:252

    print(json.dumps({key: report[key] for key in ("metrics", "samples", "splits")},
                     indent=2, sort_keys=True))
    manifest = load_json(args.thresholds)
    manifest_errors = validate_manifest(manifest, fingerprint, oracle)
    if manifest_errors and not args.no_thresholds:
        raise SystemExit("Invalid threshold manifest:\n- " + "\n- ".join(manifest_errors))
    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)
    if failures:
        baseline = load_json(args.baseline)
        old = {record["id"]: record for record in baseline["cases"]}
        suspect = [record for record in report["cases"] if old.get(record["id"]) != record]
        if not suspect:
            suspect = [record for record in report["cases"]
                       if (record.get("grades") and record["grades"][0] < 2)
                       or record.get("falsePositives", 0)
                       or ("coherence" in record and not all(record["coherence"]))]
        detail = "\n".join(f"  {item['id']}: {item.get('actual', item)}"
                           for item in suspect)
        raise SystemExit("Relevance gate failed:\n- " + "\n- ".join(failures)
                         + ("\nProblem cases:\n" + detail if detail else ""))


if __name__ == "__main__":
    main()

View on GitHub (pinned to a38d04c3d5)

Solutions

1. Start from the printed problem-case ids: re-run those queries via search.py and inspect why expected results no longer rank.
2. If a legitimate improvement changed ordering, update the baseline (--write-baseline) and regenerate thresholds so gates track the new behavior.
3. If a regression, bisect recent changes to core.py / CSVs and fix the ranking regression.
4. Use --split calibration vs held_out to narrow whether calibration or generalization is failing.

# preview metrics against thresholds before gating the CI job
report = evaluate(fixture, oracle)
for failure in check_thresholds(report, manifest, selected_split):
    print('will fail:', failure)

When it happens

Trigger: Changing search logic in core.py, CSV content, or reasoning weights so ranking quality drops below the recorded floors; a case newly graded poorly (grades[0] < 2), accumulating falsePositives, or failing coherence checks; running on a split whose thresholds are tighter than the whole-set results.

Common situations: Refactoring BM25 scoring or tokenization and accidentally degrading ranking; new CSV rows diluting results for existing queries; thresholds regenerated on a lucky run so normal runs fail.

Related errors

• Invalid threshold manifest:\n- {manifest_errors}
• Invalid JSON file {path}: {error}
• Invalid relevance fixture:\n- {errors}
• verified-at has suspicious date {value!r}
• {name} catalog counts are stale: {', '.join(missing)}

AI-assisted analysis of nextlevelbuilder/ui-ux-pro-max-skill@a38d04c3d5 (2026-08-14). Data as JSON: /api/errors/f929ca5e0bafeaa5. Report an issue: GitHub.