{"record":{"id":"67e1262a6d2ed46a","repo":"nextlevelbuilder/ui-ux-pro-max-skill","slug":"invalid-relevance-fixture-n-errors","errorCode":null,"errorMessage":"Invalid relevance fixture:\\n- {errors}","messagePattern":"Invalid relevance fixture:\\\\n- (.+?)","errorType":"console","errorClass":"SystemExit","httpStatus":null,"severity":"error","filePath":"scripts/evaluate-relevance.py","lineNumber":226,"sourceCode":"            \"metrics\": summary[\"metrics\"], \"samples\": samples,\n            \"splits\": splits, \"cases\": records}\n\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--cases\", type=Path, default=FIXTURE_DIR / \"relevance-cases.json\")\n    parser.add_argument(\"--thresholds\", type=Path, default=FIXTURE_DIR / \"relevance-thresholds.json\")\n    parser.add_argument(\"--baseline\", type=Path, default=FIXTURE_DIR / \"relevance-baseline.json\")\n    parser.add_argument(\"--split\", choices=(\"all\", \"calibration\", \"held_out\"), default=\"all\")\n    parser.add_argument(\"--write-baseline\", type=Path)\n    parser.add_argument(\"--no-thresholds\", action=\"store_true\")\n    args = parser.parse_args()\n    fixture = load_json(args.cases)\n    fingerprint = runtime_fingerprint()\n    oracle = oracle_fingerprint(args.cases)\n    errors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)\n    if errors:\n        raise SystemExit(\"Invalid relevance fixture:\\n- \" + \"\\n- \".join(errors))\n    selected_split = args.split\n    if selected_split != \"all\":\n        fixture = {**fixture, \"cases\": [case for case in fixture[\"cases\"]\n                                        if case[\"split\"] == selected_split]}\n    report = evaluate(fixture, oracle)\n    if args.write_baseline:\n        args.write_baseline.write_text(json.dumps(report, indent=2, sort_keys=True) + \"\\n\", encoding=\"utf-8\")\n    print(json.dumps({key: report[key] for key in (\"metrics\", \"samples\", \"splits\")},\n                     indent=2, sort_keys=True))\n    manifest = load_json(args.thresholds)\n    manifest_errors = validate_manifest(manifest, fingerprint, oracle)\n    if manifest_errors and not args.no_thresholds:\n        raise SystemExit(\"Invalid threshold manifest:\\n- \" + \"\\n- \".join(manifest_errors))\n    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)\n    if failures:\n        baseline = load_json(args.baseline)\n        old = {record[\"id\"]: record for record in baseline[\"cases\"]}\n        suspect = [record for record in report[\"cases\"] if old.get(record[\"id\"]) != record]","sourceCodeStart":208,"sourceCodeEnd":244,"githubUrl":"https://github.com/nextlevelbuilder/ui-ux-pro-max-skill/blob/a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5/scripts/evaluate-relevance.py#L208-L244","documentation":"Before evaluating, the harness runs validate_fixture() (schema/structure checks against CSV_CONFIG and AVAILABLE_STACKS) plus validate_identities(); all collected problems are listed and the run aborts with SystemExit. This is a data-contract gate: each case must reference existing domains/stacks and satisfy identity invariants so results are comparable across runs.","triggerScenarios":"A relevance-cases.json case using a domain that no longer exists in the CSV config (e.g. a renamed domain), a stack not in AVAILABLE_STACKS, missing required fields, duplicate case ids, or an expected-id set that violates the identity rules in validate_identities().","commonSituations":"Domains/stacks renamed in the CSVs without updating fixtures; adding new cases by copy-paste and leaving placeholder values; the fixture and code at different versions after a partial pull.","solutions":["Read the listed per-case errors — each line names the exact violation; fix those fields in relevance-cases.json.","Cross-check every case's domain against CSV_CONFIG and every stack against AVAILABLE_STACKS in the current code.","Ensure each case has required keys (id, query, split among calibration/held_out, expectations) and unique ids.","After CSV renames, sweep the fixture for the old identifiers."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"# fail fast with actionable output before the full run\nfrom relevance_metrics import validate_fixture, validate_identities\n\nfixture = load_json(args.cases)\nerrors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)\nif errors:\n    for e in errors:\n        print(e)\n    raise SystemExit(1)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["After renaming domains/stacks in CSVs, grep the fixtures for the old identifiers in the same change.","Add a fixture schema check to CI independent of the full evaluation run.","When copy-pasting cases, always update id, split, and the expected set."],"tags":["python","testing","validation","fixtures"],"backgroundTag":null,"analyzedSha":"a38d04c3d5c298c851dbe5e6ee1965ee3de42cb5","analyzedAt":"2026-08-14T18:51:02.321Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}