What causes "Invalid relevance fixture:\n- {errors}" in nextlevelbuilder/ui-ux-pro-max-skill?

Read the listed per-case errors — each line names the exact violation; fix those fields in relevance-cases.json.

nextlevelbuilder/ui-ux-pro-max-skill · error · SystemExit

Invalid relevance fixture:\n- {errors}

Error message

Invalid relevance fixture:\n- {errors}

What it means

Before evaluating, the harness runs validate_fixture() (schema/structure checks against CSV_CONFIG and AVAILABLE_STACKS) plus validate_identities(); all collected problems are listed and the run aborts with SystemExit. This is a data-contract gate: each case must reference existing domains/stacks and satisfy identity invariants so results are comparable across runs.

Source

Thrown at scripts/evaluate-relevance.py:226

            "metrics": summary["metrics"], "samples": samples,
            "splits": splits, "cases": records}


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", type=Path, default=FIXTURE_DIR / "relevance-cases.json")
    parser.add_argument("--thresholds", type=Path, default=FIXTURE_DIR / "relevance-thresholds.json")
    parser.add_argument("--baseline", type=Path, default=FIXTURE_DIR / "relevance-baseline.json")
    parser.add_argument("--split", choices=("all", "calibration", "held_out"), default="all")
    parser.add_argument("--write-baseline", type=Path)
    parser.add_argument("--no-thresholds", action="store_true")
    args = parser.parse_args()
    fixture = load_json(args.cases)
    fingerprint = runtime_fingerprint()
    oracle = oracle_fingerprint(args.cases)
    errors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)
    if errors:
        raise SystemExit("Invalid relevance fixture:\n- " + "\n- ".join(errors))
    selected_split = args.split
    if selected_split != "all":
        fixture = {**fixture, "cases": [case for case in fixture["cases"]
                                        if case["split"] == selected_split]}
    report = evaluate(fixture, oracle)
    if args.write_baseline:
        args.write_baseline.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    print(json.dumps({key: report[key] for key in ("metrics", "samples", "splits")},
                     indent=2, sort_keys=True))
    manifest = load_json(args.thresholds)
    manifest_errors = validate_manifest(manifest, fingerprint, oracle)
    if manifest_errors and not args.no_thresholds:
        raise SystemExit("Invalid threshold manifest:\n- " + "\n- ".join(manifest_errors))
    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)
    if failures:
        baseline = load_json(args.baseline)
        old = {record["id"]: record for record in baseline["cases"]}
        suspect = [record for record in report["cases"] if old.get(record["id"]) != record]

View on GitHub (pinned to a38d04c3d5)

Solutions

Read the listed per-case errors — each line names the exact violation; fix those fields in relevance-cases.json.
Cross-check every case's domain against CSV_CONFIG and every stack against AVAILABLE_STACKS in the current code.
Ensure each case has required keys (id, query, split among calibration/held_out, expectations) and unique ids.
After CSV renames, sweep the fixture for the old identifiers.

Defensive patterns

Strategy: validation

Validate before calling

# fail fast with actionable output before the full run
from relevance_metrics import validate_fixture, validate_identities

fixture = load_json(args.cases)
errors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)
if errors:
    for e in errors:
        print(e)
    raise SystemExit(1)

Prevention

After renaming domains/stacks in CSVs, grep the fixtures for the old identifiers in the same change.
Add a fixture schema check to CI independent of the full evaluation run.
When copy-pasting cases, always update id, split, and the expected set.

When it happens

Trigger: A relevance-cases.json case using a domain that no longer exists in the CSV config (e.g. a renamed domain), a stack not in AVAILABLE_STACKS, missing required fields, duplicate case ids, or an expected-id set that violates the identity rules in validate_identities().

Common situations: Domains/stacks renamed in the CSVs without updating fixtures; adding new cases by copy-paste and leaving placeholder values; the fixture and code at different versions after a partial pull.

Related errors

AI-assisted analysis of nextlevelbuilder/ui-ux-pro-max-skill@a38d04c3d5 (2026-08-14). Data as JSON: /api/errors/67e1262a6d2ed46a. Report an issue: GitHub.