nextlevelbuilder/ui-ux-pro-max-skill · error · SystemExit

Invalid threshold manifest:\n- {manifest_errors}

Error message

Invalid threshold manifest:\n- {manifest_errors}

What it means

After evaluation, the thresholds manifest (relevance-thresholds.json) is validated against the runtime fingerprint (hash of core.py, design_system.py, reasoning_contract.py, and all CSVs) and the oracle fingerprint of the cases file. Errors mean the manifest was generated for different code/data than what's running — the thresholds can't be trusted for this tree — so the gate refuses rather than compare apples to oranges.

Source

Thrown at scripts/evaluate-relevance.py:239

    fixture = load_json(args.cases)
    fingerprint = runtime_fingerprint()
    oracle = oracle_fingerprint(args.cases)
    errors = validate_fixture(fixture, CSV_CONFIG, AVAILABLE_STACKS) + validate_identities(fixture)
    if errors:
        raise SystemExit("Invalid relevance fixture:\n- " + "\n- ".join(errors))
    selected_split = args.split
    if selected_split != "all":
        fixture = {**fixture, "cases": [case for case in fixture["cases"]
                                        if case["split"] == selected_split]}
    report = evaluate(fixture, oracle)
    if args.write_baseline:
        args.write_baseline.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    print(json.dumps({key: report[key] for key in ("metrics", "samples", "splits")},
                     indent=2, sort_keys=True))
    manifest = load_json(args.thresholds)
    manifest_errors = validate_manifest(manifest, fingerprint, oracle)
    if manifest_errors and not args.no_thresholds:
        raise SystemExit("Invalid threshold manifest:\n- " + "\n- ".join(manifest_errors))
    failures = [] if args.no_thresholds else check_thresholds(report, manifest, selected_split)
    if failures:
        baseline = load_json(args.baseline)
        old = {record["id"]: record for record in baseline["cases"]}
        suspect = [record for record in report["cases"] if old.get(record["id"]) != record]
        if not suspect:
            suspect = [record for record in report["cases"]
                       if (record.get("grades") and record["grades"][0] < 2)
                       or record.get("falsePositives", 0)
                       or ("coherence" in record and not all(record["coherence"]))]
        detail = "\n".join(f"  {item['id']}: {item.get('actual', item)}"
                           for item in suspect)
        raise SystemExit("Relevance gate failed:\n- " + "\n- ".join(failures)
                         + ("\nProblem cases:\n" + detail if detail else ""))


if __name__ == "__main__":
    main()

View on GitHub (pinned to a38d04c3d5)

Solutions

1. Regenerate the manifest with the harness's own generation flow so it records the current fingerprints.
2. If the change is intentional, commit the regenerated relevance-thresholds.json together with the code/CSV changes.
3. If you only want metrics without gating, pass --no-thresholds explicitly — but don't ship that result as a pass.
4. Verify both fingerprints (runtime + oracle) match after regeneration; the error lines tell you which one mismatched.

# detect drift before evaluating
manifest = load_json(args.thresholds)
if manifest.get('runtimeFingerprint') != fingerprint or manifest.get('oracleFingerprint') != oracle:
    raise SystemExit('Threshold manifest out of date - regenerate it for this tree')

When it happens

Trigger: Editing any CSV, core.py, design_system.py, or reasoning_contract.py after the manifest was written; editing relevance-cases.json (changes oracle fingerprint) without regenerating thresholds; checking out a branch whose data differs from the branch that produced the manifest.

Common situations: Data updates merged without re-running threshold generation; a stale manifest committed on one branch and evaluated on another; local CSV tweaks during search-quality experiments.

Related errors

• Relevance gate failed:\n- {failures}
• Invalid JSON file {path}: {error}
• Invalid relevance fixture:\n- {errors}
• verified-at has suspicious date {value!r}
• {name} catalog counts are stale: {', '.join(missing)}

AI-assisted analysis of nextlevelbuilder/ui-ux-pro-max-skill@a38d04c3d5 (2026-08-14). Data as JSON: /api/errors/49c727ca2cb1e05d. Report an issue: GitHub.