zed-industries/zed · error · ValueError

unknown judge preset '{name}' (valid: {valid})

Error message

unknown judge preset '{name}' (valid: {valid})

What it means

Raised by config.get_judge() when the requested judge preset name is not a key in the JUDGES dict (crates/eval_cli/zed_eval/config.py:84-124). The lookup is case-sensitive, and the error message lists every valid preset (deepseek, deepseek-v4-pro, gpt55, kimi, kimi-k2.7-code, leaderboard, opus-leaderboard). Validation runs eagerly at launch time (resolve_benchmark_judge, build_rejudge_request) and again when the controller builds the harness command (harness_command.py:157), so no remote Modal work starts with a bad judge name.

Source

Thrown at crates/eval_cli/zed_eval/config.py:152

    "baseten:deepseek-v4-pro": "baseten/deepseek-ai/DeepSeek-V4-Pro",
    "baseten:deepseek": "baseten/deepseek-ai/DeepSeek-V4-Pro",
}


def orchestration_info() -> dict[str, str]:
    return {
        "controller_image_recipe_version": CONTROLLER_IMAGE_RECIPE_VERSION,
        "harbor_version": HARBOR_VERSION,
        "modal_client_version": MODAL_VERSION,
    }


def get_judge(name: str) -> JudgeConfig:
    try:
        return JUDGES[name]
    except KeyError as error:
        valid = ", ".join(sorted(JUDGES))
        raise ValueError(f"unknown judge preset '{name}' (valid: {valid})") from error


def canonical_part(part: str) -> str:
    normalized = part.strip().lower()
    if normalized in ("qna", "rf", "tw"):
        return normalized
    valid = ", ".join(("qna", "rf", "tw"))
    raise ValueError(f"unknown SWE-Atlas part '{part}' (valid: {valid})")


def resolve_model_preset(model: str) -> str:
    return MODEL_PRESETS.get(model, model)

View on GitHub (pinned to bc538def45)

Solutions

1. Re-run with a preset name taken verbatim from the error message's valid list (exact case, e.g. --judge kimi-k2.7-code).
2. Open crates/eval_cli/zed_eval/config.py and check the JUDGES dict for the current canonical names.
3. If you must keep an old name in a script, add an alias entry to JUDGES (like the existing deepseek/deepseek-v4-pro pairs) and redeploy the Modal app.
4. For a genuinely new judge, add a JudgeConfig(model, upstream, auth_env, max_tokens) entry to JUDGES and run zed-eval deploy.

Example fix

# before
zed-eval run rf --judge Kimi-K2.7-Code
# -> ValueError: unknown judge preset 'Kimi-K2.7-Code' (valid: ...)

# after
zed-eval run rf --judge kimi-k2.7-code

from zed_eval import config

if judge_name not in config.JUDGES:
    raise SystemExit(
        f"unknown judge '{judge_name}'; valid: {', '.join(sorted(config.JUDGES))}"
    )
judge = config.get_judge(judge_name)

def is_valid_judge(name: str) -> bool:
    return isinstance(name, str) and name in config.JUDGES

try:
    judge = config.get_judge(args.judge)
except ValueError as error:
    raise SystemExit(str(error))  # message already lists valid presets

When it happens

Trigger: Passing --judge with a typo or wrong case (e.g. --judge Kimi or --judge kimi-k2.7) to zed-eval run/rejudge/swe-atlas; setting judge_preset by hand in a programmatically built run request dict; re-running an old script after a preset was renamed or removed from JUDGES in a newer zed_eval.

Common situations: Case mismatch is the classic hit: SWE-Atlas part names and model presets tolerate case/whitespace, but judge names do not. Also stale shell history or CI scripts referencing presets that no longer exist after upgrading eval_cli, or copying a judge name from an outdated README.

Related errors

• unknown benchmark '{benchmark_id}' (valid: {valid})
• unknown benchmark '{selector}' (valid: {valid})
• unknown SWE-Atlas part '{part}' (valid: {valid})
• registry dataset requires a name
• path dataset requires data_dir

AI-assisted analysis of zed-industries/zed@bc538def45 (2026-08-16). Data as JSON: /api/errors/3bfff467c7999529. Report an issue: GitHub.