{"record":{"id":"3bfff467c7999529","repo":"zed-industries/zed","slug":"unknown-judge-preset-name-valid-valid","errorCode":null,"errorMessage":"unknown judge preset '{name}' (valid: {valid})","messagePattern":"unknown judge preset '(.+?)' \\(valid: (.+?)\\)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"crates/eval_cli/zed_eval/config.py","lineNumber":152,"sourceCode":"    \"baseten:deepseek-v4-pro\": \"baseten/deepseek-ai/DeepSeek-V4-Pro\",\n    \"baseten:deepseek\": \"baseten/deepseek-ai/DeepSeek-V4-Pro\",\n}\n\n\ndef orchestration_info() -> dict[str, str]:\n    return {\n        \"controller_image_recipe_version\": CONTROLLER_IMAGE_RECIPE_VERSION,\n        \"harbor_version\": HARBOR_VERSION,\n        \"modal_client_version\": MODAL_VERSION,\n    }\n\n\ndef get_judge(name: str) -> JudgeConfig:\n    try:\n        return JUDGES[name]\n    except KeyError as error:\n        valid = \", \".join(sorted(JUDGES))\n        raise ValueError(f\"unknown judge preset '{name}' (valid: {valid})\") from error\n\n\ndef canonical_part(part: str) -> str:\n    normalized = part.strip().lower()\n    if normalized in (\"qna\", \"rf\", \"tw\"):\n        return normalized\n    valid = \", \".join((\"qna\", \"rf\", \"tw\"))\n    raise ValueError(f\"unknown SWE-Atlas part '{part}' (valid: {valid})\")\n\n\ndef resolve_model_preset(model: str) -> str:\n    return MODEL_PRESETS.get(model, model)\n","sourceCodeStart":134,"sourceCodeEnd":165,"githubUrl":"https://github.com/zed-industries/zed/blob/bc538def4545534201bbfcac4e95ac34ea6501b6/crates/eval_cli/zed_eval/config.py#L134-L165","documentation":"Raised by config.get_judge() when the requested judge preset name is not a key in the JUDGES dict (crates/eval_cli/zed_eval/config.py:84-124). The lookup is case-sensitive, and the error message lists every valid preset (deepseek, deepseek-v4-pro, gpt55, kimi, kimi-k2.7-code, leaderboard, opus-leaderboard). Validation runs eagerly at launch time (resolve_benchmark_judge, build_rejudge_request) and again when the controller builds the harness command (harness_command.py:157), so no remote Modal work starts with a bad judge name.","triggerScenarios":"Passing --judge with a typo or wrong case (e.g. --judge Kimi or --judge kimi-k2.7) to zed-eval run/rejudge/swe-atlas; setting judge_preset by hand in a programmatically built run request dict; re-running an old script after a preset was renamed or removed from JUDGES in a newer zed_eval.","commonSituations":"Case mismatch is the classic hit: SWE-Atlas part names and model presets tolerate case/whitespace, but judge names do not. Also stale shell history or CI scripts referencing presets that no longer exist after upgrading eval_cli, or copying a judge name from an outdated README.","solutions":["Re-run with a preset name taken verbatim from the error message's valid list (exact case, e.g. --judge kimi-k2.7-code).","Open crates/eval_cli/zed_eval/config.py and check the JUDGES dict for the current canonical names.","If you must keep an old name in a script, add an alias entry to JUDGES (like the existing deepseek/deepseek-v4-pro pairs) and redeploy the Modal app.","For a genuinely new judge, add a JudgeConfig(model, upstream, auth_env, max_tokens) entry to JUDGES and run zed-eval deploy."],"exampleFix":"# before\nzed-eval run rf --judge Kimi-K2.7-Code\n# -> ValueError: unknown judge preset 'Kimi-K2.7-Code' (valid: ...)\n\n# after\nzed-eval run rf --judge kimi-k2.7-code","handlingStrategy":"validation","validationCode":"from zed_eval import config\n\nif judge_name not in config.JUDGES:\n    raise SystemExit(\n        f\"unknown judge '{judge_name}'; valid: {', '.join(sorted(config.JUDGES))}\"\n    )\njudge = config.get_judge(judge_name)","typeGuard":"def is_valid_judge(name: str) -> bool:\n    return isinstance(name, str) and name in config.JUDGES","tryCatchPattern":"try:\n    judge = config.get_judge(args.judge)\nexcept ValueError as error:\n    raise SystemExit(str(error))  # message already lists valid presets","preventionTips":["Copy preset names verbatim from the error's valid list; matching is case-sensitive.","Validate --judge before launching so no Modal build/controller work is spawned for a typo.","Re-check pinned preset names in scripts after upgrading zed_eval (JUDGES changes over time)."],"tags":["config","validation","llm-judge","cli"],"backgroundTag":null,"analyzedSha":"bc538def4545534201bbfcac4e95ac34ea6501b6","analyzedAt":"2026-08-16T07:30:46.435Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}