{"record":{"id":"1467c20f01dfa315","repo":"huggingface/open-r1","slug":"unknown-benchmark-benchmark","errorCode":null,"errorMessage":"Unknown benchmark {benchmark}","messagePattern":"Unknown benchmark (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/open_r1/utils/evaluation.py","lineNumber":118,"sourceCode":"        prompt_encoded = base64.b64encode(training_args.system_prompt.encode()).decode()\n        cmd_args.append(prompt_encoded)\n    cmd[-1] += \" \" + \" \".join(cmd_args)\n    subprocess.run(cmd, check=True)\n\n\ndef run_benchmark_jobs(training_args: Union[\"SFTConfig\", \"GRPOConfig\"], model_args: \"ModelConfig\") -> None:\n    benchmarks = training_args.benchmarks\n    if len(benchmarks) == 1 and benchmarks[0] == \"all\":\n        benchmarks = get_lighteval_tasks()\n        # Evaluate on all supported benchmarks. Later we may want to include a `chat` option\n        # that just evaluates on `ifeval` and `mt_bench` etc.\n\n    for benchmark in benchmarks:\n        print(f\"Launching benchmark `{benchmark}`\")\n        if benchmark in get_lighteval_tasks():\n            run_lighteval_job(benchmark, training_args, model_args)\n        else:\n            raise ValueError(f\"Unknown benchmark {benchmark}\")\n","sourceCodeStart":100,"sourceCodeEnd":119,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/utils/evaluation.py#L100-L119","documentation":"run_benchmark_jobs raises ValueError('Unknown benchmark {benchmark}') when a requested benchmark name is not found in get_lighteval_tasks(). Only benchmarks resolvable to a LightEval task set can be launched; anything else is rejected before job submission. This prevents silently launching empty/invalid evaluation jobs on the cluster.","triggerScenarios":"Calling run_benchmark_jobs(benchmarks=[...]) or --benchmark <name> with a name that get_lighteval_tasks() doesn't contain — misspelled task, custom benchmark not registered in LightEval, or a task added in a newer lighteval version than installed.","commonSituations":"Typo in benchmark name (e.g. 'mmlu' vs the exact lighteval task id), copying a benchmark name from another repo's docs, stale lighteval package missing recently added tasks, custom task suite not installed in the environment.","solutions":["List valid benchmarks via get_lighteval_tasks() (or lighteval's task registry) and use an exact name","Fix the typo / casing in the --benchmark argument","pip install -U lighteval if the benchmark exists only in newer versions","Register your custom task suite with lighteval before referencing it","Validate benchmark names against the registry before submitting jobs"],"exampleFix":"// before\nrun_benchmark_jobs(benchmarks=[\"MMLU\"], training_args=ta, model_args=ma)\n// after\nrun_benchmark_jobs(benchmarks=[\"lighteval|mmlu\"], training_args=ta, model_args=ma)  # exact task id from get_lighteval_tasks()","handlingStrategy":"validation","validationCode":"from open_r1.utils.evaluation import get_lighteval_tasks\n\ndef validate_benchmarks(benchmarks):\n    known = get_lighteval_tasks()\n    unknown = [b for b in benchmarks if b not in known]\n    if unknown:\n        raise ValueError(f\"Unknown benchmarks: {unknown}. Valid: {sorted(known)}\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Copy benchmark names directly from get_lighteval_tasks() output","Keep lighteval up to date so newly added tasks resolve","Validate --benchmark values before submitting expensive SLURM/cluster jobs","Maintain a checked-in list of approved benchmark ids for your project"],"tags":["benchmark","configuration","lighteval","validation"],"backgroundTag":"unknown-benchmark-name","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}