What causes "Unknown benchmark {benchmark}" in huggingface/open-r1?

List valid benchmarks via get_lighteval_tasks() (or lighteval's task registry) and use an exact name

Unknown benchmark {benchmark}

Error message

Unknown benchmark {benchmark}

What it means

run_benchmark_jobs raises ValueError('Unknown benchmark {benchmark}') when a requested benchmark name is not found in get_lighteval_tasks(). Only benchmarks resolvable to a LightEval task set can be launched; anything else is rejected before job submission. This prevents silently launching empty/invalid evaluation jobs on the cluster.

Source

Thrown at src/open_r1/utils/evaluation.py:118

        prompt_encoded = base64.b64encode(training_args.system_prompt.encode()).decode()
        cmd_args.append(prompt_encoded)
    cmd[-1] += " " + " ".join(cmd_args)
    subprocess.run(cmd, check=True)


def run_benchmark_jobs(training_args: Union["SFTConfig", "GRPOConfig"], model_args: "ModelConfig") -> None:
    benchmarks = training_args.benchmarks
    if len(benchmarks) == 1 and benchmarks[0] == "all":
        benchmarks = get_lighteval_tasks()
        # Evaluate on all supported benchmarks. Later we may want to include a `chat` option
        # that just evaluates on `ifeval` and `mt_bench` etc.

    for benchmark in benchmarks:
        print(f"Launching benchmark `{benchmark}`")
        if benchmark in get_lighteval_tasks():
            run_lighteval_job(benchmark, training_args, model_args)
        else:
            raise ValueError(f"Unknown benchmark {benchmark}")

View on GitHub (pinned to 1416fa0cf2)

Solutions

List valid benchmarks via get_lighteval_tasks() (or lighteval's task registry) and use an exact name
Fix the typo / casing in the --benchmark argument
pip install -U lighteval if the benchmark exists only in newer versions
Register your custom task suite with lighteval before referencing it
Validate benchmark names against the registry before submitting jobs

Example fix

// before
run_benchmark_jobs(benchmarks=["MMLU"], training_args=ta, model_args=ma)
// after
run_benchmark_jobs(benchmarks=["lighteval|mmlu"], training_args=ta, model_args=ma)  # exact task id from get_lighteval_tasks()

Defensive patterns

Strategy: validation

Validate before calling

from open_r1.utils.evaluation import get_lighteval_tasks

def validate_benchmarks(benchmarks):
    known = get_lighteval_tasks()
    unknown = [b for b in benchmarks if b not in known]
    if unknown:
        raise ValueError(f"Unknown benchmarks: {unknown}. Valid: {sorted(known)}")

Prevention

Copy benchmark names directly from get_lighteval_tasks() output
Keep lighteval up to date so newly added tasks resolve
Validate --benchmark values before submitting expensive SLURM/cluster jobs
Maintain a checked-in list of approved benchmark ids for your project

When it happens

Trigger: Calling run_benchmark_jobs(benchmarks=[...]) or --benchmark <name> with a name that get_lighteval_tasks() doesn't contain — misspelled task, custom benchmark not registered in LightEval, or a task added in a newer lighteval version than installed.

Common situations: Typo in benchmark name (e.g. 'mmlu' vs the exact lighteval task id), copying a benchmark name from another repo's docs, stale lighteval package missing recently added tasks, custom task suite not installed in the environment.

Related errors

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/1467c20f01dfa315. Report an issue: GitHub.