huggingface/open-r1 · error

All verification_info must have the same language

Error message

All verification_info must have the same language

What it means

code_reward evaluates model code completions against verification_info entries (test cases with a 'language' field, e.g. 'python', 'javascript'). When enforce_same_language is true, it verifies every entry matches the first entry's language and raises this ValueError on any mismatch, because a single execution provider cannot run mixed-language sandboxes.

Source

Thrown at src/open_r1/rewards.py:584

    evaluate_code(code_snippet, test_cases)
    """

    code_snippets = [extract_code(completion[-1]["content"]) for completion in completions]
    verification_info = kwargs["verification_info"]

    template = evaluation_script_template

    scripts = [
        template.format(code=json.dumps(code), test_cases=json.dumps(json.dumps(info["test_cases"])))
        for code, info in zip(code_snippets, verification_info)
    ]

    language = verification_info[0]["language"]

    if enforce_same_language:
        all_same_language = all(v["language"] == language for v in verification_info)
        if not all_same_language:
            raise ValueError("All verification_info must have the same language", verification_info)

    execution_provider = get_provider(
        provider_type=provider_type,
        num_parallel=num_parallel,
        **kwargs,
    )

    return execution_provider.execute_scripts(scripts, ["python"] * len(scripts))


def get_code_format_reward(language: str = "python"):
    """Format reward function specifically for code responses.

    Args:
        language: Programming language supported by E2B https://e2b.dev/docs/code-interpreting/supported-languages
    """

    def code_format_reward(completions, **kwargs):

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Split your evaluation batch by language and call code_reward once per language group.
2. Ensure every verification_info dict has a correct, consistent 'language' value.
3. Set enforce_same_language=False only if your execution provider genuinely supports the mixed languages.

Example fix

// before
rewards = code_reward(completions, verification_info=[{"language": "python"}, {"language": "js"}])
// after
py = [i for i in info if i["language"] == "python"]; js = [i for i in info if i["language"] == "js"]
rewards_py = code_reward(comps_py, verification_info=py); rewards_js = code_reward(comps_js, verification_info=js)

langs = {v.get("language") for v in verification_info}
if enforce_same_language and len(langs) != 1:
    raise SystemExit(f"Mixed verification languages: {langs}; batch by language first")

def same_language(info) -> bool:
    return len({v.get("language") for v in info}) <= 1 and all("language" in v for v in info)

try:
    rewards = code_reward(completions=completions, verification_info=info)
except ValueError as e:
    if "same language" in str(e):
        rewards = []
        for lang in {v["language"] for v in info}:
            idx = [i for i, v in enumerate(info) if v["language"] == lang]
            rewards += code_reward(completions=[completions[i] for i in idx], verification_info=[info[i] for i in idx])
    else:
        raise

When it happens

Trigger: Calling code_reward / get_code_format_reward pipeline with a batch where verification_info[0]['language']='python' but some other entry says 'javascript' or is missing 'language', with enforce_same_language=True (default).

Common situations: Mixed-language coding datasets (e.g. MultiPLE-style) fed to the code reward; prompts from different languages batched together; verification_info built with inconsistent or missing 'language' keys.

Related errors

• max_penalty {max_penalty} should not be positive
• Word splitting for language `{language}` is not yet implemen
• Either `dataset_name` or `dataset_mixture` must be provided
• Please install jieba to use Chinese language
• Invalid submission language: {submission_language}

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/02f0cea3db8aa3d2. Report an issue: GitHub.