{"record":{"id":"02f0cea3db8aa3d2","repo":"huggingface/open-r1","slug":"all-verification-info-must-have-the-same-language","errorCode":null,"errorMessage":"All verification_info must have the same language","messagePattern":"All verification_info must have the same language","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/open_r1/rewards.py","lineNumber":584,"sourceCode":"    evaluate_code(code_snippet, test_cases)\n    \"\"\"\n\n    code_snippets = [extract_code(completion[-1][\"content\"]) for completion in completions]\n    verification_info = kwargs[\"verification_info\"]\n\n    template = evaluation_script_template\n\n    scripts = [\n        template.format(code=json.dumps(code), test_cases=json.dumps(json.dumps(info[\"test_cases\"])))\n        for code, info in zip(code_snippets, verification_info)\n    ]\n\n    language = verification_info[0][\"language\"]\n\n    if enforce_same_language:\n        all_same_language = all(v[\"language\"] == language for v in verification_info)\n        if not all_same_language:\n            raise ValueError(\"All verification_info must have the same language\", verification_info)\n\n    execution_provider = get_provider(\n        provider_type=provider_type,\n        num_parallel=num_parallel,\n        **kwargs,\n    )\n\n    return execution_provider.execute_scripts(scripts, [\"python\"] * len(scripts))\n\n\ndef get_code_format_reward(language: str = \"python\"):\n    \"\"\"Format reward function specifically for code responses.\n\n    Args:\n        language: Programming language supported by E2B https://e2b.dev/docs/code-interpreting/supported-languages\n    \"\"\"\n\n    def code_format_reward(completions, **kwargs):","sourceCodeStart":566,"sourceCodeEnd":602,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/rewards.py#L566-L602","documentation":"code_reward evaluates model code completions against verification_info entries (test cases with a 'language' field, e.g. 'python', 'javascript'). When enforce_same_language is true, it verifies every entry matches the first entry's language and raises this ValueError on any mismatch, because a single execution provider cannot run mixed-language sandboxes.","triggerScenarios":"Calling code_reward / get_code_format_reward pipeline with a batch where verification_info[0]['language']='python' but some other entry says 'javascript' or is missing 'language', with enforce_same_language=True (default).","commonSituations":"Mixed-language coding datasets (e.g. MultiPLE-style) fed to the code reward; prompts from different languages batched together; verification_info built with inconsistent or missing 'language' keys.","solutions":["Split your evaluation batch by language and call code_reward once per language group.","Ensure every verification_info dict has a correct, consistent 'language' value.","Set enforce_same_language=False only if your execution provider genuinely supports the mixed languages."],"exampleFix":"// before\nrewards = code_reward(completions, verification_info=[{\"language\": \"python\"}, {\"language\": \"js\"}])\n// after\npy = [i for i in info if i[\"language\"] == \"python\"]; js = [i for i in info if i[\"language\"] == \"js\"]\nrewards_py = code_reward(comps_py, verification_info=py); rewards_js = code_reward(comps_js, verification_info=js)","handlingStrategy":"validation","validationCode":"langs = {v.get(\"language\") for v in verification_info}\nif enforce_same_language and len(langs) != 1:\n    raise SystemExit(f\"Mixed verification languages: {langs}; batch by language first\")","typeGuard":"def same_language(info) -> bool:\n    return len({v.get(\"language\") for v in info}) <= 1 and all(\"language\" in v for v in info)","tryCatchPattern":"try:\n    rewards = code_reward(completions=completions, verification_info=info)\nexcept ValueError as e:\n    if \"same language\" in str(e):\n        rewards = []\n        for lang in {v[\"language\"] for v in info}:\n            idx = [i for i, v in enumerate(info) if v[\"language\"] == lang]\n            rewards += code_reward(completions=[completions[i] for i in idx], verification_info=[info[i] for i in idx])\n    else:\n        raise","preventionTips":["Group completions by language before code reward evaluation","Always include a correct 'language' key in every verification_info dict","Enforce a per-dataset language field at data-prep time"],"tags":["python","validation","rewards","code-evaluation"],"backgroundTag":"inconsistent-batch-language","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}