huggingface/open-r1 · error

Invalid submission language: {submission_language}

Error message

Invalid submission language: {submission_language}

What it means

score_single_test_case only supports "python" and "cpp" submissions; any other submission_language raises ValueError before attempting remote execution. The language also determines the file extension (main.{language}) sent to the execution backend, so unsupported values cannot proceed.

Source

Thrown at src/open_r1/utils/competitive_programming/cf_scoring.py:21

from io import BytesIO
from typing import Literal

from async_lru import alru_cache

from .piston_client import PistonClient
from .utils import batched


async def score_single_test_case(
    client: PistonClient,
    problem_data: dict,
    test_input: str,
    test_output: str,
    submission: str,
    submission_language: str = "cpp",
) -> tuple[str, str]:
    if submission_language not in ["python", "cpp"]:
        raise ValueError(f"Invalid submission language: {submission_language}")
    try:
        result = await client.send_execute(
            {
                "files": [
                    {"name": f"main.{submission_language}", "content": submission},
                    *(
                        [{"name": "checker.py", "content": problem_data["generated_checker"]}]
                        if problem_data["generated_checker"]
                        else []
                    ),
                    {"name": "input.txt", "content": test_input},
                    {"name": "correct_output.txt", "content": test_output},
                    {
                        "name": "grader_config",
                        "content": "\n".join(
                            f"{key}={value}"
                            for key, value in {
                                "TIME_LIMIT": problem_data["time_limit"],

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Set submission_language to "python" or "cpp" (exact strings)
2. Normalize dataset language labels before scoring (map "python3"/"py" -> "python")
3. Filter or skip samples whose language isn't supported
4. Extend the allowed list in cf_scoring.py if the execution backend actually supports more languages

Example fix

// before
score = await score_single_test_case(case, inp, out, sub, "python3")  # ValueError
// after
lang = "python" if submission_language.startswith("py") else "cpp"
score = await score_single_test_case(case, inp, out, sub, lang)

ALLOWED_LANGS = {"python", "cpp"}
LANG_ALIASES = {"py": "python", "python3": "python", "c++": "cpp", "cxx": "cpp"}

def normalize_lang(lang: str) -> str:
    lang = LANG_ALIASES.get(lang.lower(), lang.lower())
    if lang not in ALLOWED_LANGS:
        raise SystemExit(f"Unsupported submission language {lang!r}; use 'python' or 'cpp'")
    return lang

def is_supported_lang(lang) -> bool:
    return isinstance(lang, str) and lang in ("python", "cpp")

try:
    result = await score_single_test_case(client, case, inp, out, sub, lang)
except ValueError as e:
    if str(e).startswith("Invalid submission language"):
        logger.error("%s — normalize language to 'python' or 'cpp'", e)
        return None
    raise

When it happens

Trigger: Calling score_single_test_case (directly or via score_submission) with submission_language set to anything outside ["python", "cpp"] — e.g. "py", "python3", "java", "rust", or a None passed through from config.

Common situations: Configuring a dataset with a language column containing "python3" or "py3"; attempting to score other Codeforces languages (Java, Rust); a pipeline default of "cpp" being overridden with a raw language name from dataset metadata.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Invalid scoring mode: {scoring_mode}
• Either `dataset_name` or `dataset_mixture` must be provided
• max_penalty {max_penalty} should not be positive
• Word splitting for language `{language}` is not yet implemen
• All verification_info must have the same language

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/e88b7b6a90524c23. Report an issue: GitHub.