huggingface/open-r1 · error

Invalid scoring mode: {scoring_mode}

Error message

Invalid scoring mode: {scoring_mode}

What it means

At the end of score_submission, the scoring_mode literal is dispatched: pass_fail, partial, or weighted_sum. Any other value reaches the final else and raises ValueError. The type hint is Literal[...] but Python doesn't enforce it at runtime, so invalid strings only fail here.

Source

Thrown at src/open_r1/utils/competitive_programming/cf_scoring.py:146

            return no_compile_reward

        tests_passed_results = [
            result and result["run"]["code"] == 0 and result["run"]["stdout"].strip() == "1" for result in results
        ]
        if scoring_mode == "pass_fail" and any(not test_passed for test_passed in tests_passed_results):
            break
        passed_test_cases += sum(1 for test_passed in tests_passed_results if test_passed)

    pass_fail_score = 1.0 if passed_test_cases == len(test_cases) else 0.0

    if scoring_mode == "pass_fail":
        return pass_fail_score
    elif scoring_mode == "partial":
        return passed_test_cases / len(test_cases)
    elif scoring_mode == "weighted_sum":
        return pass_fail_score + 0.1 * (passed_test_cases / len(test_cases))
    else:
        raise ValueError(f"Invalid scoring mode: {scoring_mode}")

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. Use one of the exact strings: "pass_fail", "partial", or "weighted_sum"
2. Fix casing in the config (matching is case-sensitive)
3. Check the library version's supported modes if migrating from a fork/older release
4. Validate scoring_mode against a Literal/Enum at config-load time to fail earlier

Example fix

// before
reward = await score_submission(problem, sub, scoring_mode="partial_credit")  # ValueError
// after
reward = await score_submission(problem, sub, scoring_mode="partial")

from typing import Literal
ScoringMode = Literal["pass_fail", "partial", "weighted_sum"]

def validate_scoring_mode(mode: str) -> str:
    if mode not in ("pass_fail", "partial", "weighted_sum"):
        raise SystemExit(f"scoring_mode must be 'pass_fail', 'partial', or 'weighted_sum', got {mode!r}")
    return mode

def is_valid_scoring_mode(mode) -> bool:
    return isinstance(mode, str) and mode in ("pass_fail", "partial", "weighted_sum")

try:
    reward = await score_submission(problem, submission, scoring_mode=mode)
except ValueError as e:
    if str(e).startswith("Invalid scoring mode"):
        logger.error("%s — use 'pass_fail', 'partial', or 'weighted_sum'", e)
        raise SystemExit(1) from e
    raise

When it happens

Trigger: Calling score_submission with scoring_mode set to an unhandled string — e.g. "Pass_Fail", "partial_credit", "binary", "all_or_nothing", or a value read from config/dataset metadata — instead of one of "pass_fail" | "partial" | "weighted_sum".

Common situations: Typos or casing mismatches in YAML/CLI configs; renaming a scoring mode across library versions while old configs persist; passing a non-string (e.g. an enum whose .name differs from the expected value).

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Invalid submission language: {submission_language}
• Either `dataset_name` or `dataset_mixture` must be provided
• max_penalty {max_penalty} should not be positive
• Word splitting for language `{language}` is not yet implemen
• All verification_info must have the same language

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/67a1285cd0ffde65. Report an issue: GitHub.