{"record":{"id":"67a1285cd0ffde65","repo":"huggingface/open-r1","slug":"invalid-scoring-mode-scoring-mode","errorCode":null,"errorMessage":"Invalid scoring mode: {scoring_mode}","messagePattern":"Invalid scoring mode: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/open_r1/utils/competitive_programming/cf_scoring.py","lineNumber":146,"sourceCode":"            return no_compile_reward\n\n        tests_passed_results = [\n            result and result[\"run\"][\"code\"] == 0 and result[\"run\"][\"stdout\"].strip() == \"1\" for result in results\n        ]\n        if scoring_mode == \"pass_fail\" and any(not test_passed for test_passed in tests_passed_results):\n            break\n        passed_test_cases += sum(1 for test_passed in tests_passed_results if test_passed)\n\n    pass_fail_score = 1.0 if passed_test_cases == len(test_cases) else 0.0\n\n    if scoring_mode == \"pass_fail\":\n        return pass_fail_score\n    elif scoring_mode == \"partial\":\n        return passed_test_cases / len(test_cases)\n    elif scoring_mode == \"weighted_sum\":\n        return pass_fail_score + 0.1 * (passed_test_cases / len(test_cases))\n    else:\n        raise ValueError(f\"Invalid scoring mode: {scoring_mode}\")\n","sourceCodeStart":128,"sourceCodeEnd":147,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/utils/competitive_programming/cf_scoring.py#L128-L147","documentation":"At the end of score_submission, the scoring_mode literal is dispatched: pass_fail, partial, or weighted_sum. Any other value reaches the final else and raises ValueError. The type hint is Literal[...] but Python doesn't enforce it at runtime, so invalid strings only fail here.","triggerScenarios":"Calling score_submission with scoring_mode set to an unhandled string — e.g. \"Pass_Fail\", \"partial_credit\", \"binary\", \"all_or_nothing\", or a value read from config/dataset metadata — instead of one of \"pass_fail\" | \"partial\" | \"weighted_sum\".","commonSituations":"Typos or casing mismatches in YAML/CLI configs; renaming a scoring mode across library versions while old configs persist; passing a non-string (e.g. an enum whose .name differs from the expected value).","solutions":["Use one of the exact strings: \"pass_fail\", \"partial\", or \"weighted_sum\"","Fix casing in the config (matching is case-sensitive)","Check the library version's supported modes if migrating from a fork/older release","Validate scoring_mode against a Literal/Enum at config-load time to fail earlier"],"exampleFix":"// before\nreward = await score_submission(problem, sub, scoring_mode=\"partial_credit\")  # ValueError\n// after\nreward = await score_submission(problem, sub, scoring_mode=\"partial\")","handlingStrategy":"validation","validationCode":"from typing import Literal\nScoringMode = Literal[\"pass_fail\", \"partial\", \"weighted_sum\"]\n\ndef validate_scoring_mode(mode: str) -> str:\n    if mode not in (\"pass_fail\", \"partial\", \"weighted_sum\"):\n        raise SystemExit(f\"scoring_mode must be 'pass_fail', 'partial', or 'weighted_sum', got {mode!r}\")\n    return mode","typeGuard":"def is_valid_scoring_mode(mode) -> bool:\n    return isinstance(mode, str) and mode in (\"pass_fail\", \"partial\", \"weighted_sum\")","tryCatchPattern":"try:\n    reward = await score_submission(problem, submission, scoring_mode=mode)\nexcept ValueError as e:\n    if str(e).startswith(\"Invalid scoring mode\"):\n        logger.error(\"%s — use 'pass_fail', 'partial', or 'weighted_sum'\", e)\n        raise SystemExit(1) from e\n    raise","preventionTips":["Annotate configs with the Literal type and validate at parse time (pydantic/argparse choices)","Use lowercase exact strings; matching is case-sensitive","Grep configs for scoring_mode when upgrading between library versions","Add a unit test that iterates allowed modes through score_submission with a stub problem"],"tags":["python","validation","invalid-argument","scoring"],"backgroundTag":"invalid-enum-value","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}