{"record":{"id":"dcfc2ceb4cb3d177","repo":"huggingface/open-r1","slug":"word-splitting-for-language-language-is-not-ye","errorCode":null,"errorMessage":"Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function.","messagePattern":"Word splitting for language `(.+?)` is not yet implemented\\. Please implement your own zip-ngram function\\.","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/open_r1/rewards.py","lineNumber":317,"sourceCode":"\n        def zipngram(text: str, ngram_size: int):\n            words = text.lower().split()\n            return zip(*[words[i:] for i in range(ngram_size)]), words\n\n    elif language == \"zh\":\n        from transformers.utils.import_utils import _is_package_available\n\n        if not _is_package_available(\"jieba\"):\n            raise ValueError(\"Please install jieba to use Chinese language\")\n\n        def zipngram(text: str, ngram_size: int):\n            import jieba\n\n            seg_list = list(jieba.cut(text))\n            return zip(*[seg_list[i:] for i in range(ngram_size)]), seg_list\n\n    else:\n        raise ValueError(\n            f\"Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function.\"\n        )\n\n    def repetition_penalty_reward(completions, **kwargs) -> float:\n        \"\"\"\n        reward function the penalizes repetitions\n        ref implementation: https://github.com/eddycmu/demystify-long-cot/blob/release/openrlhf/openrlhf/reward/repetition.py\n\n        Args:\n            completions: List of model completions\n        \"\"\"\n\n        contents = [completion[0][\"content\"] for completion in completions]\n        rewards = []\n        for completion in contents:\n            if completion == \"\":\n                rewards.append(0.0)\n                continue","sourceCodeStart":299,"sourceCodeEnd":335,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/rewards.py#L299-L335","documentation":"get_repetition_penalty_reward only implements word segmentation for 'en' (whitespace split) and 'zh' (jieba). Any other language string falls through to the else branch and raises this ValueError telling you to supply your own zip-ngram tokenizer.","triggerScenarios":"get_repetition_penalty_reward(..., language=\"fr\"|\"de\"|\"ja\"|any unsupported code).","commonSituations":"Training multilingual/other-language models and assuming more languages are supported; passing a locale like 'en_US' or 'english' instead of 'en'; typo in the language code.","solutions":["Use language=\"en\" or \"zh\" (exact lowercase codes).","For another language, copy the function and add an elif branch with your own segmentation in zipngram.","Normalize your language config value to the supported codes (e.g. strip region suffixes)."],"exampleFix":"// before\nreward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language=\"fr\")\n// after\nreward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language=\"en\")","handlingStrategy":"validation","validationCode":"SUPPORTED = {\"en\", \"zh\"}\nif language not in SUPPORTED:\n    raise SystemExit(f\"language={language!r} unsupported; use one of {sorted(SUPPORTED)}\")","typeGuard":"def is_supported_rep_penalty_language(lang: str) -> bool:\n    return lang in {\"en\", \"zh\"}","tryCatchPattern":"try:\n    reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language=lang)\nexcept ValueError as e:\n    if \"not yet implemented\" in str(e):\n        reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language=\"en\")\n    else:\n        raise","preventionTips":["Normalize locale strings ('en_US' -> 'en') before passing language","Extend the function with your own zipngram for other languages","Keep a registry of supported language codes in config"],"tags":["python","validation","rewards","i18n"],"backgroundTag":"unsupported-language-code","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}