{"record":{"id":"6b566f3071efd162","repo":"huggingface/open-r1","slug":"please-install-jieba-to-use-chinese-language","errorCode":null,"errorMessage":"Please install jieba to use Chinese language","messagePattern":"Please install jieba to use Chinese language","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/open_r1/rewards.py","lineNumber":308,"sourceCode":"    Args:\n    ngram_size: size of the n-grams\n    max_penalty: Maximum (negative) penalty for wrong answers\n    language: Language of the text, defaults to `en`. Used to choose the way to split the text into n-grams.\n    \"\"\"\n    if max_penalty > 0:\n        raise ValueError(f\"max_penalty {max_penalty} should not be positive\")\n\n    if language == \"en\":\n\n        def zipngram(text: str, ngram_size: int):\n            words = text.lower().split()\n            return zip(*[words[i:] for i in range(ngram_size)]), words\n\n    elif language == \"zh\":\n        from transformers.utils.import_utils import _is_package_available\n\n        if not _is_package_available(\"jieba\"):\n            raise ValueError(\"Please install jieba to use Chinese language\")\n\n        def zipngram(text: str, ngram_size: int):\n            import jieba\n\n            seg_list = list(jieba.cut(text))\n            return zip(*[seg_list[i:] for i in range(ngram_size)]), seg_list\n\n    else:\n        raise ValueError(\n            f\"Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function.\"\n        )\n\n    def repetition_penalty_reward(completions, **kwargs) -> float:\n        \"\"\"\n        reward function the penalizes repetitions\n        ref implementation: https://github.com/eddycmu/demystify-long-cot/blob/release/openrlhf/openrlhf/reward/repetition.py\n\n        Args:","sourceCodeStart":290,"sourceCodeEnd":326,"githubUrl":"https://github.com/huggingface/open-r1/blob/1416fa0cf21595d2083b399a2a0bbddd7f6e9563/src/open_r1/rewards.py#L290-L326","documentation":"For language='zh', the repetition-penalty reward segments text with jieba, which is an optional dependency not installed with the library. It checks transformers' _is_package_available('jieba') and raises this ValueError if missing.","triggerScenarios":"get_repetition_penalty_reward(..., language=\"zh\") in an environment where jieba is not pip-installed.","commonSituations":"Training Chinese models after copying an English reward config and only changing language to 'zh'; minimal Docker images without optional extras; CI environments that install only core requirements.","solutions":["pip install jieba (add it to requirements/environment).","Or use language=\"en\" if your completions are actually English.","If jieba is installed but the error persists, verify it is in the same Python env/venv the trainer runs in (pip show jieba)."],"exampleFix":"// before\nreward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language=\"zh\")  # jieba missing\n// after\n# pip install jieba\nreward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language=\"zh\")","handlingStrategy":"fallback","validationCode":"from transformers.utils.import_utils import _is_package_available\nif lang == \"zh\" and not _is_package_available(\"jieba\"):\n    raise SystemExit(\"Install jieba for Chinese repetition penalty: pip install jieba\")","typeGuard":"def zh_supported(lang: str) -> bool:\n    return lang != \"zh\" or _is_package_available(\"jieba\")","tryCatchPattern":"try:\n    reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language=\"zh\")\nexcept ValueError as e:\n    if \"jieba\" in str(e):\n        logger.warning(\"jieba missing; falling back to English splitter\")\n        reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language=\"en\")\n    else:\n        raise","preventionTips":["pip install jieba in requirements/Dockerfile when training Chinese models","Pin the same venv for training as for dependency checks","Add an import check to job startup scripts"],"tags":["python","dependency","optional-dependency","rewards"],"backgroundTag":"missing-optional-dependency","analyzedSha":"1416fa0cf21595d2083b399a2a0bbddd7f6e9563","analyzedAt":"2026-08-30T08:56:53.400Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}