huggingface/open-r1 · error

Please install jieba to use Chinese language

Error message

Please install jieba to use Chinese language

What it means

For language='zh', the repetition-penalty reward segments text with jieba, which is an optional dependency not installed with the library. It checks transformers' _is_package_available('jieba') and raises this ValueError if missing.

Source

Thrown at src/open_r1/rewards.py:308

    Args:
    ngram_size: size of the n-grams
    max_penalty: Maximum (negative) penalty for wrong answers
    language: Language of the text, defaults to `en`. Used to choose the way to split the text into n-grams.
    """
    if max_penalty > 0:
        raise ValueError(f"max_penalty {max_penalty} should not be positive")

    if language == "en":

        def zipngram(text: str, ngram_size: int):
            words = text.lower().split()
            return zip(*[words[i:] for i in range(ngram_size)]), words

    elif language == "zh":
        from transformers.utils.import_utils import _is_package_available

        if not _is_package_available("jieba"):
            raise ValueError("Please install jieba to use Chinese language")

        def zipngram(text: str, ngram_size: int):
            import jieba

            seg_list = list(jieba.cut(text))
            return zip(*[seg_list[i:] for i in range(ngram_size)]), seg_list

    else:
        raise ValueError(
            f"Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function."
        )

    def repetition_penalty_reward(completions, **kwargs) -> float:
        """
        reward function the penalizes repetitions
        ref implementation: https://github.com/eddycmu/demystify-long-cot/blob/release/openrlhf/openrlhf/reward/repetition.py

        Args:

View on GitHub (pinned to 1416fa0cf2)

Solutions

1. pip install jieba (add it to requirements/environment).
2. Or use language="en" if your completions are actually English.
3. If jieba is installed but the error persists, verify it is in the same Python env/venv the trainer runs in (pip show jieba).

Example fix

// before
reward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language="zh")  # jieba missing
// after
# pip install jieba
reward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language="zh")

from transformers.utils.import_utils import _is_package_available
if lang == "zh" and not _is_package_available("jieba"):
    raise SystemExit("Install jieba for Chinese repetition penalty: pip install jieba")

def zh_supported(lang: str) -> bool:
    return lang != "zh" or _is_package_available("jieba")

try:
    reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language="zh")
except ValueError as e:
    if "jieba" in str(e):
        logger.warning("jieba missing; falling back to English splitter")
        reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language="en")
    else:
        raise

When it happens

Trigger: get_repetition_penalty_reward(..., language="zh") in an environment where jieba is not pip-installed.

Common situations: Training Chinese models after copying an English reward config and only changing language to 'zh'; minimal Docker images without optional extras; CI environments that install only core requirements.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• max_penalty {max_penalty} should not be positive
• Word splitting for language `{language}` is not yet implemen
• All verification_info must have the same language
• Either `dataset_name` or `dataset_mixture` must be provided
• dataset_mixture must be a dictionary with a 'datasets' key.

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/6b566f3071efd162. Report an issue: GitHub.