Word splitting for language `{language}` is not yet implemen

Error message

Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function.

What it means

get_repetition_penalty_reward only implements word segmentation for 'en' (whitespace split) and 'zh' (jieba). Any other language string falls through to the else branch and raises this ValueError telling you to supply your own zip-ngram tokenizer.

Source

Thrown at src/open_r1/rewards.py:317

        def zipngram(text: str, ngram_size: int):
            words = text.lower().split()
            return zip(*[words[i:] for i in range(ngram_size)]), words

    elif language == "zh":
        from transformers.utils.import_utils import _is_package_available

        if not _is_package_available("jieba"):
            raise ValueError("Please install jieba to use Chinese language")

        def zipngram(text: str, ngram_size: int):
            import jieba

            seg_list = list(jieba.cut(text))
            return zip(*[seg_list[i:] for i in range(ngram_size)]), seg_list

    else:
        raise ValueError(
            f"Word splitting for language `{language}` is not yet implemented. Please implement your own zip-ngram function."
        )

    def repetition_penalty_reward(completions, **kwargs) -> float:
        """
        reward function the penalizes repetitions
        ref implementation: https://github.com/eddycmu/demystify-long-cot/blob/release/openrlhf/openrlhf/reward/repetition.py

        Args:
            completions: List of model completions
        """

        contents = [completion[0]["content"] for completion in completions]
        rewards = []
        for completion in contents:
            if completion == "":
                rewards.append(0.0)
                continue

View on GitHub (pinned to 1416fa0cf2)

Solutions

Use language="en" or "zh" (exact lowercase codes).
For another language, copy the function and add an elif branch with your own segmentation in zipngram.
Normalize your language config value to the supported codes (e.g. strip region suffixes).

Example fix

// before
reward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language="fr")
// after
reward = get_repetition_penalty_reward(ngram_size=2, max_penalty=-1.0, language="en")

Defensive patterns

Strategy: validation

Validate before calling

SUPPORTED = {"en", "zh"}
if language not in SUPPORTED:
    raise SystemExit(f"language={language!r} unsupported; use one of {sorted(SUPPORTED)}")

Type guard

def is_supported_rep_penalty_language(lang: str) -> bool:
    return lang in {"en", "zh"}

Try / catch

try:
    reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language=lang)
except ValueError as e:
    if "not yet implemented" in str(e):
        reward = get_repetition_penalty_reward(ngram_size=n, max_penalty=p, language="en")
    else:
        raise

Prevention

Normalize locale strings ('en_US' -> 'en') before passing language
Extend the function with your own zipngram for other languages
Keep a registry of supported language codes in config

When it happens

Trigger: get_repetition_penalty_reward(..., language="fr"|"de"|"ja"|any unsupported code).

Common situations: Training multilingual/other-language models and assuming more languages are supported; passing a locale like 'en_US' or 'english' instead of 'en'; typo in the language code.

Related errors

AI-assisted analysis of huggingface/open-r1@1416fa0cf2 (2026-08-30). Data as JSON: /api/errors/dcfc2ceb4cb3d177. Report an issue: GitHub.