huggingface/transformers · error · ValueError

The sum of mask_replace_prob and random_replace_prob should

Error message

The sum of mask_replace_prob and random_replace_prob should not exceed 1

What it means

Raised by DataCollatorForLanguageModeling.__post_init__ when mask_replace_prob + random_replace_prob > 1. The collator splits masked tokens into three disjoint groups (replaced by [MASK], replaced by a random token, left unchanged); if the two replacement probabilities sum above 1 there is no valid remainder, so the configuration is rejected.

Source

Thrown at src/transformers/data/data_collator.py:708

    seed: int | None = None

    def __post_init__(self):
        if self.mlm:
            if self.tokenizer.mask_token is None:
                raise ValueError(
                    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
                    "You should pass `mlm=False` to train on causal language modeling instead."
                )
            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:
                raise ValueError("mlm_probability should be between 0 and 1.")
            self.mlm_probability = float(self.mlm_probability)
        elif self.whole_word_mask:
            raise ValueError(
                "Whole word masking can only be used with mlm=True."
                "If you want to use whole word masking, please set mlm=True."
            )
        if self.mask_replace_prob + self.random_replace_prob > 1:
            raise ValueError("The sum of mask_replace_prob and random_replace_prob should not exceed 1")
        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:
            raise ValueError("mask_replace_prob should be between 0 and 1.")
        if self.random_replace_prob < 0 or self.random_replace_prob > 1:
            raise ValueError("random_replace_prob should be between 0 and 1.")

        if self.whole_word_mask:
            if not self.tokenizer.is_fast:
                warnings.warn(
                    "Whole word masking depends on offset mapping which is only natively available with fast tokenizers.",
                    UserWarning,
                )

            if self.mask_replace_prob < 1:
                warnings.warn(
                    "Random token replacement is not supported with whole word masking. "
                    "Setting mask_replace_prob to 1.",
                )
                self.mask_replace_prob = 1

View on GitHub (pinned to a597f97485)

Solutions

1. Lower one or both probabilities so mask_replace_prob + random_replace_prob <= 1 (e.g. defaults 0.8 + 0.1).
2. If you want every masked token changed, use exactly mask_replace_prob=1.0, random_replace_prob=0.0 (or 0.0/1.0).
3. Double-check that you passed fractions in [0,1], not integer percentages.

Example fix

# before
collator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=0.9, random_replace_prob=0.3)

# after
collator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=0.8, random_replace_prob=0.2)

def check_mask_probs(mask_replace_prob, random_replace_prob):
    assert 0 <= mask_replace_prob <= 1 and 0 <= random_replace_prob <= 1
    assert mask_replace_prob + random_replace_prob <= 1, 'replacement probs must sum to <= 1'

check_mask_probs(0.8, 0.2)  # call before constructing the collator

When it happens

Trigger: Constructing DataCollatorForLanguageModeling with e.g. mask_replace_prob=0.9 and random_replace_prob=0.2 (defaults are 0.8 and 0.1). Validated on every instantiation, before any batching happens.

Common situations: Tuning the 80/10/10 BERT masking ratios and forgetting the third 'unchanged' bucket; passing percentages (e.g. 80 and 30 instead of 0.8 and 0.3).

Related errors

• Whole word masking can only be used with mlm=True.If you wan
• mask_replace_prob should be between 0 and 1.
• random_replace_prob should be between 0 and 1.
• db_range must be greater than zero
• Stage_names must be set for transformers backbones

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/fc5c3a3de941ad7b. Report an issue: GitHub.