huggingface/transformers · error · ValueError

mask_replace_prob should be between 0 and 1.

Error message

mask_replace_prob should be between 0 and 1.

What it means

Raised by DataCollatorForLanguageModeling.__post_init__ when mask_replace_prob is outside [0,1]. This parameter is the probability that a masked token is replaced by the tokenizer's mask token, so like any probability it must lie between 0 and 1; anything else indicates a misconfigured ratio (typically percentages passed instead of fractions).

Source

Thrown at src/transformers/data/data_collator.py:710

    def __post_init__(self):
        if self.mlm:
            if self.tokenizer.mask_token is None:
                raise ValueError(
                    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
                    "You should pass `mlm=False` to train on causal language modeling instead."
                )
            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:
                raise ValueError("mlm_probability should be between 0 and 1.")
            self.mlm_probability = float(self.mlm_probability)
        elif self.whole_word_mask:
            raise ValueError(
                "Whole word masking can only be used with mlm=True."
                "If you want to use whole word masking, please set mlm=True."
            )
        if self.mask_replace_prob + self.random_replace_prob > 1:
            raise ValueError("The sum of mask_replace_prob and random_replace_prob should not exceed 1")
        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:
            raise ValueError("mask_replace_prob should be between 0 and 1.")
        if self.random_replace_prob < 0 or self.random_replace_prob > 1:
            raise ValueError("random_replace_prob should be between 0 and 1.")

        if self.whole_word_mask:
            if not self.tokenizer.is_fast:
                warnings.warn(
                    "Whole word masking depends on offset mapping which is only natively available with fast tokenizers.",
                    UserWarning,
                )

            if self.mask_replace_prob < 1:
                warnings.warn(
                    "Random token replacement is not supported with whole word masking. "
                    "Setting mask_replace_prob to 1.",
                )
                self.mask_replace_prob = 1
                self.random_replace_prob = 0

View on GitHub (pinned to a597f97485)

Solutions

1. Pass a fraction in [0,1], e.g. mask_replace_prob=0.8 for the standard BERT behavior.
2. Fix the upstream config file / CLI value if the wrong number flows into the constructor.
3. Sanity-check all probability arguments (mlm_probability, mask_replace_prob, random_replace_prob) before building the collator.

Example fix

# before
collator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=80)

# after
collator = DataCollatorForLanguageModeling(tokenizer=tok, mask_replace_prob=0.8)

mask_replace_prob = float(cfg['mask_replace_prob'])
if not 0.0 <= mask_replace_prob <= 1.0:
    raise ValueError(f'mask_replace_prob={mask_replace_prob} outside [0,1] (percent vs fraction bug?)')

When it happens

Trigger: Constructing DataCollatorForLanguageModeling(tokenizer, mask_replace_prob=80) or any negative value; also triggered by NaN values introduced through config parsing.

Common situations: Porting a paper's '80% mask replacement' setting as 80 instead of 0.8; loading the value from a YAML/JSON config where the wrong field or scale was used.

Related errors

• The sum of mask_replace_prob and random_replace_prob should
• random_replace_prob should be between 0 and 1.
• db_range must be greater than zero
• Stage_names must be set for transformers backbones
• out_features must be a list got {type(self._out_features)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a65e8c4bf560c23e. Report an issue: GitHub.