huggingface/transformers · error · ValueError

random_replace_prob should be between 0 and 1.

Error message

random_replace_prob should be between 0 and 1.

What it means

Raised by DataCollatorForLanguageModeling.__post_init__ when random_replace_prob is outside [0,1]. This parameter is the probability that a masked token is replaced by a random vocabulary token (the classic 10% branch of BERT masking), so it must be a valid probability; values like 10 (percent instead of 0.1) or negatives are rejected.

Source

Thrown at src/transformers/data/data_collator.py:712

            if self.tokenizer.mask_token is None:
                raise ValueError(
                    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
                    "You should pass `mlm=False` to train on causal language modeling instead."
                )
            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:
                raise ValueError("mlm_probability should be between 0 and 1.")
            self.mlm_probability = float(self.mlm_probability)
        elif self.whole_word_mask:
            raise ValueError(
                "Whole word masking can only be used with mlm=True."
                "If you want to use whole word masking, please set mlm=True."
            )
        if self.mask_replace_prob + self.random_replace_prob > 1:
            raise ValueError("The sum of mask_replace_prob and random_replace_prob should not exceed 1")
        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:
            raise ValueError("mask_replace_prob should be between 0 and 1.")
        if self.random_replace_prob < 0 or self.random_replace_prob > 1:
            raise ValueError("random_replace_prob should be between 0 and 1.")

        if self.whole_word_mask:
            if not self.tokenizer.is_fast:
                warnings.warn(
                    "Whole word masking depends on offset mapping which is only natively available with fast tokenizers.",
                    UserWarning,
                )

            if self.mask_replace_prob < 1:
                warnings.warn(
                    "Random token replacement is not supported with whole word masking. "
                    "Setting mask_replace_prob to 1.",
                )
                self.mask_replace_prob = 1
                self.random_replace_prob = 0

        self.mask_replace_prob = float(self.mask_replace_prob)
        self.random_replace_prob = float(self.random_replace_prob)

View on GitHub (pinned to a597f97485)

Solutions

1. Pass a fraction in [0,1], e.g. random_replace_prob=0.1.
2. Fix the config/CLI source of the value if it was scaled incorrectly.
3. Verify the sum rule at the same time: mask_replace_prob + random_replace_prob must not exceed 1.

Example fix

# before
collator = DataCollatorForLanguageModeling(tokenizer=tok, random_replace_prob=10)

# after
collator = DataCollatorForLanguageModeling(tokenizer=tok, random_replace_prob=0.1)

random_replace_prob = float(cfg['random_replace_prob'])
if not 0.0 <= random_replace_prob <= 1.0:
    raise ValueError(f'random_replace_prob={random_replace_prob} outside [0,1]')

When it happens

Trigger: Constructing DataCollatorForLanguageModeling(tokenizer, random_replace_prob=10) or any value < 0 or > 1.

Common situations: Transcribing '10% random token' from the BERT paper as 10 rather than 0.1; config-file scale mismatches; copy-paste between scripts that use different units.

Related errors

• The sum of mask_replace_prob and random_replace_prob should
• mask_replace_prob should be between 0 and 1.
• db_range must be greater than zero
• Stage_names must be set for transformers backbones
• out_features must be a list got {type(self._out_features)}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/dff1bcc8bc69db56. Report an issue: GitHub.