huggingface/transformers · error · ValueError

Whole word masking can only be used with mlm=True.If you wan

Error message

Whole word masking can only be used with mlm=True.If you want to use whole word masking, please set mlm=True.

What it means

Raised by DataCollatorForLanguageModeling.__post_init__ when whole_word_mask=True is combined with mlm=False. Whole-word masking works by grouping token-level mask decisions into words, which only makes sense when tokens are actually being masked for the masked-language-modeling objective, so the collator refuses this contradictory configuration at construction time.

Source

Thrown at src/transformers/data/data_collator.py:703

    mlm_probability: float | None = 0.15
    mask_replace_prob: float = 0.8
    random_replace_prob: float = 0.1
    pad_to_multiple_of: int | None = None
    return_tensors: str = "pt"
    seed: int | None = None

    def __post_init__(self):
        if self.mlm:
            if self.tokenizer.mask_token is None:
                raise ValueError(
                    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
                    "You should pass `mlm=False` to train on causal language modeling instead."
                )
            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:
                raise ValueError("mlm_probability should be between 0 and 1.")
            self.mlm_probability = float(self.mlm_probability)
        elif self.whole_word_mask:
            raise ValueError(
                "Whole word masking can only be used with mlm=True."
                "If you want to use whole word masking, please set mlm=True."
            )
        if self.mask_replace_prob + self.random_replace_prob > 1:
            raise ValueError("The sum of mask_replace_prob and random_replace_prob should not exceed 1")
        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:
            raise ValueError("mask_replace_prob should be between 0 and 1.")
        if self.random_replace_prob < 0 or self.random_replace_prob > 1:
            raise ValueError("random_replace_prob should be between 0 and 1.")

        if self.whole_word_mask:
            if not self.tokenizer.is_fast:
                warnings.warn(
                    "Whole word masking depends on offset mapping which is only natively available with fast tokenizers.",
                    UserWarning,
                )

            if self.mask_replace_prob < 1:

View on GitHub (pinned to a597f97485)

Solutions

Set mlm=True if you actually want whole-word masked language modeling.
Set whole_word_mask=False (or omit it) if you want causal language modeling with mlm=False.
For whole-word masking use the dedicated DataCollatorForWholeWordMask class with its default mlm=True.

Example fix

# before
collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, whole_word_mask=True)

# after (causal LM)
collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
# after (whole-word MLM)
collator = DataCollatorForWholeWordMask(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

Defensive patterns

Strategy: validation

Validate before calling

def build_lm_collator(tokenizer, mlm, whole_word_mask=False, **kw):
    if whole_word_mask and not mlm:
        raise ValueError('whole_word_mask requires mlm=True; refusing to construct collator')
    return DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=mlm, whole_word_mask=whole_word_mask, **kw)

Try / catch

try:
    collator = DataCollatorForLanguageModeling(tok, mlm=mlm, whole_word_mask=wwm)
except ValueError as e:
    if 'Whole word masking' in str(e):
        logger.error('Fix config: whole_word_mask=True needs mlm=True')
    raise

Prevention

Validate flag combinations (whole_word_mask implies mlm) in your training config schema before constructing collators.
Prefer DataCollatorForWholeWordMask for WWM instead of hand-setting the flag.

When it happens

Trigger: Constructing DataCollatorForLanguageModeling(tokenizer, mlm=False, whole_word_mask=True); also instantiating DataCollatorForWholeWordMask and then overriding mlm=False, since that subclass sets whole_word_mask=True.

Common situations: A developer switches an MLM pretraining script to causal LM by flipping mlm=False but leaves a whole-word-masking flag enabled; or copies a DataCollatorForWholeWordMask config into a DataCollatorForLanguageModeling instantiation.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/76f19b790ed6e308. Report an issue: GitHub.