{"record":{"id":"76f19b790ed6e308","repo":"huggingface/transformers","slug":"whole-word-masking-can-only-be-used-with-mlm-true","errorCode":null,"errorMessage":"Whole word masking can only be used with mlm=True.If you want to use whole word masking, please set mlm=True.","messagePattern":"Whole word masking can only be used with mlm=True\\.If you want to use whole word masking, please set mlm=True\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":703,"sourceCode":"    mlm_probability: float | None = 0.15\n    mask_replace_prob: float = 0.8\n    random_replace_prob: float = 0.1\n    pad_to_multiple_of: int | None = None\n    return_tensors: str = \"pt\"\n    seed: int | None = None\n\n    def __post_init__(self):\n        if self.mlm:\n            if self.tokenizer.mask_token is None:\n                raise ValueError(\n                    \"This tokenizer does not have a mask token which is necessary for masked language modeling. \"\n                    \"You should pass `mlm=False` to train on causal language modeling instead.\"\n                )\n            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:\n                raise ValueError(\"mlm_probability should be between 0 and 1.\")\n            self.mlm_probability = float(self.mlm_probability)\n        elif self.whole_word_mask:\n            raise ValueError(\n                \"Whole word masking can only be used with mlm=True.\"\n                \"If you want to use whole word masking, please set mlm=True.\"\n            )\n        if self.mask_replace_prob + self.random_replace_prob > 1:\n            raise ValueError(\"The sum of mask_replace_prob and random_replace_prob should not exceed 1\")\n        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:\n            raise ValueError(\"mask_replace_prob should be between 0 and 1.\")\n        if self.random_replace_prob < 0 or self.random_replace_prob > 1:\n            raise ValueError(\"random_replace_prob should be between 0 and 1.\")\n\n        if self.whole_word_mask:\n            if not self.tokenizer.is_fast:\n                warnings.warn(\n                    \"Whole word masking depends on offset mapping which is only natively available with fast tokenizers.\",\n                    UserWarning,\n                )\n\n            if self.mask_replace_prob < 1:","sourceCodeStart":685,"sourceCodeEnd":721,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L685-L721","documentation":"Raised by DataCollatorForLanguageModeling.__post_init__ when whole_word_mask=True is combined with mlm=False. Whole-word masking works by grouping token-level mask decisions into words, which only makes sense when tokens are actually being masked for the masked-language-modeling objective, so the collator refuses this contradictory configuration at construction time.","triggerScenarios":"Constructing DataCollatorForLanguageModeling(tokenizer, mlm=False, whole_word_mask=True); also instantiating DataCollatorForWholeWordMask and then overriding mlm=False, since that subclass sets whole_word_mask=True.","commonSituations":"A developer switches an MLM pretraining script to causal LM by flipping mlm=False but leaves a whole-word-masking flag enabled; or copies a DataCollatorForWholeWordMask config into a DataCollatorForLanguageModeling instantiation.","solutions":["Set mlm=True if you actually want whole-word masked language modeling.","Set whole_word_mask=False (or omit it) if you want causal language modeling with mlm=False.","For whole-word masking use the dedicated DataCollatorForWholeWordMask class with its default mlm=True."],"exampleFix":"# before\ncollator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, whole_word_mask=True)\n\n# after (causal LM)\ncollator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)\n# after (whole-word MLM)\ncollator = DataCollatorForWholeWordMask(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)","handlingStrategy":"validation","validationCode":"def build_lm_collator(tokenizer, mlm, whole_word_mask=False, **kw):\n    if whole_word_mask and not mlm:\n        raise ValueError('whole_word_mask requires mlm=True; refusing to construct collator')\n    return DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=mlm, whole_word_mask=whole_word_mask, **kw)","typeGuard":null,"tryCatchPattern":"try:\n    collator = DataCollatorForLanguageModeling(tok, mlm=mlm, whole_word_mask=wwm)\nexcept ValueError as e:\n    if 'Whole word masking' in str(e):\n        logger.error('Fix config: whole_word_mask=True needs mlm=True')\n    raise","preventionTips":["Validate flag combinations (whole_word_mask implies mlm) in your training config schema before constructing collators.","Prefer DataCollatorForWholeWordMask for WWM instead of hand-setting the flag."],"tags":["data-collator","configuration","masked-language-modeling"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}