{"record":{"id":"b1a21a7105e467f5","repo":"huggingface/transformers","slug":"this-tokenizer-does-not-have-a-mask-token-which-is","errorCode":null,"errorMessage":"This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.","messagePattern":"This tokenizer does not have a mask token which is necessary for masked language modeling\\. You should pass `mlm=False` to train on causal language modeling instead\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":695,"sourceCode":"        remaining proportion will consist of masked tokens left unchanged.\n\n    </Tip>\n    \"\"\"\n\n    tokenizer: PreTrainedTokenizerBase\n    mlm: bool = True\n    whole_word_mask: bool = False\n    mlm_probability: float | None = 0.15\n    mask_replace_prob: float = 0.8\n    random_replace_prob: float = 0.1\n    pad_to_multiple_of: int | None = None\n    return_tensors: str = \"pt\"\n    seed: int | None = None\n\n    def __post_init__(self):\n        if self.mlm:\n            if self.tokenizer.mask_token is None:\n                raise ValueError(\n                    \"This tokenizer does not have a mask token which is necessary for masked language modeling. \"\n                    \"You should pass `mlm=False` to train on causal language modeling instead.\"\n                )\n            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:\n                raise ValueError(\"mlm_probability should be between 0 and 1.\")\n            self.mlm_probability = float(self.mlm_probability)\n        elif self.whole_word_mask:\n            raise ValueError(\n                \"Whole word masking can only be used with mlm=True.\"\n                \"If you want to use whole word masking, please set mlm=True.\"\n            )\n        if self.mask_replace_prob + self.random_replace_prob > 1:\n            raise ValueError(\"The sum of mask_replace_prob and random_replace_prob should not exceed 1\")\n        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:\n            raise ValueError(\"mask_replace_prob should be between 0 and 1.\")\n        if self.random_replace_prob < 0 or self.random_replace_prob > 1:\n            raise ValueError(\"random_replace_prob should be between 0 and 1.\")\n","sourceCodeStart":677,"sourceCodeEnd":713,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L677-L713","documentation":"Raised in DataCollatorForLanguageModeling.__post_init__ (data_collator.py:695). With mlm=True (the default) the collator masks tokens using tokenizer.mask_token; if the tokenizer has none (typical for causal LMs like GPT-2/Llama), masked-language-model collation is impossible and the dataclass refuses to construct, pointing you to mlm=False for causal training.","triggerScenarios":"DataCollatorForLanguageModeling(tokenizer=llama_or_gpt_tokenizer) with default mlm=True; raised at construction time in __post_init__, before any data is seen.","commonSituations":"Copy-pasting an MLM training snippet (BERT-style) into a causal-LM fine-tuning script; or fine-tuning a BERT-family model with a tokenizer that dropped the mask token after customization.","solutions":["For causal LM training pass mlm=False: DataCollatorForLanguageModeling(tokenizer, mlm=False).","If you truly need MLM with this tokenizer, give it a mask token (tokenizer.add_special_tokens({'mask_token': '[MASK]'}) plus embedding resize).","Double-check you loaded a BERT/RoBERTa-style tokenizer when MLM was intended."],"exampleFix":"# before\ncollator = DataCollatorForLanguageModeling(tokenizer=llama_tokenizer)  # mlm=True default -> raises\n\n# after\ncollator = DataCollatorForLanguageModeling(tokenizer=llama_tokenizer, mlm=False)","handlingStrategy":"validation","validationCode":"if mlm:\n    assert tokenizer.mask_token is not None, (\n        'tokenizer has no mask token; pass mlm=False for causal LM or add a mask token'\n    )\ncollator = DataCollatorForLanguageModeling(tokenizer, mlm=mlm)","typeGuard":"def supports_mlm(tokenizer) -> bool:\n    return tokenizer.mask_token is not None","tryCatchPattern":null,"preventionTips":["Default to mlm=False for causal LMs; only use mlm=True with BERT/RoBERTa-style tokenizers.","Check tokenizer.mask_token programmatically when MLM support depends on the model config.","Construct collators in a try block during app startup so config errors surface before training begins."],"tags":["data-collator","mlm","tokenizer","mask-token","training"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}