{"record":{"id":"f5a59796d363be6c","repo":"huggingface/transformers","slug":"this-tokenizer-does-not-have-a-mask-token-which-is-f5a597","errorCode":null,"errorMessage":"This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.","messagePattern":"This tokenizer does not have a mask token which is necessary for masked language modeling\\. Remove the --mlm flag if you want to use this tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":1102,"sourceCode":"        sentence_order_label = torch.stack(sop_label_list)\n\n        return {\n            \"input_ids\": input_ids,\n            \"labels\": labels,\n            \"attention_mask\": attention_mask,\n            \"token_type_ids\": token_type_ids,\n            \"sentence_order_label\": sentence_order_label,\n        }\n\n    def mask_tokens(self, inputs: Any) -> tuple[Any, Any, Any]:\n        \"\"\"\n        Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10%\n        original. N-gram not applied yet.\n        \"\"\"\n        import torch\n\n        if self.tokenizer.mask_token is None:\n            raise ValueError(\n                \"This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the\"\n                \" --mlm flag if you want to use this tokenizer.\"\n            )\n\n        labels = inputs.clone()\n        # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)\n        probability_matrix = torch.full(labels.shape, self.mlm_probability)\n        special_tokens_mask = [\n            self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()\n        ]\n        probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)\n        if self.tokenizer.pad_token is not None:\n            padding_mask = labels.eq(self.tokenizer.pad_token_id)\n            probability_matrix.masked_fill_(padding_mask, value=0.0)\n        masked_indices = torch.bernoulli(probability_matrix).bool()\n        # probability be `1` (masked), however in albert model attention mask `0` means masked, revert the value\n        attention_mask = (~masked_indices).float()\n        if self.tokenizer.pad_token is not None:","sourceCodeStart":1084,"sourceCodeEnd":1120,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L1084-L1120","documentation":"Raised by DataCollatorForSOP.mask_tokens when the tokenizer has no mask_token. The sentence-order-prediction collator still masks tokens for its MLM head, which requires a mask token to substitute; tokenizers trained without a mask token (e.g. GPT-2) cannot support it. The message references the legacy --mlm flag from the run_pretraining-style example scripts.","triggerScenarios":"Constructing DataCollatorForSOP(tokenizer=gpt2_tokenizer, ...) where tokenizer.mask_token is None and then calling it on a batch (mask_tokens runs inside torch_call).","commonSituations":"Reusing an ALBERT-style SOP pretraining pipeline with a causal-LM tokenizer that has no [MASK]; swapping tokenizers in an experiment; using a tokenizer whose mask token was deliberately removed.","solutions":["Add a mask token before tokenizing: tokenizer.add_special_tokens({'mask_token': '[MASK]}) and resize model embeddings accordingly.","Switch to a tokenizer that already has a mask token (BERT, ALBERT, RoBERTa) for SOP/MLM training.","If SOP is not needed, use DataCollatorForLanguageModeling with mlm=False or a causal-LM setup instead."],"exampleFix":"# before\ntok = AutoTokenizer.from_pretrained('gpt2')\ncollator = DataCollatorForSOP(tokenizer=tok)  # later raises in mask_tokens\n\n# after\ntok = AutoTokenizer.from_pretrained('gpt2')\ntok.add_special_tokens({'mask_token': '[MASK]'})\nmodel.resize_token_embeddings(len(tok))\ncollator = DataCollatorForSOP(tokenizer=tok)","handlingStrategy":"validation","validationCode":"if tokenizer.mask_token is None:\n    tokenizer.add_special_tokens({'mask_token': '[MASK]'})\n    model.resize_token_embeddings(len(tokenizer))\ncollator = DataCollatorForSOP(tokenizer=tokenizer)","typeGuard":"def supports_masking(tokenizer) -> bool:\n    return getattr(tokenizer, 'mask_token', None) is not None","tryCatchPattern":null,"preventionTips":["Check tokenizer.mask_token before constructing any MLM/SOP collator.","After adding special tokens, always resize model embeddings to the new vocab size."],"tags":["data-collator","tokenizer","masked-language-modeling","sop"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}