{"record":{"id":"897f4cc9dfa192f0","repo":"huggingface/transformers","slug":"this-tokenizer-does-not-have-a-mask-token-which-is-897f4c","errorCode":null,"errorMessage":"This tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.","messagePattern":"This tokenizer does not have a mask token which is necessary for permutation language modeling\\. Please add a mask token if you want to use this tokenizer\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":1182,"sourceCode":"        return {\"input_ids\": inputs, \"perm_mask\": perm_mask, \"target_mapping\": target_mapping, \"labels\": labels}\n\n    def torch_mask_tokens(self, inputs: Any) -> tuple[Any, Any, Any, Any]:\n        \"\"\"\n        The masked tokens to be predicted for a particular sequence are determined by the following algorithm:\n\n            0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).\n            1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)\n            2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be\n               masked\n            3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -\n               span_length]` and mask tokens `start_index:start_index + span_length`\n            4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the\n               sequence to be processed), repeat from Step 1.\n        \"\"\"\n        import torch\n\n        if self.tokenizer.mask_token is None:\n            raise ValueError(\n                \"This tokenizer does not have a mask token which is necessary for permutation language modeling.\"\n                \" Please add a mask token if you want to use this tokenizer.\"\n            )\n\n        if inputs.size(1) % 2 != 0:\n            raise ValueError(\n                \"This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see\"\n                \" relevant comments in source code for details.\"\n            )\n\n        labels = inputs.clone()\n        # Creating the mask and target_mapping tensors\n        masked_indices = torch.full(labels.shape, 0, dtype=torch.bool)\n        target_mapping = torch.zeros((labels.size(0), labels.size(1), labels.size(1)), dtype=torch.float32)\n\n        for i in range(labels.size(0)):\n            # Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).\n            cur_len = 0","sourceCodeStart":1164,"sourceCodeEnd":1200,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L1164-L1200","documentation":"Raised by DataCollatorForPermutationLanguageModeling.torch_call (via its mask_tokens) when the tokenizer has no mask token. The XLNet-style permutation LM objective still masks target spans with a mask token, so a tokenizer without one (GPT-2, most causal-LM tokenizers) cannot be used with this collator.","triggerScenarios":"DataCollatorForPermutationLanguageModeling(tokenizer=gpt2) called on a batch of tokenized examples; the check runs on the first collate call.","commonSituations":"Adapting an XLNet pretraining script to a different checkpoint whose tokenizer lacks a mask token; mixing up collators between causal and permutation LM pipelines.","solutions":["Use a tokenizer with a mask token (XLNet, BERT, RoBERTa families) with this collator.","Add a mask token: tokenizer.add_special_tokens({'mask_token': '<mask>'}) and call model.resize_token_embeddings(len(tokenizer)).","If you are actually training a causal model, use DataCollatorForLanguageModeling(mlm=False) instead."],"exampleFix":"# before\ntok = AutoTokenizer.from_pretrained('gpt2')\ncollator = DataCollatorForPermutationLanguageModeling(tokenizer=tok)\nbatch = collator(encoded)  # raises\n\n# after\ntok.add_special_tokens({'mask_token': '<mask>'})\nmodel.resize_token_embeddings(len(tok))\nbatch = collator(encoded)","handlingStrategy":"validation","validationCode":"if tokenizer.mask_token is None:\n    raise ValueError(f'{type(tokenizer).__name__} has no mask token; permutation LM requires one. '\n                     'Call tokenizer.add_special_tokens({\"mask_token\": \"<mask>\"}) first.')","typeGuard":"def can_do_permutation_lm(tokenizer) -> bool:\n    return tokenizer.mask_token is not None","tryCatchPattern":null,"preventionTips":["Pair DataCollatorForPermutationLanguageModeling only with mask-token tokenizers (XLNet, BERT, RoBERTa).","Add a startup assertion in training scripts: assert tokenizer.mask_token is not None for MLM/permutation objectives."],"tags":["data-collator","tokenizer","permutation-language-modeling","xlnet"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}