huggingface/transformers · error · ValueError

This tokenizer does not have a mask token which is necessary

Error message

This tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.

What it means

Raised by DataCollatorForPermutationLanguageModeling.torch_call (via its mask_tokens) when the tokenizer has no mask token. The XLNet-style permutation LM objective still masks target spans with a mask token, so a tokenizer without one (GPT-2, most causal-LM tokenizers) cannot be used with this collator.

Source

Thrown at src/transformers/data/data_collator.py:1182

        return {"input_ids": inputs, "perm_mask": perm_mask, "target_mapping": target_mapping, "labels": labels}

    def torch_mask_tokens(self, inputs: Any) -> tuple[Any, Any, Any, Any]:
        """
        The masked tokens to be predicted for a particular sequence are determined by the following algorithm:

            0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
            1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
            2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be
               masked
            3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -
               span_length]` and mask tokens `start_index:start_index + span_length`
            4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the
               sequence to be processed), repeat from Step 1.
        """
        import torch

        if self.tokenizer.mask_token is None:
            raise ValueError(
                "This tokenizer does not have a mask token which is necessary for permutation language modeling."
                " Please add a mask token if you want to use this tokenizer."
            )

        if inputs.size(1) % 2 != 0:
            raise ValueError(
                "This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see"
                " relevant comments in source code for details."
            )

        labels = inputs.clone()
        # Creating the mask and target_mapping tensors
        masked_indices = torch.full(labels.shape, 0, dtype=torch.bool)
        target_mapping = torch.zeros((labels.size(0), labels.size(1), labels.size(1)), dtype=torch.float32)

        for i in range(labels.size(0)):
            # Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
            cur_len = 0

View on GitHub (pinned to a597f97485)

Solutions

1. Use a tokenizer with a mask token (XLNet, BERT, RoBERTa families) with this collator.
2. Add a mask token: tokenizer.add_special_tokens({'mask_token': '<mask>'}) and call model.resize_token_embeddings(len(tokenizer)).
3. If you are actually training a causal model, use DataCollatorForLanguageModeling(mlm=False) instead.

Example fix

# before
tok = AutoTokenizer.from_pretrained('gpt2')
collator = DataCollatorForPermutationLanguageModeling(tokenizer=tok)
batch = collator(encoded)  # raises

# after
tok.add_special_tokens({'mask_token': '<mask>'})
model.resize_token_embeddings(len(tok))
batch = collator(encoded)

if tokenizer.mask_token is None:
    raise ValueError(f'{type(tokenizer).__name__} has no mask token; permutation LM requires one. '
                     'Call tokenizer.add_special_tokens({"mask_token": "<mask>"}) first.')

def can_do_permutation_lm(tokenizer) -> bool:
    return tokenizer.mask_token is not None

When it happens

Trigger: DataCollatorForPermutationLanguageModeling(tokenizer=gpt2) called on a batch of tokenized examples; the check runs on the first collate call.

Common situations: Adapting an XLNet pretraining script to a different checkpoint whose tokenizer lacks a mask token; mixing up collators between causal and permutation LM pipelines.

Related errors

• You are attempting to pad samples but the tokenizer you are
• This tokenizer does not have a mask token which is necessary
• This tokenizer does not have a mask token which is necessary
• This collator requires that sequence lengths be even to crea
• {error_message} requires the protobuf library but it was not

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/897f4cc9dfa192f0. Report an issue: GitHub.