huggingface/transformers · error · ValueError

This tokenizer does not have a mask token which is necessary

Error message

This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.

What it means

Raised by DataCollatorForSOP.mask_tokens when the tokenizer has no mask_token. The sentence-order-prediction collator still masks tokens for its MLM head, which requires a mask token to substitute; tokenizers trained without a mask token (e.g. GPT-2) cannot support it. The message references the legacy --mlm flag from the run_pretraining-style example scripts.

Source

Thrown at src/transformers/data/data_collator.py:1102

        sentence_order_label = torch.stack(sop_label_list)

        return {
            "input_ids": input_ids,
            "labels": labels,
            "attention_mask": attention_mask,
            "token_type_ids": token_type_ids,
            "sentence_order_label": sentence_order_label,
        }

    def mask_tokens(self, inputs: Any) -> tuple[Any, Any, Any]:
        """
        Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10%
        original. N-gram not applied yet.
        """
        import torch

        if self.tokenizer.mask_token is None:
            raise ValueError(
                "This tokenizer does not have a mask token which is necessary for masked language modeling. Remove the"
                " --mlm flag if you want to use this tokenizer."
            )

        labels = inputs.clone()
        # We sample a few tokens in each sequence for masked-LM training (with probability args.mlm_probability defaults to 0.15 in Bert/RoBERTa)
        probability_matrix = torch.full(labels.shape, self.mlm_probability)
        special_tokens_mask = [
            self.tokenizer.get_special_tokens_mask(val, already_has_special_tokens=True) for val in labels.tolist()
        ]
        probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
        if self.tokenizer.pad_token is not None:
            padding_mask = labels.eq(self.tokenizer.pad_token_id)
            probability_matrix.masked_fill_(padding_mask, value=0.0)
        masked_indices = torch.bernoulli(probability_matrix).bool()
        # probability be `1` (masked), however in albert model attention mask `0` means masked, revert the value
        attention_mask = (~masked_indices).float()
        if self.tokenizer.pad_token is not None:

View on GitHub (pinned to a597f97485)

Solutions

1. Add a mask token before tokenizing: tokenizer.add_special_tokens({'mask_token': '[MASK]}) and resize model embeddings accordingly.
2. Switch to a tokenizer that already has a mask token (BERT, ALBERT, RoBERTa) for SOP/MLM training.
3. If SOP is not needed, use DataCollatorForLanguageModeling with mlm=False or a causal-LM setup instead.

Example fix

# before
tok = AutoTokenizer.from_pretrained('gpt2')
collator = DataCollatorForSOP(tokenizer=tok)  # later raises in mask_tokens

# after
tok = AutoTokenizer.from_pretrained('gpt2')
tok.add_special_tokens({'mask_token': '[MASK]'})
model.resize_token_embeddings(len(tok))
collator = DataCollatorForSOP(tokenizer=tok)

if tokenizer.mask_token is None:
    tokenizer.add_special_tokens({'mask_token': '[MASK]'})
    model.resize_token_embeddings(len(tokenizer))
collator = DataCollatorForSOP(tokenizer=tokenizer)

def supports_masking(tokenizer) -> bool:
    return getattr(tokenizer, 'mask_token', None) is not None

When it happens

Trigger: Constructing DataCollatorForSOP(tokenizer=gpt2_tokenizer, ...) where tokenizer.mask_token is None and then calling it on a batch (mask_tokens runs inside torch_call).

Common situations: Reusing an ALBERT-style SOP pretraining pipeline with a causal-LM tokenizer that has no [MASK]; swapping tokenizers in an experiment; using a tokenizer whose mask token was deliberately removed.

Related errors

• You are attempting to pad samples but the tokenizer you are
• This tokenizer does not have a mask token which is necessary
• Whole word masking can only be used with mlm=True.If you wan
• The sum of mask_replace_prob and random_replace_prob should
• This tokenizer does not have a mask token which is necessary

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/f5a59796d363be6c. Report an issue: GitHub.