huggingface/transformers · error · ValueError

This tokenizer does not have a mask token which is necessary

Error message

This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.

What it means

Raised in DataCollatorForLanguageModeling.__post_init__ (data_collator.py:695). With mlm=True (the default) the collator masks tokens using tokenizer.mask_token; if the tokenizer has none (typical for causal LMs like GPT-2/Llama), masked-language-model collation is impossible and the dataclass refuses to construct, pointing you to mlm=False for causal training.

Source

Thrown at src/transformers/data/data_collator.py:695

        remaining proportion will consist of masked tokens left unchanged.

    </Tip>
    """

    tokenizer: PreTrainedTokenizerBase
    mlm: bool = True
    whole_word_mask: bool = False
    mlm_probability: float | None = 0.15
    mask_replace_prob: float = 0.8
    random_replace_prob: float = 0.1
    pad_to_multiple_of: int | None = None
    return_tensors: str = "pt"
    seed: int | None = None

    def __post_init__(self):
        if self.mlm:
            if self.tokenizer.mask_token is None:
                raise ValueError(
                    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
                    "You should pass `mlm=False` to train on causal language modeling instead."
                )
            if self.mlm_probability is None or self.mlm_probability < 0 or self.mlm_probability > 1:
                raise ValueError("mlm_probability should be between 0 and 1.")
            self.mlm_probability = float(self.mlm_probability)
        elif self.whole_word_mask:
            raise ValueError(
                "Whole word masking can only be used with mlm=True."
                "If you want to use whole word masking, please set mlm=True."
            )
        if self.mask_replace_prob + self.random_replace_prob > 1:
            raise ValueError("The sum of mask_replace_prob and random_replace_prob should not exceed 1")
        if self.mask_replace_prob < 0 or self.mask_replace_prob > 1:
            raise ValueError("mask_replace_prob should be between 0 and 1.")
        if self.random_replace_prob < 0 or self.random_replace_prob > 1:
            raise ValueError("random_replace_prob should be between 0 and 1.")

View on GitHub (pinned to a597f97485)

Solutions

1. For causal LM training pass mlm=False: DataCollatorForLanguageModeling(tokenizer, mlm=False).
2. If you truly need MLM with this tokenizer, give it a mask token (tokenizer.add_special_tokens({'mask_token': '[MASK]'}) plus embedding resize).
3. Double-check you loaded a BERT/RoBERTa-style tokenizer when MLM was intended.

Example fix

# before
collator = DataCollatorForLanguageModeling(tokenizer=llama_tokenizer)  # mlm=True default -> raises

# after
collator = DataCollatorForLanguageModeling(tokenizer=llama_tokenizer, mlm=False)

if mlm:
    assert tokenizer.mask_token is not None, (
        'tokenizer has no mask token; pass mlm=False for causal LM or add a mask token'
    )
collator = DataCollatorForLanguageModeling(tokenizer, mlm=mlm)

def supports_mlm(tokenizer) -> bool:
    return tokenizer.mask_token is not None

When it happens

Trigger: DataCollatorForLanguageModeling(tokenizer=llama_or_gpt_tokenizer) with default mlm=True; raised at construction time in __post_init__, before any data is seen.

Common situations: Copy-pasting an MLM training snippet (BERT-style) into a causal-LM fine-tuning script; or fine-tuning a BERT-family model with a tokenizer that dropped the mask token after customization.

Related errors

• mlm_probability should be between 0 and 1.
• You are attempting to pad samples but the tokenizer you are
• This tokenizer does not have a mask token which is necessary
• This tokenizer does not have a mask token which is necessary
• File not found: {audio}

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/b1a21a7105e467f5. Report an issue: GitHub.