huggingface/transformers · error · ValueError

You are attempting to pad samples but the tokenizer you are

Error message

You are attempting to pad samples but the tokenizer you are using ({tokenizer.__class__.__name__}) does not have a pad token.

What it means

Raised by the torch-side _torch_collate_batch pad path (data_collator.py:369). When batched sequences have different lengths (or pad_to_multiple_of forces growth), the collator must pad using tokenizer.pad_token_id; if tokenizer.pad_token is None there is no valid fill value, so it raises with the tokenizer class name. Same-length batches short-circuit before this check, which is why it can appear intermittent.

Source

Thrown at src/transformers/data/data_collator.py:369

    """Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary."""
    import torch

    # Tensorize if necessary.
    if isinstance(examples[0], (list, tuple, np.ndarray)):
        examples = [torch.tensor(e, dtype=torch.long) for e in examples]

    length_of_first = examples[0].size(0)

    # Check if padding is necessary.

    are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)
    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):
        if not isinstance(examples, torch.Tensor):
            return torch.stack(examples, dim=0)

    # If yes, check if we have a `pad_token`.
    if tokenizer.pad_token is None:
        raise ValueError(
            "You are attempting to pad samples but the tokenizer you are using"
            f" ({tokenizer.__class__.__name__}) does not have a pad token."
        )

    # Creating the full tensor and filling it with our data.
    max_length = max(x.size(0) for x in examples)
    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
    result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)
    for i, example in enumerate(examples):
        if tokenizer.padding_side == "right":
            result[i, : example.shape[0]] = example
        else:
            result[i, -example.shape[0] :] = example
    return result


def _numpy_collate_batch(examples, tokenizer, pad_to_multiple_of: int | None = None):

View on GitHub (pinned to a597f97485)

Solutions

1. Set a pad token before training: tokenizer.pad_token = tokenizer.eos_token (quickest), or add a dedicated '[PAD]' token and resize embeddings accordingly.
2. Or rely on processing_class/collator configs that pass an explicit pad token id if the collator variant supports it.
3. As a last resort make all sequences the same length (padding='max_length' at tokenization time) so the collator never needs to pad — but this wastes compute.

Example fix

# before
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
collator = DataCollatorForLanguageModeling(tokenizer)  # raises on ragged batch

# after
tokenizer.pad_token = tokenizer.eos_token
collator = DataCollatorForLanguageModeling(tokenizer)

lengths = {x.size(0) for x in examples}
needs_pad = len(lengths) > 1 or (pad_to_multiple_of and max(lengths) % pad_to_multiple_of)
if needs_pad:
    assert tokenizer.pad_token is not None, (
        f'{type(tokenizer).__name__} has no pad token; set tokenizer.pad_token = tokenizer.eos_token first'
    )

def tokenizer_can_pad(tokenizer) -> bool:
    return tokenizer.pad_token is not None

When it happens

Trigger: DataCollatorForLanguageModeling / DataCollatorForSeq2Seq / DataCollatorWithPadding over a dataset with unequal sequence lengths, using a tokenizer without a pad token (GPT-2/GPT-J/Llama/Mistral style tokenizers historically ship none). Batches that happen to be uniform length skip the error.

Common situations: Fine-tuning causal LMs (llama, mistral, qwen early versions) without first setting a pad token. Works on toy fixed-length data, then fails on the first ragged batch — a classic intermittent surprise.

Related errors

• This tokenizer does not have a mask token which is necessary
• This tokenizer does not have a mask token which is necessary
• This tokenizer does not have a mask token which is necessary
• This collator requires that sequence lengths be even to crea
• {error_message} requires the protobuf library but it was not

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a2eb4aeec1e90d1e. Report an issue: GitHub.