{"record":{"id":"a2eb4aeec1e90d1e","repo":"huggingface/transformers","slug":"you-are-attempting-to-pad-samples-but-the-tokenize","errorCode":null,"errorMessage":"You are attempting to pad samples but the tokenizer you are using ({tokenizer.__class__.__name__}) does not have a pad token.","messagePattern":"You are attempting to pad samples but the tokenizer you are using \\((.+?)\\) does not have a pad token\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/data/data_collator.py","lineNumber":369,"sourceCode":"    \"\"\"Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary.\"\"\"\n    import torch\n\n    # Tensorize if necessary.\n    if isinstance(examples[0], (list, tuple, np.ndarray)):\n        examples = [torch.tensor(e, dtype=torch.long) for e in examples]\n\n    length_of_first = examples[0].size(0)\n\n    # Check if padding is necessary.\n\n    are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)\n    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):\n        if not isinstance(examples, torch.Tensor):\n            return torch.stack(examples, dim=0)\n\n    # If yes, check if we have a `pad_token`.\n    if tokenizer.pad_token is None:\n        raise ValueError(\n            \"You are attempting to pad samples but the tokenizer you are using\"\n            f\" ({tokenizer.__class__.__name__}) does not have a pad token.\"\n        )\n\n    # Creating the full tensor and filling it with our data.\n    max_length = max(x.size(0) for x in examples)\n    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):\n        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of\n    result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)\n    for i, example in enumerate(examples):\n        if tokenizer.padding_side == \"right\":\n            result[i, : example.shape[0]] = example\n        else:\n            result[i, -example.shape[0] :] = example\n    return result\n\n\ndef _numpy_collate_batch(examples, tokenizer, pad_to_multiple_of: int | None = None):","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/data/data_collator.py#L351-L387","documentation":"Raised by the torch-side _torch_collate_batch pad path (data_collator.py:369). When batched sequences have different lengths (or pad_to_multiple_of forces growth), the collator must pad using tokenizer.pad_token_id; if tokenizer.pad_token is None there is no valid fill value, so it raises with the tokenizer class name. Same-length batches short-circuit before this check, which is why it can appear intermittent.","triggerScenarios":"DataCollatorForLanguageModeling / DataCollatorForSeq2Seq / DataCollatorWithPadding over a dataset with unequal sequence lengths, using a tokenizer without a pad token (GPT-2/GPT-J/Llama/Mistral style tokenizers historically ship none). Batches that happen to be uniform length skip the error.","commonSituations":"Fine-tuning causal LMs (llama, mistral, qwen early versions) without first setting a pad token. Works on toy fixed-length data, then fails on the first ragged batch — a classic intermittent surprise.","solutions":["Set a pad token before training: tokenizer.pad_token = tokenizer.eos_token (quickest), or add a dedicated '[PAD]' token and resize embeddings accordingly.","Or rely on processing_class/collator configs that pass an explicit pad token id if the collator variant supports it.","As a last resort make all sequences the same length (padding='max_length' at tokenization time) so the collator never needs to pad — but this wastes compute."],"exampleFix":"# before\ntokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')\ncollator = DataCollatorForLanguageModeling(tokenizer)  # raises on ragged batch\n\n# after\ntokenizer.pad_token = tokenizer.eos_token\ncollator = DataCollatorForLanguageModeling(tokenizer)","handlingStrategy":"validation","validationCode":"lengths = {x.size(0) for x in examples}\nneeds_pad = len(lengths) > 1 or (pad_to_multiple_of and max(lengths) % pad_to_multiple_of)\nif needs_pad:\n    assert tokenizer.pad_token is not None, (\n        f'{type(tokenizer).__name__} has no pad token; set tokenizer.pad_token = tokenizer.eos_token first'\n    )","typeGuard":"def tokenizer_can_pad(tokenizer) -> bool:\n    return tokenizer.pad_token is not None","tryCatchPattern":null,"preventionTips":["Set tokenizer.pad_token (e.g. = eos_token) immediately after loading any causal-LM tokenizer.","Standardize on one tokenizer setup helper used by every script so pad tokens are never forgotten.","Include one ragged batch in smoke tests so pad-token issues fail fast, not mid-epoch."],"tags":["data-collator","padding","tokenizer","pad-token"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}