huggingface/transformers · error · ValueError

This collator requires that sequence lengths be even to crea

Error message

This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.

What it means

Raised by DataCollatorForPermutationLanguageModeling.torch_call when inputs.size(1) (sequence length) is odd. The collator builds a leakage-free perm_mask/target_mapping pair for the two-stream XLNet attention; the construction as implemented requires an even number of positions per sequence, so odd-length batches are rejected rather than producing a subtly wrong mask.

Source

Thrown at src/transformers/data/data_collator.py:1188

            0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
            1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
            2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be
               masked
            3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length -
               span_length]` and mask tokens `start_index:start_index + span_length`
            4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the
               sequence to be processed), repeat from Step 1.
        """
        import torch

        if self.tokenizer.mask_token is None:
            raise ValueError(
                "This tokenizer does not have a mask token which is necessary for permutation language modeling."
                " Please add a mask token if you want to use this tokenizer."
            )

        if inputs.size(1) % 2 != 0:
            raise ValueError(
                "This collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see"
                " relevant comments in source code for details."
            )

        labels = inputs.clone()
        # Creating the mask and target_mapping tensors
        masked_indices = torch.full(labels.shape, 0, dtype=torch.bool)
        target_mapping = torch.zeros((labels.size(0), labels.size(1), labels.size(1)), dtype=torch.float32)

        for i in range(labels.size(0)):
            # Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far).
            cur_len = 0
            max_len = labels.size(1)

            while cur_len < max_len:
                # Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked)
                span_length = torch.randint(1, self.max_span_length + 1, (1,)).item()
                # Reserve a context of length `context_length = span_length / plm_probability` to surround the span to be masked

View on GitHub (pinned to a597f97485)

Solutions

1. Set tokenizer.padding_side='right' and pad to an even length, e.g. DataCollator... with padding='max_length', max_length=128, or pad_to_multiple_of=2 on the tokenizer.
2. Use a tokenizer with a pad token (tokenizer.pad_token) so the collator pads the batch to a common even length.
3. Truncate/pad inputs to an even max_length before collating.

Example fix

# before
tok = AutoTokenizer.from_pretrained('xlnet-base-cased')  # no padding configured
collator = DataCollatorForPermutationLanguageModeling(tokenizer=tok)
batch = collator(samples)  # odd seq len -> raises

# after
from transformers import DataCollatorForPermutationLanguageModeling
collator = DataCollatorForPermutationLanguageModeling(tok)
encoded = tok(texts, padding='max_length', max_length=128, truncation=True)
batch = collator([dict(e) for e in encoded])

def pad_to_even(tokenizer, texts, max_length):
    max_length += max_length % 2
    return tokenizer(texts, padding='max_length', max_length=max_length, truncation=True)

enc = pad_to_even(tokenizer, texts, max_length=127)  # becomes 128

When it happens

Trigger: Calling the collator on a batch whose padded sequence length is odd, e.g. max length 15 after padding, or a tokenizer with no padding that yields odd-length samples.

Common situations: Tokenizing without pad_to_max_length and letting dynamic padding produce an odd max length; truncating to max_length=255; batching a single odd-length sample.

Related errors

• You are attempting to pad samples but the tokenizer you are
• This tokenizer does not have a mask token which is necessary
• Framework '{return_tensors}' not recognized!
• This tokenizer does not have a mask token which is necessary
• mlm_probability should be between 0 and 1.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a8bc9b6d1e6769ca. Report an issue: GitHub.