huggingface/transformers · error · ValueError

Input ids should be of shape (batch_size, input_len), but is

Error message

Input ids should be of shape (batch_size, input_len), but is {input_ids.shape}

What it means

Raised by the n-gram-based logits processor in transformers generation (the one exposing _check_input_ids_shape / compute_g_values) when the input_ids tensor passed to the processor is not 2-D. The processor builds n-gram keys per (batch, sequence) position, so it requires input_ids of shape (batch_size, input_len). Any extra or missing dimension (e.g. a single unbatched sequence of shape (input_len,) or a (batch, seq, 1) tensor) triggers it.

Source

Thrown at src/transformers/generation/logits_process.py:2896

        we pre-compute a random sampling table, and use apply modulo table size to
        map from ngram keys (int64) to g values.

        Args:
            ngram_keys (`torch.LongTensor`):
                Random keys (batch_size, num_ngrams, depth).

        Returns:
            G values (batch_size, num_ngrams, depth).
        """
        (sampling_table_size,) = self.sampling_table.shape
        sampling_table = self.sampling_table.reshape((1, 1, sampling_table_size))
        ngram_keys = ngram_keys % sampling_table_size
        return torch.take_along_dim(sampling_table, indices=ngram_keys, dim=2)

    def _check_input_ids_shape(self, input_ids: torch.LongTensor):
        """Checks the shape of input ids."""
        if len(input_ids.shape) != 2:
            raise ValueError(f"Input ids should be of shape (batch_size, input_len), but is {input_ids.shape}")

    def compute_g_values(self, input_ids: torch.LongTensor) -> torch.LongTensor:
        """
        Computes g values for each ngram from the given sequence of tokens.

        Args:
            input_ids (`torch.LongTensor`):
                Input token ids (batch_size, input_len).

        Returns:
            G values (batch_size, input_len - (ngram_len - 1), depth).
        """
        self._check_input_ids_shape(input_ids)
        ngrams = input_ids.unfold(dimension=1, size=self.ngram_len, step=1)
        ngram_keys = self.compute_ngram_keys(ngrams)
        return self.sample_g_values(ngram_keys)

    def compute_context_repetition_mask(self, input_ids: torch.LongTensor) -> torch.LongTensor:

View on GitHub (pinned to a597f97485)

Solutions

  1. Reshape input_ids to 2-D before calling the API: input_ids = input_ids.reshape(-1, input_ids.shape[-1]) or input_ids[None, :] for a single sequence.
  2. If you already have (batch, seq, X), squeeze the last dim only if it is size 1 and inspect where the extra dim was introduced.
  3. Check upstream code that produced input_ids (tokenizer output is always 2-D; something after tokenization altered the shape).

Example fix

// before
seq = tokenizer(text, return_tensors="pt").input_ids
processor.compute_g_values(seq[0])  # rank-1, raises

// after
g_values = processor.compute_g_values(seq)  # keep (batch, seq) 2-D
Defensive patterns

Strategy: validation

Validate before calling

def as_2d_input_ids(input_ids: torch.Tensor) -> torch.Tensor:
    if input_ids.dim() == 1:
        return input_ids.unsqueeze(0)
    if input_ids.dim() == 3 and input_ids.size(-1) == 1:
        return input_ids.squeeze(-1)
    assert input_ids.dim() == 2, f"expected (batch, seq), got {tuple(input_ids.shape)}"
    return input_ids

Type guard

def is_valid_input_ids(t: torch.Tensor) -> bool:
    return isinstance(t, torch.Tensor) and t.dim() == 2 and t.dtype in (torch.long, torch.int)

Prevention

When it happens

Trigger: Calling compute_g_values (or a generate() path that routes through this processor) with input_ids of rank != 2; passing unbatched token ids like torch.tensor([1,2,3]) instead of [[1,2,3]]; passing input_ids with a trailing dimension from a custom forward hook.

Common situations: Custom generation loops or watermarking/n-gram sampling experiments where the user slices or unsqueezes input_ids manually; multimodal pipelines where input_ids come shaped (batch, seq, extra).

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/1c73c28feb743948. Report an issue: GitHub.