huggingface/transformers · error · ValueError

inputs must be a 1D or 2D tensor, got {inputs.dim() = }

Error message

inputs must be a 1D or 2D tensor, got {inputs.dim() = }

What it means

When switching to continuous batching, generate converts the input tensor into a Python list of per-request token lists and only understands 1D (single unpadded sequence) or 2D (batch of sequences) tensors. A tensor with any other rank cannot be mapped to requests and raises, echoing the offending dimension count.

Source

Thrown at src/transformers/generation/utils.py:2408

        # 0.b. If requested, switched to continuous batching generation
        if kwargs.get("cache_implementation") == "paged":
            logger.warning(
                "Detected cache_implementation=paged: switching to continuous batching. You should consider using "
                "generate_batch directly instead."
            )

            # generate_batch expects a list of lists of ints, so we create it from the inputs or input_ids
            inputs = inputs if inputs is not None else kwargs.get("input_ids")
            if inputs is None:
                raise ValueError("inputs or input_ids must be provided for CB generation.")

            if inputs.dim() == 1:
                inputs = inputs.unsqueeze(0).tolist()
            elif inputs.dim() == 2:
                inputs = inputs.tolist()
            else:
                raise ValueError(f"inputs must be a 1D or 2D tensor, got {inputs.dim() = }")

            # some arguments are not supported for continuous batching
            if stopping_criteria is not None:
                raise NotImplementedError(
                    f"stopping_criteria is not supported for continuous batching. Got {stopping_criteria = }"
                )
            if prefix_allowed_tokens_fn is not None:
                raise NotImplementedError(
                    f"prefix_allowed_tokens_fn is not supported for continuous batching. Got {prefix_allowed_tokens_fn = }"
                )
            if assistant_model is not None:
                raise NotImplementedError(
                    f"assistant_model is not supported for continuous batching. Got {assistant_model = }"
                )
            if streamer is not None:  # TODO: actually this could be supported
                raise NotImplementedError(f"streaming is not supported for continuous batching. Got {streamer = }")
            if negative_prompt_ids is not None:
                raise NotImplementedError(

View on GitHub (pinned to a597f97485)

Solutions

1. Squeeze spurious axes: `input_ids = input_ids.squeeze(1)` (verify shape is [batch, seq] or [seq]) before the call.
2. If the extra dim is beams, drop beam shaping — continuous batching manages its own scheduling.
3. Pass what the backend expects: token ids only, not embeddings/hidden states.
4. Log `inputs.shape` right before generate to catch rank drift early.

Example fix

# before
input_ids = input_ids.unsqueeze(0)  # -> shape [1, 1, seq], dim()==3
out = model.generate(inputs=input_ids, cache_implementation="paged")  # ValueError: got dim = 3

# after
input_ids = input_ids.squeeze(0)  # ensure [batch, seq] or [seq]
out = model.generate(inputs=input_ids, cache_implementation="paged")

if inputs.dim() > 2:
    raise ValueError(f"continuous batching needs 1D/2D input_ids, got shape {tuple(inputs.shape)}")

def is_cb_ready(t) -> bool:
    return t.dim() in (1, 2)

When it happens

Trigger: `model.generate(inputs=input_ids, cache_implementation="paged")` where `input_ids.dim()` is 3+ — e.g. tensors shaped [batch, num_beams, seq] from beam prep, extra leading axes from an encoder pass, or mistakenly passed encoder hidden states instead of token ids.

Common situations: Reusing tensors produced by earlier beam-search or shaping code (extra beam/axis dimension); passing `decoder_input_ids` reshaped for multi-beam; passing embeddings or hidden states where token ids are expected; pre-processing pipelines that add a spurious axis.

Related errors

• inputs or input_ids must be provided for CB generation.
• stopping_criteria is not supported for continuous batching.
• PUSH_TO_HUB_TOKEN is not set, cannot push results to the Hub
• Unknown backend {backend!r}; expected 'auto', 'torchcodec',
• `crop` was called, but the current layer does not track past

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/81022472a36e2f88. Report an issue: GitHub.