huggingface/transformers · error · ValueError

`decoder_start_token_id` expected to have length {batch_size

Error message

`decoder_start_token_id` expected to have length {batch_size} but got {decoder_start_token_id.shape[0]}

What it means

ValueError in _prepare_decoder_args_for_generation (encoder-decoder path): when decoder_start_token_id is given as a 1-D tensor, its length must equal batch_size, because each sequence in the batch needs its own start token. A mismatch means the per-batch start tokens and the batched encoder inputs disagree.

Source

Thrown at src/transformers/generation/utils.py:873

        decoder_start_token_id: torch.Tensor,
        device: torch.device | None = None,
    ) -> tuple[torch.LongTensor, dict[str, torch.Tensor]]:
        """Prepares `decoder_input_ids` for generation with encoder-decoder models"""
        # 1. Check whether the user has defined `decoder_input_ids` manually. To facilitate in terms of input naming,
        # we also allow the user to pass it under `input_ids`, if the encoder does not use it as the main input.
        if model_kwargs is not None and "decoder_input_ids" in model_kwargs:
            decoder_input_ids = model_kwargs.pop("decoder_input_ids")
        elif "input_ids" in model_kwargs and model_input_name != "input_ids":
            decoder_input_ids = model_kwargs.pop("input_ids")
        else:
            decoder_input_ids = None

        # 2. `decoder_start_token_id` must have shape (batch_size, 1)
        if device is None:
            device = self.device
        if decoder_start_token_id.ndim == 1:
            if decoder_start_token_id.shape[0] != batch_size:
                raise ValueError(
                    f"`decoder_start_token_id` expected to have length {batch_size} but got {decoder_start_token_id.shape[0]}"
                )
            decoder_start_token_id = decoder_start_token_id.view(-1, 1)
        else:
            decoder_start_token_id = (
                torch.ones((batch_size, 1), dtype=torch.long, device=device) * decoder_start_token_id
            )

        # 3. Encoder-decoder models expect the `decoder_input_ids` to start with a special token. Let's ensure that.
        # no user input -> use decoder_start_token_id as decoder_input_ids
        if decoder_input_ids is None:
            decoder_input_ids = decoder_start_token_id
        # exception: Donut checkpoints have task-specific decoder starts and don't expect a BOS token. Note that the
        # original checkpoints can't be detected through `self.__class__.__name__.lower()`, needing custom logic.
        # See: https://github.com/huggingface/transformers/pull/31470
        elif "donut" in self.__class__.__name__.lower() or (
            self.config.model_type == "vision-encoder-decoder" and "donut" in self.config.encoder.model_type.lower()
        ):

View on GitHub (pinned to a597f97485)

Solutions

1. Repeat the tensor to the batch size: decoder_start_token_id=tensor.repeat(batch_size).
2. Or pass a scalar/int start token and let generate broadcast it (torch.ones((batch,1)) * start is created automatically).
3. Derive batch_size from the same source (inputs/encoder_outputs) used for the rest of the call.

Example fix

# before
out = model.generate(**enc_inputs, decoder_start_token_id=start2)  # batch is 4, tensor len 2

# after
out = model.generate(**enc_inputs, decoder_start_token_id=start2.repeat(enc_inputs['input_ids'].shape[0]))

batch_size = inputs.input_ids.shape[0]
if decoder_start_token_id.ndim == 1:
    assert decoder_start_token_id.shape[0] == batch_size, (
        f"decoder_start_token_id len {decoder_start_token_id.shape[0]} != batch {batch_size}"
    )
    # or broadcast: decoder_start_token_id = decoder_start_token_id[:1].repeat(batch_size)

When it happens

Trigger: model.generate(..., batch_size=4, decoder_start_token_id=torch.tensor([a, b])) with a batch of 4; passing one start token per prompt when the encoder batch was expanded (e.g. after num_beams expansion or repeat_interleave); leftover decoder_start_token_id from a previous single-example call.

Common situations: Batch prompt-free generation with per-class start tokens; beam-search code that expands inputs but not the start-token tensor; reusing a cached start tensor across different batch sizes.

Related errors

• You passed `inputs_embeds` and `input_ids` to `.generate()`.
• If `is_encoder_decoder` is True, make sure that `encoder_out
• `decoder_start_token_id` or `bos_token_id` has to be defined
• The batch size is not consistent across layers: {values}
• Expected {len(combined_cache_data) = } to be 4 or 6. {combin

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/d774fe621a48b57d. Report an issue: GitHub.