huggingface/transformers · error · ValueError

`decoder_start_token_id` or `bos_token_id` has to be defined

Error message

`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation.

What it means

Encoder-decoder generation must know which token starts the decoder. It resolves `decoder_start_token_id` (falling back to `bos_token_id`) from the generation config/model config; if the resulting `decoder_start_token_tensor` is None, `generate` cannot construct `decoder_input_ids` and raises.

Source

Thrown at src/transformers/generation/utils.py:2091

            )

        # We can have more than one eos token. Always treat it as a 1D tensor (when it exists).
        if eos_token_tensor is not None and eos_token_tensor.ndim == 0:
            eos_token_tensor = eos_token_tensor.unsqueeze(0)

        # Set pad token if unset (and there are conditions to do so)
        if pad_token_tensor is None and eos_token_tensor is not None:
            # Only emits the warnings if batch_size>1, as batch_size==1 means no padding, thus no problems
            if kwargs_has_attention_mask is not None and not kwargs_has_attention_mask and is_batched_sequence:
                logger.warning(
                    "The attention mask and the pad token id were not set, with a batched input. As a consequence, you may "
                    "observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results."
                )
            pad_token_tensor = eos_token_tensor[0]

        # Sanity checks/warnings
        if self.config.is_encoder_decoder and decoder_start_token_tensor is None:
            raise ValueError(
                "`decoder_start_token_id` or `bos_token_id` has to be defined for encoder-decoder generation."
            )
        if eos_token_tensor is not None and torch.isin(eos_token_tensor, pad_token_tensor).any():
            # Only emits the warning if batch_size>1, as batch_size==1 means no padding, thus no problems
            if kwargs_has_attention_mask is not None and not kwargs_has_attention_mask and is_batched_sequence:
                logger.warning_once(
                    "The attention mask is not set with a batched input, and cannot be inferred from input because pad token "
                    "is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's "
                    "`attention_mask` to obtain reliable results."
                )
        if eos_token_tensor is not None and (
            torch.is_floating_point(eos_token_tensor) or (eos_token_tensor < 0).any()
        ):
            logger.warning(
                f"`eos_token_id` should consist of positive integers, but is {eos_token_tensor}. Your generation "
                "will not stop until the maximum length is reached. Depending on other flags, it may even crash."
            )

View on GitHub (pinned to a597f97485)

Solutions

Set it on the generation config: `model.generation_config.decoder_start_token_id = tokenizer.pad_token_id` (or the correct start id, e.g. `tokenizer.bos_token_id` / `tokenizer.convert_tokens_to_ids(tokenizer.lang_code_to_token["en"])` for mBART-style models).
Or pass it per call: `model.generate(**enc, decoder_start_token_id=start_id)`.
Or supply `decoder_input_ids` explicitly so the start token is not needed.
Persist the fix: update the checkpoint's `generation_config.json` so reloads work.

Example fix

# before
out = model.generate(input_ids=encoder_input_ids)  # decoder start undefined -> ValueError

# after
model.generation_config.decoder_start_token_id = tokenizer.bos_token_id
out = model.generate(input_ids=encoder_input_ids)

Defensive patterns

Strategy: validation

Validate before calling

if model.config.is_encoder_decoder:
    gc = model.generation_config
    if gc.decoder_start_token_id is None and gc.bos_token_id is None:
        gc.decoder_start_token_id = tokenizer.bos_token_id if tokenizer.bos_token_id is not None else tokenizer.pad_token_id

Prevention

For seq2seq checkpoints you train/export yourself, always set decoder_start_token_id in generation_config.json.
Smoke-test generate() after loading a fine-tuned encoder-decoder model.
Know your model's start token convention (T5 pad, mBART language code) and pin it explicitly.

When it happens

Trigger: `model.generate(**encoder_inputs)` on an encoder-decoder model whose `generation_config.decoder_start_token_id` and `.bos_token_id` are both None — common with custom-trained seq2seq models or hand-built `GenerationConfig`s; calling `generate(input_ids=...)` without `decoder_input_ids`.

Common situations: Fine-tuned T5/BART/Whisper-style checkpoints where the start token was implied by trainer code but never stored in the config; programmatic `GenerationConfig(...)` omitting it; converting checkpoints from other frameworks that drop the field.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/299bf79a0cf7cac8. Report an issue: GitHub.