huggingface/transformers · error · ValueError

`bos_token_id` has to be defined when no `input_ids` are pro

Error message

`bos_token_id` has to be defined when no `input_ids` are provided.

What it means

ValueError from _maybe_initialize_input_ids_for_generation: no input_ids were given (and no usable inputs_embeds), so generate wants to bootstrap a (batch_size, 1) tensor filled with bos_token_id — but the config/generation_config has bos_token_id=None. Without either a prompt or a BOS id there is nothing valid to start decoding from.

Source

Thrown at src/transformers/generation/utils.py:747

        # soft-prompting or in multimodal implementations built on top of decoder-only language models.
        batch_size = 1
        for value in model_kwargs.values():
            if isinstance(value, torch.Tensor):
                batch_size = value.shape[0]
                break

        if "inputs_embeds" in model_kwargs:
            return torch.ones(
                (batch_size, 0),
                dtype=torch.long,
                # Use the device of the existing tensor to avoid any potential `meta` device issue, which is likely
                # linked to the offloading behavior (keeping it on meta device). See PR #44848. Previously, it used
                # `self.device`.
                device=self.device if self.device.type != "meta" else model_kwargs["inputs_embeds"].device,
            )

        if bos_token_id is None:
            raise ValueError("`bos_token_id` has to be defined when no `input_ids` are provided.")

        return torch.ones((batch_size, 1), dtype=torch.long, device=self.device) * bos_token_id

    def _prepare_position_ids_for_generation(self, inputs_tensor, model_kwargs):
        """
        Tries to infer position ids given attention mask and past kv cache length. All instances when
        `position_ids=None` should call this method.
        """
        # `input_ids` may be present in the model kwargs, instead of being the main input (e.g. multimodal model)
        if "input_ids" in model_kwargs and model_kwargs["input_ids"].shape[1] > 0:
            inputs_tensor = model_kwargs["input_ids"]

        seq_length = inputs_tensor.shape[1]

        if (attention_mask := model_kwargs.get("attention_mask")) is not None:
            position_ids = attention_mask.long().cumsum(-1) - 1
            # We need this as otherwise padding tokens appear as -1 in position
            position_ids = position_ids.masked_fill(attention_mask == 0, 0)

View on GitHub (pinned to a597f97485)

Solutions

1. Pass input_ids explicitly, e.g. input_ids=torch.full((1,1), tokenizer.bos_token_id, ...).
2. Or set model.generation_config.bos_token_id = tokenizer.bos_token_id before calling generate.
3. If you meant conditioned generation, make sure the prompt tensor is actually forwarded (not dropped by a wrapper).

Example fix

# before
out = model.generate(batch_size=1, max_new_tokens=32)  # bos_token_id is None

# after
out = model.generate(input_ids=torch.tensor([[tokenizer.bos_token_id]]), max_new_tokens=32)

bos = getattr(model.generation_config, "bos_token_id", None)
if input_ids is None and bos is None:
    input_ids = torch.full((batch_size, 1), tokenizer.bos_token_id, dtype=torch.long, device=model.device)

When it happens

Trigger: model.generate(batch_size=2, max_new_tokens=50) (prompt-free generation) on a model whose generation_config.bos_token_id is None; models like Whisper or some Gemma configs where bos is not set; passing only attention_mask/model_kwargs without ids.

Common situations: Unconditional sampling scripts; models with stripped generation configs after conversion/quantization; assuming generate() can start from a learned start token that the config does not declare.

Related errors

• File not found: {audio}
• out_indices must be a list, got {type(self._out_indices)}
• out_indices must be valid indices for stage_names {self.stag
• out_indices must not contain any duplicates, got {self._out_
• out_indices must be in the same order as stage_names, expect

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/04bd285b3e992c60. Report an issue: GitHub.