{"record":{"id":"d774fe621a48b57d","repo":"huggingface/transformers","slug":"decoder-start-token-id-expected-to-have-length","errorCode":null,"errorMessage":"`decoder_start_token_id` expected to have length {batch_size} but got {decoder_start_token_id.shape[0]}","messagePattern":"`decoder_start_token_id` expected to have length (.+?) but got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/transformers/generation/utils.py","lineNumber":873,"sourceCode":"        decoder_start_token_id: torch.Tensor,\n        device: torch.device | None = None,\n    ) -> tuple[torch.LongTensor, dict[str, torch.Tensor]]:\n        \"\"\"Prepares `decoder_input_ids` for generation with encoder-decoder models\"\"\"\n        # 1. Check whether the user has defined `decoder_input_ids` manually. To facilitate in terms of input naming,\n        # we also allow the user to pass it under `input_ids`, if the encoder does not use it as the main input.\n        if model_kwargs is not None and \"decoder_input_ids\" in model_kwargs:\n            decoder_input_ids = model_kwargs.pop(\"decoder_input_ids\")\n        elif \"input_ids\" in model_kwargs and model_input_name != \"input_ids\":\n            decoder_input_ids = model_kwargs.pop(\"input_ids\")\n        else:\n            decoder_input_ids = None\n\n        # 2. `decoder_start_token_id` must have shape (batch_size, 1)\n        if device is None:\n            device = self.device\n        if decoder_start_token_id.ndim == 1:\n            if decoder_start_token_id.shape[0] != batch_size:\n                raise ValueError(\n                    f\"`decoder_start_token_id` expected to have length {batch_size} but got {decoder_start_token_id.shape[0]}\"\n                )\n            decoder_start_token_id = decoder_start_token_id.view(-1, 1)\n        else:\n            decoder_start_token_id = (\n                torch.ones((batch_size, 1), dtype=torch.long, device=device) * decoder_start_token_id\n            )\n\n        # 3. Encoder-decoder models expect the `decoder_input_ids` to start with a special token. Let's ensure that.\n        # no user input -> use decoder_start_token_id as decoder_input_ids\n        if decoder_input_ids is None:\n            decoder_input_ids = decoder_start_token_id\n        # exception: Donut checkpoints have task-specific decoder starts and don't expect a BOS token. Note that the\n        # original checkpoints can't be detected through `self.__class__.__name__.lower()`, needing custom logic.\n        # See: https://github.com/huggingface/transformers/pull/31470\n        elif \"donut\" in self.__class__.__name__.lower() or (\n            self.config.model_type == \"vision-encoder-decoder\" and \"donut\" in self.config.encoder.model_type.lower()\n        ):","sourceCodeStart":855,"sourceCodeEnd":891,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/generation/utils.py#L855-L891","documentation":"ValueError in _prepare_decoder_args_for_generation (encoder-decoder path): when decoder_start_token_id is given as a 1-D tensor, its length must equal batch_size, because each sequence in the batch needs its own start token. A mismatch means the per-batch start tokens and the batched encoder inputs disagree.","triggerScenarios":"model.generate(..., batch_size=4, decoder_start_token_id=torch.tensor([a, b])) with a batch of 4; passing one start token per prompt when the encoder batch was expanded (e.g. after num_beams expansion or repeat_interleave); leftover decoder_start_token_id from a previous single-example call.","commonSituations":"Batch prompt-free generation with per-class start tokens; beam-search code that expands inputs but not the start-token tensor; reusing a cached start tensor across different batch sizes.","solutions":["Repeat the tensor to the batch size: decoder_start_token_id=tensor.repeat(batch_size).","Or pass a scalar/int start token and let generate broadcast it (torch.ones((batch,1)) * start is created automatically).","Derive batch_size from the same source (inputs/encoder_outputs) used for the rest of the call."],"exampleFix":"# before\nout = model.generate(**enc_inputs, decoder_start_token_id=start2)  # batch is 4, tensor len 2\n\n# after\nout = model.generate(**enc_inputs, decoder_start_token_id=start2.repeat(enc_inputs['input_ids'].shape[0]))","handlingStrategy":"validation","validationCode":"batch_size = inputs.input_ids.shape[0]\nif decoder_start_token_id.ndim == 1:\n    assert decoder_start_token_id.shape[0] == batch_size, (\n        f\"decoder_start_token_id len {decoder_start_token_id.shape[0]} != batch {batch_size}\"\n    )\n    # or broadcast: decoder_start_token_id = decoder_start_token_id[:1].repeat(batch_size)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pass a scalar decoder_start_token_id when all sequences share it; generate broadcasts it.","Derive any per-batch tensor from the same batch_size used for encoder inputs, after any beam expansion."],"tags":["generate","encoder-decoder","batch-size","decoder-start-token"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}