sgl-project/sglang · error · ValueError

input contains {len(positions)} occurrences of embed_overrid

Error message

input contains {len(positions)} occurrences of embed_override_token_id={token_id}, but embed_overrides has {len(embeds)} entries.

What it means

Raised by _resolve_embed_overrides when the count of placeholder tokens equal to embed_override_token_id in input_ids does not match the number of tensors in embed_overrides. Each override tensor must replace exactly one placeholder occurrence, so counts must match 1:1.

Source

Thrown at python/sglang/srt/managers/tokenizer_manager.py:1461

                http_worker_ipc=obj.http_worker_ipc,
                return_pooled_hidden_states=obj.return_pooled_hidden_states,
                multi_item_delimiter_indices=obj.multi_item_delimiter_indices,
            )

        tokenized_obj.time_stats = self.rid_to_state[obj.rid].time_stats
        self.rid_to_state[obj.rid].time_stats.set_tokenize_finish_time()

        return tokenized_obj

    @staticmethod
    def _resolve_embed_overrides(
        input_ids: array[int],
        token_id: int,
        embeds: List[torch.Tensor],
    ) -> PositionalEmbeds:
        positions = [idx for idx, tok in enumerate(input_ids) if tok == token_id]
        if len(positions) != len(embeds):
            raise ValueError(
                f"input contains {len(positions)} occurrences of "
                f"embed_override_token_id={token_id}, "
                f"but embed_overrides has {len(embeds)} entries."
            )
        return PositionalEmbeds(embeds=embeds, positions=positions)

    async def _batch_tokenize_and_process(
        self, batch_size: int, obj: Union[GenerateReqInput, EmbeddingReqInput]
    ) -> List[Union[TokenizedGenerateReqInput, TokenizedEmbeddingReqInput]]:
        """Handle batch tokenization for text inputs only."""
        logger.debug(f"Starting batch tokenization for {batch_size} text requests")

        # If batch does not have text nothing to tokenize
        # so lets construct the return object
        if not self._batch_has_text(batch_size, obj):
            # All requests already have input_ids, no need to tokenize
            return [await self._tokenize_one_request(obj[i]) for i in range(batch_size)]

View on GitHub (pinned to 0132848349)

Solutions

1. Count placeholders: input_ids.count(embed_override_token_id) must equal len(embed_overrides)
2. Adjust the prompt/template so placeholder count matches the number of override embeddings
3. Regenerate embed_overrides from the same source that produced the tokenized input

Example fix

# before
input_ids=[1, 999, 999, 2]; embed_overrides=[img_emb]  # 1 embed, 2 placeholders
# after
input_ids=[1, 999, 2]; embed_overrides=[img_emb]

n_placeholders = sum(input_ids.count(t) if isinstance(input_ids[0], list) else input_ids.count(t) for t in [embed_override_token_id])
assert n_placeholders == len(embed_overrides), f'{n_placeholders} placeholders vs {len(embed_overrides)} embeds'

except ValueError as e: if 'embed_override_token_id' in str(e): recount placeholders and rebuild overrides

When it happens

Trigger: Calling generate with input_embeds overrides where embed_overrides has fewer/more tensors than the number of embed_override_token_id tokens present in the tokenized input, e.g. num image patches changed but the placeholder token count in the text did not.

Common situations: Editing prompt templates that contain a different number of placeholder tokens than the multimodal encoder produced embeddings for; chunking embeddings without adjusting placeholders; stale template after model changes.

Related errors

• {selection_error}{component_suffix}
• No compatible attention backend is available{component_suffi
• output_ws should be prepared for cuda-graph mode
• sequence_lengths should be prepared for vision flashinfer_cu
• MM inputs where only some items are precomputed.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a136d84d1ec7ef3c. Report an issue: GitHub.