sgl-project/sglang · error · ValueError

{label} contains {len(positions)} occurrences of embed_overr

Error message

{label} contains {len(positions)} occurrences of embed_override_token_id={embed_override_token_id}, but {len(embeds)} override embeddings were provided.

What it means

When embed overrides are used, every occurrence of embed_override_token_id in the tokenized sequence must have exactly one matching embedding vector. This error reports the count mismatch between placeholder token occurrences and provided embeddings, per label (query/item).

Source

Thrown at python/sglang/srt/managers/tokenizer_manager_score_mixin.py:288

    def _resolve_overrides_for_sequence(
        self,
        token_ids: List[int],
        embeds: Optional[List[torch.Tensor]],
        embed_override_token_id: int,
        position_offset: int = 0,
        label: str = "input",
    ) -> Tuple[List[torch.Tensor], List[int]]:
        """Scan token_ids for placeholder occurrences and pair with embeddings.
        Returns empty lists when embeds is None."""
        if embeds is None:
            return [], []
        positions = [
            idx + position_offset
            for idx, tok in enumerate(token_ids)
            if tok == embed_override_token_id
        ]
        if len(positions) != len(embeds):
            raise ValueError(
                f"{label} contains {len(positions)} occurrences of "
                f"embed_override_token_id={embed_override_token_id}, "
                f"but {len(embeds)} override embeddings were provided."
            )
        return embeds, positions

    def _resolve_embed_overrides_for_request(
        self,
        query: List[int],
        item: List[int],
        embed_override_token_id: int,
        query_embed_overrides: Optional[List[torch.Tensor]],
        item_embeds: Optional[List[torch.Tensor]],
        item_position_offset: int,
        item_label: str,
    ) -> Optional[PositionalEmbeds]:
        """Resolve embed overrides for a query+item pair; None when no overrides exist."""
        q_embeds, q_positions = self._resolve_overrides_for_sequence(

View on GitHub (pinned to 0132848349)

Solutions

1. Count occurrences of embed_override_token_id in your tokenized query/items and match embeds length exactly
2. Re-tokenize with the same tokenizer/settings used to build the embeddings
3. Ensure separate overrides for query vs item each match their own segment counts

Example fix

# before
embeds = torch.randn(3, 4096)
result = engine.score_request(..., query_embed_overrides=embeds,
    items=["text <placeholder>"], embed_override_token_id=PH_TOK)  # 1 occurrence != 3

# after
n = query_token_ids.count(PH_TOK)
result = engine.score_request(..., query_embed_overrides=embeds[:n], ...)

n_q = sum(1 for t in query_token_ids if t == EMBED_TOK)
n_i = sum(1 for t in item_token_ids if t == EMBED_TOK) if items is not None else 0
assert len(query_embeds or []) == n_q
assert len(item_embeds or []) == n_i

def overrides_match(query_ids, q_embeds, item_ids, i_embeds, tok) -> bool:
    return (len(q_embeds or []) == sum(1 for t in query_ids if t == tok)
            and len(i_embeds or []) == sum(1 for t in (item_ids or []) if t == tok))

try:
    r = engine.score_request(...)
except ValueError as e:
    if "override embeddings were provided" in str(e):
        raise ValueError(f"embed count mismatch: {e}") from e
    raise

When it happens

Trigger: Calling scoring APIs with query_embed_overrides or item_embed_overrides whose length differs from the number of embed_override_token_id placeholders present in query/items token ids; also when position_offset shifts or a tokenizer inserts extra placeholder tokens.

Common situations: Embedding lists built from a different tokenization than the one used server-side; forgetting that both query and item can contain the placeholder; off-by-one after truncation; multimodal embeddings pipelines updated independently of prompts.

Related errors

• embed_override_token_id is required when query_embed_overrid
• Value error, parameter top_n should be larger than 0.
• v_cache must be provided
• q can only be None when only_qv=True
• q must be provided unless qv is provided with only_qv=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8eec1ef267b7a7d4. Report an issue: GitHub.