sgl-project/sglang · error · ValueError

Token ID {token_id} is out of vocabulary (vocab size: {vocab

Error message

Token ID {token_id} is out of vocabulary (vocab size: {vocab_size})

What it means

Raised by score_request when a token id in label_token_ids is >= tokenizer.vocab_size (note: negative ids are NOT checked here and fail elsewhere). Label tokens index the model vocabulary directly, so any id outside [0, vocab_size) is invalid.

Source

Thrown at python/sglang/srt/managers/tokenizer_manager_score_mixin.py:507

            query_embed_overrides is not None or item_embed_overrides is not None
        )
        if has_embeds and embed_override_token_id is None:
            raise ValueError(
                "embed_override_token_id is required when query_embed_overrides "
                "or item_embed_overrides are supplied."
            )
        if item_first and has_embeds:
            raise ValueError("item_first is not supported when embeddings are supplied")
        if item_embed_overrides is not None and len(item_embed_overrides) != len(items):
            raise ValueError(
                f"item_embed_overrides length ({len(item_embed_overrides)}) "
                f"must match items length ({len(items)})."
            )
        if self.tokenizer is not None and label_token_ids is not None:
            vocab_size = self.tokenizer.vocab_size
            for token_id in label_token_ids:
                if token_id >= vocab_size:
                    raise ValueError(
                        f"Token ID {token_id} is out of vocabulary (vocab size: {vocab_size})"
                    )

        # Check if multi-item scoring is enabled
        use_multi_item_scoring = self.server_args.enable_mis

        input_ids = None
        text_prompts = None
        positional_embed_overrides = None
        delimiter_indices = None

        use_text_prompts = isinstance(query, str) and not has_embeds

        if use_text_prompts:
            # Both query and items are text
            items_list = [items] if isinstance(items, str) else items
            if use_multi_item_scoring:
                # Tokenize separately, then combine at token level with placeholder

View on GitHub (pinned to 0132848349)

Solutions

Re-derive label ids with self.tokenizer.convert_tokens_to_ids(label) for the loaded model
Print tokenizer.vocab_size and the offending ids to find the mismatch
If ids come from a config file, regenerate it for the current model checkpoint

Example fix

# before
label_token_ids=[151665, 151666]  # hardcoded from another model
# after
label_token_ids=[tok.convert_tokens_to_ids(t) for t in ["<|good|>", "<|bad|>"]]

Defensive patterns

Strategy: validation

Validate before calling

vocab = engine.tokenizer.vocab_size
assert all(0 <= t < vocab for t in label_token_ids), f"ids outside [0, {vocab})"

Type guard

def valid_label_ids(ids: list[int], tokenizer) -> bool:
    return all(0 <= t < tokenizer.vocab_size for t in ids)

Try / catch

try:
    await engine.async_score(q, d, label_token_ids=ids)
except ValueError as e:
    if "out of vocabulary" in str(e):
        ids = [tok.convert_tokens_to_ids(t) for t in labels]  # re-derive

Prevention

Always derive label ids via convert_tokens_to_ids on the served model's tokenizer
Never hardcode token ids across model versions

When it happens

Trigger: Calling score with label_token_ids containing ids from a different tokenizer/vocabulary, ids computed after special-token offsets, or hardcoded ids copied from another model.

Common situations: Switching the base model but keeping hardcoded label token ids from the previous model; using ids from an added-tokens table that extends past vocab_size; converting a token string with the wrong tokenizer instance.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f8690d202da64a20. Report an issue: GitHub.