sgl-project/sglang · error · ValueError

token_ids_logprob contains out-of-vocabulary token id {token

Error message

token_ids_logprob contains out-of-vocabulary token id {token_id}; valid range is [0, {vocab_size}).

What it means

Raised when a token id in token_ids_logprob falls outside [0, vocab_size) for the loaded model. SGLang validates requested logprob tokens against model_config.vocab_size so the sampler does not index nonexistent rows.

Source

Thrown at python/sglang/srt/managers/tokenizer_manager.py:1314

                f"Provided dimensions are greater than max embedding dimension: {self.model_config.hidden_size}"
            )

    def _validate_token_ids_logprob(self, obj: GenerateReqInput) -> None:
        # Batch requests are split into per-request sub-objects before this
        # runs (normalize_batch_and_arguments + __getitem__), so the only
        # legal shape here is the per-request contract of
        # TokenizedGenerateReqInput.token_ids_logprob: a flat list of ints.
        token_ids_logprob = obj.token_ids_logprob
        if not token_ids_logprob:
            return
        if not isinstance(token_ids_logprob, list):
            raise ValueError("token_ids_logprob must be a flat list of integers.")
        vocab_size = self.model_config.vocab_size
        for token_id in token_ids_logprob:
            if not isinstance(token_id, int):
                raise ValueError("token_ids_logprob must be a flat list of integers.")
            if token_id < 0 or token_id >= vocab_size:
                raise ValueError(
                    f"token_ids_logprob contains out-of-vocabulary token id "
                    f"{token_id}; valid range is [0, {vocab_size})."
                )

    def _validate_input_ids_in_vocab(
        self, input_ids: Union[List[int], List[List[int]]], vocab_size: int
    ) -> None:
        # Handle both single sequence and batch of sequences
        if isinstance(input_ids[0], list):
            # Batch of sequences
            for seq in input_ids:
                if any(id >= vocab_size for id in seq):
                    raise ValueError(
                        f"The input_ids {seq} contains values greater than the vocab size ({vocab_size})."
                    )
        else:
            # Single sequence
            if any(id >= vocab_size for id in input_ids):

View on GitHub (pinned to 0132848349)

Solutions

  1. Re-derive the ids with the tokenizer of the served model (tokenizer.encode or decode round-trip)
  2. Print model_config.vocab_size and clamp/filter ids to [0, vocab_size)
  3. Drop negative or oversized ids from token_ids_logprob

Example fix

# before
token_ids_logprob=[200000, 5]  # model vocab_size=32000
# after
token_ids_logprob=[t for t in candidates if 0 <= t < vocab_size]
Defensive patterns

Strategy: validation

Validate before calling

vocab_size = client.get_server_info()['model_config']['vocab_size']
token_ids_logprob = [t for t in token_ids_logprob if 0 <= t < vocab_size]

Type guard

def in_vocab(ids, vocab_size): return all(0 <= t < vocab_size for t in ids)

Try / catch

except ValueError as e: if 'out-of-vocabulary' in str(e): filter ids and retry

Prevention

When it happens

Trigger: Requesting logprobs for token ids from a different tokenizer/vocabulary than the served model, e.g. ids near 151k for a 32k-vocab model, or negative ids.

Common situations: Reusing ids computed with another tokenizer or model revision, hardcoding ids copied from a different model's output, off-by-one vocab size assumptions after a tokenizer upgrade.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ff286bc18e953f00. Report an issue: GitHub.