sgl-project/sglang · error · ValueError

logit_bias must has keys in [0, {vocab_size - 1}], got {toke

Error message

logit_bias must has keys in [0, {vocab_size - 1}], got {token_id}.

What it means

Every key of logit_bias must be a token id in [0, vocab_size). verify() checks each key against the model's vocabulary size because a bias on a nonexistent token id would silently do nothing or crash the sampler.

Source

Thrown at python/sglang/srt/sampling/sampling_params.py:213

                    f"min_new_tokens must be in [0, max_new_tokens({self.max_new_tokens})], got "
                    f"{self.min_new_tokens}."
                )
        if self.logit_bias is not None:
            for token_id in self.logit_bias:
                if not 0 <= int(token_id) < vocab_size:
                    raise ValueError(
                        f"logit_bias must has keys in [0, {vocab_size - 1}], got "
                        f"{token_id}."
                    )

        grammars = [
            self.json_schema,
            self.regex,
            self.ebnf,
            self.structural_tag,
        ]  # since mutually exclusive, only one can be set
        if sum(x is not None for x in grammars) > 1:
            raise ValueError(
                "Only one of json_schema, regex, ebnf, or structural_tag can be set."
            )

    def normalize(self, tokenizer):
        # Process stop strings
        if self.stop_strs is None:
            self.stop_strs = []
            self.stop_str_max_len = 0
        else:
            if isinstance(self.stop_strs, str):
                self.stop_strs = [self.stop_strs]

            stop_str_max_len = 0
            for stop_str in self.stop_strs:
                if tokenizer is not None:
                    stop_str_ids = tokenizer.encode(stop_str, add_special_tokens=False)
                    stop_str_max_len = max(stop_str_max_len, len(stop_str_ids))
                else:

View on GitHub (pinned to 0132848349)

Solutions

1. Re-encode the bias keys with the tokenizer of the served model (tokenizer.encode(token)).
2. Ensure every key satisfies 0 <= key < vocab_size; drop out-of-range ids.
3. Verify you are hitting the intended model endpoint.

Example fix

# before
logit_bias = {130000: -100}  # id from a 256k-vocab tokenizer
# after
logit_bias = {tid: -100 for tok in ["foo"] if (tid := tokenizer.encode(tok, add_special_tokens=False)[0]) < tokenizer.vocab_size}

VOCAB = tokenizer.vocab_size
logit_bias = {int(k): v for k, v in logit_bias.items() if 0 <= int(k) < VOCAB}
if not logit_bias:
    logit_bias = None
sp = SamplingParams(logit_bias=logit_bias)

def valid_logit_bias(bias: dict, vocab_size: int) -> bool:
    return all(0 <= int(k) < vocab_size for k in bias)

try:
    sp = SamplingParams(logit_bias=raw).normalize(tokenizer)
except ValueError as e:
    if 'logit_bias' in str(e):
        log.warning('dropping invalid logit_bias keys'); raw = None
    else:
        raise

When it happens

Trigger: Passing SamplingParams(logit_bias={128300: -100}) with a model whose vocab size is smaller (e.g. ~128k for Llama vs 32k for another tokenizer); ids obtained from a different tokenizer than the served model.

Common situations: Reusing logit_bias dicts tuned on one model against a different model/tokenizer; computing ids via tokenizer A while serving model B; off-by-one using id == vocab_size.

Related errors

• beam_width must be at least 1, got {self.beam_width}.
• temperature must be a non-negative finite number, got {self.
• top_p must be in (0, 1], got {self.top_p}.
• min_p must be in [0, 1], got {self.min_p}.
• top_k must be -1 (disable) or at least 1, got {self.top_k}.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/dbde6445feeaab3a. Report an issue: GitHub.