{"record":{"id":"dbde6445feeaab3a","repo":"sgl-project/sglang","slug":"logit-bias-must-has-keys-in-0-vocab-size-1","errorCode":null,"errorMessage":"logit_bias must has keys in [0, {vocab_size - 1}], got {token_id}.","messagePattern":"logit_bias must has keys in \\[0, (.+?)\\], got (.+?)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/sampling/sampling_params.py","lineNumber":213,"sourceCode":"                    f\"min_new_tokens must be in [0, max_new_tokens({self.max_new_tokens})], got \"\n                    f\"{self.min_new_tokens}.\"\n                )\n        if self.logit_bias is not None:\n            for token_id in self.logit_bias:\n                if not 0 <= int(token_id) < vocab_size:\n                    raise ValueError(\n                        f\"logit_bias must has keys in [0, {vocab_size - 1}], got \"\n                        f\"{token_id}.\"\n                    )\n\n        grammars = [\n            self.json_schema,\n            self.regex,\n            self.ebnf,\n            self.structural_tag,\n        ]  # since mutually exclusive, only one can be set\n        if sum(x is not None for x in grammars) > 1:\n            raise ValueError(\n                \"Only one of json_schema, regex, ebnf, or structural_tag can be set.\"\n            )\n\n    def normalize(self, tokenizer):\n        # Process stop strings\n        if self.stop_strs is None:\n            self.stop_strs = []\n            self.stop_str_max_len = 0\n        else:\n            if isinstance(self.stop_strs, str):\n                self.stop_strs = [self.stop_strs]\n\n            stop_str_max_len = 0\n            for stop_str in self.stop_strs:\n                if tokenizer is not None:\n                    stop_str_ids = tokenizer.encode(stop_str, add_special_tokens=False)\n                    stop_str_max_len = max(stop_str_max_len, len(stop_str_ids))\n                else:","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/sampling/sampling_params.py#L195-L231","documentation":"Every key of logit_bias must be a token id in [0, vocab_size). verify() checks each key against the model's vocabulary size because a bias on a nonexistent token id would silently do nothing or crash the sampler.","triggerScenarios":"Passing SamplingParams(logit_bias={128300: -100}) with a model whose vocab size is smaller (e.g. ~128k for Llama vs 32k for another tokenizer); ids obtained from a different tokenizer than the served model.","commonSituations":"Reusing logit_bias dicts tuned on one model against a different model/tokenizer; computing ids via tokenizer A while serving model B; off-by-one using id == vocab_size.","solutions":["Re-encode the bias keys with the tokenizer of the served model (tokenizer.encode(token)).","Ensure every key satisfies 0 <= key < vocab_size; drop out-of-range ids.","Verify you are hitting the intended model endpoint."],"exampleFix":"# before\nlogit_bias = {130000: -100}  # id from a 256k-vocab tokenizer\n# after\nlogit_bias = {tid: -100 for tok in [\"foo\"] if (tid := tokenizer.encode(tok, add_special_tokens=False)[0]) < tokenizer.vocab_size}","handlingStrategy":"validation","validationCode":"VOCAB = tokenizer.vocab_size\nlogit_bias = {int(k): v for k, v in logit_bias.items() if 0 <= int(k) < VOCAB}\nif not logit_bias:\n    logit_bias = None\nsp = SamplingParams(logit_bias=logit_bias)","typeGuard":"def valid_logit_bias(bias: dict, vocab_size: int) -> bool:\n    return all(0 <= int(k) < vocab_size for k in bias)","tryCatchPattern":"try:\n    sp = SamplingParams(logit_bias=raw).normalize(tokenizer)\nexcept ValueError as e:\n    if 'logit_bias' in str(e):\n        log.warning('dropping invalid logit_bias keys'); raw = None\n    else:\n        raise","preventionTips":["Always derive bias ids with the served model's tokenizer.","Add a vocab-size assertion in test suites that pin logit_bias dicts."],"tags":["sampling-params","logit-bias","vocab-size","validation","sglang"],"backgroundTag":"token-id-out-of-vocab","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}