{"record":{"id":"27d3ea170ecc7ba1","repo":"sgl-project/sglang","slug":"the-parameter-max-tokens-will-be-overwritten-by-sp","errorCode":null,"errorMessage":"The parameter max_tokens will be overwritten by speculated number of tokens.","messagePattern":"The parameter max_tokens will be overwritten by speculated number of tokens\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"python/sglang/lang/backend/openai.py","lineNumber":125,"sourceCode":"        return self.chat_template\n\n    def _prepare_spec_execution(\n        self,\n        sampling_params: SglSamplingParams,\n        num_api_spec_tokens: int,\n        spec_var_name: str,\n    ):\n        if \"max_tokens\" not in self.spec_kwargs:\n            self.spec_kwargs[\"max_tokens\"] = num_api_spec_tokens\n        else:\n            assert self.spec_kwargs[\"max_tokens\"] == num_api_spec_tokens\n\n        params = sampling_params.to_openai_kwargs()\n        for key, value in params.items():\n            if key in [\"stop\"]:\n                continue\n            if key in [\"max_tokens\"]:\n                warnings.warn(\n                    \"The parameter max_tokens will be overwritten by speculated number of tokens.\"\n                )\n                continue\n            if key not in self.spec_kwargs:\n                self.spec_kwargs[key] = value\n            else:\n                assert (\n                    value == self.spec_kwargs[key]\n                ), \"sampling parameters should be consistent if turn on api speculative execution.\"\n        self.spec_format.append(\n            {\"text\": \"\", \"stop\": params[\"stop\"], \"name\": spec_var_name}\n        )\n        return \"\", {}\n\n    def generate(\n        self,\n        s: StreamExecutor,\n        sampling_params: SglSamplingParams,","sourceCodeStart":107,"sourceCodeEnd":143,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/lang/backend/openai.py#L107-L143","documentation":"Warning from the OpenAI backend's speculative-execution preparation: when building spec_kwargs from your sampling_params, the max_tokens key is deliberately dropped because the runtime overwrites it with the speculated number of tokens. It tells you any max_tokens you set will not take effect.","triggerScenarios":"Calling generate() with speculative execution enabled on the OpenAI backend while sampling_params includes max_tokens (which to_openai_kwargs always includes).","commonSituations":"Writing SGL frontend programs with the OpenAI backend and setting max_tokens expecting it to bound generation length; behavior differs from the non-spec path where max_tokens is honored.","solutions":["Remove max_tokens from sampling_params when using spec execution and control length via stop strings or the speculation budget instead","If you need a hard token cap, use stop conditions or post-truncate the output","Pin the speculated token count if the API exposes it (spec-level / num_speculative_tokens) rather than max_tokens"],"exampleFix":"# before\ngen(\"Hi\", max_tokens=100)  # ignored under spec execution\n# after\ngen(\"Hi\", stop=[\"\\n\\n\"])  # length controlled by speculation budget + stop","handlingStrategy":"validation","validationCode":"speculative = backend is openai_spec\nif speculative and \"max_tokens\" in sampling_params.to_openai_kwargs():\n    del sampling_params.max_tokens  # will be overwritten anyway","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Don't rely on max_tokens for length control under speculative execution","Use stop sequences or external truncation for output length limits","Document which sampling params are honored per backend in your wrapper"],"tags":["openai-backend","speculative-decoding","sampling-params","max-tokens","ignored-parameter"],"backgroundTag":"ignored-parameter-warning","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}