The parameter max_tokens will be overwritten by speculated n

Error message

The parameter max_tokens will be overwritten by speculated number of tokens.

What it means

Warning from the OpenAI backend's speculative-execution preparation: when building spec_kwargs from your sampling_params, the max_tokens key is deliberately dropped because the runtime overwrites it with the speculated number of tokens. It tells you any max_tokens you set will not take effect.

Source

Thrown at python/sglang/lang/backend/openai.py:125

        return self.chat_template

    def _prepare_spec_execution(
        self,
        sampling_params: SglSamplingParams,
        num_api_spec_tokens: int,
        spec_var_name: str,
    ):
        if "max_tokens" not in self.spec_kwargs:
            self.spec_kwargs["max_tokens"] = num_api_spec_tokens
        else:
            assert self.spec_kwargs["max_tokens"] == num_api_spec_tokens

        params = sampling_params.to_openai_kwargs()
        for key, value in params.items():
            if key in ["stop"]:
                continue
            if key in ["max_tokens"]:
                warnings.warn(
                    "The parameter max_tokens will be overwritten by speculated number of tokens."
                )
                continue
            if key not in self.spec_kwargs:
                self.spec_kwargs[key] = value
            else:
                assert (
                    value == self.spec_kwargs[key]
                ), "sampling parameters should be consistent if turn on api speculative execution."
        self.spec_format.append(
            {"text": "", "stop": params["stop"], "name": spec_var_name}
        )
        return "", {}

    def generate(
        self,
        s: StreamExecutor,
        sampling_params: SglSamplingParams,

View on GitHub (pinned to 0132848349)

Solutions

Remove max_tokens from sampling_params when using spec execution and control length via stop strings or the speculation budget instead
If you need a hard token cap, use stop conditions or post-truncate the output
Pin the speculated token count if the API exposes it (spec-level / num_speculative_tokens) rather than max_tokens

Example fix

# before
gen("Hi", max_tokens=100)  # ignored under spec execution
# after
gen("Hi", stop=["\n\n"])  # length controlled by speculation budget + stop

Defensive patterns

Strategy: validation

Validate before calling

speculative = backend is openai_spec
if speculative and "max_tokens" in sampling_params.to_openai_kwargs():
    del sampling_params.max_tokens  # will be overwritten anyway

Prevention

Don't rely on max_tokens for length control under speculative execution
Use stop sequences or external truncation for output length limits
Document which sampling params are honored per backend in your wrapper

When it happens

Trigger: Calling generate() with speculative execution enabled on the OpenAI backend while sampling_params includes max_tokens (which to_openai_kwargs always includes).

Common situations: Writing SGL frontend programs with the OpenAI backend and setting max_tokens expecting it to bound generation length; behavior differs from the non-spec path where max_tokens is honored.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/27d3ea170ecc7ba1. Report an issue: GitHub.