sgl-project/sglang · error · ValueError

SGLANG_RUST_SERVER does not yet apply --preferred-sampling-p

Error message

SGLANG_RUST_SERVER does not yet apply --preferred-sampling-params (the Python TokenizerManager merges it into every request; the rust ingress has no equivalent). Launch without SGLANG_RUST_SERVER, or drop --preferred-sampling-params and send those values per request.

What it means

The experimental Rust HTTP ingress replaces the Python TokenizerManager, which is what merges --preferred-sampling-params into every request. Since the rust path has no equivalent, SGLang refuses to start rather than silently dropping server-configured sampling defaults.

Source

Thrown at python/sglang/srt/managers/rust_server.py:395

        The caller gates this (``SGLANG_RUST_SERVER`` + rank 0); this always
        creates.
        """
        from sglang.srt.rust_extensions import load_rust_extension

        Server = load_rust_extension("sglang.srt.rust_extensions._server").Server

        # Force turn off HF tokenizers rayon's unpinned global thread pool.
        os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

        server_args = scheduler.server_args
        # `TokenizerManager` merges these under each request's own sampling params
        # (`{**preferred, **obj.sampling_params}`), and this server replaces that
        # manager wholesale — so honouring the flag is not implemented here yet.
        # Refuse rather than run: silently dropping it means generating with
        # sampling the operator did not configure, and `/get_model_info` would go on
        # advertising values no request ever receives.
        if get_serving().preferred_sampling_params:
            raise ValueError(
                "SGLANG_RUST_SERVER does not yet apply --preferred-sampling-params "
                "(the Python TokenizerManager merges it into every request; the rust "
                "ingress has no equivalent). Launch without SGLANG_RUST_SERVER, or "
                "drop --preferred-sampling-params and send those values per request."
            )
        http_addr = f"{get_serving().host}:{server_args.port}"

        # Per-DP-rank HTTP port with client load balancing. `None` when DP is off,
        # so the rank is not conflated with rank 0 of a one-rank group.
        dp_rank = scheduler.ps.attn_dp_rank if scheduler.ps.dp_size > 1 else None
        if dp_rank is not None:
            http_addr = f"{get_serving().host}:{server_args.port + dp_rank}"

        launch_cores, server_cores = cls._partition_cores(
            mm_workers=(
                (server_args.mm_processor_worker_num or NativeMmHost.AUTO_MM_WORKERS)
                if scheduler.model_config.is_multimodal
                else 0

View on GitHub (pinned to 0132848349)

Solutions

1. Drop --preferred-sampling-params and send sampling values per request
2. Or unset SGLANG_RUST_SERVER and use the Python ingress

Example fix

# before
SGLANG_RUST_SERVER=1 python -m sglang.launch_server --model ... --preferred-sampling-params '{"temperature":0.7}'
# after
SGLANG_RUST_SERVER=1 python -m sglang.launch_server --model ...

import os
if os.environ.get('SGLANG_RUST_SERVER'):
    assert not preferred_sampling_params, 'incompatible with rust ingress'

When it happens

Trigger: Launching with SGLANG_RUST_SERVER=1 together with --preferred-sampling-params (e.g. preferred_sampling_params JSON on the CLI/ServerArgs).

Common situations: Copy-pasting a Python-server launch command and adding the rust flag; benchmarking scripts carrying both options.

Related errors

• SGLANG_RUST_SERVER=1: no native Rust MM pipeline for model_t
• Dual chunk attention is enabled, but attention backend is se
• --quantization nvfp4_online supports only --moe-runner-backe
• PD decode DCP currently requires chunk cache; --disaggregati
• PD decode DCP currently requires chunk cache; --enable-hiera

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8f976f33c9437cad. Report an issue: GitHub.