hiyouga/LlamaFactory · error · RuntimeError

SGLang server error: {response.status_code}, {response.text}

Error message

SGLang server error: {response.status_code}, {response.text}

What it means

Raised when the POST to the local SGLang server's /generate endpoint returns a non-200 status. The engine posts input_ids plus a sampling_params dict (and optionally lora_request) with stream=True; any HTTP error body returned by the server (invalid sampling param, token mismatch, LoRA not loaded, internal crash) is surfaced verbatim as RuntimeError with the status code and response text.

Source

Thrown at src/llamafactory/chat/sglang_engine.py:222

            or 1.0,  # repetition_penalty must > 0
            "skip_special_tokens": skip_special_tokens
            if skip_special_tokens is not None
            else self.generating_args["skip_special_tokens"],
        }
        if seed is not None:
            sampling_params["seed"] = seed

        def stream_request():
            json_data = {
                "input_ids": prompt_ids,
                "sampling_params": sampling_params,
                "stream": True,
            }
            if self.lora_request:
                json_data["lora_request"] = ["lora0"]
            response = requests.post(f"{self.base_url}/generate", json=json_data, stream=True)
            if response.status_code != 200:
                raise RuntimeError(f"SGLang server error: {response.status_code}, {response.text}")

            for chunk in response.iter_lines(decode_unicode=False):
                chunk = str(chunk.decode("utf-8"))
                if chunk == "data: [DONE]":
                    break

                if chunk and chunk.startswith("data:"):
                    yield json.loads(chunk[5:].strip("\n"))

        return await asyncio.to_thread(stream_request)

    @override
    async def chat(
        self,
        messages: Sequence[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[Sequence["ImageInput"]] = None,

View on GitHub (pinned to f28afaf635)

Solutions

1. Inspect response.text embedded in the message — it is the SGLang server's own error body and names the exact bad parameter or fault.
2. Cross-check every key placed into sampling_params (temperature, top_p, top_k, repetition_penalty, max_tokens, stop, seed...) against your installed sglang version's supported SamplingParams.
3. If a LoRA is involved, confirm the adapter was passed at server start and the name matches ("lora0").
4. Reduce prompt length or raise the server's max_model_len if the error is a context-length rejection.
5. Restart the engine if the server died mid-run (see the init log for the crash).

Example fix

# before
response = requests.post(f"{self.base_url}/generate", json=json_data, stream=True)

# after (caller-side preflight: validate params against the server)
info = requests.get(f"{self.base_url}/get_model_info", timeout=5).json()
# strip unsupported sampling keys before calling engine.chat(...)

try:
    async for delta in engine.chat(messages, **kwargs):
        ...
except RuntimeError as e:
    if "SGLang server error" in str(e):
        status, _, body = str(e).partition(",")
        logger.error("sglang %s body=%s", status, body)  # body names the bad param
        # fix sampling kwargs or restart engine; do not blind-retry 4xx-style errors

When it happens

Trigger: Calling generate/chat on SGLangEngine when: a sampling_params key is rejected by the installed sglang version; the LoRA adapter named in lora_request was not served; input_ids contain token ids outside the model vocab; or the server hit an internal error (e.g. OOM during decode) after startup.

Common situations: Version skew between the client-built sampling params and the running sglang server; using --adapter_name_or_path with an engine where the LoRA failed to load; extremely long prompts exceeding max_model_len causing a 400/500 from the server.

Related errors

• SGLang not install, you may need to run `pip install sglang[
• SGLang server initialization failed: {str(e)}.
• SGLang only supports n=1.
• SGLang engine does not support `get_scores`.
• KTransformers inference requires `infer_backend: huggingface

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/17d5c2cfa055f050. Report an issue: GitHub.