hiyouga/LlamaFactory · error · RuntimeError

SGLang server initialization failed: {str(e)}.

Error message

SGLang server initialization failed: {str(e)}.

What it means

Raised when the embedded SGLang inference server fails to start inside SGLangEngine initialization. Any exception thrown while launching the server subprocess (import failure, bad arguments, port conflict, model load error, or the server crashing/exiting before becoming healthy) is caught, the partially started process is cleaned up via _cleanup_server(), and the original exception text is rewrapped in a RuntimeError. The underlying cause is visible in the preceding 'Failed to start SGLang server' log line.

Source

Thrown at src/llamafactory/chat/sglang_engine.py:128

            self.server_process, port = launch_server_cmd(launch_cmd)
            self.base_url = f"http://localhost:{port}"
            atexit.register(self._cleanup_server)

            logger.info_rank0(f"Waiting for SGLang server to be ready at {self.base_url}")
            wait_for_server(self.base_url, timeout=300)
            logger.info_rank0(f"SGLang server initialized successfully at {self.base_url}")
            try:
                response = requests.get(f"{self.base_url}/get_model_info", timeout=5)
                if response.status_code == 200:
                    model_info = response.json()
                    logger.info(f"SGLang server model info: {model_info}")
            except Exception as e:
                logger.debug(f"Note: could not get model info: {str(e)}")

        except Exception as e:
            logger.error(f"Failed to start SGLang server: {str(e)}")
            self._cleanup_server()  # make sure to clean up any started process
            raise RuntimeError(f"SGLang server initialization failed: {str(e)}.")

    def _cleanup_server(self):
        r"""Clean up the server process when the engine is destroyed."""
        if hasattr(self, "server_process") and self.server_process:
            try:
                logger.info("Terminating SGLang server process")
                terminate_process(self.server_process)
                logger.info("SGLang server process terminated")
            except Exception as e:
                logger.warning(f"Error terminating SGLang server: {str(e)}")

    async def _generate(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,

View on GitHub (pinned to f28afaf635)

Solutions

1. Read the 'Failed to start SGLang server: ...' error line above the traceback — it carries the root cause from the sglang subprocess.
2. Verify sglang is installed and importable in the same Python env: python -c "import sglang; print(sglang.__version__)".
3. Check GPU memory and lower max_model_len / disable quantization mismatch; watch the sglang server log output for CUDA OOM.
4. Free the port or let the engine pick another (kill stale sglang processes: pkill -f sglang).
5. Align versions: reinstall sglang matching the LlamaFactory requirement for your transformers version.

Example fix

# before
engine = SGLangEngine(model_args)  # RuntimeError: SGLang server initialization failed

# after
import subprocess
subprocess.run(["pkill", "-f", "sglang"], check=False)  # free stale port/process
engine = SGLangEngine(model_args)

import importlib.util, socket
spec = importlib.util.find_spec("sglang")
assert spec is not None, "sglang is not installed"
with socket.socket() as s:
    s.bind(("127.0.0.1", 0))  # probe: machine can allocate sockets
print("sglang importable:", spec.origin)

try:
    engine = SGLangEngine(model_args)
except RuntimeError as e:
    if "initialization failed" not in str(e):
        raise
    logger.error("sglang start failed: %s", e)  # message embeds root cause
    # stale-process cleanup then one manual retry, else fall back to hf engine

When it happens

Trigger: Constructing SGLangEngine(model_args) when: sglang is not installed or version-mismatched; the model path is wrong or the model OOMs on load; the chosen port is already in use; an invalid engine argument is passed to the sglang server launch command; or the server process dies during its startup wait.

Common situations: Running llamafactory-cli chat/api with an sglang engine on a machine without a compatible sglang build or CUDA driver; specifying a quantization or max-model-len the GPU cannot satisfy; leftover process holding the server port; sglang/transformers version skew after upgrading LlamaFactory.

Related errors

• SGLang not install, you may need to run `pip install sglang[
• SGLang only supports n=1.
• SGLang server error: {response.status_code}, {response.text}
• SGLang engine does not support `get_scores`.
• KTransformers inference requires `infer_backend: huggingface

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/a9a5663f14c78791. Report an issue: GitHub.