What causes "Sync request failed: {error}" in sgl-project/sglang?

Inspect the {error} JSON body and the server process logs (proc output) for the root cause.

sgl-project/sglang · critical · RuntimeError

Sync request failed: {error}

Error message

Sync request failed: {error}

What it means

Raised inside launch_server_process_and_send_one_request during DeepGEMM kernel compilation: after starting a server process, the rank-0 node POSTs a warmup /generate request, and a non-200 response triggers this RuntimeError carrying the server's JSON error body. It almost always reflects a crash/invalid-request in the compilation server, not in the HTTP client.

Source

Thrown at python/sglang/compile_deep_gemm.py:164

                        payload["bootstrap_room"] = [
                            i * (2**63 // dp_size) + (i % cfg.tp_size)
                            for i in range(dp_size)
                        ]
                    else:
                        payload["input_ids"] = (
                            base_ids
                            if dp_size == 1
                            else [list(base_ids) for _ in range(dp_size)]
                        )

                    response = requests.post(
                        f"{base_url}/generate",
                        json=payload,
                        timeout=600,
                    )
                    if response.status_code != 200:
                        error = response.json()
                        raise RuntimeError(f"Sync request failed: {error}")
                # Other nodes should wait for the exit signal from Rank-0 node.
                else:
                    start_time_waiting = time.perf_counter()
                    while proc.is_alive():
                        if time.perf_counter() - start_time_waiting < timeout:
                            time.sleep(10)
                        else:
                            raise TimeoutError("Waiting for main node timeout!")
                return proc
        except requests.RequestException:
            pass
        time.sleep(10)
    raise TimeoutError(
        "DeepGEMM Kernels compilation timeout."
        "\n\nFeel free and please restart the command."
    )

View on GitHub (pinned to 0132848349)

Solutions

Inspect the {error} JSON body and the server process logs (proc output) for the root cause.
Verify GPU/SM compatibility for DeepGEMM and CUDA_HOME/nvcc availability.
Retry with a clean DeepGEMM cache (remove ~/.cache/deep_gemm or DG_CACHE_DIR) to rule out a corrupted partial compile.
Ensure server args (tp size, dtype, context length) in compile_args match what the server supports.

Defensive patterns

Strategy: try-catch

Validate before calling

resp = requests.post(f"{base_url}/generate", json=payload, timeout=600)
if resp.status_code != 200:
    log.error("server error body: %s", resp.text)  # inspect before it becomes an exception

Try / catch

try:
    run_compile(...)
except RuntimeError as e:
    if "Sync request failed" in str(e):
        inspect_server_logs(); clear_deep_gemm_cache(); retry_once()

Prevention

Run compile_deep_gemm on a node with a working nvcc and supported SM architecture first to validate.
Keep server args consistent with the compile payload (tp, dtype, context length).
Warm the DeepGEMM cache on one node before multi-node runs.

When it happens

Trigger: Running sglang.compile_deep_gemm.run_compile where the launched server fails while handling the compilation-triggering generate request (kernel compile error, bad args, OOM); response.status_code != 200 from /generate.

Common situations: DeepGEMM JIT compilation failing on unsupported shapes/SM architecture; server args mismatched with the compile payload; CUDA toolchain issues during kernel build.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fcf99257b3e2570f. Report an issue: GitHub.