sgl-project/sglang · error · ValueError

Ngram speculative decoding only supports CUDA or CPU devices

Error message

Ngram speculative decoding only supports CUDA or CPU devices.

What it means

NGRAM speculative decoding (matching token n-grams from the prompt/request history) is implemented only for CUDA and CPU execution devices. The ngram worker's kernels and code paths have not been ported to other devices (ROCm/HIP, XPU, NPU, etc.), so the args hook rejects the combination early with ValueError.

Source

Thrown at python/sglang/srt/arg_groups/speculative_hook.py:848

    # it; flashmla / trtllm_mla / cutlass_mla can't express the per-branch tree, so reject.
    _PAGE_TREE_SPEC_BACKENDS = ("flashinfer", "fa3", "triton")
    view = resolved_view(server_args)
    if (
        cfg.speculative_eagle_topk > 1
        and view.page_size > 1
        and view.attention_backend not in _PAGE_TREE_SPEC_BACKENDS
    ):
        raise ValueError(
            f"speculative_eagle_topk > 1 with page_size > 1 is only supported on "
            f"{_PAGE_TREE_SPEC_BACKENDS}; got attention_backend="
            f"{view.attention_backend!r}. Use page_size == 1 or one of those backends."
        )


def _handle_ngram(server_args: ServerArgs) -> None:
    cfg = resolving_view(server_args)
    if cfg.device not in ("cuda", "cpu"):
        raise ValueError(
            "Ngram speculative decoding only supports CUDA or CPU devices."
        )

    _disable_overlap_schedule_for_cpu(server_args)

    if cfg.max_running_requests is None:
        declare_resolution(
            server_args,
            "_handle_ngram",
            max_running_requests=48,
        )
        logger.warning(
            "Max running requests is reset to 48 for speculative decoding. You can override this by explicitly setting --max-running-requests."
        )

    declare_resolution(
        server_args,
        "_handle_ngram",

View on GitHub (pinned to 0132848349)

Solutions

1. Remove --speculative-algorithm NGRAM on non-CUDA/CPU devices
2. Or run on a CUDA or CPU deployment
3. Check for a port/newer SGLang release if you need NGRAM spec decoding on ROCm/XPU

Example fix

# before (ROCm build)
--device rocm --speculative-algorithm NGRAM
# after
--device rocm   # no ngram speculative decoding

if server_args.speculative_algorithm == "NGRAM":
    assert server_args.device in ("cuda", "cpu"), "ngram spec decoding is cuda/cpu only"

When it happens

Trigger: Launching with --speculative-algorithm NGRAM while the resolved device is neither 'cuda' nor 'cpu' — e.g. setting it to 'rocm'/'hip', 'xpu', or running on non-NVIDIA accelerators where the device string differs.

Common situations: Trying NGRAM spec decoding on AMD GPUs via a ROCm build, Intel GPUs, or Ascend NPU builds of SGLang; mis-setting the --device flag explicitly (e.g. --device gpu instead of cuda).

Related errors

• --speculative-ngram-external-sam-budget must be positive whe
• --speculative-ngram-external-corpus-max-tokens must be posit
• speculative_ngram_external_sam_budget must be less than or e
• This browser cannot encode H.264 MP4
• Unknown match_type: '{match_type}'. Must be 'BFS' or 'PROB'.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fff408c1cbae5e35. Report an issue: GitHub.