{"record":{"id":"fff408c1cbae5e35","repo":"sgl-project/sglang","slug":"ngram-speculative-decoding-only-supports-cuda-or-c","errorCode":null,"errorMessage":"Ngram speculative decoding only supports CUDA or CPU devices.","messagePattern":"Ngram speculative decoding only supports CUDA or CPU devices\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/arg_groups/speculative_hook.py","lineNumber":848,"sourceCode":"    # it; flashmla / trtllm_mla / cutlass_mla can't express the per-branch tree, so reject.\n    _PAGE_TREE_SPEC_BACKENDS = (\"flashinfer\", \"fa3\", \"triton\")\n    view = resolved_view(server_args)\n    if (\n        cfg.speculative_eagle_topk > 1\n        and view.page_size > 1\n        and view.attention_backend not in _PAGE_TREE_SPEC_BACKENDS\n    ):\n        raise ValueError(\n            f\"speculative_eagle_topk > 1 with page_size > 1 is only supported on \"\n            f\"{_PAGE_TREE_SPEC_BACKENDS}; got attention_backend=\"\n            f\"{view.attention_backend!r}. Use page_size == 1 or one of those backends.\"\n        )\n\n\ndef _handle_ngram(server_args: ServerArgs) -> None:\n    cfg = resolving_view(server_args)\n    if cfg.device not in (\"cuda\", \"cpu\"):\n        raise ValueError(\n            \"Ngram speculative decoding only supports CUDA or CPU devices.\"\n        )\n\n    _disable_overlap_schedule_for_cpu(server_args)\n\n    if cfg.max_running_requests is None:\n        declare_resolution(\n            server_args,\n            \"_handle_ngram\",\n            max_running_requests=48,\n        )\n        logger.warning(\n            \"Max running requests is reset to 48 for speculative decoding. You can override this by explicitly setting --max-running-requests.\"\n        )\n\n    declare_resolution(\n        server_args,\n        \"_handle_ngram\",","sourceCodeStart":830,"sourceCodeEnd":866,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/arg_groups/speculative_hook.py#L830-L866","documentation":"NGRAM speculative decoding (matching token n-grams from the prompt/request history) is implemented only for CUDA and CPU execution devices. The ngram worker's kernels and code paths have not been ported to other devices (ROCm/HIP, XPU, NPU, etc.), so the args hook rejects the combination early with ValueError.","triggerScenarios":"Launching with --speculative-algorithm NGRAM while the resolved device is neither 'cuda' nor 'cpu' — e.g. setting it to 'rocm'/'hip', 'xpu', or running on non-NVIDIA accelerators where the device string differs.","commonSituations":"Trying NGRAM spec decoding on AMD GPUs via a ROCm build, Intel GPUs, or Ascend NPU builds of SGLang; mis-setting the --device flag explicitly (e.g. --device gpu instead of cuda).","solutions":["Remove --speculative-algorithm NGRAM on non-CUDA/CPU devices","Or run on a CUDA or CPU deployment","Check for a port/newer SGLang release if you need NGRAM spec decoding on ROCm/XPU"],"exampleFix":"# before (ROCm build)\n--device rocm --speculative-algorithm NGRAM\n# after\n--device rocm   # no ngram speculative decoding","handlingStrategy":"validation","validationCode":"if server_args.speculative_algorithm == \"NGRAM\":\n    assert server_args.device in (\"cuda\", \"cpu\"), \"ngram spec decoding is cuda/cpu only\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Gate speculative flags by device in multi-platform launch scripts","Avoid NGRAM on ROCm/XPU/NPU builds"],"tags":["speculative-decoding","ngram","device-support","rocm","server-args"],"backgroundTag":"unsupported-device-backend","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}