sgl-project/sglang · error · ValueError

DFLASH speculative decoding only supports CUDA and NPU devic

Error message

DFLASH speculative decoding only supports CUDA and NPU devices.

What it means

The DFLASH speculative decoding implementation only has kernels for CUDA and NPU devices; selecting it on CPU, ROCm/HIP, or other backends fails during server-args handling before launch.

Source

Thrown at python/sglang/srt/arg_groups/speculative_hook.py:190

        if cfg.speculative_adaptive:
            _init_adaptive_speculative_params(server_args)

    if algo is not None:
        # A registered algorithm's callback lives outside this tree and sets
        # fields on the record, so the writes are captured around the call.
        declare_direct_writes(
            server_args,
            "handle_speculative_decoding.custom_algo",
            algo.handle_server_args,
        )


def _handle_dflash(server_args: ServerArgs) -> None:
    cfg = resolving_view(server_args)
    from sglang.srt.arg_groups.overrides import resolved_view

    if not (cfg.device.startswith("cuda") or cfg.device == "npu"):
        raise ValueError(
            "DFLASH speculative decoding only supports CUDA and NPU devices."
        )

    if resolved_view(server_args).enable_dp_attention:
        raise ValueError(
            "Currently DFLASH speculative decoding does not support dp attention."
        )

    if cfg.pp_size != 1:
        raise ValueError(
            "Currently DFLASH speculative decoding only supports pp_size == 1."
        )

    if cfg.speculative_draft_model_path is None:
        raise ValueError(
            "DFLASH speculative decoding requires setting --speculative-draft-model-path."
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Run on CUDA GPUs or NPU
2. Or choose a speculative algorithm supported on your device (or disable speculation)

Example fix

# before (on ROCm)
--speculative-algorithm DFLASH
# after
--speculative-algorithm EAGLE

if args.speculative_algorithm == "DFLASH":
    assert args.device.startswith("cuda") or args.device == "npu"

When it happens

Trigger: --speculative-algorithm DFLASH with --device not starting with 'cuda' and not 'npu' (e.g. rocm, cpu).

Common situations: Running a DFLASH-configured launcher on AMD GPUs or CPU-only environments/CI.

Related errors

• next_token_logits row count mismatch. Expected {bs * draft_t
• candidates and next_token_logits must be on the same device,
• uniform_samples shape mismatch. Expected {(bs, draft_token_n
• uniform_samples_for_final_sampling shape mismatch. Expected
• Nemotron 3.5 DFLASH draft requires its checkpoint embedding.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f2ab1086485ba66c. Report an issue: GitHub.