sgl-project/sglang · error · ValueError

DFLASH speculative decoding only supports CUDA and NPU…

Error message

DFLASH speculative decoding only supports CUDA and NPU devices.

What it means

Configuration validation error: the DFLASH speculative decoding implementation only supports CUDA and NPU devices and is rejected on other device types.

Solutions

1. Run on CUDA GPUs or NPU
2. Or choose a speculative algorithm supported on your device (or disable speculation)

Example fix

# before (on ROCm)
--speculative-algorithm DFLASH
# after
--speculative-algorithm EAGLE

if args.speculative_algorithm == "DFLASH":
    assert args.device.startswith("cuda") or args.device == "npu"

When it happens

Trigger: --speculative-algorithm DFLASH with --device not starting with 'cuda' and not 'npu' (e.g. rocm, cpu).

Common situations: Running a DFLASH-configured launcher on AMD GPUs or CPU-only environments/CI.

Related errors

• candidates and next_token_logits must be on the same…
• expected 2D positions, got shape= .
• mask/position shape mismatch: vs .
• req_to_token table is empty but gather mask is non-empty.
• DFLASH mask_token_id is outside the target vocab size…

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f2ab1086485ba66c. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/arg_groups/speculative_hook.py:190

        if cfg.speculative_adaptive:
            _init_adaptive_speculative_params(server_args)

    if algo is not None:
        # A registered algorithm's callback lives outside this tree and sets
        # fields on the record, so the writes are captured around the call.
        declare_direct_writes(
            server_args,
            "handle_speculative_decoding.custom_algo",
            algo.handle_server_args,
        )


def _handle_dflash(server_args: ServerArgs) -> None:
    cfg = resolving_view(server_args)
    from sglang.srt.arg_groups.overrides import resolved_view

    if not (cfg.device.startswith("cuda") or cfg.device == "npu"):
        raise ValueError(
            "DFLASH speculative decoding only supports CUDA and NPU devices."
        )

    if resolved_view(server_args).enable_dp_attention:
        raise ValueError(
            "Currently DFLASH speculative decoding does not support dp attention."
        )

    if cfg.pp_size != 1:
        raise ValueError(
            "Currently DFLASH speculative decoding only supports pp_size == 1."
        )

    if cfg.speculative_draft_model_path is None:
        raise ValueError(
            "DFLASH speculative decoding requires setting --speculative-draft-model-path."
        )

View on GitHub (pinned to 0132848349)