sgl-project/sglang · error · ValueError

trtllm_mha backend only supports topk = 1 for speculative de

Error message

trtllm_mha backend only supports topk = 1 for speculative decoding.

What it means

The trtllm_mha attention backend only supports EAGLE-family speculation with topk = 1 (linear single-chain drafting). Multi-branch drafting with speculative_eagle_topk > 1 is incompatible with that kernel's verify path.

Source

Thrown at python/sglang/srt/arg_groups/speculative_hook.py:763

        assert (
            cfg.speculative_eagle_topk is None
            and cfg.speculative_num_draft_tokens is None
        )

        steps, topk, draft_tokens = _auto_choose_speculative_params(
            server_args, model_arch
        )
        declare_resolution(
            server_args,
            "_handle_eagle_family.auto_params",
            speculative_num_steps=steps,
            speculative_eagle_topk=topk,
            speculative_num_draft_tokens=draft_tokens,
        )

    if "trtllm_mha" in attention_backends_of(resolved_view(server_args)):
        if cfg.speculative_eagle_topk > 1:
            raise ValueError(
                "trtllm_mha backend only supports topk = 1 for speculative decoding."
            )

    if cfg.speculative_use_rejection_sampling:
        # Resolved alias by now: NEXTN -> EAGLE, Gemma4 draft -> FROZEN_KV_MTP.
        # Only the EAGLE/EAGLE3 draft workers emit a target-vocab proposal that
        # the rejection-sampling kernel consumes; everything else (STANDALONE,
        # FROZEN_KV_MTP, NGRAM, DFLASH) is unsupported.
        if cfg.speculative_algorithm not in ("EAGLE", "EAGLE3"):
            raise NotImplementedError(
                "--speculative-use-rejection-sampling is only supported for "
                "EAGLE / EAGLE3 / NEXTN, not "
                f"speculative_algorithm={cfg.speculative_algorithm}."
            )
        if cfg.speculative_eagle_topk != 1:
            raise ValueError(
                "--speculative-use-rejection-sampling requires --speculative-eagle-topk=1."
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Set --speculative-eagle-topk 1 when using trtllm_mha
2. Switch attention backend (e.g. flashinfer/fa3) if topk > 1 is desired
3. Increase --speculative-num-steps instead of topk to raise speculation depth

Example fix

# before
--attention-backend trtllm_mha --speculative-eagle-topk 4
# after
--attention-backend trtllm_mha --speculative-eagle-topk 1 --speculative-num-steps 4

if 'trtllm_mha' in (args.attention_backend or '') and (args.speculative_eagle_topk or 1) > 1:
    raise SystemExit('trtllm_mha only supports speculative topk = 1')

When it happens

Trigger: Selecting the trtllm_mha attention backend (directly or via --attention-backend) together with --speculative-eagle-topk > 1 on an EAGLE-family run.

Common situations: Performance-tuned launch configs that raise topk/steps for EAGLE3 while the backend is pinned to trtllm_mha (e.g. on Blackwell/TRT-LLM kernels).

Related errors

• --speculative-use-rejection-sampling is only supported for E
• speculative_eagle_topk > 1 with page_size > 1 is only suppor
• trtllm_mla cannot serve decode context parallelism with spec
• trtllm_mha backend can only be used with non-MLA models.
• hpc_ops backend does not support speculative decoding for no

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fee0bdae4b2d3400. Report an issue: GitHub.