sgl-project/sglang · error · ValueError

TRTLLM MHA backend for decode is only supported on Hopper (S

Error message

TRTLLM MHA backend for decode is only supported on Hopper (SM90), Blackwell (SM100) and (SM120) GPUs. Please use a different decode backend.

What it means

The trtllm_mha decode (context/gen) kernels only exist for Hopper (SM90), Blackwell (SM100), and SM120. Resolution-time checks reject selecting trtllm_mha as the decode backend on any other CUDA architecture.

Source

Thrown at python/sglang/srt/server_args.py:6580

                prefill_backend == "trtllm_mha"
                and is_sm120_supported()
                and (
                    cfg.kv_cache_dtype == "fp8_e4m3"
                    or (
                        envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get()
                        or 0.0
                    )
                    > 0
                )
            ):
                raise ValueError(
                    "TRTLLM FMHAv2 prefill on SM120 does not support "
                    "fp8_e4m3 KV cache or skip-softmax."
                )
            if decode_backend == "trtllm_mha" and not (
                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()
            ):
                raise ValueError(
                    "TRTLLM MHA backend for decode is only supported on Hopper (SM90), Blackwell (SM100) and (SM120) GPUs. Please use a different decode backend."
                )
            if (
                prefill_backend == "trtllm_mha"
                and not is_sm100_supported()
                and (cfg.enable_prefill_context_parallel or cfg.attn_cp_size > 1)
            ):
                raise ValueError(
                    "Prefill context parallelism with the TRTLLM MHA prefill backend "
                    "requires SM100 (trtllm-gen context kernel): the SM90/SM120 "
                    "fmha_v2 prefill path does not implement CP shard masking."
                )

        run_post_process_pass(self, _attention_backend_fa3_fp8_fallback)

        run_post_process_pass(self, _fa4_page_constraint)

        # AMD platforms backends

View on GitHub (pinned to 0132848349)

Solutions

Change decode backend to flashinfer/triton (or another backend supported on your GPU)
Confirm the architecture first: python -c "import torch;print(torch.cuda.get_device_capability())"
Move the workload to SM90/SM100/SM120 hardware if trtllm_mha decode is required

Example fix

# before
python -m sglang.launch_server --model M --decode-attention-backend trtllm_mha
# after
python -m sglang.launch_server --model M --decode-attention-backend flashinfer

Defensive patterns

Strategy: validation

Validate before calling

import torch
cap = torch.cuda.get_device_capability(0)
sm = cap[0] * 10 + cap[1]
if args.decode_attention_backend == "trtllm_mha" and sm not in (90, 100, 120):
    args.decode_attention_backend = "flashinfer"

Try / catch

try:
    ServerArgs(**kwargs)
except ValueError as e:
    if "TRTLLM MHA backend for decode" in str(e):
        kwargs["decode_attention_backend"] = "flashinfer"
        ServerArgs(**kwargs)
    else:
        raise

Prevention

Derive backend selection from detected compute capability in orchestration scripts
Pin backend choices per hardware tier rather than copying configs across clusters
Smoke-test ServerArgs construction in CI on each GPU tier

When it happens

Trigger: Passing --decode-attention-backend trtllm_mha (or a unified --attention-backend trtllm_mha) on a GPU where is_sm90_supported(), is_sm100_supported(), and is_sm120_supported() are all false.

Common situations: Deploying configs written for H100/B200 onto A100 or RTX Ada nodes; backend profiles that silently resolve decode to trtllm_mha on unsupported hardware.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d80b737d9b5546b2. Report an issue: GitHub.