sgl-project/sglang · error · ValueError

Prefill context parallelism with the TRTLLM MHA prefill back

Error message

Prefill context parallelism with the TRTLLM MHA prefill backend requires SM100 (trtllm-gen context kernel): the SM90/SM120 fmha_v2 prefill path does not implement CP shard masking.

What it means

When prefill context parallelism (CP) is enabled with the trtllm_mha prefill backend, SGLang relies on the SM100 trtllm-gen context kernel, which implements CP shard masking. The SM90/SM120 fmha_v2 prefill path does not mask CP shards, so combining CP (enable_prefill_context_parallel or attn_cp_size > 1) with trtllm_mha prefill on non-SM100 GPUs is rejected at startup.

Source

Thrown at python/sglang/srt/server_args.py:6588

                    > 0
                )
            ):
                raise ValueError(
                    "TRTLLM FMHAv2 prefill on SM120 does not support "
                    "fp8_e4m3 KV cache or skip-softmax."
                )
            if decode_backend == "trtllm_mha" and not (
                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()
            ):
                raise ValueError(
                    "TRTLLM MHA backend for decode is only supported on Hopper (SM90), Blackwell (SM100) and (SM120) GPUs. Please use a different decode backend."
                )
            if (
                prefill_backend == "trtllm_mha"
                and not is_sm100_supported()
                and (cfg.enable_prefill_context_parallel or cfg.attn_cp_size > 1)
            ):
                raise ValueError(
                    "Prefill context parallelism with the TRTLLM MHA prefill backend "
                    "requires SM100 (trtllm-gen context kernel): the SM90/SM120 "
                    "fmha_v2 prefill path does not implement CP shard masking."
                )

        run_post_process_pass(self, _attention_backend_fa3_fp8_fallback)

        run_post_process_pass(self, _fa4_page_constraint)

        # AMD platforms backends
        if resolved_view(self).attention_backend == "aiter":
            if model_config.context_len > 8192:
                self._declare(
                    "_handle_attention_backend_compatibility",
                    mem_fraction_static=cfg.mem_fraction_static * 0.85,
                )

        # Other platforms backends

View on GitHub (pinned to 0132848349)

Solutions

1. Run on SM100 (Blackwell B200-class) hardware where the trtllm-gen context kernel implements CP masking
2. Use a different prefill backend (e.g. fa3 on Hopper) when CP is required
3. Disable prefill context parallelism (attn_cp_size=1, no --enable-prefill-context-parallel) with trtllm_mha prefill

Example fix

# before
python -m sglang.launch_server --model M --prefill-attention-backend trtllm_mha --enable-prefill-context-parallel
# after (on Hopper)
python -m sglang.launch_server --model M --prefill-attention-backend fa3 --enable-prefill-context-parallel

import torch
cap = torch.cuda.get_device_capability(0)
sm = cap[0] * 10 + cap[1]
uses_cp = args.enable_prefill_context_parallel or (args.attn_cp_size or 1) > 1
if uses_cp and args.prefill_attention_backend == "trtllm_mha" and sm != 100:
    args.prefill_attention_backend = "fa3" if sm == 90 else "triton"

try:
    ServerArgs(**kwargs)
except ValueError as e:
    if "CP shard masking" in str(e):
        kwargs["prefill_attention_backend"] = "fa3"
        ServerArgs(**kwargs)
    else:
        raise

When it happens

Trigger: Setting --enable-prefill-context-parallel or attn_cp_size > 1 with prefill backend trtllm_mha on a GPU where is_sm100_supported() is false (SM90 Hopper or SM120).

Common situations: Long-context prefill sharding attempts on H100 clusters; fleet configs enabling CP globally while also using trtllm_mha prefill; adopting attn_cp_size without checking backend support.

Related errors

• TRTLLM MHA backend for prefill requires Hopper (SM90), Black
• TRTLLM FMHAv2 prefill on SM120 does not support fp8_e4m3 KV
• TRTLLM MHA backend for decode is only supported on Hopper (S
• {selection_error}{component_suffix}
• No compatible attention backend is available{component_suffi

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/9a5329eb6961a335. Report an issue: GitHub.