sgl-project/sglang · error · ValueError

TRTLLM FMHAv2 prefill on SM120 does not support fp8_e4m3 KV

Error message

TRTLLM FMHAv2 prefill on SM120 does not support fp8_e4m3 KV cache or skip-softmax.

What it means

On SM120 GPUs the trtllm_mha prefill path uses the FMHA v2 kernels, which lack support for fp8_e4m3 block-scaled KV cache operands and for the skip-softmax optimization. SGLang validates this combination at startup when KV cache dtype is fp8_e4m3 or the skip-softmax env threshold is > 0.

Source

Thrown at python/sglang/srt/server_args.py:6573

                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()
            ):
                raise ValueError(
                    "TRTLLM MHA backend for prefill requires Hopper (SM90), Blackwell (SM100), or SM120 GPUs. "
                    "Please use a different prefill backend."
                )
            if (
                prefill_backend == "trtllm_mha"
                and is_sm120_supported()
                and (
                    cfg.kv_cache_dtype == "fp8_e4m3"
                    or (
                        envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get()
                        or 0.0
                    )
                    > 0
                )
            ):
                raise ValueError(
                    "TRTLLM FMHAv2 prefill on SM120 does not support "
                    "fp8_e4m3 KV cache or skip-softmax."
                )
            if decode_backend == "trtllm_mha" and not (
                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()
            ):
                raise ValueError(
                    "TRTLLM MHA backend for decode is only supported on Hopper (SM90), Blackwell (SM100) and (SM120) GPUs. Please use a different decode backend."
                )
            if (
                prefill_backend == "trtllm_mha"
                and not is_sm100_supported()
                and (cfg.enable_prefill_context_parallel or cfg.attn_cp_size > 1)
            ):
                raise ValueError(
                    "Prefill context parallelism with the TRTLLM MHA prefill backend "
                    "requires SM100 (trtllm-gen context kernel): the SM90/SM120 "
                    "fmha_v2 prefill path does not implement CP shard masking."

View on GitHub (pinned to 0132848349)

Solutions

Use a different KV cache dtype (e.g. bf16/fp16) on SM120 with trtllm_mha prefill
Set SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR to 0/unset it on SM120
Switch prefill backend to fa4/triton on SM120 if FP8 KV cache is required

Example fix

# before
export SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR=2.0
python -m sglang.launch_server --model M --prefill-attention-backend trtllm_mha --kv-cache-dtype fp8_e4m3
# after
unset SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR
python -m sglang.launch_server --model M --prefill-attention-backend trtllm_mha --kv-cache-dtype bf16

Defensive patterns

Strategy: validation

Validate before calling

import os
import torch
cap = torch.cuda.get_device_capability(0)
sm120 = cap == (12, 0)
if sm120 and args.prefill_attention_backend == "trtllm_mha":
    assert args.kv_cache_dtype != "fp8_e4m3", "fp8_e4m3 KV unsupported for trtllm_mha prefill on SM120"
    assert float(os.environ.get("SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR", 0) or 0) <= 0

Try / catch

try:
    ServerArgs(**kwargs)
except ValueError as e:
    if "fp8_e4m3 KV cache or skip-softmax" in str(e):
        kwargs["kv_cache_dtype"] = "bf16"
        ServerArgs(**kwargs)
    else:
        raise

Prevention

Don't set SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR globally; scope it to SM100 nodes
Maintain per-arch config profiles (SM100 vs SM120) instead of one shared config
Validate kv_cache_dtype against GPU arch in a preflight script

When it happens

Trigger: Running with --prefill-attention-backend trtllm_mha on an SM120 GPU together with --kv-cache-dtype fp8_e4m3, or with envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get() > 0.0.

Common situations: Porting a Blackwell SM100 (B200) config with FP8 KV cache onto SM120 consumer GPUs; enabling skip-softmax via environment variable globally when it only applies to SM100-class parts.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5dac4f89ffb80a8f. Report an issue: GitHub.