{"record":{"id":"5dac4f89ffb80a8f","repo":"sgl-project/sglang","slug":"trtllm-fmhav2-prefill-on-sm120-does-not-support-fp","errorCode":null,"errorMessage":"TRTLLM FMHAv2 prefill on SM120 does not support fp8_e4m3 KV cache or skip-softmax.","messagePattern":"TRTLLM FMHAv2 prefill on SM120 does not support fp8_e4m3 KV cache or skip-softmax\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/server_args.py","lineNumber":6573,"sourceCode":"                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()\n            ):\n                raise ValueError(\n                    \"TRTLLM MHA backend for prefill requires Hopper (SM90), Blackwell (SM100), or SM120 GPUs. \"\n                    \"Please use a different prefill backend.\"\n                )\n            if (\n                prefill_backend == \"trtllm_mha\"\n                and is_sm120_supported()\n                and (\n                    cfg.kv_cache_dtype == \"fp8_e4m3\"\n                    or (\n                        envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get()\n                        or 0.0\n                    )\n                    > 0\n                )\n            ):\n                raise ValueError(\n                    \"TRTLLM FMHAv2 prefill on SM120 does not support \"\n                    \"fp8_e4m3 KV cache or skip-softmax.\"\n                )\n            if decode_backend == \"trtllm_mha\" and not (\n                is_sm90_supported() or is_sm100_supported() or is_sm120_supported()\n            ):\n                raise ValueError(\n                    \"TRTLLM MHA backend for decode is only supported on Hopper (SM90), Blackwell (SM100) and (SM120) GPUs. Please use a different decode backend.\"\n                )\n            if (\n                prefill_backend == \"trtllm_mha\"\n                and not is_sm100_supported()\n                and (cfg.enable_prefill_context_parallel or cfg.attn_cp_size > 1)\n            ):\n                raise ValueError(\n                    \"Prefill context parallelism with the TRTLLM MHA prefill backend \"\n                    \"requires SM100 (trtllm-gen context kernel): the SM90/SM120 \"\n                    \"fmha_v2 prefill path does not implement CP shard masking.\"","sourceCodeStart":6555,"sourceCodeEnd":6591,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/server_args.py#L6555-L6591","documentation":"On SM120 GPUs the trtllm_mha prefill path uses the FMHA v2 kernels, which lack support for fp8_e4m3 block-scaled KV cache operands and for the skip-softmax optimization. SGLang validates this combination at startup when KV cache dtype is fp8_e4m3 or the skip-softmax env threshold is > 0.","triggerScenarios":"Running with --prefill-attention-backend trtllm_mha on an SM120 GPU together with --kv-cache-dtype fp8_e4m3, or with envs.SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR.get() > 0.0.","commonSituations":"Porting a Blackwell SM100 (B200) config with FP8 KV cache onto SM120 consumer GPUs; enabling skip-softmax via environment variable globally when it only applies to SM100-class parts.","solutions":["Use a different KV cache dtype (e.g. bf16/fp16) on SM120 with trtllm_mha prefill","Set SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR to 0/unset it on SM120","Switch prefill backend to fa4/triton on SM120 if FP8 KV cache is required"],"exampleFix":"# before\nexport SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR=2.0\npython -m sglang.launch_server --model M --prefill-attention-backend trtllm_mha --kv-cache-dtype fp8_e4m3\n# after\nunset SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR\npython -m sglang.launch_server --model M --prefill-attention-backend trtllm_mha --kv-cache-dtype bf16","handlingStrategy":"validation","validationCode":"import os\nimport torch\ncap = torch.cuda.get_device_capability(0)\nsm120 = cap == (12, 0)\nif sm120 and args.prefill_attention_backend == \"trtllm_mha\":\n    assert args.kv_cache_dtype != \"fp8_e4m3\", \"fp8_e4m3 KV unsupported for trtllm_mha prefill on SM120\"\n    assert float(os.environ.get(\"SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR\", 0) or 0) <= 0","typeGuard":null,"tryCatchPattern":"try:\n    ServerArgs(**kwargs)\nexcept ValueError as e:\n    if \"fp8_e4m3 KV cache or skip-softmax\" in str(e):\n        kwargs[\"kv_cache_dtype\"] = \"bf16\"\n        ServerArgs(**kwargs)\n    else:\n        raise","preventionTips":["Don't set SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR globally; scope it to SM100 nodes","Maintain per-arch config profiles (SM100 vs SM120) instead of one shared config","Validate kv_cache_dtype against GPU arch in a preflight script"],"tags":["sglang","sm120","fp8-kv-cache","attention-backend","trtllm","env-var"],"backgroundTag":"kv-cache-dtype-incompatible","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}