sgl-project/sglang · error · ValueError

TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8

Error message

TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8_e4m3, fp4_e2m1, bf16, or auto.

What it means

Raised by SGLang's server-args validation when the TensorRT-LLM MLA attention backend is selected but --kv-cache-dtype is not one of fp8_e4m3, fp4_e2m1, bf16, or auto. The TRTLLM MLA kernels are only compiled for those cache element types, so any other dtype (e.g. fp16, fp8_e5m2) is rejected at startup.

Source

Thrown at python/sglang/srt/arg_groups/overrides.py:2495

    return {}


@register_post_process
def _mla_kv_cache_dtype_checks(view: Any) -> dict:
    """Read-only validation pass in the attention-backend compatibility
    handler: the TRT-LLM and tokenspeed MLA backends constrain the resolved
    kv-cache dtype (declarations never reach the field, so the checks read
    the view)."""
    if (
        view.attention_backend == "trtllm_mla"
        or view.decode_attention_backend == "trtllm_mla"
    ):
        if not is_blackwell_supported():
            raise ValueError(
                "TRTLLM MLA backend is only supported on Blackwell GPUs (SM100/SM12x). Please use a different backend."
            )
        if view.kv_cache_dtype not in ["fp8_e4m3", "fp4_e2m1", "bf16", "auto"]:
            raise ValueError(
                "TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8_e4m3, fp4_e2m1, bf16, or auto."
            )
    if (
        view.attention_backend == "tokenspeed_mla"
        or view.decode_attention_backend == "tokenspeed_mla"
    ):
        if not is_blackwell_supported():
            raise ValueError(
                "tokenspeed_mla backend is only supported on Blackwell GPUs (SM100/SM12x)."
            )
        if view.kv_cache_dtype not in ["fp8_e4m3"]:
            raise ValueError(
                "tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, "
                f"got {view.kv_cache_dtype}."
            )
    return {}

View on GitHub (pinned to 0132848349)

Solutions

1. Set --kv-cache-dtype fp8_e4m3 (or bf16, fp4_e2m1, or auto) when using trtllm_mla
2. Drop the explicit --kv-cache-dtype and let 'auto' resolve it
3. If you need another dtype, switch to a different attention backend (e.g. flashmla or fa3)

Example fix

# before
--attention-backend trtllm_mla --kv-cache-dtype fp8_e5m2
# after
--attention-backend trtllm_mla --kv-cache-dtype fp8_e4m3

allowed = {"fp8_e4m3","fp4_e2m1","bf16","auto"}
if args.attention_backend == "trtllm_mla" and args.kv_cache_dtype not in allowed:
    args.kv_cache_dtype = "fp8_e4m3"

When it happens

Trigger: Setting --attention-backend trtllm_mla (or --decode-attention-backend trtllm_mla) on a Blackwell GPU together with --kv-cache-dtype set to anything besides fp8_e4m3/fp4_e2m1/bf16/auto.

Common situations: Copying a DeepSeek serving config tuned for a different backend (fa3, flashmla) that used fp16 or e5m2 KV cache while switching to trtllm_mla on B200/GB200.

Related errors

• tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, got
• CuteDSL MLA backend only supports kv-cache-dtype of fp8_e4m3
• HiSparse requires one of {HISPARSE_KV_CACHE_DTYPES} KV cache
• SGLANG_DISAGG_STAGING_BUFFER is designed for non-MLA models
• DeepSeekV4 only supports interleave CP strategy, got {cfg.cp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f18c7949652c1042. Report an issue: GitHub.