sgl-project/sglang · error · ValueError

tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, got

Error message

tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, got {}.

What it means

The tokenspeed_mla backend in SGLang requires the KV cache to be stored in fp8_e4m3; its fused kernels do not support bf16/fp16/fp4 or auto resolution to those types. Validation fails at startup when any other dtype is configured.

Source

Thrown at python/sglang/srt/arg_groups/overrides.py:2507

    ):
        if not is_blackwell_supported():
            raise ValueError(
                "TRTLLM MLA backend is only supported on Blackwell GPUs (SM100/SM12x). Please use a different backend."
            )
        if view.kv_cache_dtype not in ["fp8_e4m3", "fp4_e2m1", "bf16", "auto"]:
            raise ValueError(
                "TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8_e4m3, fp4_e2m1, bf16, or auto."
            )
    if (
        view.attention_backend == "tokenspeed_mla"
        or view.decode_attention_backend == "tokenspeed_mla"
    ):
        if not is_blackwell_supported():
            raise ValueError(
                "tokenspeed_mla backend is only supported on Blackwell GPUs (SM100/SM12x)."
            )
        if view.kv_cache_dtype not in ["fp8_e4m3"]:
            raise ValueError(
                "tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, "
                f"got {view.kv_cache_dtype}."
            )
    return {}


@register_post_process
def _hisparse_validation(view: Any) -> dict:
    """Read-only validation pass: --enable-hisparse constraints (model class,
    radix cache, kv dtype, DSA backends) read the resolved values through the
    view."""
    from sglang.srt.arg_groups.hisparse_hook import validate_hisparse

    validate_hisparse(view)
    return {}


@register_post_process

View on GitHub (pinned to 0132848349)

Solutions

1. Set --kv-cache-dtype fp8_e4m3 explicitly
2. If fp8 KV cache is unacceptable (accuracy concerns), use a different MLA backend such as flashmla or trtllm_mla with bf16

Example fix

# before
--attention-backend tokenspeed_mla --kv-cache-dtype bf16
# after
--attention-backend tokenspeed_mla --kv-cache-dtype fp8_e4m3

if "tokenspeed_mla" in (args.attention_backend, args.decode_attention_backend):
    args.kv_cache_dtype = "fp8_e4m3"

When it happens

Trigger: --attention-backend tokenspeed_mla (or --decode-attention-backend tokenspeed_mla) on Blackwell with --kv-cache-dtype set to bf16, fp16, auto-resolved-bf16, or fp4.

Common situations: Reusing a bf16 DeepSeek MLA config when experimenting with tokenspeed_mla; leaving kv-cache-dtype to 'auto' when the model default resolves to bf16.

Related errors

• TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8
• CuteDSL MLA backend only supports kv-cache-dtype of fp8_e4m3
• HiSparse requires one of {HISPARSE_KV_CACHE_DTYPES} KV cache
• tokenspeed_mla backend is only supported on Blackwell GPUs (
• SGLANG_DISAGG_STAGING_BUFFER is designed for non-MLA models

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/50f82a4b5a705b9a. Report an issue: GitHub.