sgl-project/sglang · error · RuntimeError

MXFP8 KV cache requires the FA4 backend.

Error message

MXFP8 KV cache requires the FA4 backend.

What it means

MXFP8-quantized KV cache can only be read by the FA4 backend's block-scaled (mxf8f6f4) kernel; _mxfp8_sf_kwargs raises if kv_cache_is_mxfp8 but fa_impl_ver != 4 during extend or decode.

Source

Thrown at python/sglang/srt/layers/attention/flashattention_backend.py:403

            qkv_dtype=self.kv_cache_dtype,
            cu_seqlens_q=cu_seqlens_q,
            page_size=self.page_size,
            causal=True,
            has_softcap=self.has_softcap,
            num_splits=self.num_splits,
        )

    def _mxfp8_sf_kwargs(self, layer, forward_batch, q_descale=None):
        """Block-scaled UE8M0 scale factors for the FA4 MXFP8 attention path.

        The pool stores K/V scales interleaved in the FA4 BlockScaledBasicChunk
        layout (page_size==128) as sfk/sfv; the per-token Q scales (q_descale
        from the model layer) ride along as sfq. All three drive the kernel's
        block-scaled QK^T (mxf8f6f4) and in-kernel V dequant."""
        if not self.kv_cache_is_mxfp8:
            return {}
        if self.fa_impl_ver != 4:
            raise RuntimeError("MXFP8 KV cache requires the FA4 backend.")
        if q_descale is None:
            raise RuntimeError(
                "MXFP8 KV cache requires per-token Q scales (q_descale) from "
                "the attention layer for the block-scaled QK^T path."
            )
        # qk_sf_vec_size / v_sf_vec_size default to 32 inside the FA4 interface
        # when sf tensors are given, so they don't need to be passed here (the
        # flash_attn_with_kvcache / varlen wrappers don't forward them anyway).
        k_sf, v_sf = self.token_to_kv_pool.get_kv_scale_buffer(layer.layer_id)
        return {"sfq": q_descale, "sfk": k_sf, "sfv": v_sf}

    def init_forward_metadata_in_graph(self, forward_batch: ForwardBatch) -> None:
        # Single-CG has no Python between steps, so one capturable kernel updates
        # the persistent metadata.
        if not forward_batch.forward_mode.is_draft_extend_v2():
            return
        bs = forward_batch.batch_size
        metadata = self.draft_extend_metadata[bs]

View on GitHub (pinned to 0132848349)

Solutions

  1. Install/enable the FA4 backend (Blackwell GPU + FA4 support) and confirm fa_impl_ver==4 at startup
  2. Or disable MXFP8 KV cache (use plain fp8 or fp16/bf16 KV cache dtype)
  3. Or switch to an attention backend that supports MXFP8 KV (e.g. FlashInfer with fp8 path) if FA4 is unavailable

Example fix

# before: FA3 active with mxfp8 kv
python -m sglang.launch_server --kv-cache-dtype fp8_e4m3 --kv-cache-mxfp8 ...
# after: force FA4 on supported hw
SGLANG_FLASHATTENTION_VERSION=4 python -m sglang.launch_server --kv-cache-dtype fp8_e4m3 --kv-cache-mxfp8 ...
Defensive patterns

Strategy: validation

Validate before calling

assert not (kv_mxfp8 and fa_impl_ver != 4), 'MXFP8 KV requires FA4; check GPU/backend'

Prevention

When it happens

Trigger: Launching a server with KV cache dtype fp8_e4m3 + MXFP8 scale-factor layout (e.g. --kv-cache-dtype fp8_e4m3 with mxfp8 scales) while FA2 or FA3 is the active implementation (no FA4 wheel / older GPU).

Common situations: Enabling MXFP8 KV cache on Hopper (FA4 needs Blackwell), forcing SGLANG_FLASHATTENTION_VERSION=3, or a host where flash-attn 4 was not installed so detection fell back.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6cf5e2058c9438cb. Report an issue: GitHub.