{"record":{"id":"6cf5e2058c9438cb","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-the-fa4-backend","errorCode":null,"errorMessage":"MXFP8 KV cache requires the FA4 backend.","messagePattern":"MXFP8 KV cache requires the FA4 backend\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/flashattention_backend.py","lineNumber":403,"sourceCode":"            qkv_dtype=self.kv_cache_dtype,\n            cu_seqlens_q=cu_seqlens_q,\n            page_size=self.page_size,\n            causal=True,\n            has_softcap=self.has_softcap,\n            num_splits=self.num_splits,\n        )\n\n    def _mxfp8_sf_kwargs(self, layer, forward_batch, q_descale=None):\n        \"\"\"Block-scaled UE8M0 scale factors for the FA4 MXFP8 attention path.\n\n        The pool stores K/V scales interleaved in the FA4 BlockScaledBasicChunk\n        layout (page_size==128) as sfk/sfv; the per-token Q scales (q_descale\n        from the model layer) ride along as sfq. All three drive the kernel's\n        block-scaled QK^T (mxf8f6f4) and in-kernel V dequant.\"\"\"\n        if not self.kv_cache_is_mxfp8:\n            return {}\n        if self.fa_impl_ver != 4:\n            raise RuntimeError(\"MXFP8 KV cache requires the FA4 backend.\")\n        if q_descale is None:\n            raise RuntimeError(\n                \"MXFP8 KV cache requires per-token Q scales (q_descale) from \"\n                \"the attention layer for the block-scaled QK^T path.\"\n            )\n        # qk_sf_vec_size / v_sf_vec_size default to 32 inside the FA4 interface\n        # when sf tensors are given, so they don't need to be passed here (the\n        # flash_attn_with_kvcache / varlen wrappers don't forward them anyway).\n        k_sf, v_sf = self.token_to_kv_pool.get_kv_scale_buffer(layer.layer_id)\n        return {\"sfq\": q_descale, \"sfk\": k_sf, \"sfv\": v_sf}\n\n    def init_forward_metadata_in_graph(self, forward_batch: ForwardBatch) -> None:\n        # Single-CG has no Python between steps, so one capturable kernel updates\n        # the persistent metadata.\n        if not forward_batch.forward_mode.is_draft_extend_v2():\n            return\n        bs = forward_batch.batch_size\n        metadata = self.draft_extend_metadata[bs]","sourceCodeStart":385,"sourceCodeEnd":421,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/flashattention_backend.py#L385-L421","documentation":"MXFP8-quantized KV cache can only be read by the FA4 backend's block-scaled (mxf8f6f4) kernel; _mxfp8_sf_kwargs raises if kv_cache_is_mxfp8 but fa_impl_ver != 4 during extend or decode.","triggerScenarios":"Launching a server with KV cache dtype fp8_e4m3 + MXFP8 scale-factor layout (e.g. --kv-cache-dtype fp8_e4m3 with mxfp8 scales) while FA2 or FA3 is the active implementation (no FA4 wheel / older GPU).","commonSituations":"Enabling MXFP8 KV cache on Hopper (FA4 needs Blackwell), forcing SGLANG_FLASHATTENTION_VERSION=3, or a host where flash-attn 4 was not installed so detection fell back.","solutions":["Install/enable the FA4 backend (Blackwell GPU + FA4 support) and confirm fa_impl_ver==4 at startup","Or disable MXFP8 KV cache (use plain fp8 or fp16/bf16 KV cache dtype)","Or switch to an attention backend that supports MXFP8 KV (e.g. FlashInfer with fp8 path) if FA4 is unavailable"],"exampleFix":"# before: FA3 active with mxfp8 kv\npython -m sglang.launch_server --kv-cache-dtype fp8_e4m3 --kv-cache-mxfp8 ...\n# after: force FA4 on supported hw\nSGLANG_FLASHATTENTION_VERSION=4 python -m sglang.launch_server --kv-cache-dtype fp8_e4m3 --kv-cache-mxfp8 ...","handlingStrategy":"validation","validationCode":"assert not (kv_mxfp8 and fa_impl_ver != 4), 'MXFP8 KV requires FA4; check GPU/backend'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pair --kv-cache-dtype mxfp8 flags only with Blackwell + FA4 setups","Check startup logs for 'fa_impl_ver' before sending traffic"],"tags":["mxfp8","kv-cache","flash-attention-4","quantization","sglang"],"backgroundTag":"quantized-kv-backend-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}