{"record":{"id":"752c20a98562cdcc","repo":"sgl-project/sglang","slug":"mxfp8-kv-cache-requires-per-token-q-scales-q-desc","errorCode":null,"errorMessage":"MXFP8 KV cache requires per-token Q scales (q_descale) from the attention layer for the block-scaled QK^T path.","messagePattern":"MXFP8 KV cache requires per-token Q scales \\(q_descale\\) from the attention layer for the block-scaled QK\\^T path\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/attention/flashattention_backend.py","lineNumber":405,"sourceCode":"            page_size=self.page_size,\n            causal=True,\n            has_softcap=self.has_softcap,\n            num_splits=self.num_splits,\n        )\n\n    def _mxfp8_sf_kwargs(self, layer, forward_batch, q_descale=None):\n        \"\"\"Block-scaled UE8M0 scale factors for the FA4 MXFP8 attention path.\n\n        The pool stores K/V scales interleaved in the FA4 BlockScaledBasicChunk\n        layout (page_size==128) as sfk/sfv; the per-token Q scales (q_descale\n        from the model layer) ride along as sfq. All three drive the kernel's\n        block-scaled QK^T (mxf8f6f4) and in-kernel V dequant.\"\"\"\n        if not self.kv_cache_is_mxfp8:\n            return {}\n        if self.fa_impl_ver != 4:\n            raise RuntimeError(\"MXFP8 KV cache requires the FA4 backend.\")\n        if q_descale is None:\n            raise RuntimeError(\n                \"MXFP8 KV cache requires per-token Q scales (q_descale) from \"\n                \"the attention layer for the block-scaled QK^T path.\"\n            )\n        # qk_sf_vec_size / v_sf_vec_size default to 32 inside the FA4 interface\n        # when sf tensors are given, so they don't need to be passed here (the\n        # flash_attn_with_kvcache / varlen wrappers don't forward them anyway).\n        k_sf, v_sf = self.token_to_kv_pool.get_kv_scale_buffer(layer.layer_id)\n        return {\"sfq\": q_descale, \"sfk\": k_sf, \"sfv\": v_sf}\n\n    def init_forward_metadata_in_graph(self, forward_batch: ForwardBatch) -> None:\n        # Single-CG has no Python between steps, so one capturable kernel updates\n        # the persistent metadata.\n        if not forward_batch.forward_mode.is_draft_extend_v2():\n            return\n        bs = forward_batch.batch_size\n        metadata = self.draft_extend_metadata[bs]\n        mapping = self._in_graph_full_to_swa_index_mapping()\n        draft_extend_set_metadata(","sourceCodeStart":387,"sourceCodeEnd":423,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/attention/flashattention_backend.py#L387-L423","documentation":"The FA4 MXFP8 path needs per-token Q scales (q_descale) supplied by the attention layer for block-scaled QK^T; if the layer did not pass them, _mxfp8_sf_kwargs raises even when FA4 is active.","triggerScenarios":"Calling forward_extend/forward_decode with kv_cache_is_mxfp8=True, fa_impl_ver==4, but q_descale=None — typically a model whose QoQ/MXFP4/MXFP8 q linear was not quantized so no per-token q scale tensor exists.","commonSituations":"Loading a checkpoint with MXFP8 KV but bf16/fp8 tensorwise Q (no per-token Q scales); mixed quant configs; custom model integration that forgets to propagate q_descale into RadixAttention.forward.","solutions":["Use a checkpoint whose Q projection also carries per-token scales so the layer passes q_descale","In a custom model, pass the q scale tensor through RadixAttention.forward(..., q_descale=q_scale)","If Q is not quantized, disable MXFP8 KV cache and use a standard fp8 KV layout"],"exampleFix":"# before\nout = self.attn(q, k, v, forward_batch)  # q_descale omitted\n# after\nout = self.attn(q, k, v, forward_batch, q_descale=self.q_scale_token)","handlingStrategy":"validation","validationCode":"if kv_cache_is_mxfp8:\n    assert fa_impl_ver == 4 and q_descale is not None, 'MXFP8 path needs FA4 + per-token q scales'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep Q quantization and KV quantization configs consistent (both per-token or neither)","Pass q_descale through custom attention layers explicitly"],"tags":["mxfp8","q-descale","quantization","flash-attention-4","sglang"],"backgroundTag":"missing-quantization-scale","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}