sgl-project/sglang · error · RuntimeError

MXFP8 KV cache requires per-token Q scales (q_descale) from

Error message

MXFP8 KV cache requires per-token Q scales (q_descale) from the attention layer for the block-scaled QK^T path.

What it means

The FA4 MXFP8 path needs per-token Q scales (q_descale) supplied by the attention layer for block-scaled QK^T; if the layer did not pass them, _mxfp8_sf_kwargs raises even when FA4 is active.

Source

Thrown at python/sglang/srt/layers/attention/flashattention_backend.py:405

            page_size=self.page_size,
            causal=True,
            has_softcap=self.has_softcap,
            num_splits=self.num_splits,
        )

    def _mxfp8_sf_kwargs(self, layer, forward_batch, q_descale=None):
        """Block-scaled UE8M0 scale factors for the FA4 MXFP8 attention path.

        The pool stores K/V scales interleaved in the FA4 BlockScaledBasicChunk
        layout (page_size==128) as sfk/sfv; the per-token Q scales (q_descale
        from the model layer) ride along as sfq. All three drive the kernel's
        block-scaled QK^T (mxf8f6f4) and in-kernel V dequant."""
        if not self.kv_cache_is_mxfp8:
            return {}
        if self.fa_impl_ver != 4:
            raise RuntimeError("MXFP8 KV cache requires the FA4 backend.")
        if q_descale is None:
            raise RuntimeError(
                "MXFP8 KV cache requires per-token Q scales (q_descale) from "
                "the attention layer for the block-scaled QK^T path."
            )
        # qk_sf_vec_size / v_sf_vec_size default to 32 inside the FA4 interface
        # when sf tensors are given, so they don't need to be passed here (the
        # flash_attn_with_kvcache / varlen wrappers don't forward them anyway).
        k_sf, v_sf = self.token_to_kv_pool.get_kv_scale_buffer(layer.layer_id)
        return {"sfq": q_descale, "sfk": k_sf, "sfv": v_sf}

    def init_forward_metadata_in_graph(self, forward_batch: ForwardBatch) -> None:
        # Single-CG has no Python between steps, so one capturable kernel updates
        # the persistent metadata.
        if not forward_batch.forward_mode.is_draft_extend_v2():
            return
        bs = forward_batch.batch_size
        metadata = self.draft_extend_metadata[bs]
        mapping = self._in_graph_full_to_swa_index_mapping()
        draft_extend_set_metadata(

View on GitHub (pinned to 0132848349)

Solutions

1. Use a checkpoint whose Q projection also carries per-token scales so the layer passes q_descale
2. In a custom model, pass the q scale tensor through RadixAttention.forward(..., q_descale=q_scale)
3. If Q is not quantized, disable MXFP8 KV cache and use a standard fp8 KV layout

Example fix

# before
out = self.attn(q, k, v, forward_batch)  # q_descale omitted
# after
out = self.attn(q, k, v, forward_batch, q_descale=self.q_scale_token)

if kv_cache_is_mxfp8:
    assert fa_impl_ver == 4 and q_descale is not None, 'MXFP8 path needs FA4 + per-token q scales'

When it happens

Trigger: Calling forward_extend/forward_decode with kv_cache_is_mxfp8=True, fa_impl_ver==4, but q_descale=None — typically a model whose QoQ/MXFP4/MXFP8 q linear was not quantized so no per-token q scale tensor exists.

Common situations: Loading a checkpoint with MXFP8 KV but bf16/fp8 tensorwise Q (no per-token Q scales); mixed quant configs; custom model integration that forgets to propagate q_descale into RadixAttention.forward.

Related errors

• MXFP8 KV cache requires the FA4 backend.
• MXFP8 KV cache requires K and V scale tensors.
• --prefill-only-disable-kv-cache does not currently support -
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/752c20a98562cdcc. Report an issue: GitHub.