{"record":{"id":"3a0544e1eca39fa1","repo":"sgl-project/sglang","slug":"mxfp8-fused-prologue-requires-k-v-scale-buffers","errorCode":null,"errorMessage":"MXFP8 fused prologue requires K/V scale buffers.","messagePattern":"MXFP8 fused prologue requires K/V scale buffers\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/inkling_attn_prologue.py","lineNumber":94,"sourceCode":"    draft_token_num: int,\n    activation: str | None = None,\n    use_residual: bool = True,\n    do_store: bool = True,\n    mxfp8_quant: bool = False,\n    sfk: torch.Tensor | None = None,\n    sfv: torch.Tensor | None = None,\n    page_size: int = 128,\n    log_scaling_tau: torch.Tensor | None = None,\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:\n    \"\"\"Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];\n    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call\n    should pass save_kv_cache=False).\"\"\"\n    t = qkvr.shape[0]\n    if mxfp8_quant:\n        if dq % 128 != 0 or dkv % 128 != 0:\n            raise ValueError(\"MXFP8 fused prologue requires head_dim-aligned Q/K/V.\")\n        if sfk is None or sfv is None:\n            raise ValueError(\"MXFP8 fused prologue requires K/V scale buffers.\")\n        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)\n        if sfk.shape != sf_shape or sfv.shape != sf_shape:\n            raise ValueError(\n                \"MXFP8 fused prologue requires interleaved K/V scale buffers \"\n                f\"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}.\"\n            )\n        if not sfk.is_contiguous() or not sfv.is_contiguous():\n            raise ValueError(\n                \"MXFP8 fused prologue requires contiguous interleaved SFK/SFV.\"\n            )\n        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)\n        sfq_u8 = torch.empty(\n            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device\n        )\n        sfk_u8 = sfk.view(torch.uint8)\n        sfv_u8 = sfv.view(torch.uint8)\n    else:\n        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)","sourceCodeStart":76,"sourceCodeEnd":112,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/inkling_attn_prologue.py#L76-L112","documentation":"In the MXFP8 path of inkling_attn_prologue_verify, K and V must be written into scale-factor buffers (sfk, sfv) that live next to the paged KV cache — the kernel writes interleaved MXFP8 scales itself, so the caller must provide both buffers. The check sfk is None or sfv is None fires when the attention backend did not allocate or pass the MXFP8 scale caches, indicating the memory pool lacks the scale-factor tiers.","triggerScenarios":"Calling inkling_attn_prologue_verify with mxfp8_quant=True but sfk=None or sfv=None — typically because the KV cache pool was created without MXFP8 scale buffers or the caller forgot to fetch them from the token allocator.","commonSituations":"Enabling MXFP8 KV-cache quantization on a server whose memory pool was built with a non-MXFP8 layout; refactoring the pool so scale buffers became optional and the fallback path silently passes None.","solutions":["Allocate/provide sfk and sfv scale buffers with shape [k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4]","Ensure the KV cache pool is created in MXFP8 mode so scale tiers exist and are passed to the prologue","If MXFP8 is not intended, pass mxfp8_quant=False"],"exampleFix":"# before\nq, k, v = inkling_attn_prologue_verify(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True)  # sfk/sfv omitted\n# after\nq, k, v = inkling_attn_prologue_verify(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True, sfk=sfk, sfv=sfv)","handlingStrategy":"validation","validationCode":"if mxfp8_quant:\\n    assert sfk is not None and sfv is not None, 'MXFP8 requires scale buffers'","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Allocate scale buffers together with k_buf/v_buf in the pool so they cannot be forgotten","Add a pool-level invariant: mxfp8 mode implies non-None sfk/sfv"],"tags":["mxfp8","quantization","scale-buffers","attention-prologue","inkling"],"backgroundTag":"missing-required-argument","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}