{"record":{"id":"e2f10eb8bbb89aea","repo":"sgl-project/sglang","slug":"mxfp8-fused-decode-prologue-requires-k-v-scale-buf","errorCode":null,"errorMessage":"MXFP8 fused decode prologue requires K/V scale buffers.","messagePattern":"MXFP8 fused decode prologue requires K/V scale buffers\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/inkling_attn_prologue.py","lineNumber":327,"sourceCode":"    mxfp8_quant: bool = False,\n    sfk: torch.Tensor | None = None,\n    sfv: torch.Tensor | None = None,\n    page_size: int = 128,\n    log_scaling_tau: torch.Tensor | None = None,\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:\n    \"\"\"Decode {k/v decode-conv + conv-cache shift-update (+track) + qk-norm\n    (+ KV store)} in one kernel. Returns fresh (q_normed, k_normed, v_conv).\n    The k/v conv caches are shift-updated in place (fused_decode_update\n    semantics). With ``do_store`` the KV rows are scattered into k_buf/v_buf at\n    ``loc``; MXFP8 mode also quantizes Q and writes interleaved K/V scales.\"\"\"\n    t = qkvr.shape[0]\n    if mxfp8_quant:\n        if dq % 128 != 0 or dkv % 128 != 0:\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.\"\n            )\n        if sfk is None or sfv is None:\n            raise ValueError(\"MXFP8 fused decode prologue requires K/V scale buffers.\")\n        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)\n        if sfk.shape != sf_shape or sfv.shape != sf_shape:\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires interleaved K/V scale buffers \"\n                f\"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}.\"\n            )\n        if not sfk.is_contiguous() or not sfv.is_contiguous():\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV.\"\n            )\n        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)\n        sfq_u8 = torch.empty(\n            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device\n        )\n        sfk_u8 = sfk.view(torch.uint8)\n        sfv_u8 = sfv.view(torch.uint8)\n    else:\n        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)","sourceCodeStart":309,"sourceCodeEnd":345,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/inkling_attn_prologue.py#L309-L345","documentation":"The decode prologue in MXFP8 mode writes interleaved K/V scale factors as it scatter-stores KV rows, so the caller must pass both sfk and sfv scale buffers. The check sfk is None or sfv is None fires when quantization is on but the scale caches were not provided — typically because the memory pool lacks the MXFP8 scale tiers or the caller did not fetch them.","triggerScenarios":"Calling inkling_attn_prologue_decode with mxfp8_quant=True and sfk=None or sfv=None, e.g. a backend that allocated k_buf/v_buf only, or a code path that dropped the scale buffers when threading arguments.","commonSituations":"Decode batches after enabling MXFP8 with a pool allocated without scale tiers; partial refactors where the decode path was not updated to pass sfk/sfv like the prefill path was.","solutions":["Allocate and pass sfk/sfv with shape (k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4)","Create the KV pool with MXFP8 support so scale buffers exist","Fall back to mxfp8_quant=False if quantized decode is not required"],"exampleFix":"# before\nq, k, v = inkling_attn_prologue_decode(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True)\n# after\nq, k, v = inkling_attn_prologue_decode(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True, sfk=sfk, sfv=sfv)","handlingStrategy":"validation","validationCode":"if mxfp8_quant:\\n    assert sfk is not None and sfv is not None","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep decode and prefill argument threading in sync during refactors","Add a smoke decode step after enabling MXFP8 to catch missing buffers early"],"tags":["mxfp8","scale-buffers","decode","kv-cache","inkling"],"backgroundTag":"missing-required-argument","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}