{"record":{"id":"3ea89fbbc0d7138e","repo":"sgl-project/sglang","slug":"mxfp8-fused-decode-prologue-requires-head-dim-alig","errorCode":null,"errorMessage":"MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.","messagePattern":"MXFP8 fused decode prologue requires head_dim-aligned Q/K/V\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/inkling_attn_prologue.py","lineNumber":323,"sourceCode":"    use_residual: bool = True,\n    track_mask: torch.Tensor | None = None,\n    track_indices: torch.Tensor | None = None,\n    do_store: bool = True,\n    mxfp8_quant: bool = False,\n    sfk: torch.Tensor | None = None,\n    sfv: torch.Tensor | None = None,\n    page_size: int = 128,\n    log_scaling_tau: torch.Tensor | None = None,\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:\n    \"\"\"Decode {k/v decode-conv + conv-cache shift-update (+track) + qk-norm\n    (+ KV store)} in one kernel. Returns fresh (q_normed, k_normed, v_conv).\n    The k/v conv caches are shift-updated in place (fused_decode_update\n    semantics). With ``do_store`` the KV rows are scattered into k_buf/v_buf at\n    ``loc``; MXFP8 mode also quantizes Q and writes interleaved K/V scales.\"\"\"\n    t = qkvr.shape[0]\n    if mxfp8_quant:\n        if dq % 128 != 0 or dkv % 128 != 0:\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.\"\n            )\n        if sfk is None or sfv is None:\n            raise ValueError(\"MXFP8 fused decode prologue requires K/V scale buffers.\")\n        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)\n        if sfk.shape != sf_shape or sfv.shape != sf_shape:\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires interleaved K/V scale buffers \"\n                f\"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}.\"\n            )\n        if not sfk.is_contiguous() or not sfv.is_contiguous():\n            raise ValueError(\n                \"MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV.\"\n            )\n        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)\n        sfq_u8 = torch.empty(\n            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device\n        )","sourceCodeStart":305,"sourceCodeEnd":341,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/inkling_attn_prologue.py#L305-L341","documentation":"The decode prologue (inkling_attn_prologue_decode) applies the same MXFP8 constraint as the verify/extend variants: with mxfp8_quant=True, dq and dkv must both be multiples of 128 to fit the MXFP8 microscaling block structure (128-element superblocks, 32-element scale groups). The decode-specific message distinguishes it from the prefill error so you know which path raised it.","triggerScenarios":"Calling inkling_attn_prologue_decode with mxfp8_quant=True on a model with head dims not divisible by 128 — the first decode step after prefill, i.e. the error appears one step later than the extend-path variant (372/370).","commonSituations":"MXFP8 enabled for models with head_dim 64; configurations where prefill ran non-quantized but decode routed into the quantized prologue.","solutions":["Disable mxfp8_quant for this model/layer","Use 128-aligned head dims if MXFP8 is required","Gate the flag per layer: mxfp8_quant and dq % 128 == 0 and dkv % 128 == 0"],"exampleFix":"# before\nq, k, v = inkling_attn_prologue_decode(qkvr, ..., mxfp8_quant=True)\n# after\naligned = dq % 128 == 0 and dkv % 128 == 0\nq, k, v = inkling_attn_prologue_decode(qkvr, ..., mxfp8_quant=mxfp8_quant and aligned)","handlingStrategy":"validation","validationCode":"mxfp8_quant = mxfp8_quant and dq % 128 == 0 and dkv % 128 == 0","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Apply the same alignment gate to prefill and decode paths so they fail (or degrade) consistently","Log once when falling back from MXFP8 due to alignment"],"tags":["mxfp8","quantization","head-dim","decode","inkling"],"backgroundTag":"quantization-shape-constraint","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}