{"record":{"id":"995213a7f3b16620","repo":"sgl-project/sglang","slug":"mxfp8-fused-prologue-requires-contiguous-interleav","errorCode":null,"errorMessage":"MXFP8 fused prologue requires contiguous interleaved SFK/SFV.","messagePattern":"MXFP8 fused prologue requires contiguous interleaved SFK/SFV\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/inkling_attn_prologue.py","lineNumber":102,"sourceCode":"    log_scaling_tau: torch.Tensor | None = None,\n) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:\n    \"\"\"Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];\n    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call\n    should pass save_kv_cache=False).\"\"\"\n    t = qkvr.shape[0]\n    if mxfp8_quant:\n        if dq % 128 != 0 or dkv % 128 != 0:\n            raise ValueError(\"MXFP8 fused prologue requires head_dim-aligned Q/K/V.\")\n        if sfk is None or sfv is None:\n            raise ValueError(\"MXFP8 fused prologue requires K/V scale buffers.\")\n        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)\n        if sfk.shape != sf_shape or sfv.shape != sf_shape:\n            raise ValueError(\n                \"MXFP8 fused prologue requires interleaved K/V scale buffers \"\n                f\"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}.\"\n            )\n        if not sfk.is_contiguous() or not sfv.is_contiguous():\n            raise ValueError(\n                \"MXFP8 fused prologue requires contiguous interleaved SFK/SFV.\"\n            )\n        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)\n        sfq_u8 = torch.empty(\n            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device\n        )\n        sfk_u8 = sfk.view(torch.uint8)\n        sfv_u8 = sfv.view(torch.uint8)\n    else:\n        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)\n        sfq_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)\n        sfk_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)\n        sfv_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)\n    k_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)\n    v_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)\n    if activation == \"swish\":\n        activation = \"silu\"\n    use_silu = activation in (\"silu\", \"swish\")","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/inkling_attn_prologue.py#L84-L120","documentation":"The MXFP8 scale buffers sfk/sfv must be contiguous because the fused prologue kernel indexes them with the interleaved layout assuming dense row-major memory; a non-contiguous view (slice/permute of a larger buffer) would make the written scales garbage. The check not sfk.is_contiguous() or not sfv.is_contiguous() runs after the shape check, right before allocating the fp8 output tensors.","triggerScenarios":"Passing sfk/sfv that are strided views — e.g. sfk = big_buffer[:, 1] or a permuted/reshaped-with-stride tensor — into inkling_attn_prologue_verify with mxfp8_quant=True.","commonSituations":"Slicing one layer's scale buffer out of a fused multi-layer pool tensor; using .view() where it produces non-contiguous strides; narrow() on the page dimension.","solutions":["Call .contiguous() on sfk/sfv before passing them (or better, fix the allocation so each buffer is dense)","Allocate per-layer scale buffers instead of slicing a concatenated pool tensor","Check sfk.is_contiguous() in your pool getter and densify once at setup, not per step"],"exampleFix":"# before\nsfk, sfv = pool.scales[:, layer_idx]  # non-contiguous views\n# after\nsfk, sfv = pool.scales[layer_idx].contiguous()","handlingStrategy":"validation","validationCode":"if mxfp8_quant:\\n    sfk = sfk.contiguous(); sfv = sfv.contiguous()","typeGuard":"def contiguous_scales(t: torch.Tensor) -> torch.Tensor:\\n    return t if t.is_contiguous() else t.contiguous()","tryCatchPattern":null,"preventionTips":["Store per-layer scale buffers as dense standalone tensors","Assert is_contiguous() when handing buffers out of the pool"],"tags":["mxfp8","scale-buffers","contiguity","kv-cache","inkling"],"backgroundTag":"non-contiguous-tensor","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}