{"record":{"id":"ac2d953af593a1ea","repo":"sgl-project/sglang","slug":"hd256-forward-varlen-expects-q-rank-3-or-5-got-ra","errorCode":null,"errorMessage":"hd256 forward varlen expects q rank 3 or 5, got rank {q_rank}","messagePattern":"hd256 forward varlen expects q rank 3 or 5, got rank (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/flash_attn/cute/sm100_hd256_2cta_fmha_forward.py","lineNumber":269,"sourceCode":"        q_tensor, k_tensor, v_tensor, o_tensor = mQ, mK, mV, mO\n        lse_tensor = mLSE\n        cum_seqlen_q = mCuSeqlensQ\n        cum_seqlen_k = mCuSeqlensK\n\n        q_rank = len(mQ.shape)\n        k_rank = len(mK.shape)\n        if cutlass.const_expr(cum_seqlen_q is not None):\n            # Varlen path accepts either legacy 5D tensors or standard 3D tensors.\n            if cutlass.const_expr(q_rank == 5):\n                s_q = mQ.shape[1]\n                h_q = mQ.shape[2] * mQ.shape[3]\n                d = mQ.shape[4]\n            elif cutlass.const_expr(q_rank == 3):\n                s_q = mQ.shape[0]\n                h_q = mQ.shape[1]\n                d = mQ.shape[2]\n            else:\n                raise RuntimeError(\n                    f\"hd256 forward varlen expects q rank 3 or 5, got rank {q_rank}\"\n                )\n        else:\n            # Non-varlen path accepts either legacy 5D tensors or standard 4D tensors.\n            if cutlass.const_expr(q_rank == 5):\n                s_q = mQ.shape[1]\n                h_q = mQ.shape[2] * mQ.shape[3]\n                d = mQ.shape[4]\n            elif cutlass.const_expr(q_rank == 4):\n                s_q = mQ.shape[1]\n                h_q = mQ.shape[2]\n                d = mQ.shape[3]\n            else:\n                raise RuntimeError(\n                    f\"hd256 forward non-varlen expects q rank 4 or 5, got rank {q_rank}\"\n                )\n\n        if cutlass.const_expr(cum_seqlen_k is not None):","sourceCodeStart":251,"sourceCodeEnd":287,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/flash_attn/cute/sm100_hd256_2cta_fmha_forward.py#L251-L287","documentation":"The SM100 head-dim-256 varlen FMHA forward kernel accepts Q only as a rank-3 (total_q, nheads, d) or rank-5 (batch, 1, seqlen, nheads, d) tensor. Any other rank raises this RuntimeError inside the kernel's __call__.","triggerScenarios":"Invoking the hd256 2-CTA forward kernel's __call__ in varlen mode with a 4D (batch, seqlen, nheads, d) Q tensor — the non-varlen shape — or a 2D/1D tensor.","commonSituations":"Feeding a standard (B, S, H, D) batched tensor while cu_seqlens is provided; reshaping input incorrectly before varlen packing (missing the pack to (total, H, D)).","solutions":["Pack Q to rank 3: q = q.reshape(total_q, nheads, head_dim) when using cu_seqlens (varlen).","Or drop cu_seqlens and pass the 4D/5D batched tensor for the non-varlen path.","Verify q.dim() == 3 (varlen) or 4/5 (non-varlen) before calling the kernel."],"exampleFix":"# before (varlen with 4D input)\nfmha(q_4d, k, v, cu_seqlen_q=..., ...)\n\n# after\nq = q_4d.reshape(-1, nheads, head_dim)  # (total_q, H, D)\nfmha(q, k, v, cu_seqlen_q=..., ...)","handlingStrategy":"validation","validationCode":"if cu_seqlen_q is not None:  # varlen path\n    assert q.dim() in (3, 5), f\"varlen Q must be rank 3 or 5, got {q.dim()}\"","typeGuard":"def is_valid_varlen_q(q):\n    return q.dim() == 3 or q.dim() == 5","tryCatchPattern":null,"preventionTips":["Pack varlen tensors to (total_tokens, nheads, head_dim) before calling.","Keep a shape-check helper in your attention wrapper."],"tags":["cuda","attention","tensor-rank","varlen","head-dim-256"],"backgroundTag":"unexpected-tensor-rank","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}