sgl-project/sglang · error · RuntimeError

hd256 forward varlen expects q rank 3 or 5, got rank {q_rank

Error message

hd256 forward varlen expects q rank 3 or 5, got rank {q_rank}

What it means

The SM100 head-dim-256 varlen FMHA forward kernel accepts Q only as a rank-3 (total_q, nheads, d) or rank-5 (batch, 1, seqlen, nheads, d) tensor. Any other rank raises this RuntimeError inside the kernel's __call__.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/sm100_hd256_2cta_fmha_forward.py:269

        q_tensor, k_tensor, v_tensor, o_tensor = mQ, mK, mV, mO
        lse_tensor = mLSE
        cum_seqlen_q = mCuSeqlensQ
        cum_seqlen_k = mCuSeqlensK

        q_rank = len(mQ.shape)
        k_rank = len(mK.shape)
        if cutlass.const_expr(cum_seqlen_q is not None):
            # Varlen path accepts either legacy 5D tensors or standard 3D tensors.
            if cutlass.const_expr(q_rank == 5):
                s_q = mQ.shape[1]
                h_q = mQ.shape[2] * mQ.shape[3]
                d = mQ.shape[4]
            elif cutlass.const_expr(q_rank == 3):
                s_q = mQ.shape[0]
                h_q = mQ.shape[1]
                d = mQ.shape[2]
            else:
                raise RuntimeError(
                    f"hd256 forward varlen expects q rank 3 or 5, got rank {q_rank}"
                )
        else:
            # Non-varlen path accepts either legacy 5D tensors or standard 4D tensors.
            if cutlass.const_expr(q_rank == 5):
                s_q = mQ.shape[1]
                h_q = mQ.shape[2] * mQ.shape[3]
                d = mQ.shape[4]
            elif cutlass.const_expr(q_rank == 4):
                s_q = mQ.shape[1]
                h_q = mQ.shape[2]
                d = mQ.shape[3]
            else:
                raise RuntimeError(
                    f"hd256 forward non-varlen expects q rank 4 or 5, got rank {q_rank}"
                )

        if cutlass.const_expr(cum_seqlen_k is not None):

View on GitHub (pinned to 0132848349)

Solutions

1. Pack Q to rank 3: q = q.reshape(total_q, nheads, head_dim) when using cu_seqlens (varlen).
2. Or drop cu_seqlens and pass the 4D/5D batched tensor for the non-varlen path.
3. Verify q.dim() == 3 (varlen) or 4/5 (non-varlen) before calling the kernel.

Example fix

# before (varlen with 4D input)
fmha(q_4d, k, v, cu_seqlen_q=..., ...)

# after
q = q_4d.reshape(-1, nheads, head_dim)  # (total_q, H, D)
fmha(q, k, v, cu_seqlen_q=..., ...)

if cu_seqlen_q is not None:  # varlen path
    assert q.dim() in (3, 5), f"varlen Q must be rank 3 or 5, got {q.dim()}"

def is_valid_varlen_q(q):
    return q.dim() == 3 or q.dim() == 5

When it happens

Trigger: Invoking the hd256 2-CTA forward kernel's __call__ in varlen mode with a 4D (batch, seqlen, nheads, d) Q tensor — the non-varlen shape — or a 2D/1D tensor.

Common situations: Feeding a standard (B, S, H, D) batched tensor while cu_seqlens is provided; reshaping input incorrectly before varlen packing (missing the pack to (total, H, D)).

Related errors

• hd256 forward non-varlen expects q rank 4 or 5, got rank {q_
• The batch size is expected to be 1 rather than {q.shape[0]}
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• cu_seqlens_q tensor must be Int32

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ac2d953af593a1ea. Report an issue: GitHub.