sgl-project/sglang · error · RuntimeError

hd256 forward non-varlen expects q rank 4 or 5, got rank {q_

Error message

hd256 forward non-varlen expects q rank 4 or 5, got rank {q_rank}

What it means

The SM100 head-dim-256 non-varlen FMHA forward kernel accepts Q only as rank 4 (batch, 1, seqlen, nheads, d minus one) standard batched or rank-5 legacy layout. Any other rank raises this RuntimeError in __call__.

Source

Thrown at python/sglang/kernels/ops/attention/flash_attn/cute/sm100_hd256_2cta_fmha_forward.py:283

                s_q = mQ.shape[0]
                h_q = mQ.shape[1]
                d = mQ.shape[2]
            else:
                raise RuntimeError(
                    f"hd256 forward varlen expects q rank 3 or 5, got rank {q_rank}"
                )
        else:
            # Non-varlen path accepts either legacy 5D tensors or standard 4D tensors.
            if cutlass.const_expr(q_rank == 5):
                s_q = mQ.shape[1]
                h_q = mQ.shape[2] * mQ.shape[3]
                d = mQ.shape[4]
            elif cutlass.const_expr(q_rank == 4):
                s_q = mQ.shape[1]
                h_q = mQ.shape[2]
                d = mQ.shape[3]
            else:
                raise RuntimeError(
                    f"hd256 forward non-varlen expects q rank 4 or 5, got rank {q_rank}"
                )

        if cutlass.const_expr(cum_seqlen_k is not None):
            if cutlass.const_expr(k_rank == 5):
                s_k = mK.shape[1]
                h_k = mK.shape[2]
            elif cutlass.const_expr(k_rank == 3):
                s_k = mK.shape[0]
                h_k = mK.shape[1]
            else:
                raise RuntimeError(
                    f"hd256 forward varlen expects k rank 3 or 5, got rank {k_rank}"
                )
        else:
            if cutlass.const_expr(k_rank == 5):
                s_k = mK.shape[1]
                h_k = mK.shape[2]

View on GitHub (pinned to 0132848349)

Solutions

1. Pass the full batched 4D tensor (batch, seqlen, nheads, head_dim) for the non-varlen path.
2. If your data is packed (total_q, H, D), supply cu_seqlen_q/cu_seqlen_k so the varlen branch is taken.
3. Add an assert q.dim() in (4, 5) guard before invoking the kernel.

Example fix

# before (packed tensor, no cu_seqlens)
fmha(q_packed, k_packed, v_packed)

# after
fmha(q_packed, k_packed, v_packed, cu_seqlen_q=cu_q, cu_seqlen_k=cu_k)

if cu_seqlen_q is None:  # non-varlen path
    assert q.dim() in (4, 5), f"batched Q must be rank 4 or 5, got {q.dim()}"

def is_valid_batched_q(q):
    return q.dim() in (4, 5)

When it happens

Trigger: Calling the hd256 forward kernel without cu_seqlens (non-varlen path) with a rank-3 (total, H, D) packed varlen tensor or a rank-2 tensor.

Common situations: Passing pre-packed varlen tensors but forgetting cu_seqlen_k/cu_seqlen_q so the kernel takes the non-varlen branch; reshaping mistakes that drop the batch dimension.

Related errors

• hd256 forward varlen expects q rank 3 or 5, got rank {q_rank
• {name}_block_cnt and {name}_block_idx must be on the same de
• {name}_block tensors must live on CUDA
• FlashAttention combine kernel cannot be implemented with giv
• q, k, and v must be CUDA tensors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/db2461bf98be072b. Report an issue: GitHub.