sgl-project/sglang · error · ValueError

Invalid packed Q size {q_dim}: must be divisible by K={K}. K

Error message

Invalid packed Q size {q_dim}: must be divisible by K={K}. KDA packed decode requires num_q_heads == num_k_heads and head_q_dim == head_k_dim.

What it means

After splitting the QK half of packed mixed_qkv, the Q size (qk_dim//2) must be divisible by K (head_k_dim). KDA packed decode only supports num_q_heads == num_k_heads and head_q_dim == head_k_dim, so q_dim must be an exact multiple of K.

Source

Thrown at python/sglang/kernels/ops/attention/fla/fused_recurrent.py:615

        raise ValueError(f"`A_log` must have {HV} elements (got {A_log.numel()}).")
    if dt_bias.numel() != HV * K:
        raise ValueError(
            f"`dt_bias` must have {HV * K} elements (got {dt_bias.numel()})."
        )
    if out.shape != (B, 1, HV, V):
        raise ValueError(
            f"`out` must have shape {(B, 1, HV, V)} (got out.shape={tuple(out.shape)})."
        )

    qkv_dim = mixed_qkv.shape[1]
    qk_dim = qkv_dim - HV * V
    if qk_dim <= 0 or qk_dim % 2 != 0:
        raise ValueError(
            f"Invalid packed `mixed_qkv` last dim={qkv_dim} for HV={HV}, V={V}."
        )
    q_dim = qk_dim // 2
    if q_dim % K != 0:
        raise ValueError(
            f"Invalid packed Q size {q_dim}: must be divisible by K={K}. "
            "KDA packed decode requires num_q_heads == num_k_heads and "
            "head_q_dim == head_k_dim."
        )
    H = q_dim // K
    if H <= 0 or HV % H != 0:
        raise ValueError(
            f"Invalid head config inferred from mixed_qkv: H={H}, HV={HV}."
        )

    # Batched-decode CUDA fast path:
    # row-streaming state update reaches the in-place R+W bandwidth of the
    # part (~9.6 TB/s) where this triton kernel tops out at ~5 TB/s holding a
    # [BV, K] register tile per warp. ULP-level output differences only
    # (reduction order); small batches keep triton (launch-bound anyway).
    if use_qk_l2norm_in_kernel:
        from sglang.kernels.ops.attention import kda_packed_decode as kda_decode_cuda

View on GitHub (pinned to 0132848349)

Solutions

1. Use the non-packed decode path (fused_recurrent_kda with unpacked q/k/v) for GQA-shaped models
2. Verify head_q_dim == head_k_dim and num_q_heads == num_k_heads in the checkpoint config
3. Repack or project q to have exactly H*K elements matching k

Example fix

// before
out = fused_recurrent_kda_packed_decode(mixed_qkv, ...)  # GQA weights, q_dim % K != 0
// after
q, k, v = unpack(mixed_qkv)  # use unpacked API for GQA geometry
out = fused_recurrent_kda(q=q, k=k, v=v, ...)

assert num_q_heads == num_k_heads and head_q_dim == head_k_dim, 'packed decode needs MHA-style q/k'

def supports_packed_decode(cfg) -> bool:
    return cfg.num_q_heads == cfg.num_k_heads and cfg.head_q_dim == cfg.head_k_dim

try:
    out = fused_recurrent_kda_packed_decode(...)
except ValueError:
    out = fused_recurrent_kda(q, k, v, ...)  # unpacked fallback

When it happens

Trigger: Calling packed decode with GQA-style KDA weights where num_q_heads is a multiple-but-not-equal of num_k_heads, or head_q_dim != head_k_dim, making q_dim % K != 0.

Common situations: Reusing a chunked prefill kernel config on the packed decode path; loading a KDA checkpoint with q/k head dim mismatch (e.g. 128 vs 64); hand-crafting mixed_qkv for tests.

Related errors

• `dt_bias` must have {HV * K} elements (got {dt_bias.numel()}
• `a` must have shape [B, HV*K] with HV={HV}, K={K} (got a.sha
• `b` must have shape [B, HV] with HV={HV} (got b.shape={tuple
• `mixed_qkv` must be 2D (got ndim={mixed_qkv.ndim}).
• `b` must be 2D (got b.ndim={b.ndim}).

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cf062935f43e2c5b. Report an issue: GitHub.