sgl-project/sglang · error · ValueError

Query heads {q_heads} not divisible by KV heads {kv_heads}

Error message

Query heads {q_heads} not divisible by KV heads {kv_heads}

What it means

When query heads outnumber KV heads in Quest retrieval, q_heads must be divisible by kv_heads so grouped heads can be averaged (GQA/MQA alignment). If not divisible, the grouping is invalid and page scores cannot be computed via the mean approximation.

Source

Thrown at python/sglang/srt/mem_cache/sparsity/algorithms/quest_algorithm.py:152

        head_dim = k_min.shape[-1]
        if queries.dim() == 2:
            bs, hidden = queries.shape
            if hidden % head_dim != 0:
                raise ValueError(
                    f"Quest query hidden size {hidden} not divisible by head_dim {head_dim}"
                )
            q_heads = hidden // head_dim
            q = queries.view(bs, q_heads, head_dim)
        elif queries.dim() == 3:
            q = queries
        else:
            raise ValueError(f"Unsupported query shape for Quest: {queries.shape}")

        kv_heads = k_min.shape[-2]
        q_heads = q.shape[1]
        if q_heads != kv_heads:
            if q_heads % kv_heads != 0:
                raise ValueError(
                    f"Query heads {q_heads} not divisible by KV heads {kv_heads}"
                )
            group = q_heads // kv_heads
            # Average grouped query heads to align with KV heads (approximation for MQA/GQA).
            q = q.view(q.shape[0], kv_heads, group, head_dim).mean(dim=2)

        q = q.to(k_min.dtype).unsqueeze(1)  # [bs, 1, kv_heads, head_dim]

        criticality = torch.where(q >= 0, q * k_max, q * k_min).sum(dim=(2, 3))
        criticality = torch.where(
            valid_mask, criticality, torch.full_like(criticality, float("-inf"))
        )

        return criticality

View on GitHub (pinned to 0132848349)

Solutions

1. Make the query tensor's head count a multiple of the KV cache head count (align GQA group size)
2. Reproduce MQA by repeating/averaging queries to exactly kv_heads before retrieval
3. Check TP settings: kv_heads after sharding must divide q_heads after sharding

Example fix

# before
# q: (bs, 12, hd), kv cache has 8 heads -> error
# after
# align GQA: build q with 8 heads (repeat-interleave averaged) or fix kv_heads=6/12

kv_heads = k_min.shape[-2]
q_heads = queries.shape[1]
if q_heads % kv_heads != 0:
    raise ValueError(f"{q_heads} q_heads not divisible by {kv_heads} kv_heads")

def gqa_aligned(q_heads: int, kv_heads: int) -> bool:
    return q_heads == kv_heads or q_heads % kv_heads == 0

When it happens

Trigger: Calling Quest retrieval with q_heads % kv_heads != 0 — e.g. 12 query heads against 8 KV heads, or a query tensor whose head count came from a different GQA group size than the KV cache.

Common situations: Custom models with unusual GQA ratios; queries built from an intermediate projection with an arbitrary head count; KV cache configured with a different kv_head count than the model (e.g. --kv-cache-quant or tensor-parallel changes to head counts).

Related errors

• Quest query hidden size {hidden} not divisible by head_dim {
• Unsupported query shape for Quest: {queries.shape}
• GQA/MQA requires query heads to be a multiple of KV heads, g
• forward_batch with seq_lens is required for TopK retrieval
• Required: indexer, forward_batch, x, q_lora, positions

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3fb4b492f43e7c11. Report an issue: GitHub.