{"record":{"id":"8780904c0e559cf2","repo":"sgl-project/sglang","slug":"sparse-mla-q8kv8-prefill-fwd-requires-h-q-padded-t","errorCode":null,"errorMessage":"sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive multiple of 64, got {h_q}","messagePattern":"sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive multiple of 64, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":350,"sourceCode":"    if kv.dtype != torch.float8_e4m3fn:\n        raise ValueError(f\"kv must be torch.float8_e4m3fn, got {kv.dtype}\")\n\n    if not q.is_contiguous():\n        raise ValueError(\"q must be contiguous\")\n    if not kv.is_contiguous():\n        raise ValueError(\"kv must be contiguous\")\n    if not indices.is_contiguous():\n        raise ValueError(\"indices must be contiguous\")\n\n    if kv_d_qk != d_qk:\n        raise ValueError(f\"kv d_qk must match q d_qk={d_qk}, got {kv_d_qk}\")\n\n    # The CUDA implementation uses B_H=64 and launches h_q / B_H CTAs.\n    # Reject unpadded TP-local head counts instead of launching zero CTAs and\n    # returning uninitialized outputs, which can appear to callers as a hang or\n    # a later collective failure.\n    if h_q == 0 or h_q % 64 != 0:\n        raise ValueError(\n            \"sparse_mla_q8kv8_prefill_fwd requires h_q padded to a positive \"\n            f\"multiple of 64, got {h_q}\"\n        )\n\n    if h_kv != 1:\n        raise ValueError(f\"sparse_mla_q8kv8_prefill_fwd requires h_kv=1, got {h_kv}\")\n\n    if d_qk not in (512, 576):\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd supports d_qk=512/576, got {d_qk}\"\n        )\n\n    if indices.shape[:2] != (s_q, h_kv):\n        raise ValueError(\n            \"indices must have shape \"\n            f\"({s_q}, {h_kv}, topk), got {tuple(indices.shape)}\"\n        )\n","sourceCodeStart":332,"sourceCodeEnd":368,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L332-L368","documentation":"The CUDA kernel uses a block size B_H=64 and launches h_q/64 CTAs; it requires the TP-local query head count h_q to be a positive multiple of 64. Zero or non-multiple head counts would launch zero CTAs and return uninitialized outputs, so they are rejected explicitly (comment notes this could otherwise look like a hang).","triggerScenarios":"Running with a TP degree that produces h_q not divisible by 64 (e.g. 128 total heads with TP=3 gives h_q≈43), or h_q==0 due to a misconfigured head allocation.","commonSituations":"Unusual tensor-parallel sizes; small models with head counts like 32; head-padding step skipped when building the per-rank q tensor.","solutions":["Choose a TP degree such that num_attention_heads / tp_size is a multiple of 64 (e.g. TP in {1,2,4,8} for 128 heads)","Pad h_q to the next multiple of 64 if your model supports head padding","Verify h_q is computed from the correct config (num_attention_heads, not num_kv_heads) and is nonzero"],"exampleFix":"# before: 128 heads, tp=3 -> h_q=43\n# after: use tp=4 -> h_q=32? no -> use tp=2 -> h_q=64 (multiple of 64)\nserver_args.tensor_parallel_size = 2","handlingStrategy":"validation","validationCode":"h_q = q.shape[1]\nassert h_q > 0 and h_q % 64 == 0, f\"h_q={h_q} must be a positive multiple of 64\"","typeGuard":"def head_count_ok(q: torch.Tensor) -> bool:\n    h = q.shape[1]\n    return h > 0 and h % 64 == 0","tryCatchPattern":null,"preventionTips":["Validate TP degree against num heads (h_q multiple of 64) at launch","Fail fast in server args validation, not at first prefill"],"tags":["shape-validation","tensor-parallel","sparse-attention"],"backgroundTag":"head-count-not-multiple-of-block","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}