{"record":{"id":"400f90472834b054","repo":"sgl-project/sglang","slug":"attn-sink-must-be-float32-with-shape-h-q-got","errorCode":null,"errorMessage":"attn_sink must be float32 with shape ({h_q},), got {tuple(attn_sink.shape)}/{attn_sink.dtype}","messagePattern":"attn_sink must be float32 with shape \\((.+?),\\), got (.+?)/(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py","lineNumber":408,"sourceCode":"            )\n        if not topk_length.is_contiguous():\n            raise ValueError(\"topk_length must be contiguous\")\n        if torch.any(topk_length < 0).item() or torch.any(topk_length > topk).item():\n            raise ValueError(\n                \"topk_length values must satisfy \" f\"0 <= topk_length <= topk ({topk})\"\n            )\n\n    if d_v != 512:\n        raise ValueError(\n            f\"sparse_mla_q8kv8_prefill_fwd only supports d_v=512, got {d_v}\"\n        )\n\n    if attn_sink is not None and topk_length is None:\n        raise ValueError(\"attn_sink requires topk_length to be provided as well\")\n\n    if attn_sink is not None:\n        if attn_sink.shape != (h_q,) or attn_sink.dtype != torch.float32:\n            raise ValueError(\n                f\"attn_sink must be float32 with shape ({h_q},), got \"\n                f\"{tuple(attn_sink.shape)}/{attn_sink.dtype}\"\n            )\n        if not attn_sink.is_cuda:\n            raise ValueError(\"attn_sink must be a CUDA tensor\")\n        if attn_sink.device != device:\n            raise ValueError(\n                f\"attn_sink must be on q's device {device}, got {attn_sink.device}\"\n            )\n        if not attn_sink.is_contiguous():\n            raise ValueError(\"attn_sink must be contiguous\")\n\n    for name, scale in ((\"q_scale\", q_scale), (\"kv_scale\", kv_scale)):\n        if not isinstance(scale, torch.Tensor):\n            raise ValueError(f\"{name} must be a torch.Tensor\")\n        if not scale.is_cuda:\n            raise ValueError(f\"{name} must be a CUDA tensor\")\n        if scale.device != device:","sourceCodeStart":390,"sourceCodeEnd":426,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/attention/sparse_mla_q8kv8_prefill_sm90.py#L390-L426","documentation":"sparse_mla_q8kv8_prefill_fwd validates the optional attn_sink tensor before launching the sparse MLA q8kv8 prefill kernel. attn_sink must be a 1-D float32 tensor of length h_q (the number of query heads); any other shape or dtype is rejected because the kernel indexes it per head. The error message reports both the offending shape and dtype.","triggerScenarios":"Calling sparse_mla_q8kv8_prefill_fwd(..., attn_sink=tensor) where tensor has shape (h_kv,), a scalar shape (), (1, h_q), or dtype torch.bfloat16/float16 instead of torch.float32.","commonSituations":"Passing a KV-head-shaped sink (forgetting it is per query head), reusing a bf16 model parameter as the sink, or passing a sink tensor that was indexed/squeezed incorrectly.","solutions":["Build the sink as torch.zeros(h_q, dtype=torch.float32, device=q.device) and fill per-head values","Check h_q from q.shape[1] before constructing attn_sink so lengths always match","Cast an existing sink with attn_sink.float() if the values are right but the dtype is wrong"],"exampleFix":"// before\nattn_sink = torch.zeros(num_kv_heads, dtype=torch.bfloat16, device=q.device)\nsparse_mla_q8kv8_prefill_fwd(q, ..., attn_sink=attn_sink)\n// after\nh_q = q.shape[1]\nattn_sink = torch.zeros(h_q, dtype=torch.float32, device=q.device)\nsparse_mla_q8kv8_prefill_fwd(q, ..., attn_sink=attn_sink)","handlingStrategy":"validation","validationCode":"assert attn_sink is None or (attn_sink.dtype == torch.float32 and attn_sink.shape == (q.shape[1],)), 'attn_sink must be float32 shape (h_q,)'","typeGuard":"def valid_attn_sink(t: torch.Tensor, h_q: int) -> bool:\n    return t.dtype == torch.float32 and tuple(t.shape) == (h_q,)","tryCatchPattern":null,"preventionTips":["Derive h_q from q.shape[1] at the call site instead of hardcoding head counts","Centralize sink construction in one helper that always uses float32/(h_q,)"],"tags":["mla","sparse-attention","tensor-validation","dtype"],"backgroundTag":"tensor-shape-validation-failed","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}