{"record":{"id":"7e65da30d85cabce","repo":"sgl-project/sglang","slug":"f-sol-attn-requires-bfloat16-activations-got-q-d","errorCode":null,"errorMessage":"f\"Sol-Attn requires bfloat16 activations, got {q.dtype}\"","messagePattern":"f\"Sol-Attn requires bfloat16 activations, got (.+?)\"","errorType":"validation","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/layers/attention/backends/sol_attn.py","lineNumber":218,"sourceCode":"                is_causal=self.causal,\n                sm_scale=self.softmax_scale,\n            )[0]\n        return output\n\n    def _run_sol_attn_thd(\n        self,\n        query: torch.Tensor,\n        key: torch.Tensor,\n        value: torch.Tensor,\n    ) -> torch.Tensor:\n        from sol_attn import sol_attn\n\n        cfg = _get_sol_attn_runtime_config()\n        q = query.unsqueeze(0).contiguous()\n        k = key.unsqueeze(0).contiguous()\n        v = value.unsqueeze(0).contiguous()\n        if q.dtype != torch.bfloat16:\n            raise TypeError(f\"Sol-Attn requires bfloat16 activations, got {q.dtype}\")\n\n        if self._sol_params is None:\n            self._sol_params = frozenset(inspect.signature(sol_attn).parameters)\n\n        kwargs = {\n            \"tau\": cfg[\"tau\"],\n            \"thresh_type\": cfg[\"thresh_type\"],\n            \"kv_splits\": _resolve_kv_splits(q, cfg[\"kv_splits\"]),\n            \"sink_start\": cfg[\"sink_start\"],\n            \"sink_tokens\": cfg[\"sink_tokens\"],\n        }\n        # Wan2GP Ada port: INT8-QK Triton; official NVlabs API has no int8_qk.\n        if \"int8_qk\" in self._sol_params and tuple(\n            torch.cuda.get_device_capability(q.device)\n        ) >= (8, 9):\n            kwargs[\"int8_qk\"] = True\n        kwargs = {k: v for k, v in kwargs.items() if k in self._sol_params}\n        return sol_attn(q, k, v, **kwargs).squeeze(0)","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/attention/backends/sol_attn.py#L200-L236","documentation":"The Sol-Attn attention backend only supports bfloat16 activations. Before invoking the sol_attn kernel, _run_sol_attn_thd checks the query dtype and raises TypeError if it is not torch.bfloat16, because the underlying Sol-Attn kernel is compiled/tuned exclusively for bf16 tensor cores.","triggerScenarios":"Calling forward or forward_varlen on the Sol-Attn attention backend with query tensors in fp16, fp32, or any non-bf16 dtype; e.g. loading a model checkpoint in float16 or running with a dtype override so q.dtype != torch.bfloat16.","commonSituations":"Running a multimodal generation model whose config specifies fp16, casting inputs to float() for debugging, or a precision-override CLI flag (--dtype float16 / half) while the attention backend is set to sol_attn.","solutions":["Ensure the model/inputs run in bfloat16: load weights with torch.bfloat16 and pass --dtype bfloat16 (or equivalent) so query/key/value arrive as bf16.","Cast tensors before calling forward: q = q.to(torch.bfloat16) (and likewise k, v) if mixed precision upstream is unavoidable.","Switch to a different attention backend that supports your dtype if bf16 is not acceptable on your hardware.","Verify GPU support: bf16 requires Ampere (sm_80)+ NVIDIA GPUs or supported AMD cards; on older GPUs choose fp16-compatible backends instead."],"exampleFix":"// before\nout = attn.forward(query, key, value)  # query is torch.float16\n\n// after\nq = query.to(torch.bfloat16)\nk = key.to(torch.bfloat16)\nv = value.to(torch.bfloat16)\nout = attn.forward(q, k, v)","handlingStrategy":"validation","validationCode":"import torch\n\ndef assert_bf16(*ts):\n    for t in ts:\n        if t.dtype != torch.bfloat16:\n            raise TypeError(f\"cast to bf16 required, got {t.dtype}\")\n\nassert_bf16(q, k, v)\nout = attn.forward(q, k, v)","typeGuard":"def is_bf16(t: torch.Tensor) -> bool:\n    return t.dtype == torch.bfloat16","tryCatchPattern":null,"preventionTips":["Standardize on bfloat16 for the whole multimodal pipeline when using Sol-Attn.","Add a dtype assert at the model entrypoint rather than deep in the attention stack.","Check GPU compute capability (sm_80+) supports bf16 before selecting this backend."],"tags":["dtype","bfloat16","attention-backend","gpu"],"backgroundTag":"unsupported-tensor-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}