sgl-project/sglang · error · TypeError

f"Sol-Attn requires bfloat16 activations, got {q.dtype}"

Error message

f"Sol-Attn requires bfloat16 activations, got {q.dtype}"

What it means

The Sol-Attn attention backend only supports bfloat16 activations. Before invoking the sol_attn kernel, _run_sol_attn_thd checks the query dtype and raises TypeError if it is not torch.bfloat16, because the underlying Sol-Attn kernel is compiled/tuned exclusively for bf16 tensor cores.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/attention/backends/sol_attn.py:218

                is_causal=self.causal,
                sm_scale=self.softmax_scale,
            )[0]
        return output

    def _run_sol_attn_thd(
        self,
        query: torch.Tensor,
        key: torch.Tensor,
        value: torch.Tensor,
    ) -> torch.Tensor:
        from sol_attn import sol_attn

        cfg = _get_sol_attn_runtime_config()
        q = query.unsqueeze(0).contiguous()
        k = key.unsqueeze(0).contiguous()
        v = value.unsqueeze(0).contiguous()
        if q.dtype != torch.bfloat16:
            raise TypeError(f"Sol-Attn requires bfloat16 activations, got {q.dtype}")

        if self._sol_params is None:
            self._sol_params = frozenset(inspect.signature(sol_attn).parameters)

        kwargs = {
            "tau": cfg["tau"],
            "thresh_type": cfg["thresh_type"],
            "kv_splits": _resolve_kv_splits(q, cfg["kv_splits"]),
            "sink_start": cfg["sink_start"],
            "sink_tokens": cfg["sink_tokens"],
        }
        # Wan2GP Ada port: INT8-QK Triton; official NVlabs API has no int8_qk.
        if "int8_qk" in self._sol_params and tuple(
            torch.cuda.get_device_capability(q.device)
        ) >= (8, 9):
            kwargs["int8_qk"] = True
        kwargs = {k: v for k, v in kwargs.items() if k in self._sol_params}
        return sol_attn(q, k, v, **kwargs).squeeze(0)

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the model/inputs run in bfloat16: load weights with torch.bfloat16 and pass --dtype bfloat16 (or equivalent) so query/key/value arrive as bf16.
2. Cast tensors before calling forward: q = q.to(torch.bfloat16) (and likewise k, v) if mixed precision upstream is unavoidable.
3. Switch to a different attention backend that supports your dtype if bf16 is not acceptable on your hardware.
4. Verify GPU support: bf16 requires Ampere (sm_80)+ NVIDIA GPUs or supported AMD cards; on older GPUs choose fp16-compatible backends instead.

Example fix

// before
out = attn.forward(query, key, value)  # query is torch.float16

// after
q = query.to(torch.bfloat16)
k = key.to(torch.bfloat16)
v = value.to(torch.bfloat16)
out = attn.forward(q, k, v)

import torch

def assert_bf16(*ts):
    for t in ts:
        if t.dtype != torch.bfloat16:
            raise TypeError(f"cast to bf16 required, got {t.dtype}")

assert_bf16(q, k, v)
out = attn.forward(q, k, v)

def is_bf16(t: torch.Tensor) -> bool:
    return t.dtype == torch.bfloat16

When it happens

Trigger: Calling forward or forward_varlen on the Sol-Attn attention backend with query tensors in fp16, fp32, or any non-bf16 dtype; e.g. loading a model checkpoint in float16 or running with a dtype override so q.dtype != torch.bfloat16.

Common situations: Running a multimodal generation model whose config specifies fp16, casting inputs to float() for debugging, or a precision-override CLI flag (--dtype float16 / half) while the attention backend is set to sol_attn.

Related errors

• sparse_attn_v4_paged_decode expects fp16/bf16 q, got {q.dtyp
• timestep must be a CUDA bfloat16 tensor
• QKV tensors must be CUDA bfloat16 tensors
• Unsupported interleaved_rope_fp64 dtype: {dtype}
• Unsupported ltx25_decoder_rope dtype: {dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7e65da30d85cabce. Report an issue: GitHub.