{"record":{"id":"15fe278fe1497474","repo":"sgl-project/sglang","slug":"fused-qk-norm-rope-kernel-only-supports-float16","errorCode":null,"errorMessage":"Fused QK-Norm + RoPE kernel only supports float16/bfloat16, but got {img_q.dtype}","messagePattern":"Fused QK-Norm \\+ RoPE kernel only supports float16/bfloat16, but got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/models/dits/joy_image.py","lineNumber":259,"sourceCode":"        ) = self.txt_mod(vec)\n\n        # Image attention\n        img_modulated = self.fused_modulate_img_norm1(\n            img, shift=img_mod1_shift, scale=img_mod1_scale\n        )\n        img_qkv, _ = self.img_attn_qkv(img_modulated)\n        img_q, img_k, img_v = rearrange(\n            img_qkv, \"B L (K H D) -> K B L H D\", K=3, H=self.local_heads_num\n        )\n\n        if vis_freqs_cis is None:\n            raise ValueError(\n                \"vis_freqs_cis is required for fused QK-Norm + RoPE kernel\"\n            )\n        if not (isinstance(vis_freqs_cis, torch.Tensor) and vis_freqs_cis.dim() == 2):\n            raise ValueError(\"vis_freqs_cis must be a 2D cos_sin_cache tensor\")\n        if img_q.dtype not in (torch.float16, torch.bfloat16):\n            raise ValueError(\n                f\"Fused QK-Norm + RoPE kernel only supports float16/bfloat16, but got {img_q.dtype}\"\n            )\n        img_q = img_q.contiguous()\n        img_k = img_k.contiguous()\n        img_q, img_k = apply_qk_norm_with_optional_rope(\n            q=img_q,\n            k=img_k,\n            q_norm=self.img_attn_q_norm,\n            k_norm=self.img_attn_k_norm,\n            head_dim=img_q.shape[-1],\n            cos_sin_cache=vis_freqs_cis,\n            is_neox=False,\n            allow_inplace=True,\n        )\n        img_q, img_k = img_q.to(img_v), img_k.to(img_v)\n\n        # Text attention\n        txt_modulated = self.fused_modulate_txt_norm1(","sourceCodeStart":241,"sourceCodeEnd":277,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/models/dits/joy_image.py#L241-L277","documentation":"JoyImage's fused QK-Norm + RoPE path only accepts half-precision query tensors. The Triton/fused kernel that applies RMSNorm and rotary position embeddings is compiled for float16/bfloat16 layouts, so any other dtype (float32, float64) is rejected before the kernel launch.","triggerScenarios":"Calling JoyImage forward where the img_q tensor (image-branch query after qkv split) has a dtype other than torch.float16 or torch.bfloat16, e.g. running the DiT in float32 (debugging, CPU fallback) or with autocast disabled while model weights were kept in fp32.","commonSituations":"Developer forces float32 for numerical debugging or runs on hardware without bf16; upcasting tensors before forward; mixed setups where text branch is fp16 but vision tensors were cast to fp32.","solutions":["Cast model/inputs to bfloat16 or float16 before forward: model.to(torch.bfloat16), img_q = img_q.to(torch.bfloat16)","Ensure autocast/sampler dtype matches the configured dtype (e.g. --dtype bfloat16 in server args)","If fp32 is required, bypass the fused apply_qk_norm_with_optional_rope path with an unfused reference implementation (compute q-norm/k-norm + RoPE in eager PyTorch)"],"exampleFix":"// before\nout = dit(hidden_states, encoder_hidden_states=cond, ...)  # img_q is float32\n\n// after\ndit = dit.to(torch.bfloat16)\nhidden_states = hidden_states.to(torch.bfloat16)\nout = dit(hidden_states, encoder_hidden_states=cond.to(torch.bfloat16), ...)","handlingStrategy":"validation","validationCode":"assert img_q.dtype in (torch.float16, torch.bfloat16), f\"need fp16/bf16, got {img_q.dtype}\"","typeGuard":"def is_half_precision(t: torch.Tensor) -> bool:\\n    return t.dtype in (torch.float16, torch.bfloat16)","tryCatchPattern":"try:\\n    out = dit(...)\\nexcept ValueError as e:\\n    if 'float16/bfloat16' in str(e):\\n        dit = dit.to(torch.bfloat16); out = dit(...)\\n    else:\\n        raise","preventionTips":["Always launch the DiT with an explicit --dtype bfloat16/float16","Assert dtypes in your sampling loop before forward","Keep autocast consistent with weight dtype"],"tags":["dtype","rope","fused-kernel","joyimage","multimodal"],"backgroundTag":"unsupported-dtype-for-kernel","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}