sgl-project/sglang · critical · Exception

QK Norm type not supported

Error message

QK Norm type not supported

What it means

Raised in WanVideo attention __init__ when the qk_norm string matches none of the supported options (only 'rms_norm_across_heads' and other listed branches are supported). The exception is raised bare after an error log, so the traceback carries no message.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/wanvideo.py:519

                supported_attention_backends=self_attn_backends,
                prefix=add_prefix("attn1", prefix),
                quant_config=quant_config,
                is_cross_attention=False,
            )

        self.hidden_dim = dim
        self.num_attention_heads = num_heads
        self.dim_head = dim // num_heads
        if qk_norm == "rms_norm":
            self.norm_q = RMSNorm(self.dim_head, eps=eps)
            self.norm_k = RMSNorm(self.dim_head, eps=eps)
        elif qk_norm == "rms_norm_across_heads":
            # LTX applies qk norm across all heads
            self.norm_q = RMSNorm(dim, eps=eps)
            self.norm_k = RMSNorm(dim, eps=eps)
        else:
            logger.error("QK Norm type not supported")
            raise Exception
        assert cross_attn_norm is True
        self.qk_norm = qk_norm
        self.tp_rmsnorm = qk_norm == "rms_norm_across_heads" and tp_size > 1
        self.self_attn_residual_norm = ScaleResidualLayerNormScaleShift(
            dim,
            eps=eps,
            elementwise_affine=True,
            dtype=torch.float32,
        )

        # 2. Cross-attention
        cross_attn_backends = {
            b for b in supported_attention_backends if not b.is_sparse
        }
        if added_kv_proj_dim is not None:
            # I2V
            self.attn2 = WanI2VCrossAttention(
                dim,

View on GitHub (pinned to 0132848349)

Solutions

1. Check supported qk_norm values in the if/elif chain in wanvideo.py around line 519 and use one of them
2. Set qk_norm = 'rms_norm_across_heads' for LTX-style models
3. Extend the chain with a new branch (plus norm_q/norm_k modules) if your model genuinely needs a new norm type

Example fix

# before
qk_norm = "layer_norm"
# after
qk_norm = "rms_norm_across_heads"

SUPPORTED_QK_NORMS = {"rms_norm_across_heads", ...}  # mirror if/elif chain
assert qk_norm in SUPPORTED_QK_NORMS, f"unsupported qk_norm {qk_norm!r}"

def is_supported_qk_norm(s: str) -> bool:
    return s in {"rms_norm_across_heads", "layer_norm_across_heads"}

except Exception as e:
    logger.error("QK norm %r unsupported during attention init", qk_norm)
    raise

When it happens

Trigger: Constructing the attention block with an unrecognized qk_norm value such as 'layer_norm', 'knorm', a typo, or None.

Common situations: Adding a new Wan/LTX variant with a different QK normalization; config carrying an empty qk_norm string.

Related errors

• Unsupported qk_norm: {qk_norm}
• bad compress_ratio {compress_ratio}
• The requested FlashAttention forward configuration exceeds S
• flashinfer_sparse_mla supports only GLM DSA with FP8 KV cach
• GLM DSA with FP8 KV cache on NVIDIA SM120/SM121 supports onl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a0a1a0eda096c3f3. Report an issue: GitHub.