sgl-project/sglang · critical · ValueError

Unsupported qk_norm: {qk_norm}

Error message

Unsupported qk_norm: {qk_norm}

What it means

LingBotWorld attention only implements two QK-norm modes: per-head RMSNorm ('rms_norm'-style on dim_head) and 'rms_norm_across_heads' (RMSNorm over the full dim). Any other qk_norm string in the config is rejected at init because no norm layers could be built.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/dits/lingbot_world.py:420

        self.tp_rmsnorm = qk_norm == "rms_norm_across_heads" and tp_size > 1

        self.attn1 = USPAttention(
            num_heads=self.local_num_heads,
            head_size=self.dim_head,
            causal=False,
            supported_attention_backends=supported_attention_backends,
            prefix=add_prefix("attn1", prefix),
            quant_config=quant_config,
            is_cross_attention=False,
        )
        if qk_norm == "rms_norm":
            self.norm_q = RMSNorm(self.dim_head, eps=eps)
            self.norm_k = RMSNorm(self.dim_head, eps=eps)
        elif qk_norm == "rms_norm_across_heads":
            self.norm_q = RMSNorm(dim, eps=eps)
            self.norm_k = RMSNorm(dim, eps=eps)
        else:
            raise ValueError(f"Unsupported qk_norm: {qk_norm}")
        if not cross_attn_norm:
            raise ValueError("LingBotWorld requires cross_attn_norm=True")
        self.self_attn_residual_norm = ScaleResidualLayerNormScaleShift(
            dim,
            eps=eps,
            elementwise_affine=True,
            dtype=torch.float32,
        )

        cross_attn_backends = {
            b for b in supported_attention_backends if not b.is_sparse
        }
        if added_kv_proj_dim is not None:
            self.attn2 = WanI2VCrossAttention(
                dim,
                num_heads,
                qk_norm=qk_norm,
                eps=eps,

View on GitHub (pinned to 0132848349)

Solutions

1. Set qk_norm to one of the supported strings: per-head mode or 'rms_norm_across_heads'
2. If your checkpoint truly uses a different norm, implement the branch in LingBotWorld attention and map the name
3. Validate the config value before model construction

Example fix

// before
config.qk_norm = "layer_norm"

// after
config.qk_norm = "rms_norm_across_heads"   // or per-head RMSNorm mode

SUPPORTED_QK_NORM = {'rms_norm', 'rms_norm_across_heads'}\nassert config.qk_norm in SUPPORTED_QK_NORM, f'qk_norm={config.qk_norm!r} not in {SUPPORTED_QK_NORM}'

def is_supported_qk_norm(v: str) -> bool:\n    return v in ('rms_norm', 'rms_norm_across_heads')

When it happens

Trigger: Constructing the attention with config.qk_norm set to something like 'layer_norm', 'qk_norm', None, or a typo such as 'rms-norm-across-heads'.

Common situations: Adapting a config from another DiT family that uses different norm names; hand-editing config.json; case or dash/slash differences in the string.

Related errors

• LingBotWorld requires cross_attn_norm=True
• Hidden size {self.hidden_size} must be divisible by num_atte
• num_heads ({self.num_heads}) must be divisible by ulysses_de
• {rope_type=} not supported. Choose between 'interleaved' and
• Modality {modality} is not supported. Supported modalities a

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b5229bea72376449. Report an issue: GitHub.