sgl-project/sglang · error · NotImplementedError

Only 'rms_norm_across_heads' is supported as a valid value f

Error message

Only 'rms_norm_across_heads' is supported as a valid value for `qk_norm`.

What it means

NotImplementedError from the LTX-2 attention connector __init__: only qk_norm='rms_norm_across_heads' is implemented. Any other qk_norm value (or None) from the model config is rejected.

Source

Thrown at python/sglang/multimodal_gen/runtime/models/adapter/ltx_2_connector.py:113

        self,
        query_dim: int,
        heads: int = 8,
        kv_heads: int = 8,
        dim_head: int = 64,
        dropout: float = 0.0,
        bias: bool = True,
        cross_attention_dim: Optional[int] = None,
        out_bias: bool = True,
        qk_norm: str = "rms_norm_across_heads",
        norm_eps: float = 1e-6,
        norm_elementwise_affine: bool = True,
        rope_type: str = "interleaved",
        apply_gated_attention: bool = False,
        processor=None,
    ):
        super().__init__()
        if qk_norm != "rms_norm_across_heads":
            raise NotImplementedError(
                "Only 'rms_norm_across_heads' is supported as a valid value for `qk_norm`."
            )

        self.head_dim = dim_head
        self.inner_dim = dim_head * heads
        self.inner_kv_dim = self.inner_dim if kv_heads is None else dim_head * kv_heads
        self.query_dim = query_dim
        self.cross_attention_dim = (
            cross_attention_dim if cross_attention_dim is not None else query_dim
        )
        self.use_bias = bias
        self.dropout = dropout
        self.out_dim = query_dim
        self.heads = heads
        self.rope_type = rope_type

        self.norm_q = torch.nn.RMSNorm(
            dim_head * heads, eps=norm_eps, elementwise_affine=norm_elementwise_affine

View on GitHub (pinned to 0132848349)

Solutions

1. Set qk_norm='rms_norm_across_heads' exactly (lowercase, underscores)
2. Check the checkpoint config for the exact string it uses and remap it before constructing the connector
3. If you truly need another norm, implement it in the connector — it is deliberately unimplemented

Example fix

// before
qk_norm="rms_norm"
// after
qk_norm="rms_norm_across_heads"

qk_norm = qk_norm or "rms_norm_across_heads"
if qk_norm != "rms_norm_across_heads":
    qk_norm = {"rms_norm": "rms_norm_across_heads"}.get(qk_norm, qk_norm)
assert qk_norm == "rms_norm_across_heads"

When it happens

Trigger: Constructing the connector with qk_norm set to anything other than 'rms_norm_across_heads' (e.g. 'rmsnorm', 'layer_norm', None) — commonly from a checkpoint config that uses a different normalization name.

Common situations: New LTX-2 checkpoint variants with different qk-norm configs; typos or case differences in config ('RMS_Norm_across_heads'); ported configs from other codebases that use None or 'rms_norm' per-head.

Related errors

• Block sparsity + paged KV not supported on SM100
• Block sparsity + sheared bias is not supported on SM90
• Custom user-provided score_mod is not supported on SM8x arch
• Only neox-style RoPE is supported.
• padding_side must be 'left' or 'right', got {padding_side}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/43c5606edf088ca0. Report an issue: GitHub.