sgl-project/sglang · error · NotImplementedError

Only neox-style RoPE is supported.

Error message

Only neox-style RoPE is supported.

What it means

The fused KV materialization kernel only implements neox-style (half-rotate) rotary position embeddings. The model's rotary_emb reports is_neox_style=False (GPT-J interleaved style), which is unsupported.

Source

Thrown at python/sglang/kernels/ops/speculative/fused_kv_materialize.py:270

        rotary_emb,
        num_kv_heads: int,
        head_dim: int,
        device: torch.device,
        max_position_hint: Optional[int] = None,
    ):
        self.num_kv_heads = num_kv_heads
        self.head_dim = head_dim
        self.rotary_emb = rotary_emb
        self.n_layers = len(layers)
        self.device = device
        self.kv_size = self.num_kv_heads * self.head_dim
        self.layer_out_dim = 2 * self.kv_size

        self.rotary_dim = int(getattr(rotary_emb, "rotary_dim", head_dim))
        self.is_neox_style = bool(getattr(rotary_emb, "is_neox_style", True))

        if not self.is_neox_style:
            raise NotImplementedError("Only neox-style RoPE is supported.")
        if self.rotary_dim <= 0 or self.rotary_dim > self.head_dim:
            raise ValueError(
                "Invalid fused KV rotary/head dim pair: "
                f"rotary_dim={self.rotary_dim}, head_dim={self.head_dim}."
            )

        self.max_position_hint = (
            max(int(max_position_hint) - 1, 0)
            if max_position_hint is not None
            else None
        )
        self._reserved_rope_cache_len = int(
            getattr(self.rotary_emb, "cos_sin_cache", torch.empty((0,))).shape[0]
        )
        self._mm_out_supported = True
        self._workspace_capacity = 0
        self._workspace_dtype: Optional[torch.dtype] = None
        self._proj_workspace: Optional[torch.Tensor] = None

View on GitHub (pinned to 0132848349)

Solutions

1. Fall back to the standard (non-fused) draft KV path for non-neox models.
2. Skip enabling fused KV materialization for GPT-J-style RoPE models.
3. If implementing support, a interleaved-to-neox transpose in the kernel would be required upstream.

Example fix

// before
mat = FusedKVMaterializer(model.layers, rotary_emb)  # GPT-J style
// after
mat = None  # use standard draft path; fused KV only for neox-style RoPE

if not bool(getattr(rotary_emb, 'is_neox_style', True)):
    use_fused_kv = False  # fall back to standard draft path

def supports_fused_kv(rotary_emb) -> bool:
    return bool(getattr(rotary_emb, 'is_neox_style', True))

try:
    mat = FusedKVMaterializer(...)
except NotImplementedError:
    mat = None  # standard path

When it happens

Trigger: Constructing FusedKVMaterializer for a model whose rotary_emb.is_neox_style is False (e.g. GPT-J, some CodeGen variants).

Common situations: Enabling the fused KV speculative path on a model family with interleaved RoPE; defaults to True via getattr so custom rotary_emb objects lacking the attribute won't trip this.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Invalid fused KV rotary/head dim pair: rotary_dim={rotary_di
• This browser cannot encode H.264 MP4
• rope_pool_fused expects q/k/v to be 3-D
• rope_pool_fused expects positions/slots to be 1-D
• rope_pool_fused expects pool tensors to be 3-D

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3fb9d85179c8d205. Report an issue: GitHub.