{"record":{"id":"3fb9d85179c8d205","repo":"sgl-project/sglang","slug":"only-neox-style-rope-is-supported","errorCode":null,"errorMessage":"Only neox-style RoPE is supported.","messagePattern":"Only neox-style RoPE is supported\\.","errorType":"validation","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/kernels/ops/speculative/fused_kv_materialize.py","lineNumber":270,"sourceCode":"        rotary_emb,\n        num_kv_heads: int,\n        head_dim: int,\n        device: torch.device,\n        max_position_hint: Optional[int] = None,\n    ):\n        self.num_kv_heads = num_kv_heads\n        self.head_dim = head_dim\n        self.rotary_emb = rotary_emb\n        self.n_layers = len(layers)\n        self.device = device\n        self.kv_size = self.num_kv_heads * self.head_dim\n        self.layer_out_dim = 2 * self.kv_size\n\n        self.rotary_dim = int(getattr(rotary_emb, \"rotary_dim\", head_dim))\n        self.is_neox_style = bool(getattr(rotary_emb, \"is_neox_style\", True))\n\n        if not self.is_neox_style:\n            raise NotImplementedError(\"Only neox-style RoPE is supported.\")\n        if self.rotary_dim <= 0 or self.rotary_dim > self.head_dim:\n            raise ValueError(\n                \"Invalid fused KV rotary/head dim pair: \"\n                f\"rotary_dim={self.rotary_dim}, head_dim={self.head_dim}.\"\n            )\n\n        self.max_position_hint = (\n            max(int(max_position_hint) - 1, 0)\n            if max_position_hint is not None\n            else None\n        )\n        self._reserved_rope_cache_len = int(\n            getattr(self.rotary_emb, \"cos_sin_cache\", torch.empty((0,))).shape[0]\n        )\n        self._mm_out_supported = True\n        self._workspace_capacity = 0\n        self._workspace_dtype: Optional[torch.dtype] = None\n        self._proj_workspace: Optional[torch.Tensor] = None","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/kernels/ops/speculative/fused_kv_materialize.py#L252-L288","documentation":"The fused KV materialization kernel only implements neox-style (half-rotate) rotary position embeddings. The model's rotary_emb reports is_neox_style=False (GPT-J interleaved style), which is unsupported.","triggerScenarios":"Constructing FusedKVMaterializer for a model whose rotary_emb.is_neox_style is False (e.g. GPT-J, some CodeGen variants).","commonSituations":"Enabling the fused KV speculative path on a model family with interleaved RoPE; defaults to True via getattr so custom rotary_emb objects lacking the attribute won't trip this.","solutions":["Fall back to the standard (non-fused) draft KV path for non-neox models.","Skip enabling fused KV materialization for GPT-J-style RoPE models.","If implementing support, a interleaved-to-neox transpose in the kernel would be required upstream."],"exampleFix":"// before\nmat = FusedKVMaterializer(model.layers, rotary_emb)  # GPT-J style\n// after\nmat = None  # use standard draft path; fused KV only for neox-style RoPE","handlingStrategy":"fallback","validationCode":"if not bool(getattr(rotary_emb, 'is_neox_style', True)):\n    use_fused_kv = False  # fall back to standard draft path","typeGuard":"def supports_fused_kv(rotary_emb) -> bool:\n    return bool(getattr(rotary_emb, 'is_neox_style', True))","tryCatchPattern":"try:\n    mat = FusedKVMaterializer(...)\nexcept NotImplementedError:\n    mat = None  # standard path","preventionTips":["Gate fused KV enablement on is_neox_style check at startup."],"tags":["unsupported-feature","rope","speculative-decoding"],"backgroundTag":"unsupported-operation","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}