sgl-project/sglang · error · NotImplementedError

NVFP4 embedding is gather-only. Reaching here means a tied l

Error message

NVFP4 embedding is gather-only. Reaching here means a tied lm_head is sharing this module; exclude the embedding from NVFP4 in the quantization recipe to serve such a checkpoint.

What it means

Error "NVFP4 embedding is gather-only. Reaching here means a tied lm_head is sharing this module; exclude the embedding from NVFP4 in the quantization recipe to serve such a checkpoint." thrown in sgl-project/sglang.

Source

Thrown at python/sglang/srt/layers/quantization/modelopt_quant.py:742

        )
        set_weight_attrs(
            weight_scale_2,
            {"weight_loader": lambda p, w: p.data.copy_(w.reshape(p.shape).float())},
        )
        layer.register_parameter("weight_scale_2", weight_scale_2)

        # A buffer; CUDA graph capture rejects host->device copies.
        layer.register_buffer(
            "e2m1_lut",
            torch.tensor(_E2M1_LUT, dtype=torch.float32),
            persistent=False,
        )

    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
        pass

    def apply(self, *args, **kwargs):
        raise NotImplementedError(
            "NVFP4 embedding is gather-only. Reaching here means a tied lm_head "
            "is sharing this module; exclude the embedding from NVFP4 in the "
            "quantization recipe to serve such a checkpoint."
        )

    def embedding(self, layer: torch.nn.Module, input_: torch.Tensor) -> torch.Tensor:
        index_shape = input_.shape
        flat = input_.reshape(-1)
        packed = layer.weight[flat]  # [T, H/2] uint8
        scale = layer.weight_scale[flat]  # [T, H/16] e4m3
        rows, half = packed.shape
        hidden = half * 2

        codes = packed.new_empty((rows, hidden))
        codes[:, 0::2] = packed & 0x0F
        codes[:, 1::2] = packed >> 4

        mag = layer.e2m1_lut[(codes & 0x7).long()]

View on GitHub (pinned to 0132848349)

When it happens

Trigger: Thrown at python/sglang/srt/layers/quantization/modelopt_quant.py:742 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/90b05e2a003347c5. Report an issue: GitHub.