sgl-project/sglang · error · NotImplementedError

Comfy INT8 embedding weights support lookup only

Error message

Comfy INT8 embedding weights support lookup only

What it means

Comfy INT8 embedding quantization stores weights as int8 codes plus scales; embeddings must be looked up and dequantized via the embedding() path. Calling apply() (the matmul interface) on such a layer is not implemented.

Source

Thrown at python/sglang/multimodal_gen/runtime/layers/quantization/comfy_nvfp4.py:86

            ),
            extra_weight_attrs,
            {"input_dim": 1, "output_dim": 0},
        )
        _register_parameter(
            layer,
            "weight_scale",
            torch.empty(output_size_per_partition, 1, dtype=torch.float32),
            extra_weight_attrs,
            {"output_dim": 0},
        )

    def apply(
        self,
        layer: nn.Module,
        x: torch.Tensor,
        bias: torch.Tensor | None = None,
    ) -> torch.Tensor:
        raise NotImplementedError("Comfy INT8 embedding weights support lookup only")

    def embedding(self, layer: nn.Module, input_: torch.Tensor) -> torch.Tensor:
        weight = F.embedding(input_, layer.weight).to(self.output_dtype)
        scale = F.embedding(input_, layer.weight_scale).to(self.output_dtype)
        return weight * scale


class ComfyFullPrecisionNvfp4LinearMethod(ModelOptFp4LinearMethod):
    """Keep NVFP4 storage and dequantize one active Linear for its matmul."""

    def __init__(
        self,
        quant_config: ComfyNvfp4Config,
        *,
        has_pre_quant_scale: bool,
    ) -> None:
        self.quant_config = quant_config
        self.has_pre_quant_scale = has_pre_quant_scale

View on GitHub (pinned to 0132848349)

Solutions

1. Call the embedding path (embedding(layer, input_ids)) instead of apply()
2. Route embedding layers to lookup, not matmul, in your dispatch logic

Example fix

// before
out = quant_method.apply(layer, input_ids)

// after
out = quant_method.embedding(layer, input_ids)

def is_embedding_method(m) -> bool:
    return hasattr(m, "embedding") and not hasattr(m, "apply_matmul")

When it happens

Trigger: Invoking apply(layer, x) on the int8 embedding quant method instead of calling layer via its embedding path (e.g. routing an embedding layer through a linear-style forward).

Common situations: Generic code that calls apply() on every quant method regardless of layer type; passing token indices to a path expecting hidden states.

Related errors

• f"The class {type(quant_method).__name__} must implement the
• Static compressed-tensors scheme is not yet supported on NPU
• QVGPackedCausalKVCache does not support pinned-sink (longliv
• bitsandbytes 4-bit TP only supports column-parallel output s
• bitsandbytes 4-bit TP does not support nested quant states.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/74cc6958a3af28de. Report an issue: GitHub.