sgl-project/sglang · error · NotImplementedError

InklingNvfp4MoEMethod is the dense shared-expert method; rou

Error message

InklingNvfp4MoEMethod is the dense shared-expert method; routed NVFP4 MoE uses ModelOptNvFp4FusedMoEMethod.

What it means

InklingNvfp4MoEMethod.apply unconditionally raises NotImplementedError; the method exists only to satisfy the FusedMoEMethodBase abstract interface. Routed NVFP4 MoE layers are served by ModelOptNvFp4FusedMoEMethod, and Inkling's dense shared-expert MLP uses its own FP4 path.

Source

Thrown at python/sglang/srt/models/inkling_common/quantization/quant.py:162

        """Process weights for the dense shared-expert NVFP4 path.

        Routed NVFP4 MoE now uses ModelOptNvFp4FusedMoEMethod; this hook is reached
        only for shared experts (InklingBatchDenseMLP), which carry an ``_fp4_strategy``
        and run their own weight preparation.
        """
        if getattr(layer, "_fp4_strategy", None) is not None:
            layer.process_weights_after_loading()

    def apply(
        self,
        layer: torch.nn.Module,
        dispatch_output,  # type: ignore[override]
    ):
        # Kept only to satisfy the FusedMoEMethodBase abstract interface.
        # InklingNvfp4MoEMethod serves the dense shared-expert path (InklingBatchDenseMLP,
        # which uses its own FP4 serving); routed NVFP4 MoE uses
        # ModelOptNvFp4FusedMoEMethod.
        raise NotImplementedError(
            "InklingNvfp4MoEMethod is the dense shared-expert method; routed NVFP4 "
            "MoE uses ModelOptNvFp4FusedMoEMethod."
        )

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure routed NVFP4 MoE layers use ModelOptNvFp4FusedMoEMethod
2. Keep InklingNvfp4MoEMethod attached only to the dense shared-expert MLP (InklingBatchDenseMLP)
3. Do not call .apply() on this method; if you hit it, audit the method-selection logic in the quant registry

Example fix

# before
method = InklingNvfp4MoEMethod(...)
method.apply(layer, output)  # raises
# after
from sglang.srt.layers.quantization.modelopt import ModelOptNvFp4FusedMoEMethod
method = ModelOptNvFp4FusedMoEMethod(...)

assert type(method) is not InklingNvfp4MoEMethod or layer.is_dense_shared_expert

def is_routed_moe_method(m) -> bool:
    return not isinstance(m, InklingNvfp4MoEMethod)

When it happens

Trigger: A FusedMoE layer being routed to InklingNvfp4MoEMethod.apply — i.e. the quant method was attached to a routed MoE weight instead of the dense shared-expert path, or generic code invoked apply().

Common situations: Wiring change that registers InklingNvfp4MoEMethod for routed experts; calling the method directly in tests or custom loaders.

Related errors

• Current platform does not support NVFP4 quantization. Please
• Online MXFP4 requantization from compressed-tensors NVFP4 ch
• Inkling only supports group size 16 for NVFP4
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/92e4c4e5c0cf7ff8. Report an issue: GitHub.