sgl-project/sglang · error · RuntimeError

{self.__class__.__name__}.apply should not be called.

Error message

{self.__class__.__name__}.apply should not be called.

What it means

BaseScalaQuantLinearMethod.apply is an intentionally unimplemented stub on the KV-cache quantization method class; the quantized KV scale is applied inside Attention.forward, not via a linear apply(). Calling apply (directly or through generic dispatch code that assumes every quant method implements it) raises this RuntimeError naming the concrete subclass.

Source

Thrown at python/sglang/srt/layers/quantization/kv_cache.py:49

    def create_weights(self, layer: torch.nn.Module):
        """
        Create "weight" (aka k_scale and v_scale) for an attention layer.
        """
        # Initialize the KV cache scales to -1.0, which is an invalid value.
        # If the k/v_scale appears in the checkpoint, it will be
        # overwritten when loading weights.
        layer.k_scale = torch.nn.Parameter(
            torch.tensor(-1.0, dtype=torch.float32), requires_grad=False
        )
        layer.v_scale = torch.nn.Parameter(
            torch.tensor(-1.0, dtype=torch.float32), requires_grad=False
        )
        layer.k_scale._skip_weight_check = True
        layer.v_scale._skip_weight_check = True

    def apply(self, layer: torch.nn.Module) -> torch.Tensor:
        raise RuntimeError(f"{self.__class__.__name__}.apply should not be called.")

    def process_weights_after_loading(self, layer) -> None:
        if layer.k_scale > 0.0 and layer.v_scale > 0.0:
            # We prefer to use separate k_scale and v_scale if present
            k_scale = layer.k_scale.to("cpu").tolist()
            v_scale = layer.v_scale.to("cpu").tolist()
            if is_fp8_fnuz():
                k_scale *= 2
                v_scale *= 2
        elif layer.k_scale <= 0.0 and layer.v_scale <= 0.0:
            # If no scales were loaded (both scales are invalid non-positive
            # values), use the default value of 1.0
            k_scale = 1.0
            v_scale = 1.0
        else:
            # If we find a single kv_scale in the checkpoint, we remap
            # kv_scale to k_scale during weight loading, and duplicate
            # k_scale to v_scale here

View on GitHub (pinned to 0132848349)

Solutions

  1. Do not call apply() on KV-cache quant methods; they only provide process_weights_after_loading and the scales are consumed by the attention backend
  2. Branch on the method type before dispatching (see typeGuard)
  3. Keep layer.quant_method usage aligned with the layer kind (LinearMethod vs KV method)

Example fix

# before
out = layer.quant_method.apply(layer, x)
# after
if isinstance(layer.quant_method, BaseScalaQuantLinearMethod):
    raise TypeError("KV-cache quant methods have no apply()")
out = layer.quant_method.apply(layer, x)
Defensive patterns

Strategy: type-guard

Validate before calling

from sglang.srt.layers.quantization.kv_cache import BaseScalaQuantLinearMethod
if isinstance(layer.quant_method, BaseScalaQuantLinearMethod):
    raise TypeError("KV-cache quant methods expose no apply(); use attention forward")

Type guard

def has_apply(quant_method) -> bool:
    return not isinstance(quant_method, BaseScalaQuantLinearMethod) and callable(getattr(quant_method, "apply", None))

Try / catch

try:
    out = layer.quant_method.apply(layer, x)
except RuntimeError as e:
    if "should not be called" in str(e):
        raise TypeError("misrouted KV quant method") from e
    raise

Prevention

When it happens

Trigger: Generic code that iterates quant methods and calls .apply(layer); porting a code path from linear-layer quantization (where apply is the main entry) to a layer whose quant_method is a KV-cache scalar method such as BaseScalaQuantLinearMethod.

Common situations: Writing model support code that treats all QuantizeMethodBase implementations uniformly; newer quant method subclasses forgetting to keep apply unreachable.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/06bbcb0598d4778d. Report an issue: GitHub.