sgl-project/sglang · error · RuntimeError

.apply should not be called.

Error message

{self.__class__.__name__}.apply should not be called.

What it means

BaseScalaQuantLinearMethod.apply is an intentionally unimplemented stub on the KV-cache quantization method class; the quantized KV scale is applied inside Attention.forward, not via a linear apply(). Calling apply (directly or through generic dispatch code that assumes every quant method implements it) raises this RuntimeError naming the concrete subclass.

Solutions

  1. Do not call apply() on KV-cache quant methods; they only provide process_weights_after_loading and the scales are consumed by the attention backend
  2. Branch on the method type before dispatching (see typeGuard)
  3. Keep layer.quant_method usage aligned with the layer kind (LinearMethod vs KV method)

Example fix

# before
out = layer.quant_method.apply(layer, x)
# after
if isinstance(layer.quant_method, BaseScalaQuantLinearMethod):
    raise TypeError("KV-cache quant methods have no apply()")
out = layer.quant_method.apply(layer, x)
Defensive patterns

Strategy: type-guard

Validate before calling

from sglang.srt.layers.quantization.kv_cache import BaseScalaQuantLinearMethod
if isinstance(layer.quant_method, BaseScalaQuantLinearMethod):
    raise TypeError("KV-cache quant methods expose no apply(); use attention forward")

Type guard

def has_apply(quant_method) -> bool:
    return not isinstance(quant_method, BaseScalaQuantLinearMethod) and callable(getattr(quant_method, "apply", None))

Try / catch

try:
    out = layer.quant_method.apply(layer, x)
except RuntimeError as e:
    if "should not be called" in str(e):
        raise TypeError("misrouted KV quant method") from e
    raise

Prevention

When it happens

Trigger: Generic code that iterates quant methods and calls .apply(layer); porting a code path from linear-layer quantization (where apply is the main entry) to a layer whose quant_method is a KV-cache scalar method such as BaseScalaQuantLinearMethod.

Common situations: Writing model support code that treats all QuantizeMethodBase implementations uniformly; newer quant method subclasses forgetting to keep apply unreachable.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/06bbcb0598d4778d. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/layers/quantization/kv_cache.py:49

    def create_weights(self, layer: torch.nn.Module):
        """
        Create "weight" (aka k_scale and v_scale) for an attention layer.
        """
        # Initialize the KV cache scales to -1.0, which is an invalid value.
        # If the k/v_scale appears in the checkpoint, it will be
        # overwritten when loading weights.
        layer.k_scale = torch.nn.Parameter(
            torch.tensor(-1.0, dtype=torch.float32), requires_grad=False
        )
        layer.v_scale = torch.nn.Parameter(
            torch.tensor(-1.0, dtype=torch.float32), requires_grad=False
        )
        layer.k_scale._skip_weight_check = True
        layer.v_scale._skip_weight_check = True

    def apply(self, layer: torch.nn.Module) -> torch.Tensor:
        raise RuntimeError(f"{self.__class__.__name__}.apply should not be called.")

    def process_weights_after_loading(self, layer) -> None:
        if layer.k_scale > 0.0 and layer.v_scale > 0.0:
            # We prefer to use separate k_scale and v_scale if present
            k_scale = layer.k_scale.to("cpu").tolist()
            v_scale = layer.v_scale.to("cpu").tolist()
            if is_fp8_fnuz():
                k_scale *= 2
                v_scale *= 2
        elif layer.k_scale <= 0.0 and layer.v_scale <= 0.0:
            # If no scales were loaded (both scales are invalid non-positive
            # values), use the default value of 1.0
            k_scale = 1.0
            v_scale = 1.0
        else:
            # If we find a single kv_scale in the checkpoint, we remap
            # kv_scale to k_scale during weight loading, and duplicate
            # k_scale to v_scale here

View on GitHub (pinned to 0132848349)