sgl-project/sglang · error · RuntimeError

Self attention has no KV cache scaling factor attribute!

Error message

Self attention has no KV cache scaling factor attribute!

What it means

Apertus's load_kv_cache_scales expects each attention module to expose k_scale (set when FP8 KV cache is enabled). If attn has no k_scale attribute, the calibration scale cannot be applied and a RuntimeError is raised.

Source

Thrown at python/sglang/srt/models/apertus.py:396

    # make sure to leave KV cache scale factors in a known good (dummy) state
    def load_kv_cache_scales(self, quantization_param_path: str) -> None:
        tp_size = get_parallel().tp_size
        tp_rank = get_parallel().tp_rank
        for layer_idx, scaling_factor in kv_cache_scales_loader(
            quantization_param_path,
            tp_rank,
            tp_size,
            self.config.num_hidden_layers,
            self.config.__class__.model_type,
        ):
            if not isinstance(self.layers[layer_idx], nn.Identity):
                layer_self_attn = self.layers[layer_idx].self_attn

            if hasattr(layer_self_attn.attn, "k_scale"):
                layer_self_attn.attn.k_scale = scaling_factor
                layer_self_attn.attn.v_scale = scaling_factor
            else:
                raise RuntimeError(
                    "Self attention has no KV cache scaling " "factor attribute!"
                )


class ApertusForCausalLM(nn.Module):
    # LoRA specific attributes
    embedding_modules = {
        "embed_tokens": "input_embeddings",
        "lm_head": "output_embeddings",
    }
    embedding_padding_modules = ["lm_head"]
    # BitandBytes specific attributes
    default_bitsandbytes_target_modules = [
        ".down_proj.",
        ".up_proj.",
        ".q_proj.",
        ".k_proj.",
        ".v_proj.",

View on GitHub (pinned to 0132848349)

Solutions

1. Ensure the model is launched with FP8 KV cache enabled so k_scale/v_scale attributes exist
2. Verify the quant_config / --kv-cache-dtype actually initializes attention with scaling attributes before loading scales.json

assert all(hasattr(l.self_attn.attn, "k_scale") for l in model.layers), "launch with FP8 KV cache before loading scales"

try: model.load_kv_cache_scales(...)
except RuntimeError as e: log.warning(f"skipping kv scales: {e}")

When it happens

Trigger: Calling load_kv_cache_scales (e.g. loading kv_cache_scales.json for FP8 KV cache quantization) when the attention backend was initialized without FP8 KV scaling support.

Common situations: Mixing --kv-cache-dtype fp8 with quantization formats/backends that don't populate k_scale/v_scale; loading scale files on a non-FP8 deployment.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• Self attention has no KV cache scaling factor attribute!
• Self attention has no KV cache scaling factor attribute!
• kv_scales must be fp32, got {kv_scales.dtype}
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/96a42df3bcaea2a4. Report an issue: GitHub.