sgl-project/sglang · error · RuntimeError

Self attention has no KV cache scaling factor attribute!

Error message

Self attention has no KV cache scaling factor attribute!

What it means

When loading kv-cache quantization scales, the attention backend must expose k_scale/v_scale attributes on attn. If absent, the scale cannot be applied and a RuntimeError is raised instead of silently ignoring quantization.

Source

Thrown at python/sglang/srt/models/mimo_v2.py:1135

    # factors (or else raise an exception). Thus, handled exceptions should
    # make sure to leave KV cache scale factors in a known good (dummy) state
    def load_kv_cache_scales(self, quantization_param_path: str) -> None:
        attn_tp_rank = get_parallel().attn_tp_rank
        attn_tp_size = get_parallel().attn_tp_size
        for layer_idx, scaling_factor in kv_cache_scales_loader(
            quantization_param_path,
            attn_tp_rank,
            attn_tp_size,
            self.config.num_hidden_layers,
            self.config.__class__.model_type,
        ):
            if not isinstance(self.layers[layer_idx], nn.Identity):
                layer_self_attn = self.layers[layer_idx].self_attn
            if hasattr(layer_self_attn.attn, "k_scale"):
                layer_self_attn.attn.k_scale = scaling_factor
                layer_self_attn.attn.v_scale = scaling_factor
            else:
                raise RuntimeError(
                    "Self attention has no KV cache scaling " "factor attribute!"
                )


class MiMoV2ForCausalLM(nn.Module, AudioEncoderMixin):
    # BitandBytes specific attributes
    default_bitsandbytes_target_modules = [
        ".gate_proj.",
        ".down_proj.",
        ".up_proj.",
        ".q_proj.",
        ".k_proj.",
        ".v_proj.",
        ".o_proj.",
    ]
    bitsandbytes_stacked_params_mapping = {
        # shard_name, weight_name, index
        "q_proj": ("qkv_proj", 0),

View on GitHub (pinned to 0132848349)

Solutions

1. Use an attention backend that supports kv cache scaling (e.g. FlashAttention/triton with fp8 kv) so attn exposes k_scale
2. Regenerate or drop the kv_scale JSON if you don't intend fp8 kv cache
3. Check that layer indices in the scale file match the model

if not hasattr(layer.self_attn.attn, 'k_scale'):
    raise SystemExit('current attention backend lacks kv-scale support; drop kv scale file or switch backend')

def backend_supports_kv_scale(model) -> bool:
    return hasattr(model.layers[0].self_attn.attn, 'k_scale')

try:
    model.load_kv_cache_scales(path)
except RuntimeError as e:
    if 'KV cache scaling' in str(e):
        logger.warning('skipping kv scales: backend unsupported')
    else:
        raise

When it happens

Trigger: Calling load_kv_cache_scales with a kv cache scaling file while the attention backend's attn object has no 'k_scale' attribute (backends without fp8 kv-cache support).

Common situations: Using --kv-cache-dtype fp8_e4m3 with a scaling JSON on an attention backend that doesn't implement k_scale/v_scale; mixing quantized-kv artifacts with a different backend.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• Self attention has no KV cache scaling factor attribute!
• Self attention has no KV cache scaling factor attribute!
• kv_scales must be fp32, got {kv_scales.dtype}
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f7be89d86e7613ba. Report an issue: GitHub.