sgl-project/sglang · error · RuntimeError

Self attention has no KV cache scaling factor attribute!

Error message

Self attention has no KV cache scaling factor attribute!

What it means

Arcee's load_kv_cache_scales requires the attention module to have a k_scale attribute, which exists only when the attention layer was built with KV-cache scaling (FP8). Its absence means the scales file cannot be applied.

Source

Thrown at python/sglang/srt/models/arcee.py:365

    def load_kv_cache_scales(self, quantization_param_path: str) -> None:
        tp_size = get_parallel().tp_size
        tp_rank = get_parallel().tp_rank
        for layer_idx, scaling_factor in kv_cache_scales_loader(
            quantization_param_path,
            tp_rank,
            tp_size,
            self.config.num_hidden_layers,
            self.config.__class__.model_type,
        ):
            if not isinstance(self.layers[layer_idx], nn.Identity):
                layer_self_attn = self.layers[layer_idx].self_attn

            if hasattr(layer_self_attn.attn, "k_scale"):
                layer_self_attn.attn.k_scale = scaling_factor
                layer_self_attn.attn.v_scale = scaling_factor
            else:
                raise RuntimeError(
                    "Self attention has no KV cache scaling factor attribute!"
                )


class ArceeForCausalLM(nn.Module):
    # BitandBytes specific attributes
    default_bitsandbytes_target_modules = [
        # Note: gate_proj is removed compared to Llama
        ".down_proj.",
        ".up_proj.",
        ".q_proj.",
        ".k_proj.",
        ".v_proj.",
        ".o_proj.",
    ]
    # in TP, these weights are partitioned along the column dimension (dim=-1)
    column_parallel_weights_modules = [".down_proj.", ".o_proj."]
    bitsandbytes_stacked_params_mapping = {

View on GitHub (pinned to 0132848349)

Solutions

1. Launch with FP8 KV cache (--kv-cache-dtype fp8_e4m3 / matching quant config) so k_scale exists
2. Skip loading kv_cache_scales.json if you are not quantizing the KV cache

assert all(hasattr(l.self_attn.attn, "k_scale") for l in model.layers)

try: model.load_kv_cache_scales(...)
except RuntimeError: pass  # non-FP8 deployment

When it happens

Trigger: Calling load_kv_cache_scales with a kv_cache_scales.json while the server was not launched with FP8 KV cache quantization.

Common situations: Applying FP8 KV calibration files to an FP16/bf16 deployment; quantization format mismatch after a config change.

Related errors

• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• Self attention has no KV cache scaling factor attribute!
• Self attention has no KV cache scaling factor attribute!
• kv_scales must be fp32, got {kv_scales.dtype}
• D={D_check} must be divisible by GROUP_SIZE={_FP8_GROUP_SIZE

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/79bbe224e95460dd. Report an issue: GitHub.