{"record":{"id":"f7be89d86e7613ba","repo":"sgl-project/sglang","slug":"self-attention-has-no-kv-cache-scaling-factor-attr-f7be89","errorCode":null,"errorMessage":"Self attention has no KV cache scaling factor attribute!","messagePattern":"Self attention has no KV cache scaling factor attribute!","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/mimo_v2.py","lineNumber":1135,"sourceCode":"    # factors (or else raise an exception). Thus, handled exceptions should\n    # make sure to leave KV cache scale factors in a known good (dummy) state\n    def load_kv_cache_scales(self, quantization_param_path: str) -> None:\n        attn_tp_rank = get_parallel().attn_tp_rank\n        attn_tp_size = get_parallel().attn_tp_size\n        for layer_idx, scaling_factor in kv_cache_scales_loader(\n            quantization_param_path,\n            attn_tp_rank,\n            attn_tp_size,\n            self.config.num_hidden_layers,\n            self.config.__class__.model_type,\n        ):\n            if not isinstance(self.layers[layer_idx], nn.Identity):\n                layer_self_attn = self.layers[layer_idx].self_attn\n            if hasattr(layer_self_attn.attn, \"k_scale\"):\n                layer_self_attn.attn.k_scale = scaling_factor\n                layer_self_attn.attn.v_scale = scaling_factor\n            else:\n                raise RuntimeError(\n                    \"Self attention has no KV cache scaling \" \"factor attribute!\"\n                )\n\n\nclass MiMoV2ForCausalLM(nn.Module, AudioEncoderMixin):\n    # BitandBytes specific attributes\n    default_bitsandbytes_target_modules = [\n        \".gate_proj.\",\n        \".down_proj.\",\n        \".up_proj.\",\n        \".q_proj.\",\n        \".k_proj.\",\n        \".v_proj.\",\n        \".o_proj.\",\n    ]\n    bitsandbytes_stacked_params_mapping = {\n        # shard_name, weight_name, index\n        \"q_proj\": (\"qkv_proj\", 0),","sourceCodeStart":1117,"sourceCodeEnd":1153,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/mimo_v2.py#L1117-L1153","documentation":"When loading kv-cache quantization scales, the attention backend must expose k_scale/v_scale attributes on attn. If absent, the scale cannot be applied and a RuntimeError is raised instead of silently ignoring quantization.","triggerScenarios":"Calling load_kv_cache_scales with a kv cache scaling file while the attention backend's attn object has no 'k_scale' attribute (backends without fp8 kv-cache support).","commonSituations":"Using --kv-cache-dtype fp8_e4m3 with a scaling JSON on an attention backend that doesn't implement k_scale/v_scale; mixing quantized-kv artifacts with a different backend.","solutions":["Use an attention backend that supports kv cache scaling (e.g. FlashAttention/triton with fp8 kv) so attn exposes k_scale","Regenerate or drop the kv_scale JSON if you don't intend fp8 kv cache","Check that layer indices in the scale file match the model"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"if not hasattr(layer.self_attn.attn, 'k_scale'):\n    raise SystemExit('current attention backend lacks kv-scale support; drop kv scale file or switch backend')","typeGuard":"def backend_supports_kv_scale(model) -> bool:\n    return hasattr(model.layers[0].self_attn.attn, 'k_scale')","tryCatchPattern":"try:\n    model.load_kv_cache_scales(path)\nexcept RuntimeError as e:\n    if 'KV cache scaling' in str(e):\n        logger.warning('skipping kv scales: backend unsupported')\n    else:\n        raise","preventionTips":["Only pass kv scale JSON when using an fp8-capable attention backend","Confirm --attention-backend matches the one used when scales were calibrated"],"tags":["kv-cache","fp8","quantization","attention-backend"],"backgroundTag":"kv-cache-scale-missing","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}