{"record":{"id":"96a42df3bcaea2a4","repo":"sgl-project/sglang","slug":"self-attention-has-no-kv-cache-scaling-factor-attr","errorCode":null,"errorMessage":"Self attention has no KV cache scaling factor attribute!","messagePattern":"Self attention has no KV cache scaling factor attribute!","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/apertus.py","lineNumber":396,"sourceCode":"    # make sure to leave KV cache scale factors in a known good (dummy) state\n    def load_kv_cache_scales(self, quantization_param_path: str) -> None:\n        tp_size = get_parallel().tp_size\n        tp_rank = get_parallel().tp_rank\n        for layer_idx, scaling_factor in kv_cache_scales_loader(\n            quantization_param_path,\n            tp_rank,\n            tp_size,\n            self.config.num_hidden_layers,\n            self.config.__class__.model_type,\n        ):\n            if not isinstance(self.layers[layer_idx], nn.Identity):\n                layer_self_attn = self.layers[layer_idx].self_attn\n\n            if hasattr(layer_self_attn.attn, \"k_scale\"):\n                layer_self_attn.attn.k_scale = scaling_factor\n                layer_self_attn.attn.v_scale = scaling_factor\n            else:\n                raise RuntimeError(\n                    \"Self attention has no KV cache scaling \" \"factor attribute!\"\n                )\n\n\nclass ApertusForCausalLM(nn.Module):\n    # LoRA specific attributes\n    embedding_modules = {\n        \"embed_tokens\": \"input_embeddings\",\n        \"lm_head\": \"output_embeddings\",\n    }\n    embedding_padding_modules = [\"lm_head\"]\n    # BitandBytes specific attributes\n    default_bitsandbytes_target_modules = [\n        \".down_proj.\",\n        \".up_proj.\",\n        \".q_proj.\",\n        \".k_proj.\",\n        \".v_proj.\",","sourceCodeStart":378,"sourceCodeEnd":414,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/apertus.py#L378-L414","documentation":"Apertus's load_kv_cache_scales expects each attention module to expose k_scale (set when FP8 KV cache is enabled). If attn has no k_scale attribute, the calibration scale cannot be applied and a RuntimeError is raised.","triggerScenarios":"Calling load_kv_cache_scales (e.g. loading kv_cache_scales.json for FP8 KV cache quantization) when the attention backend was initialized without FP8 KV scaling support.","commonSituations":"Mixing --kv-cache-dtype fp8 with quantization formats/backends that don't populate k_scale/v_scale; loading scale files on a non-FP8 deployment.","solutions":["Ensure the model is launched with FP8 KV cache enabled so k_scale/v_scale attributes exist","Verify the quant_config / --kv-cache-dtype actually initializes attention with scaling attributes before loading scales.json"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"assert all(hasattr(l.self_attn.attn, \"k_scale\") for l in model.layers), \"launch with FP8 KV cache before loading scales\"","typeGuard":null,"tryCatchPattern":"try: model.load_kv_cache_scales(...)\nexcept RuntimeError as e: log.warning(f\"skipping kv scales: {e}\")","preventionTips":["Only load kv_cache_scales.json when running FP8 KV cache","Gate scale loading on server_args.kv_cache_dtype"],"tags":["fp8","kv-cache","quantization","apertus"],"backgroundTag":"fp8-kv-cache-scale-missing","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}