{"record":{"id":"79bbe224e95460dd","repo":"sgl-project/sglang","slug":"self-attention-has-no-kv-cache-scaling-factor-attr-79bbe2","errorCode":null,"errorMessage":"Self attention has no KV cache scaling factor attribute!","messagePattern":"Self attention has no KV cache scaling factor attribute!","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/arcee.py","lineNumber":365,"sourceCode":"\n    def load_kv_cache_scales(self, quantization_param_path: str) -> None:\n        tp_size = get_parallel().tp_size\n        tp_rank = get_parallel().tp_rank\n        for layer_idx, scaling_factor in kv_cache_scales_loader(\n            quantization_param_path,\n            tp_rank,\n            tp_size,\n            self.config.num_hidden_layers,\n            self.config.__class__.model_type,\n        ):\n            if not isinstance(self.layers[layer_idx], nn.Identity):\n                layer_self_attn = self.layers[layer_idx].self_attn\n\n            if hasattr(layer_self_attn.attn, \"k_scale\"):\n                layer_self_attn.attn.k_scale = scaling_factor\n                layer_self_attn.attn.v_scale = scaling_factor\n            else:\n                raise RuntimeError(\n                    \"Self attention has no KV cache scaling factor attribute!\"\n                )\n\n\nclass ArceeForCausalLM(nn.Module):\n    # BitandBytes specific attributes\n    default_bitsandbytes_target_modules = [\n        # Note: gate_proj is removed compared to Llama\n        \".down_proj.\",\n        \".up_proj.\",\n        \".q_proj.\",\n        \".k_proj.\",\n        \".v_proj.\",\n        \".o_proj.\",\n    ]\n    # in TP, these weights are partitioned along the column dimension (dim=-1)\n    column_parallel_weights_modules = [\".down_proj.\", \".o_proj.\"]\n    bitsandbytes_stacked_params_mapping = {","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/arcee.py#L347-L383","documentation":"Arcee's load_kv_cache_scales requires the attention module to have a k_scale attribute, which exists only when the attention layer was built with KV-cache scaling (FP8). Its absence means the scales file cannot be applied.","triggerScenarios":"Calling load_kv_cache_scales with a kv_cache_scales.json while the server was not launched with FP8 KV cache quantization.","commonSituations":"Applying FP8 KV calibration files to an FP16/bf16 deployment; quantization format mismatch after a config change.","solutions":["Launch with FP8 KV cache (--kv-cache-dtype fp8_e4m3 / matching quant config) so k_scale exists","Skip loading kv_cache_scales.json if you are not quantizing the KV cache"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"assert all(hasattr(l.self_attn.attn, \"k_scale\") for l in model.layers)","typeGuard":null,"tryCatchPattern":"try: model.load_kv_cache_scales(...)\nexcept RuntimeError: pass  # non-FP8 deployment","preventionTips":["Match scale files to the deployed KV dtype"],"tags":["fp8","kv-cache","quantization","arcee"],"backgroundTag":"fp8-kv-cache-scale-missing","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}