{"record":{"id":"6d77d2783da9c800","repo":"sgl-project/sglang","slug":"kimi-k3-mla-k-projection-must-remain-gguf-q4-0","errorCode":null,"errorMessage":"Kimi-K3 MLA K projection must remain GGUF Q4_0","messagePattern":"Kimi-K3 MLA K projection must remain GGUF Q4_0","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/kimi_k3.py","lineNumber":3152,"sourceCode":"        self.post_load_weights()\n        return loaded_params\n\n    def post_load_weights(self):\n        # Also invoked by loader post-load hooks (DummyModelLoader,\n        # ShardedStateLoader, remote-instance flows -- none of which call\n        # load_weights), so e.g. dummy-weight benchmarks get w_kc/w_vc and\n        # the fused buffers too. Same pattern as deepseek_v4.\n        # Post-load: absorb kv_b_proj into w_kc and w_vc for MLA layers\n        for layer_id in self.config.full_attention_layer_ids:\n            if layer_id >= len(self.model.layers):\n                continue  # truncated config (e.g. num_hidden_layers override)\n            layer = self.model.layers[layer_id]\n            if isinstance(layer, PPMissingLayer):\n                continue\n            self_attn = layer.self_attn\n            if getattr(self_attn, \"_kimi_split_gguf_kv_b\", False):\n                if int(self_attn.k_b_qweight_type.weight_type) != 2:\n                    raise ValueError(\"Kimi-K3 MLA K projection must remain GGUF Q4_0\")\n                if int(self_attn.v_b_qweight_type.weight_type) != 10:\n                    raise ValueError(\"Kimi-K3 MLA V projection must remain GGUF Q2_K\")\n                self_attn.use_deep_gemm_bmm = False\n                continue\n            kv_b_weight = _get_k3_dense_weight(self_attn.kv_b_proj)\n            w_kc, w_vc = kv_b_weight.unflatten(\n                0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim)\n            ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1)\n            self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2)\n            self_attn.w_vc = w_vc.contiguous().transpose(1, 2)\n            if hasattr(self_attn.kv_b_proj, \"weight_scale\"):\n                self_attn.w_scale = self_attn.kv_b_proj.weight_scale\n\n        # Post-load: precompute the attn-res combined score weights BEFORE\n        # cuda graph capture (a lazy first call inside get_cw would bake the\n        # multiply into every captured graph replay otherwise). Warm both\n        # dtypes: the fast kernel consumes bf16, the triton fallback fp32.\n        def _warm_cw(proj, norm):","sourceCodeStart":3134,"sourceCodeEnd":3170,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_k3.py#L3134-L3170","documentation":"For Kimi K3 checkpoints with a split GGUF kv_b_proj (flag _kimi_split_gguf_kv_b), post_load_weights asserts the K part of the split projection stays GGUF type 2 (Q4_0). If any re-quantization, conversion, or weight surgery changed the K projection's ggml weight type, this ValueError fires, because the fast MLA path assumes Q4_0 K weights.","triggerScenarios":"Loading a Kimi K3 GGUF checkpoint where the split kv_b K projection's k_b_qweight_type.weight_type != 2 — e.g. the checkpoint was re-quantized with a different K quant (Q8_0/Q4_K) or the split tooling wrote the wrong type field.","commonSituations":"Re-quantizing Kimi K3 GGUF weights with llama.cpp at non-original settings; using a converted/mixed GGUF where kv_b was split with mismatched types; corrupted type metadata in the GGUF file.","solutions":["Re-obtain the original Kimi K3 GGUF checkpoint (K projection quantized Q4_0) instead of a re-quantized one.","Re-run the KV split tooling so k_b stays Q4_0 (weight_type==2) and v_b stays Q2_K (10).","If you must use a different K quant, extend post_load_weights to handle that weight_type and bypass use_deep_gemm_bmm appropriately."],"exampleFix":"# before: re-quantized with Q8_0 K -> weight_type 8 -> ValueError\n# after: use original checkpoint\n# k_b Q4_0 (weight_type == 2), v_b Q2_K (weight_type == 10)","handlingStrategy":"validation","validationCode":"from gguf import GGUFReader\nr = GGUFReader(ckpt)\nfor t in r.tensors:\n    if t.name.endswith(\"k_b.weight\"):\n        assert int.from_bytes(t.tensor_type_bytes, \"little\") == 2, \"K must be Q4_0\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never re-quantize Kimi K3 GGUF kv_b tensors; keep the original checkpoint.","Verify ggml weight types after any GGUF merge/split operation."],"tags":["gguf","quantization","weight-loading","kimi","mla"],"backgroundTag":"checkpoint-quantization-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}