{"record":{"id":"6e997316b6404564","repo":"sgl-project/sglang","slug":"kimi-k3-mla-v-projection-must-remain-gguf-q2-k","errorCode":null,"errorMessage":"Kimi-K3 MLA V projection must remain GGUF Q2_K","messagePattern":"Kimi-K3 MLA V projection must remain GGUF Q2_K","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/kimi_k3.py","lineNumber":3154,"sourceCode":"\n    def post_load_weights(self):\n        # Also invoked by loader post-load hooks (DummyModelLoader,\n        # ShardedStateLoader, remote-instance flows -- none of which call\n        # load_weights), so e.g. dummy-weight benchmarks get w_kc/w_vc and\n        # the fused buffers too. Same pattern as deepseek_v4.\n        # Post-load: absorb kv_b_proj into w_kc and w_vc for MLA layers\n        for layer_id in self.config.full_attention_layer_ids:\n            if layer_id >= len(self.model.layers):\n                continue  # truncated config (e.g. num_hidden_layers override)\n            layer = self.model.layers[layer_id]\n            if isinstance(layer, PPMissingLayer):\n                continue\n            self_attn = layer.self_attn\n            if getattr(self_attn, \"_kimi_split_gguf_kv_b\", False):\n                if int(self_attn.k_b_qweight_type.weight_type) != 2:\n                    raise ValueError(\"Kimi-K3 MLA K projection must remain GGUF Q4_0\")\n                if int(self_attn.v_b_qweight_type.weight_type) != 10:\n                    raise ValueError(\"Kimi-K3 MLA V projection must remain GGUF Q2_K\")\n                self_attn.use_deep_gemm_bmm = False\n                continue\n            kv_b_weight = _get_k3_dense_weight(self_attn.kv_b_proj)\n            w_kc, w_vc = kv_b_weight.unflatten(\n                0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim)\n            ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1)\n            self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2)\n            self_attn.w_vc = w_vc.contiguous().transpose(1, 2)\n            if hasattr(self_attn.kv_b_proj, \"weight_scale\"):\n                self_attn.w_scale = self_attn.kv_b_proj.weight_scale\n\n        # Post-load: precompute the attn-res combined score weights BEFORE\n        # cuda graph capture (a lazy first call inside get_cw would bake the\n        # multiply into every captured graph replay otherwise). Warm both\n        # dtypes: the fast kernel consumes bf16, the triton fallback fp32.\n        def _warm_cw(proj, norm):\n            get_cw(proj, norm, dtype=torch.bfloat16)\n            get_cw(proj, norm)","sourceCodeStart":3136,"sourceCodeEnd":3172,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_k3.py#L3136-L3172","documentation":"Companion to the Q4_0 K check: for split-GGUF Kimi K3 MLA layers, post_load_weights also asserts the V projection stays GGUF type 10 (Q2_K). A mismatch raises ValueError because the optimized MLA BMM path is only correct for the original Q2_K V layout.","triggerScenarios":"Loading a split GGUF Kimi K3 checkpoint where v_b_qweight_type.weight_type != 10 — V projection re-quantized away from Q2_K during GGUF conversion or mixing checkpoints from different quant recipes.","commonSituations":"Same as 5428: re-quantization with llama.cpp, mixed-recipe GGUF merges, or conversion tools writing wrong ggml type ids for the split kv_b tensors.","solutions":["Use the original Kimi K3 GGUF checkpoint whose V projection is Q2_K (weight_type 10).","Regenerate the split kv_b tensors with the correct types: k_b Q4_0 (2) and v_b Q2_K (10).","Patch post_load_weights to support the alternative V quant type if you control the serving stack."],"exampleFix":"# before: v_b re-quantized as Q4_K (weight_type 12) -> ValueError\n# after: v_b Q2_K (weight_type == 10), k_b Q4_0 (weight_type == 2)","handlingStrategy":"validation","validationCode":"for t in GGUFReader(ckpt).tensors:\n    if t.name.endswith(\"v_b.weight\"):\n        assert gguf_type(t) == 10, \"V must be Q2_K\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep the original Q2_K V quantization when preparing split GGUF checkpoints.","Validate both k_b (2) and v_b (10) types in a pre-load check script."],"tags":["gguf","quantization","weight-loading","kimi","mla"],"backgroundTag":"checkpoint-quantization-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}