sgl-project/sglang · error · ValueError

Kimi-K3 MLA V projection must remain GGUF Q2_K

Error message

Kimi-K3 MLA V projection must remain GGUF Q2_K

What it means

Companion to the Q4_0 K check: for split-GGUF Kimi K3 MLA layers, post_load_weights also asserts the V projection stays GGUF type 10 (Q2_K). A mismatch raises ValueError because the optimized MLA BMM path is only correct for the original Q2_K V layout.

Source

Thrown at python/sglang/srt/models/kimi_k3.py:3154

    def post_load_weights(self):
        # Also invoked by loader post-load hooks (DummyModelLoader,
        # ShardedStateLoader, remote-instance flows -- none of which call
        # load_weights), so e.g. dummy-weight benchmarks get w_kc/w_vc and
        # the fused buffers too. Same pattern as deepseek_v4.
        # Post-load: absorb kv_b_proj into w_kc and w_vc for MLA layers
        for layer_id in self.config.full_attention_layer_ids:
            if layer_id >= len(self.model.layers):
                continue  # truncated config (e.g. num_hidden_layers override)
            layer = self.model.layers[layer_id]
            if isinstance(layer, PPMissingLayer):
                continue
            self_attn = layer.self_attn
            if getattr(self_attn, "_kimi_split_gguf_kv_b", False):
                if int(self_attn.k_b_qweight_type.weight_type) != 2:
                    raise ValueError("Kimi-K3 MLA K projection must remain GGUF Q4_0")
                if int(self_attn.v_b_qweight_type.weight_type) != 10:
                    raise ValueError("Kimi-K3 MLA V projection must remain GGUF Q2_K")
                self_attn.use_deep_gemm_bmm = False
                continue
            kv_b_weight = _get_k3_dense_weight(self_attn.kv_b_proj)
            w_kc, w_vc = kv_b_weight.unflatten(
                0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim)
            ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1)
            self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2)
            self_attn.w_vc = w_vc.contiguous().transpose(1, 2)
            if hasattr(self_attn.kv_b_proj, "weight_scale"):
                self_attn.w_scale = self_attn.kv_b_proj.weight_scale

        # Post-load: precompute the attn-res combined score weights BEFORE
        # cuda graph capture (a lazy first call inside get_cw would bake the
        # multiply into every captured graph replay otherwise). Warm both
        # dtypes: the fast kernel consumes bf16, the triton fallback fp32.
        def _warm_cw(proj, norm):
            get_cw(proj, norm, dtype=torch.bfloat16)
            get_cw(proj, norm)

View on GitHub (pinned to 0132848349)

Solutions

1. Use the original Kimi K3 GGUF checkpoint whose V projection is Q2_K (weight_type 10).
2. Regenerate the split kv_b tensors with the correct types: k_b Q4_0 (2) and v_b Q2_K (10).
3. Patch post_load_weights to support the alternative V quant type if you control the serving stack.

Example fix

# before: v_b re-quantized as Q4_K (weight_type 12) -> ValueError
# after: v_b Q2_K (weight_type == 10), k_b Q4_0 (weight_type == 2)

for t in GGUFReader(ckpt).tensors:
    if t.name.endswith("v_b.weight"):
        assert gguf_type(t) == 10, "V must be Q2_K"

When it happens

Trigger: Loading a split GGUF Kimi K3 checkpoint where v_b_qweight_type.weight_type != 10 — V projection re-quantized away from Q2_K during GGUF conversion or mixing checkpoints from different quant recipes.

Common situations: Same as 5428: re-quantization with llama.cpp, mixed-recipe GGUF merges, or conversion tools writing wrong ggml type ids for the split kv_b tensors.

Related errors

• Kimi-K3 MLA K projection must remain GGUF Q4_0
• {name} must be a torch.Tensor
• {name} must be a CUDA tensor
• {name} must be on q's device {device}, got {scale.device}
• {name} must be float32, got {scale.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6e997316b6404564. Report an issue: GitHub.