sgl-project/sglang · error · ValueError

Kimi-K3 MLA K projection must remain GGUF Q4_0

Error message

Kimi-K3 MLA K projection must remain GGUF Q4_0

What it means

For Kimi K3 checkpoints with a split GGUF kv_b_proj (flag _kimi_split_gguf_kv_b), post_load_weights asserts the K part of the split projection stays GGUF type 2 (Q4_0). If any re-quantization, conversion, or weight surgery changed the K projection's ggml weight type, this ValueError fires, because the fast MLA path assumes Q4_0 K weights.

Source

Thrown at python/sglang/srt/models/kimi_k3.py:3152

        self.post_load_weights()
        return loaded_params

    def post_load_weights(self):
        # Also invoked by loader post-load hooks (DummyModelLoader,
        # ShardedStateLoader, remote-instance flows -- none of which call
        # load_weights), so e.g. dummy-weight benchmarks get w_kc/w_vc and
        # the fused buffers too. Same pattern as deepseek_v4.
        # Post-load: absorb kv_b_proj into w_kc and w_vc for MLA layers
        for layer_id in self.config.full_attention_layer_ids:
            if layer_id >= len(self.model.layers):
                continue  # truncated config (e.g. num_hidden_layers override)
            layer = self.model.layers[layer_id]
            if isinstance(layer, PPMissingLayer):
                continue
            self_attn = layer.self_attn
            if getattr(self_attn, "_kimi_split_gguf_kv_b", False):
                if int(self_attn.k_b_qweight_type.weight_type) != 2:
                    raise ValueError("Kimi-K3 MLA K projection must remain GGUF Q4_0")
                if int(self_attn.v_b_qweight_type.weight_type) != 10:
                    raise ValueError("Kimi-K3 MLA V projection must remain GGUF Q2_K")
                self_attn.use_deep_gemm_bmm = False
                continue
            kv_b_weight = _get_k3_dense_weight(self_attn.kv_b_proj)
            w_kc, w_vc = kv_b_weight.unflatten(
                0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim)
            ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1)
            self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2)
            self_attn.w_vc = w_vc.contiguous().transpose(1, 2)
            if hasattr(self_attn.kv_b_proj, "weight_scale"):
                self_attn.w_scale = self_attn.kv_b_proj.weight_scale

        # Post-load: precompute the attn-res combined score weights BEFORE
        # cuda graph capture (a lazy first call inside get_cw would bake the
        # multiply into every captured graph replay otherwise). Warm both
        # dtypes: the fast kernel consumes bf16, the triton fallback fp32.
        def _warm_cw(proj, norm):

View on GitHub (pinned to 0132848349)

Solutions

1. Re-obtain the original Kimi K3 GGUF checkpoint (K projection quantized Q4_0) instead of a re-quantized one.
2. Re-run the KV split tooling so k_b stays Q4_0 (weight_type==2) and v_b stays Q2_K (10).
3. If you must use a different K quant, extend post_load_weights to handle that weight_type and bypass use_deep_gemm_bmm appropriately.

Example fix

# before: re-quantized with Q8_0 K -> weight_type 8 -> ValueError
# after: use original checkpoint
# k_b Q4_0 (weight_type == 2), v_b Q2_K (weight_type == 10)

from gguf import GGUFReader
r = GGUFReader(ckpt)
for t in r.tensors:
    if t.name.endswith("k_b.weight"):
        assert int.from_bytes(t.tensor_type_bytes, "little") == 2, "K must be Q4_0"

When it happens

Trigger: Loading a Kimi K3 GGUF checkpoint where the split kv_b K projection's k_b_qweight_type.weight_type != 2 — e.g. the checkpoint was re-quantized with a different K quant (Q8_0/Q4_K) or the split tooling wrote the wrong type field.

Common situations: Re-quantizing Kimi K3 GGUF weights with llama.cpp at non-original settings; using a converted/mixed GGUF where kv_b was split with mismatched types; corrupted type metadata in the GGUF file.

Related errors

• Kimi-K3 MLA V projection must remain GGUF Q2_K
• {name} must be a torch.Tensor
• {name} must be a CUDA tensor
• {name} must be on q's device {device}, got {scale.device}
• {name} must be float32, got {scale.dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6d77d2783da9c800. Report an issue: GitHub.