{"record":{"id":"9d105f45e93c01d9","repo":"sgl-project/sglang","slug":"eagle3-mla-draft-post-load-weights-only-supports-f","errorCode":null,"errorMessage":"Eagle3 MLA draft post_load_weights only supports float dtypes, got {w.dtype}","messagePattern":"Eagle3 MLA draft post_load_weights only supports float dtypes, got (.+?)","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/models/kimi_k25_eagle3.py","lineNumber":510,"sourceCode":"                logger.warning(\"Eagle3 MLA: skipping unexpected weight %s\", name)\n                continue\n            param = params_dict[mapped_name]\n            weight_loader = getattr(param, \"weight_loader\", default_weight_loader)\n            weight_loader(param, loaded_weight)\n\n        self.post_load_weights()\n\n    def post_load_weights(self) -> None:\n        \"\"\"Split kv_b_proj into w_kc / w_vc tensors used by MLA absorb_core.\n\n        DeepseekV2 normally does this in DeepseekV2WeightLoaderMixin.post_load_weights;\n        we re-implement the bf16 fast-path directly here to keep the eagle3 draft\n        path independent of the full DeepseekV2 weight loader.\n        \"\"\"\n        self_attn = self.model.midlayer.self_attn\n        w = self_attn.kv_b_proj.weight\n        if w.dtype not in (torch.bfloat16, torch.float16, torch.float32):\n            raise NotImplementedError(\n                f\"Eagle3 MLA draft post_load_weights only supports float dtypes, got {w.dtype}\"\n            )\n        w_kc, w_vc = w.unflatten(\n            0, (-1, self_attn.qk_nope_head_dim + self_attn.v_head_dim)\n        ).split([self_attn.qk_nope_head_dim, self_attn.v_head_dim], dim=1)\n        self_attn.w_kc = w_kc.transpose(1, 2).contiguous().transpose(1, 2)\n        self_attn.w_vc = w_vc.contiguous().transpose(1, 2)\n\n\nEntryClass = [Eagle3DeepseekV2ForCausalLM]\n","sourceCodeStart":492,"sourceCodeEnd":521,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/models/kimi_k25_eagle3.py#L492-L521","documentation":"Eagle3MLA.post_load_weights re-implements the DeepseekV2 bf16 fast path for splitting kv_b_proj into w_kc/w_vc, and it requires the weight to be a float dtype (bf16/fp16/fp32) because it calls .unflatten/.split on the raw tensor. Quantized or integer-packed kv_b_proj weights (GGUF/awq/marlin tensors) hit this NotImplementedError.","triggerScenarios":"Loading an EAGLE3 MLA draft checkpoint whose midlayer self_attn.kv_b_proj.weight is not bfloat16/float16/float32 — e.g. a GGUF-quantized or otherwise integer-dtype kv_b_proj — during load_weights -> post_load_weights.","commonSituations":"Loading a quantized (GGUF/awq/GPTQ) Kimi K2.5 EAGLE3 draft; mixing a quantized target-model loader with the eagle3 draft path; exotic torch versions/serialization that round-trip weights as uint8/int8.","solutions":["Use an unquantized (bf16/fp16) EAGLE3 draft checkpoint for kv_b_proj.","Pre-cast kv_b_proj to torch.bfloat16 before load if you authored the checkpoint.","Extend post_load_weights to dequantize integer dtypes before the split if quantized drafts must be supported."],"exampleFix":"// before\n# kv_b_proj.weight loaded as GGUF/quantized dtype -> NotImplementedError\n\n// after\n# convert to bf16 when preparing the draft checkpoint\nw = w.to(torch.bfloat16)","handlingStrategy":"validation","validationCode":"import torch, safetensors\nw = safetensors.torch.load_file(draft_path + \"/kv_b_proj.safetensors\")[\"kv_b_proj.weight\"]\nassert w.dtype in (torch.bfloat16, torch.float16, torch.float32), f\"bad dtype {w.dtype}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use bf16/fp16 EAGLE3 draft checkpoints; avoid quantized draft weights.","Inspect checkpoint dtypes with safetensors before loading."],"tags":["eagle3","weight-loading","dtype","quantization","speculative-decoding"],"backgroundTag":"unsupported-weight-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}