sgl-project/sglang · error · ValueError

Eagle3 MLA layer requires q_lora_rank in the draft config

Error message

Eagle3 MLA layer requires q_lora_rank in the draft config

What it means

The EAGLE3 draft layer for Kimi K2.5 rebuilds the MLA fused QKV-down projection to accept 2*hidden_size inputs (concatenated embed + target hidden states). This requires a q_lora_rank in the draft config because the projection output size is derived from it. Without it, DeepseekV2AttentionMLA built an MQA-style projection and this ValueError fires at construction.

Source

Thrown at python/sglang/srt/models/kimi_k25_eagle3.py:114

            v_head_dim=config.v_head_dim,
            q_lora_rank=config.q_lora_rank,
            kv_lora_rank=config.kv_lora_rank,
            rope_theta=rope_theta,
            rope_scaling=rope_scaling,
            max_position_embeddings=max_position_embeddings,
            quant_config=quant_config,
            layer_id=layer_id,
            reduce_results=True,
            prefix=add_prefix("self_attn", prefix),
        )

        # EAGLE3 doubles MLA's QKV-down input by concatenating
        # input_layernorm(embed) and hidden_norm(target_hidden) along the
        # feature dim. Replace the projection that DeepseekV2AttentionMLA
        # built for a single-hidden input.
        attn = self.self_attn
        if attn.q_lora_rank is None:
            raise ValueError(
                "Eagle3 MLA layer requires q_lora_rank in the draft config"
            )
        attn.fused_qkv_a_proj_with_mqa = ReplicatedLinear(
            2 * config.hidden_size,
            attn.q_lora_rank + attn.kv_lora_rank + attn.qk_rope_head_dim,
            bias=False,
            quant_config=quant_config,
            prefix=add_prefix("self_attn.fused_qkv_a_proj_with_mqa", prefix),
        )
        # Recompute fused-proj-dependent flags so they reflect the new input dim.
        attn.has_fused_proj = True
        attn._use_min_latency_fused_a_gemm = False
        quant_method = getattr(attn.fused_qkv_a_proj_with_mqa, "quant_method", None)
        attn.is_packed_weight = (
            quant_method is not None
            and hasattr(quant_method, "quant_config")
            and quant_method.quant_config is not None
            and quant_method.quant_config.get_name()

View on GitHub (pinned to 0132848349)

Solutions

1. Set q_lora_rank (a positive int, e.g. 1536) in the draft model's config.json when using the EAGLE3 MLA draft.
2. Use a draft checkpoint that was built for EAGLE3 (has q_lora_rank) rather than the plain MTP head.
3. Verify the draft config loads with q_lora_rank != None before launching: json.load(config)['q_lora_rank'] is not None.

Example fix

// before (draft config.json)
{"hidden_size": 7168, "kv_lora_rank": 512, ...}

// after
{"hidden_size": 7168, "q_lora_rank": 1536, "kv_lora_rank": 512, ...}

import json
cfg = json.load(open(draft_config_path))
if cfg.get("q_lora_rank") is None:
    raise ValueError("EAGLE3 MLA draft requires q_lora_rank in draft config")

When it happens

Trigger: Instantiating Eagle3MLADecoderLayer (Eagle3MLA draft model __init__) with a draft config where config.q_lora_rank is None — i.e. a pure-MLA (MQA) draft model without Q low-rank compression, such as a K2.5 MTP head used as the EAGLE3 draft.

Common situations: Pointing --speculative-draft-model-path at a Kimi MTP draft checkpoint whose config lacks q_lora_rank; hand-written EAGLE3 draft configs missing the q_lora_rank field; mixing a DeepSeek-V2-style MLA config with an EAGLE3 wrapper that assumes Q-LoRA is present.

Related errors

• EAGLE3 currently only supports 1 layer
• Invalid fused KV rotary/head dim pair: rotary_dim={rotary_di
• num_kv_heads mismatch across layers for fused KV path: expec
• TensorRT-LLM MLA backend only supports kv-cache-dtype of fp8
• tokenspeed_mla backend requires kv-cache-dtype=fp8_e4m3, got

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/1ef9b77d5d495def. Report an issue: GitHub.