sgl-project/sglang · critical · ValueError

MiMoV2ForCausalLM requires effective attention TP size

Error message

MiMoV2ForCausalLM requires effective attention TP size {expected_attn_tp_size} because its fused qkv_proj weights are TP={expected_attn_tp_size}-interleaved; got {effective_attn_tp_size} (tp_size={cfg.tp_size}, dp_size={cfg.dp_size}, enable_dp_attention={view.enable_dp_attention}, attn_cp_size={view.attn_cp_size}).

What it means

MiMoV2ForCausalLM ships its fused qkv_proj weights pre-sharded/interleaved for a fixed attention tensor-parallel degree. SGLang computes the effective attention TP size as tp_size // attn_dp_size // attn_cp_size and rejects any value that does not evenly divide the weight's expected interleaving factor, since loading would produce corrupt attention projections.

Solutions

  1. Reshape the topology so effective attention TP equals the expected factor: drop --enable-dp-attention, reduce --dp-size, or raise --tp-size so tp_size // attn_dp_size // attn_cp_size divides the expected size
  2. Check the model config/weights for the qkv TP interleaving factor and match tp_size to it
  3. If DP attention is required, increase tp_size so per-rank attention TP stays at the expected degree

Example fix

# before
python -m sglang.launch_server --model MiMoV2 --tp-size 8 --dp-size 8 --enable-dp-attention
# after (effective attn TP = 8 // 8 = 1 → invalid; keep attn TP aligned)
python -m sglang.launch_server --model MiMoV2 --tp-size 8
Defensive patterns

Strategy: validation

Validate before calling

tp, dp, attn_cp = cfg.tp_size, cfg.dp_size, (cfg.attn_cp_size or 1)
attn_dp = dp if cfg.enable_dp_attention else 1
effective = tp // attn_dp // attn_cp
expected = EXPECTED_MIMO_V2_ATTN_TP  # from model config / fused qkv interleaving
assert expected is None or expected % effective == 0, (
    f"effective attn TP {effective} invalid, need divisor of {expected}")

Try / catch

try:
    server_args = ServerArgs.from_cli_args(cli)
except ValueError as e:
    if "effective attention TP" in str(e):
        raise SystemExit(f"TP/DP topology mismatch for MiMoV2: {e}")
    raise

Prevention

When it happens

Trigger: Launching MiMoV2 with --tp-size / --dp_size / --enable-dp-attention / attn_cp_size combinations such that tp_size // attn_dp_size // attn_cp_size is not a divisor of expected_attn_tp_size (e.g. dp attention collapsing per-rank attention TP below the fused qkv interleaving).

Common situations: Enabling DP attention or context parallelism on a topology the checkpoint wasn't sharded for; mixing dp_size with tp_size so effective per-rank attention TP shrinks; using a MiMoV2 checkpoint with a different fused-qkv interleaving than a prior launch config.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/319d761cedf5096a. Report an issue: GitHub.

Appendix: source

Thrown at python/sglang/srt/server_args.py:6136

                assert (
                    self._resolved().ep_size == 1
                ), "Triton kernel MoE is only supported when ep_size == 1"

        elif model_arch in ("MiMoV2ForCausalLM", "MiMoV2FlashForCausalLM"):
            if model_arch == "MiMoV2ForCausalLM" and not cfg.encoder_only:
                expected_attn_tp_size = get_mimo_v2_fused_qkv_expected_tp_size(
                    hf_config
                )
                view = self._resolved()
                attn_dp_size = cfg.dp_size if view.enable_dp_attention else 1
                effective_attn_tp_size = (
                    cfg.tp_size // attn_dp_size // view.attn_cp_size
                )
                if (
                    expected_attn_tp_size is not None
                    and expected_attn_tp_size % effective_attn_tp_size != 0
                ):
                    raise ValueError(
                        "MiMoV2ForCausalLM requires effective attention TP "
                        f"size {expected_attn_tp_size} because its fused "
                        "qkv_proj weights are "
                        f"TP={expected_attn_tp_size}-interleaved; got "
                        f"{effective_attn_tp_size} "
                        f"(tp_size={cfg.tp_size}, dp_size={cfg.dp_size}, "
                        f"enable_dp_attention={view.enable_dp_attention}, "
                        f"attn_cp_size={view.attn_cp_size}). "
                        "Set --tp, --dp, --enable-dp-attention, and "
                        "--attention-context-parallel-size so the effective "
                        f"attention TP size is {expected_attn_tp_size}."
                    )

            # enable_multi_layer_eagle for EAGLE moved to the override registry
            # (arg_groups/overrides.py: _mimo_v2_overrides).

            # MiMoV2 hierarchical cache runs on the unified radix tree, which
            # is the default tree cache now. MiMoV2 has head_dim != v_head_dim,

View on GitHub (pinned to 0132848349)