hiyouga/LlamaFactory · error · ValueError

`moe_token_dispatcher_type` must be 'allgather', 'alltoall',

Error message

`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.

What it means

Raised in MegatronBridgeArguments.__post_init__ (megatron_bridge_args.py:176) when moe_token_dispatcher_type is set to a string outside {'allgather', 'alltoall', 'flex'}. The dispatcher moves tokens between attention and expert groups in mixture-of-experts models; these three are the Megatron-core types this build bridges. Validation is a case-sensitive membership check that runs at config-parse time.

Source

Thrown at src/llamafactory/hparams/megatron_bridge_args.py:176

            raise ValueError("`expert_model_parallel_size` must be >= 1.")
        if self.context_parallel_size < 1:
            raise ValueError("`context_parallel_size` must be >= 1.")
        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:
            raise ValueError("`virtual_pipeline_model_parallel_size` must be >= 1 when set.")
        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:
            raise ValueError("`sequence_parallel` requires `tensor_model_parallel_size` > 1.")
        if self.recompute_granularity is not None and self.recompute_granularity not in ("full", "selective"):
            raise ValueError("`recompute_granularity` must be 'full' or 'selective'.")
        if self.recompute_method is not None and self.recompute_method not in ("uniform", "block"):
            raise ValueError("`recompute_method` must be 'uniform' or 'block'.")
        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:
            raise ValueError("`recompute_num_layers` must be >= 1 when set.")
        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (
            "allgather",
            "alltoall",
            "flex",
        ):
            raise ValueError("`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.")

        if isinstance(self.extra_config, str):
            config_str = self.extra_config.strip()
            if config_str.startswith("{"):
                self.extra_config = _convert_str_dict(json.loads(config_str))
            else:
                self.extra_config = config_str

    def load_extra_config(self) -> dict:
        if self.extra_config is None:
            return {}
        if isinstance(self.extra_config, dict):
            return self.extra_config
        if not os.path.isfile(self.extra_config):
            raise ValueError(f"`extra_config` file not found: {self.extra_config}")
        with open(self.extra_config, encoding="utf-8") as f:
            return json.load(f)

View on GitHub (pinned to f28afaf635)

Solutions

1. Use one of exactly: 'allgather', 'alltoall', 'flex'
2. Check the Megatron-core version pinned by this LlamaFactory release for newly supported dispatchers before using exotic names
3. Remove the key to use the default dispatcher if you have no specific throughput reason to tune it

Example fix

# before
moe_token_dispatcher_type: standard

# after
moe_token_dispatcher_type: alltoall

if cfg.get('moe_token_dispatcher_type') not in (None, 'allgather', 'alltoall', 'flex'):
    raise SystemExit(f"unsupported dispatcher: {cfg['moe_token_dispatcher_type']!r}")

def is_supported_dispatcher(v: str | None) -> bool:
    return v in (None, 'allgather', 'alltoall', 'flex')

When it happens

Trigger: Setting moe_token_dispatcher_type: allgather_ltc, 'AllGather', 'a2a', or a Megatron version's newer type not yet supported here; enabling MoE via extra_config on a non-MoE checkpoint with an experimental dispatcher name.

Common situations: Porting configs from newer Megatron-core (which has more dispatcher variants) into an older LlamaFactory; casing/typo mistakes; copying DeepSpeed or vLLM MoE option names (e.g. 'standard') into the Megatron field.

Related errors

• `expert_model_parallel_size` must be >= 1.
• `virtual_pipeline_model_parallel_size` must be >= 1 when set
• `sequence_parallel` requires `tensor_model_parallel_size` >
• `recompute_granularity` must be 'full' or 'selective'.
• `recompute_method` must be 'uniform' or 'block'.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/19d68d15b4b6fb30. Report an issue: GitHub.