{"record":{"id":"19d68d15b4b6fb30","repo":"hiyouga/LlamaFactory","slug":"moe-token-dispatcher-type-must-be-allgather","errorCode":null,"errorMessage":"`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.","messagePattern":"`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/megatron_bridge_args.py","lineNumber":176,"sourceCode":"            raise ValueError(\"`expert_model_parallel_size` must be >= 1.\")\n        if self.context_parallel_size < 1:\n            raise ValueError(\"`context_parallel_size` must be >= 1.\")\n        if self.virtual_pipeline_model_parallel_size is not None and self.virtual_pipeline_model_parallel_size < 1:\n            raise ValueError(\"`virtual_pipeline_model_parallel_size` must be >= 1 when set.\")\n        if self.sequence_parallel and self.tensor_model_parallel_size <= 1:\n            raise ValueError(\"`sequence_parallel` requires `tensor_model_parallel_size` > 1.\")\n        if self.recompute_granularity is not None and self.recompute_granularity not in (\"full\", \"selective\"):\n            raise ValueError(\"`recompute_granularity` must be 'full' or 'selective'.\")\n        if self.recompute_method is not None and self.recompute_method not in (\"uniform\", \"block\"):\n            raise ValueError(\"`recompute_method` must be 'uniform' or 'block'.\")\n        if self.recompute_num_layers is not None and self.recompute_num_layers < 1:\n            raise ValueError(\"`recompute_num_layers` must be >= 1 when set.\")\n        if self.moe_token_dispatcher_type is not None and self.moe_token_dispatcher_type not in (\n            \"allgather\",\n            \"alltoall\",\n            \"flex\",\n        ):\n            raise ValueError(\"`moe_token_dispatcher_type` must be 'allgather', 'alltoall', or 'flex'.\")\n\n        if isinstance(self.extra_config, str):\n            config_str = self.extra_config.strip()\n            if config_str.startswith(\"{\"):\n                self.extra_config = _convert_str_dict(json.loads(config_str))\n            else:\n                self.extra_config = config_str\n\n    def load_extra_config(self) -> dict:\n        if self.extra_config is None:\n            return {}\n        if isinstance(self.extra_config, dict):\n            return self.extra_config\n        if not os.path.isfile(self.extra_config):\n            raise ValueError(f\"`extra_config` file not found: {self.extra_config}\")\n        with open(self.extra_config, encoding=\"utf-8\") as f:\n            return json.load(f)\n","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/megatron_bridge_args.py#L158-L194","documentation":"Raised in MegatronBridgeArguments.__post_init__ (megatron_bridge_args.py:176) when moe_token_dispatcher_type is set to a string outside {'allgather', 'alltoall', 'flex'}. The dispatcher moves tokens between attention and expert groups in mixture-of-experts models; these three are the Megatron-core types this build bridges. Validation is a case-sensitive membership check that runs at config-parse time.","triggerScenarios":"Setting moe_token_dispatcher_type: allgather_ltc, 'AllGather', 'a2a', or a Megatron version's newer type not yet supported here; enabling MoE via extra_config on a non-MoE checkpoint with an experimental dispatcher name.","commonSituations":"Porting configs from newer Megatron-core (which has more dispatcher variants) into an older LlamaFactory; casing/typo mistakes; copying DeepSpeed or vLLM MoE option names (e.g. 'standard') into the Megatron field.","solutions":["Use one of exactly: 'allgather', 'alltoall', 'flex'","Check the Megatron-core version pinned by this LlamaFactory release for newly supported dispatchers before using exotic names","Remove the key to use the default dispatcher if you have no specific throughput reason to tune it"],"exampleFix":"# before\nmoe_token_dispatcher_type: standard\n\n# after\nmoe_token_dispatcher_type: alltoall","handlingStrategy":"validation","validationCode":"if cfg.get('moe_token_dispatcher_type') not in (None, 'allgather', 'alltoall', 'flex'):\n    raise SystemExit(f\"unsupported dispatcher: {cfg['moe_token_dispatcher_type']!r}\")","typeGuard":"def is_supported_dispatcher(v: str | None) -> bool:\n    return v in (None, 'allgather', 'alltoall', 'flex')","tryCatchPattern":null,"preventionTips":["Pin MoE configs to the dispatcher whitelist of the LlamaFactory version you run","Do not mix vLLM/DeepSpeed MoE option names into megatron fields"],"tags":["megatron","moe","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}