{"record":{"id":"e19a1a1965a16e5b","repo":"huggingface/transformers","slug":"weight-conversions-e-g-moe-expert-fusion-with","errorCode":null,"errorMessage":"Weight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.","messagePattern":"Weight conversions \\(e\\.g\\., MoE expert fusion\\) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon\\. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/deepspeed.py","lineNumber":344,"sourceCode":"\n\ndef _apply_weight_conversions_to_state_dict(model, state_dict, weight_mapping):\n    \"\"\"\n    Apply weight conversions (renaming and merging/splitting operations) to a state dict.\n    This is a simplified version that handles the conversion without loading into the model.\n    \"\"\"\n    # Check for Tensor Parallelism - weight conversions are not tested with TP\n    # TP uses ReplaceWithTensorSlicing which may conflict with our weight conversions\n    ds_config = deepspeed_config()\n    if ds_config is not None:\n        # Check training config (tensor_parallel.autotp_size)\n        tp_size = ds_config.get(\"tensor_parallel\", {}).get(\"autotp_size\", 1)\n        # Check inference config (inference.tensor_parallel.tp_size)\n        inference_config = ds_config.get(\"inference\", {})\n        if isinstance(inference_config, dict):\n            tp_size = max(tp_size, inference_config.get(\"tensor_parallel\", {}).get(\"tp_size\", 1))\n        if tp_size > 1:\n            raise NotImplementedError(\n                \"Weight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism \"\n                \"are not yet implemented but support is coming soon. Please disable tensor_parallel \"\n                \"in your DeepSpeed config or convert your checkpoint to the expected format first.\"\n            )\n\n    from ..core_model_loading import WeightConverter, WeightRenaming, dot_natural_key, rename_source_key\n\n    # Preserve metadata from the original state dict\n    metadata = getattr(state_dict, \"_metadata\", None)\n\n    base_model_prefix = model.base_model_prefix\n\n    # Build a meta state dict for matching - only keys/shapes, no actual tensor data\n    # This minimizes memory since we don't duplicate the model's parameters\n    model_state_dict = {}\n    for key, param in model.state_dict().items():\n        model_state_dict[key] = torch.empty(param.shape, dtype=param.dtype, device=\"meta\")\n","sourceCodeStart":326,"sourceCodeEnd":362,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/deepspeed.py#L326-L362","documentation":"Error \"Weight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.\" thrown in huggingface/transformers.","triggerScenarios":"Raised when weight conversions (e.g. MoE expert fusion) are requested under DeepSpeed Tensor Parallelism.","commonSituations":"Loading a checkpoint needing format conversion with tensor_parallel enabled in the DeepSpeed config.","solutions":["Disable tensor_parallel in the DeepSpeed config.","Convert the checkpoint to the expected format before loading with DeepSpeed TP."],"exampleFix":null,"handlingStrategy":null,"validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}