hiyouga/LlamaFactory · error · ValueError

Megatron Bridge is incompatible with DeepSpeed.

Error message

Megatron Bridge is incompatible with DeepSpeed.

What it means

DeepSpeed ZeRO sharding and Megatron's own distributed runtime both try to own the process group and optimizer state; running both is impossible. When use_megatron_bridge is true and training_args.deepspeed is set (a DeepSpeed config file path), the parser raises ValueError.

Source

Thrown at src/llamafactory/hparams/parser.py:441

            raise ValueError("`neat_packing` cannot be set as True except SFT.")

        if data_args.train_on_prompt or data_args.mask_history:
            raise ValueError("`train_on_prompt` or `mask_history` cannot be set as True except SFT.")

    if finetuning_args.stage == "sft" and training_args.do_predict and not training_args.predict_with_generate:
        raise ValueError("Please enable `predict_with_generate` to save model predictions.")

    if finetuning_args.use_megatron_bridge:
        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:
            raise ValueError("Megatron Bridge cannot be used together with MCA or HyperParallel.")
        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Delete the deepspeed key (or pass null) from the config when using Megatron Bridge; use Megatron parallel sizes (TP/PP/CP/EP) for sharding instead.
2. If you specifically need ZeRO optimizations, unset USE_MEGATRON_BRIDGE and stay on the DeepSpeed path.
3. Check launcher scripts for hardcoded --deepspeed flags.

Example fix

# before
export USE_MEGATRON_BRIDGE=1
deepspeed: ds_z3_config.json

# after
export USE_MEGATRON_BRIDGE=1
# deepspeed removed; use tensor/pipeline parallelism instead
tensor_model_parallel_size: 2
pipeline_model_parallel_size: 2

import os

if os.environ.get("USE_MEGATRON_BRIDGE") == "1" and cfg.get("deepspeed"):
    raise SystemExit("Megatron Bridge is incompatible with DeepSpeed; remove the deepspeed config")

When it happens

Trigger: USE_MEGATRON_BRIDGE=1 together with deepspeed: ds_z2_config.json (or --deepspeed on the CLI) in the training args of a train config.

Common situations: Cluster templates that always inject a deepspeed block; migrating a DeepSpeed SFT recipe to Megatron Bridge and forgetting to strip the deepspeed key; CI passing --deepspeed globally.

Related errors

• Megatron Bridge cannot be used together with MCA or HyperPar
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6758943991ea20de. Report an issue: GitHub.