hiyouga/LlamaFactory · error · ValueError

Megatron Bridge cannot be used together with MCA or HyperPar

Error message

Megatron Bridge cannot be used together with MCA or HyperParallel.

What it means

Megatron Bridge replaces the HuggingFace distributed stack with Megatron's, so it cannot coexist with MCA (Megatron-core via USE_MCA) or HyperParallel FSDP2, which are alternative distributed backends. The parser rejects the combination when finetuning_args.use_megatron_bridge is true and either use_mca or use_hyper_parallel is true.

Source

Thrown at src/llamafactory/hparams/parser.py:433

        _set_transformers_logging()

    # Check arguments
    if finetuning_args.stage != "sft":
        if training_args.predict_with_generate:
            raise ValueError("`predict_with_generate` cannot be set as True except SFT.")

        if data_args.neat_packing:
            raise ValueError("`neat_packing` cannot be set as True except SFT.")

        if data_args.train_on_prompt or data_args.mask_history:
            raise ValueError("`train_on_prompt` or `mask_history` cannot be set as True except SFT.")

    if finetuning_args.stage == "sft" and training_args.do_predict and not training_args.predict_with_generate:
        raise ValueError("Please enable `predict_with_generate` to save model predictions.")

    if finetuning_args.use_megatron_bridge:
        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:
            raise ValueError("Megatron Bridge cannot be used together with MCA or HyperParallel.")
        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:

View on GitHub (pinned to f28afaf635)

Solutions

1. Unset conflicting backend env/flags: unset USE_MCA, and set hyper_parallel options off (use_hyper_parallel: false) in the config.
2. Pick exactly one backend per run: Megatron Bridge (USE_MEGATRON_BRIDGE=1), MCA (USE_MCA=1), or HF+HyperParallel (default path).
3. Audit exported variables before launching: env | grep -E 'USE_MCA|USE_MEGATRON_BRIDGE'.

Example fix

# before
export USE_MCA=1
export USE_MEGATRON_BRIDGE=1
# config: hyper_parallel: true

# after (Megatron Bridge only)
unset USE_MCA
export USE_MEGATRON_BRIDGE=1
# config: hyper_parallel removed / false

import os

backends = sum([
    os.environ.get("USE_MCA") == "1",
    os.environ.get("USE_MEGATRON_BRIDGE") == "1",
    bool(cfg.get("use_hyper_parallel")),
])
if backends > 1:
    raise SystemExit("Enable exactly one distributed backend: MCA, Megatron Bridge, or HyperParallel")

When it happens

Trigger: USE_MEGATRON_BRIDGE=1 in the environment while the config sets hyper_parallel/fspd2 options (use_hyper_parallel: true), or both USE_MCA and USE_MEGATRON_BRIDGE env vars set (USE_MCA is checked first, so this fires when mbridge won but the flag is still set), or the YAML itself sets use_megatron_bridge plus hyper-parallel flags.

Common situations: Shared cluster shells where USE_MCA or USE_MEGATRON_BRIDGE are exported globally in .bashrc; teams migrating between the three distributed backends leaving stale env vars; config templates that stack all performance flags.

Related errors

• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• Megatron Bridge is incompatible with DeepSpeed.
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/6f16e71373d80a65. Report an issue: GitHub.