hiyouga/LlamaFactory · error · ValueError

Megatron Bridge only supports `full` and `lora` finetuning.

Error message

Megatron Bridge only supports `full` and `lora` finetuning.

What it means

Megatron Bridge supports full fine-tuning and LoRA only. Other finetuning types such as freeze (and OFT or PiSSA variants) have no Megatron-side implementation, so the parser validates finetuning_type is in ['full', 'lora'] when the bridge is enabled.

Source

Thrown at src/llamafactory/hparams/parser.py:437

        if training_args.predict_with_generate:
            raise ValueError("`predict_with_generate` cannot be set as True except SFT.")

        if data_args.neat_packing:
            raise ValueError("`neat_packing` cannot be set as True except SFT.")

        if data_args.train_on_prompt or data_args.mask_history:
            raise ValueError("`train_on_prompt` or `mask_history` cannot be set as True except SFT.")

    if finetuning_args.stage == "sft" and training_args.do_predict and not training_args.predict_with_generate:
        raise ValueError("Please enable `predict_with_generate` to save model predictions.")

    if finetuning_args.use_megatron_bridge:
        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:
            raise ValueError("Megatron Bridge cannot be used together with MCA or HyperParallel.")
        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")
        if mb_args is None:
            raise ValueError("Megatron Bridge arguments are missing. Please set USE_MEGATRON_BRIDGE=1.")
        _validate_megatron_bridge_parallel_args(mb_args, training_args.world_size)
        finetuning_args.megatron_bridge_args = mb_args

    if finetuning_args.stage in ["rm", "ppo"] and training_args.load_best_model_at_end:
        raise ValueError("RM and PPO stages do not support `load_best_model_at_end`.")

    if finetuning_args.stage == "ppo":
        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Set finetuning_type: lora (or full) in the config when using Megatron Bridge.
2. If freeze-tuning is required, unset USE_MEGATRON_BRIDGE and run on the standard backend.
3. For LoRA on bridge, also provide the usual lora_rank/lora_target settings.

Example fix

# before
export USE_MEGATRON_BRIDGE=1
finetuning_type: freeze

# after
export USE_MEGATRON_BRIDGE=1
finetuning_type: lora

import os

if os.environ.get("USE_MEGATRON_BRIDGE") == "1" and cfg.get("finetuning_type") not in ("full", "lora"):
    raise SystemExit("Megatron Bridge supports only full/lora finetuning")

When it happens

Trigger: USE_MEGATRON_BRIDGE=1 with finetuning_type: freeze (or any value other than full/lora) in the YAML, submitted via llamafactory-cli train.

Common situations: Users coming from freeze-tuning workflows on the HF path who switch to Megatron for speed; configs that default finetuning_type: freeze for parameter-efficient experiments.

Related errors

• Total Megatron Bridge parallel size ({parallel_size}) exceed
• Total Megatron Bridge parallel size ({parallel_size}) must d
• Megatron Bridge cannot be used together with MCA or HyperPar
• Megatron Bridge only supports the `pt` and `sft` stages.
• Quantized models are not supported with Megatron Bridge.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/7e171fa9af75b45d. Report an issue: GitHub.