hiyouga/LlamaFactory · error · ValueError

Please launch distributed training with `llamafactory-cli` o

Error message

Please launch distributed training with `llamafactory-cli` or `torchrun`.

What it means

Raised in parser.py:469 when training is launched in NOT_DISTRIBUTED mode while KTransformers (use_kt) is off. LlamaFactory requires its distributed launcher for training so that environment detection, device placement, and launcher patching work correctly; a bare `python` invocation bypasses that setup.

Source

Thrown at src/llamafactory/hparams/parser.py:469

        if not training_args.do_train:
            raise ValueError("PPO training does not support evaluation, use the SFT stage to evaluate models.")

        if model_args.shift_attn:
            raise ValueError("PPO training is incompatible with S^2-Attn.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_kt:
            raise ValueError("KTransformers does not support lora reward model.")

        if finetuning_args.reward_model_type == "lora" and model_args.use_unsloth:
            raise ValueError("Unsloth does not support lora reward model.")

        if training_args.report_to and any(
            logger not in ("wandb", "tensorboard", "trackio", "none") for logger in training_args.report_to
        ):
            raise ValueError("PPO only accepts wandb, tensorboard, or trackio logger.")

    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:
        raise ValueError("Please launch distributed training with `llamafactory-cli` or `torchrun`.")

    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:
        raise ValueError("Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.")

    if training_args.max_steps == -1 and data_args.streaming:
        raise ValueError("Please specify `max_steps` in streaming mode.")

    if training_args.do_train and data_args.dataset is None:
        raise ValueError("Please specify dataset for training.")

    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (
        data_args.eval_dataset is None and data_args.val_size < 1e-6
    ):
        raise ValueError("Please make sure eval_dataset be provided or val_size >1e-6")

    if training_args.predict_with_generate:
        if is_deepspeed_zero3_enabled():
            raise ValueError("`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Launch with the CLI: `llamafactory-cli train config.yaml` (it auto-wraps with torchrun when needed)
2. Or launch explicitly with `FORCE_TORCHRUN=1 llamafactory-cli train config.yaml`
3. If using a custom script, import and call llamafactory.cli:main / launcher so parallel_mode is set up, instead of calling tuner functions raw

Example fix

# before
python src/llamafactory/... my_train.py  # NOT_DISTRIBUTED -> raises

# after
llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

import torch.distributed as dist
if not model_args.use_kt and not (dist.is_available() and dist.is_initialized()):
    raise SystemExit("Launch via llamafactory-cli / torchrun, not bare python")

When it happens

Trigger: Calling the training entry point directly, e.g. `python src/llamafactory/tuner.py ...` or a custom script that calls run_exp/run_sft without torchrun, so training_args.parallel_mode resolves to ParallelMode.NOT_DISTRIBUTED while model_args.use_kt is False.

Common situations: Porting LlamaFactory into a custom Python driver script; running the module via `python -m`; a wrapper (IDE run button, notebook) that spawns the process without the distributed environment variables.

Related errors

• Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.
• `tensor_model_parallel_size` must be >= 1.
• `pipeline_model_parallel_size` must be >= 1.
• `virtual_pipeline_model_parallel_size` must be >= 1 when set
• `sequence_parallel` requires `tensor_model_parallel_size` >

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/1aee8e47d2d65e1b. Report an issue: GitHub.