{"record":{"id":"fc7090405c16c3b4","repo":"hiyouga/LlamaFactory","slug":"please-use-force-torchrun-1-to-launch-deepspeed","errorCode":null,"errorMessage":"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.","messagePattern":"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/hparams/parser.py","lineNumber":472,"sourceCode":"        if model_args.shift_attn:\n            raise ValueError(\"PPO training is incompatible with S^2-Attn.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_kt:\n            raise ValueError(\"KTransformers does not support lora reward model.\")\n\n        if finetuning_args.reward_model_type == \"lora\" and model_args.use_unsloth:\n            raise ValueError(\"Unsloth does not support lora reward model.\")\n\n        if training_args.report_to and any(\n            logger not in (\"wandb\", \"tensorboard\", \"trackio\", \"none\") for logger in training_args.report_to\n        ):\n            raise ValueError(\"PPO only accepts wandb, tensorboard, or trackio logger.\")\n\n    if not model_args.use_kt and training_args.parallel_mode == ParallelMode.NOT_DISTRIBUTED:\n        raise ValueError(\"Please launch distributed training with `llamafactory-cli` or `torchrun`.\")\n\n    if training_args.deepspeed and training_args.parallel_mode != ParallelMode.DISTRIBUTED:\n        raise ValueError(\"Please use `FORCE_TORCHRUN=1` to launch DeepSpeed training.\")\n\n    if training_args.max_steps == -1 and data_args.streaming:\n        raise ValueError(\"Please specify `max_steps` in streaming mode.\")\n\n    if training_args.do_train and data_args.dataset is None:\n        raise ValueError(\"Please specify dataset for training.\")\n\n    if (training_args.do_eval or training_args.do_predict or training_args.predict_with_generate) and (\n        data_args.eval_dataset is None and data_args.val_size < 1e-6\n    ):\n        raise ValueError(\"Please make sure eval_dataset be provided or val_size >1e-6\")\n\n    if training_args.predict_with_generate:\n        if is_deepspeed_zero3_enabled():\n            raise ValueError(\"`predict_with_generate` is incompatible with DeepSpeed ZeRO-3.\")\n\n        if finetuning_args.compute_accuracy:\n            raise ValueError(\"Cannot use `predict_with_generate` and `compute_accuracy` together.\")","sourceCodeStart":454,"sourceCodeEnd":490,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/hparams/parser.py#L454-L490","documentation":"Raised in parser.py:472 when training_args.deepspeed is set but parallel_mode is not DISTRIBUTED. DeepSpeed integration in LlamaFactory depends on torchrun's distributed environment (RANK, LOCAL_RANK, WORLD_SIZE); without it, DeepSpeed hooks cannot initialize.","triggerScenarios":"Passing `--deepspeed ds_z2_config.json` (or the YAML equivalent) while launching the process so that ParallelMode stays NOT_DISTRIBUTED — typically running via plain python, or forgetting FORCE_TORCHRUN=1 which suppresses the torchrun wrapper.","commonSituations":"Adding a deepspeed section to the YAML but launching with a plain python entry point; CI scripts that call the tuner API directly; single-GPU users who assume DeepSpeed ZeRO works without a distributed launcher.","solutions":["Relaunch with `FORCE_TORCHRUN=1 llamafactory-cli train config.yaml`","Ensure you go through `llamafactory-cli` / `lmf` so torchrun wraps the process when deepspeed is configured","Verify no wrapper strips distributed env vars (e.g. a custom docker entrypoint or python -m path)"],"exampleFix":"# before\nllamafactory-cli train my_deepspeed.yaml  # missing FORCE_TORCHRUN\n\n# after\nFORCE_TORCHRUN=1 llamafactory-cli train my_deepspeed.yaml","handlingStrategy":"validation","validationCode":"if config.get(\"deepspeed\") and not os.environ.get(\"FORCE_TORCHRUN\"):\n    raise SystemExit(\"DeepSpeed configs require FORCE_TORCHRUN=1 llamafactory-cli train\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Standardize on FORCE_TORCHRUN=1 for every DeepSpeed run","Never call tuner functions directly when deepspeed is set in args","Wrap launches in a make target or shell script that sets the env var"],"tags":["deepspeed","distributed","torchrun","config-validation"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}