{"record":{"id":"1ff6f7b3e6aa51d3","repo":"huggingface/transformers","slug":"only-support-ddp-optuna-hpo-for-parallelmode-distr","errorCode":null,"errorMessage":"only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.","messagePattern":"only support DDP optuna HPO for ParallelMode\\.DISTRIBUTED currently\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"src/transformers/integrations/integration_utils.py","lineNumber":250,"sourceCode":"    raise RuntimeError(f\"Unknown type for trial {trial.__class__}\")\n\n\ndef run_hp_search_optuna(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:\n    import optuna\n    from accelerate.utils.memory import release_memory\n\n    if trainer.args.process_index == 0:\n\n        def _objective(trial: optuna.Trial, checkpoint_dir=None):\n            checkpoint = None\n            if checkpoint_dir:\n                for subdir in os.listdir(checkpoint_dir):\n                    if subdir.startswith(PREFIX_CHECKPOINT_DIR):\n                        checkpoint = os.path.join(checkpoint_dir, subdir)\n            trainer.objective = None\n            if trainer.args.world_size > 1:\n                if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:\n                    raise RuntimeError(\"only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.\")\n                trainer.hp_space(trial)\n                fixed_trial = optuna.trial.FixedTrial(trial.params, trial.number)\n                trial_main_rank_list = [fixed_trial]\n                torch.distributed.broadcast_object_list(trial_main_rank_list, src=0)\n                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)\n            else:\n                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)\n            # If there hasn't been any evaluation during the training loop.\n            if getattr(trainer, \"objective\", None) is None:\n                metrics = trainer.evaluate()\n                trainer.objective = trainer.compute_objective(metrics)\n\n            # Free GPU memory\n            trainer.model_wrapped, trainer.model = release_memory(trainer.model_wrapped, trainer.model)\n            trainer.accelerator.clear()\n\n            return trainer.objective\n","sourceCodeStart":232,"sourceCodeEnd":268,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/integrations/integration_utils.py#L232-L268","documentation":"In run_hp_search_optuna's objective on rank 0: when trainer.args.world_size > 1 (multi-process), transformers supports only standard multi-GPU DDP (ParallelMode.DISTRIBUTED) for optuna HPO, because it coordinates ranks via torch.distributed.broadcast_object_list. Any other parallel mode (TPU, MPS, DeepSpeed not in distributed mode, FSDP variants reported differently) raises this RuntimeError.","triggerScenarios":"Calling trainer.hyperparameter_search(backend='optuna', ...) (or run_hp_search_optuna directly) with world_size > 1 while args.parallel_mode resolves to something other than ParallelMode.DISTRIBUTED — e.g. launching on TPU (ParallelMode.TPU), with MPS devices, or a launcher that did not initialize torch.distributed.","commonSituations":"Switching an existing single-node multi-GPU optuna sweep onto TPU pods or Apple MPS; running under accelerate with a TPU config; torchrun env vars missing so accelerate mis-detects the parallel mode.","solutions":["Run the optuna HPO on a standard distributed launch: torchrun / accelerate launch so parallel_mode == ParallelMode.DISTRIBUTED (plain multi-GPU DDP).","If you cannot change the parallel mode (e.g. TPU), drop optuna and use a supported HPO path for that hardware, or set world_size to 1 (single-process per trial with multi-device inside the model).","Verify torch.distributed.is_initialized() and env vars (RANK, WORLD_SIZE, MASTER_ADDR) before starting the search so the Trainer classifies the run as DISTRIBUTED."],"exampleFix":"# before (TPU / non-DDP multi-process)\ntrainer.hyperparameter_search(backend=\"optuna\", n_trials=10)\n\n# after (plain DDP launch)\n# torchrun --nproc_per_node=4 train.py\ntrainer.hyperparameter_search(backend=\"optuna\", n_trials=10)","handlingStrategy":"validation","validationCode":"from transformers.trainer_utils import ParallelMode\n\ndef optuna_hpo_supported(trainer) -> bool:\n    a = trainer.args\n    return a.world_size == 1 or a.parallel_mode == ParallelMode.DISTRIBUTED","typeGuard":null,"tryCatchPattern":"try:\n    best = trainer.hyperparameter_search(backend=\"optuna\", n_trials=n, direction=\"minimize\")\nexcept RuntimeError as e:\n    if \"only support DDP optuna HPO\" in str(e):\n        raise SystemExit(\"relaunch with torchrun/accelerate DDP, or use a single process\") from e\n    raise","preventionTips":["Guard hyperparameter_search with a parallel_mode check before launching long jobs.","Launch optuna sweeps with torchrun or accelerate launch on multi-GPU DDP only.","Do not attempt optuna HPO on TPU/MPS with this Trainer."],"tags":["hpo","optuna","distributed","trainer"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}