huggingface/transformers · error · RuntimeError

only support DDP optuna HPO for ParallelMode.DISTRIBUTED cur

Error message

only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.

What it means

In run_hp_search_optuna's objective on rank 0: when trainer.args.world_size > 1 (multi-process), transformers supports only standard multi-GPU DDP (ParallelMode.DISTRIBUTED) for optuna HPO, because it coordinates ranks via torch.distributed.broadcast_object_list. Any other parallel mode (TPU, MPS, DeepSpeed not in distributed mode, FSDP variants reported differently) raises this RuntimeError.

Source

Thrown at src/transformers/integrations/integration_utils.py:250

    raise RuntimeError(f"Unknown type for trial {trial.__class__}")


def run_hp_search_optuna(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:
    import optuna
    from accelerate.utils.memory import release_memory

    if trainer.args.process_index == 0:

        def _objective(trial: optuna.Trial, checkpoint_dir=None):
            checkpoint = None
            if checkpoint_dir:
                for subdir in os.listdir(checkpoint_dir):
                    if subdir.startswith(PREFIX_CHECKPOINT_DIR):
                        checkpoint = os.path.join(checkpoint_dir, subdir)
            trainer.objective = None
            if trainer.args.world_size > 1:
                if trainer.args.parallel_mode != ParallelMode.DISTRIBUTED:
                    raise RuntimeError("only support DDP optuna HPO for ParallelMode.DISTRIBUTED currently.")
                trainer.hp_space(trial)
                fixed_trial = optuna.trial.FixedTrial(trial.params, trial.number)
                trial_main_rank_list = [fixed_trial]
                torch.distributed.broadcast_object_list(trial_main_rank_list, src=0)
                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
            else:
                trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
            # If there hasn't been any evaluation during the training loop.
            if getattr(trainer, "objective", None) is None:
                metrics = trainer.evaluate()
                trainer.objective = trainer.compute_objective(metrics)

            # Free GPU memory
            trainer.model_wrapped, trainer.model = release_memory(trainer.model_wrapped, trainer.model)
            trainer.accelerator.clear()

            return trainer.objective

View on GitHub (pinned to a597f97485)

Solutions

Run the optuna HPO on a standard distributed launch: torchrun / accelerate launch so parallel_mode == ParallelMode.DISTRIBUTED (plain multi-GPU DDP).
If you cannot change the parallel mode (e.g. TPU), drop optuna and use a supported HPO path for that hardware, or set world_size to 1 (single-process per trial with multi-device inside the model).
Verify torch.distributed.is_initialized() and env vars (RANK, WORLD_SIZE, MASTER_ADDR) before starting the search so the Trainer classifies the run as DISTRIBUTED.

Example fix

# before (TPU / non-DDP multi-process)
trainer.hyperparameter_search(backend="optuna", n_trials=10)

# after (plain DDP launch)
# torchrun --nproc_per_node=4 train.py
trainer.hyperparameter_search(backend="optuna", n_trials=10)

Defensive patterns

Strategy: validation

Validate before calling

from transformers.trainer_utils import ParallelMode

def optuna_hpo_supported(trainer) -> bool:
    a = trainer.args
    return a.world_size == 1 or a.parallel_mode == ParallelMode.DISTRIBUTED

Try / catch

try:
    best = trainer.hyperparameter_search(backend="optuna", n_trials=n, direction="minimize")
except RuntimeError as e:
    if "only support DDP optuna HPO" in str(e):
        raise SystemExit("relaunch with torchrun/accelerate DDP, or use a single process") from e
    raise

Prevention

Guard hyperparameter_search with a parallel_mode check before launching long jobs.
Launch optuna sweeps with torchrun or accelerate launch on multi-GPU DDP only.
Do not attempt optuna HPO on TPU/MPS with this Trainer.

When it happens

Trigger: Calling trainer.hyperparameter_search(backend='optuna', ...) (or run_hp_search_optuna directly) with world_size > 1 while args.parallel_mode resolves to something other than ParallelMode.DISTRIBUTED — e.g. launching on TPU (ParallelMode.TPU), with MPS devices, or a launcher that did not initialize torch.distributed.

Common situations: Switching an existing single-node multi-GPU optuna sweep onto TPU pods or Apple MPS; running under accelerate with a TPU config; torchrun env vars missing so accelerate mis-detects the parallel mode.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/1ff6f7b3e6aa51d3. Report an issue: GitHub.