hiyouga/LlamaFactory · error · ValueError

HyperParallel trainer requires Accelerate FSDP2 mode to be e

Error message

HyperParallel trainer requires Accelerate FSDP2 mode to be enabled.

What it means

Error "HyperParallel trainer requires Accelerate FSDP2 mode to be enabled." thrown in hiyouga/LlamaFactory.

Source

Thrown at src/llamafactory/train/hyper_parallel/trainer.py:163

        finetuning_args=None,
        processor=None,
        ref_model: Optional[nn.Module] = None,
        **kwargs,
    ):
        self._hp_args = hp_args

        # Let CustomSeq2SeqTrainer handle everything except ref_model —
        # Custom would prepare it with accelerate's fsdp2_prepare_model,
        # but we need HP's version instead.
        super().__init__(
            finetuning_args=finetuning_args,
            processor=processor,
            ref_model=None,
            **kwargs,
        )

        if not getattr(self.accelerator, "is_fsdp2", False):
            raise ValueError("HyperParallel trainer requires Accelerate FSDP2 mode to be enabled.")

        self._cp_size = hp_args.cp_size
        self._cp_rank = get_cp_rank(hp_args) if self._cp_size > 1 else 0
        self._dp_rank = get_dp_rank(hp_args) if self._cp_size > 1 else get_platform().get_rank()

        # Prepare ref_model with the same CP + HSDP path as the train model.
        self.ref_model = ref_model
        if self.ref_model is not None:
            self.ref_model = self._prepare_model_for_hyper_parallel(self.ref_model)

        self._orig_accelerator_clip_grad_norm = self.accelerator.clip_grad_norm_
        self._orig_fsdp2_prepare_model = None
        self._accelerator_patches_active = False

    def _prepare_model_for_hyper_parallel(self, model: nn.Module) -> nn.Module:
        """Apply CP runtime hooks before delegating to HyperParallel FSDP2 preparation."""
        if self._cp_size > 1:
            model = cp_prepare_model(model, self.accelerator, self._hp_args)

View on GitHub (pinned to f28afaf635)

Solutions

1. Enable FSDP2 in the Accelerate config (fsdp_config with fsdp_version: 2) so accelerator.is_fsdp2 is true before running HyperParallel training.

Example fix

# accelerate config
fsdp_config:
  fsdp_version: 2

When it happens

Trigger: Thrown at src/llamafactory/train/hyper_parallel/trainer.py:163 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/678167bf0f7b416c. Report an issue: GitHub.