hiyouga/LlamaFactory · error · ValueError

HyperParallel trainer requires Accelerate FSDP2 mode to be…

Error message

HyperParallel trainer requires Accelerate FSDP2 mode to be enabled.

What it means

Error "HyperParallel trainer requires Accelerate FSDP2 mode to be enabled." thrown in hiyouga/LlamaFactory.

Solutions

1. Enable FSDP2 in the Accelerate config (fsdp_config with fsdp_version: 2) so accelerator.is_fsdp2 is true before running HyperParallel training.

Example fix

# accelerate config
fsdp_config:
  fsdp_version: 2

When it happens

Trigger: Thrown at src/llamafactory/train/hyper_parallel/trainer.py:163 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/678167bf0f7b416c. Report an issue: GitHub.

Appendix: source

Thrown at src/llamafactory/train/hyper_parallel/trainer.py:163

        finetuning_args=None,
        processor=None,
        ref_model: Optional[nn.Module] = None,
        **kwargs,
    ):
        self._hp_args = hp_args

        # Let CustomSeq2SeqTrainer handle everything except ref_model —
        # Custom would prepare it with accelerate's fsdp2_prepare_model,
        # but we need HP's version instead.
        super().__init__(
            finetuning_args=finetuning_args,
            processor=processor,
            ref_model=None,
            **kwargs,
        )

        if not getattr(self.accelerator, "is_fsdp2", False):
            raise ValueError("HyperParallel trainer requires Accelerate FSDP2 mode to be enabled.")

        self._cp_size = hp_args.cp_size
        self._cp_rank = get_cp_rank(hp_args) if self._cp_size > 1 else 0
        self._dp_rank = get_dp_rank(hp_args) if self._cp_size > 1 else get_platform().get_rank()

        # Prepare ref_model with the same CP + HSDP path as the train model.
        self.ref_model = ref_model
        if self.ref_model is not None:
            self.ref_model = self._prepare_model_for_hyper_parallel(self.ref_model)

        self._orig_accelerator_clip_grad_norm = self.accelerator.clip_grad_norm_
        self._orig_fsdp2_prepare_model = None
        self._accelerator_patches_active = False

    def _prepare_model_for_hyper_parallel(self, model: nn.Module) -> nn.Module:
        """Apply CP runtime hooks before delegating to HyperParallel FSDP2 preparation."""
        if self._cp_size > 1:
            model = cp_prepare_model(model, self.accelerator, self._hp_args)

View on GitHub (pinned to f28afaf635)