hiyouga/LlamaFactory · error · NotImplementedError

HyperParallel CP training requires a map-style dataset becau

Error message

HyperParallel CP training requires a map-style dataset because iterable datasets cannot repeat logical batches across CP ranks.

What it means

Error "HyperParallel CP training requires a map-style dataset because iterable datasets cannot repeat logical batches across CP ranks." thrown in hiyouga/LlamaFactory.

Source

Thrown at src/llamafactory/train/hyper_parallel/trainer.py:257

        if getattr(self.finetuning_args, "disable_shuffling", False):
            return torch.utils.data.SequentialSampler(train_dataset)
        return super()._get_train_sampler(train_dataset)

    def _build_cp_batch_sampler(self, dataset, shuffle: bool, batch_size: int, drop_last: bool):
        """Repeat complete logical batches so CP groups consume the same baseline batch."""
        sampler = self._get_train_sampler(dataset) if shuffle else torch.utils.data.SequentialSampler(dataset)
        return _CPBatchRepeatedBatchSampler(
            sampler,
            batch_size=batch_size,
            drop_last=drop_last,
            repeat_factor=self._cp_size,
            logical_group_size=max(1, get_platform().get_world_size() // self._cp_size),
        )

    def _get_cp_dataloader(self, dataset, batch_size: int, shuffle: bool):
        """Create a train dataloader whose logical batches are shared within each CP group."""
        if isinstance(dataset, torch.utils.data.IterableDataset):
            raise NotImplementedError(
                "HyperParallel CP training requires a map-style dataset because iterable datasets cannot "
                "repeat logical batches across CP ranks."
            )

        try:
            import datasets  # pylint: disable=C0415
        except ImportError:  # pragma: no cover
            datasets = None

        if datasets is not None and isinstance(dataset, datasets.Dataset):
            dataset = self._remove_unused_columns(dataset, description="Training")
            data_collator = self.data_collator
        else:
            data_collator = self._get_collator_with_removed_columns(self.data_collator, description="Training")

        batch_sampler = self._build_cp_batch_sampler(
            dataset,
            shuffle=shuffle,

View on GitHub (pinned to f28afaf635)

Solutions

1. Disable streaming (streaming: false) so the dataset is loaded as a map-style dataset before HyperParallel CP training.

Example fix

streaming: false

When it happens

Trigger: Thrown at src/llamafactory/train/hyper_parallel/trainer.py:257 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/cae1aa71b81b0cd4. Report an issue: GitHub.