hiyouga/LlamaFactory · error · NotImplementedError

Iterable dataset is not supported yet.

Error message

Iterable dataset is not supported yet.

What it means

_init_data_provider uses len(self.dataset) != -1 to detect map-style datasets; an iterable dataset reports length -1 in this convention, and the stateful distributed sampler + checkpoint-resume design requires indexable, length-known data. Iterable datasets are therefore not implemented yet and raise NotImplementedError.

Source

Thrown at src/llamafactory/v1/core/utils/batching.py:232

            f"micro batch size {self.micro_batch_size}, "
            f"num micro batch {self.num_micro_batch}, "
            f"cutoff len {self.cutoff_len}, "
            f"batching workers {self.batching_workers}, "
            f"batching strategy {self.batching_strategy}."
        )

    def _init_data_provider(self) -> None:
        if len(self.dataset) != -1:
            sampler = StatefulDistributedSampler(
                self.dataset,
                num_replicas=DistributedInterface().get_world_size(Dim.DP),
                rank=DistributedInterface().get_rank(Dim.DP),
                shuffle=True,
                seed=self.seed,
                drop_last=self.drop_last,
            )
        else:
            raise NotImplementedError("Iterable dataset is not supported yet.")

        if self.batching_strategy == BatchingStrategy.NORMAL:
            batch_size = self.micro_batch_size * self.num_micro_batch
        else:
            from ...plugins.trainer_plugins.batching import BatchingPlugin

            batch_size = BatchingPlugin(self.batching_strategy).get_data_provider_batch_size(self._batch_info)

        generator_seed = torch.Generator()
        generator_seed.manual_seed(self.seed)

        self._data_provider = StatefulDataLoader(
            self.dataset,
            batch_size=batch_size,
            sampler=sampler,
            num_workers=self.batching_workers,
            collate_fn=self.renderer.process_samples,
            pin_memory=self.pin_memory,

View on GitHub (pinned to f28afaf635)

Solutions

1. Materialize the dataset to a map-style dataset (e.g. list(streamed_items) or datasets.Dataset.from_generator) before training
2. Cache the streamed data to disk (arrow/parquet) and load it as a regular dataset
3. Use the v0 pipeline, which supports iterable datasets, until v1 adds support

Example fix

# before
 ds = load_dataset("big_corpus", split="train", streaming=True)  # iterable -> NotImplementedError

# after
 ds = load_dataset("big_corpus", split="train").map(identity)  # map-style, len() known

def is_map_style(dataset) -> bool:
    try:
        return len(dataset) != -1 and len(dataset) >= 0
    except TypeError:
        return False

When it happens

Trigger: Passing a torch IterableDataset (or a dataset wrapper whose __len__ returns -1) into the v1 batching scheduler during trainer initialization.

Common situations: Streaming datasets (HF datasets load_dataset(streaming=True), webdataset, Kafka/tf.data-style sources); custom generator-based corpora.

Related errors

• The model does not have a submodule named '{submodule_name}'
• Turn off `streaming` when saving dataset to disk.
• Please specify `max_steps` in streaming mode.
• All datasets must be streaming or non-streaming.
• Streaming dataset does not support index access.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/453c3951de4916c7. Report an issue: GitHub.