{"record":{"id":"453c3951de4916c7","repo":"hiyouga/LlamaFactory","slug":"iterable-dataset-is-not-supported-yet","errorCode":null,"errorMessage":"Iterable dataset is not supported yet.","messagePattern":"Iterable dataset is not supported yet\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/utils/batching.py","lineNumber":232,"sourceCode":"            f\"micro batch size {self.micro_batch_size}, \"\n            f\"num micro batch {self.num_micro_batch}, \"\n            f\"cutoff len {self.cutoff_len}, \"\n            f\"batching workers {self.batching_workers}, \"\n            f\"batching strategy {self.batching_strategy}.\"\n        )\n\n    def _init_data_provider(self) -> None:\n        if len(self.dataset) != -1:\n            sampler = StatefulDistributedSampler(\n                self.dataset,\n                num_replicas=DistributedInterface().get_world_size(Dim.DP),\n                rank=DistributedInterface().get_rank(Dim.DP),\n                shuffle=True,\n                seed=self.seed,\n                drop_last=self.drop_last,\n            )\n        else:\n            raise NotImplementedError(\"Iterable dataset is not supported yet.\")\n\n        if self.batching_strategy == BatchingStrategy.NORMAL:\n            batch_size = self.micro_batch_size * self.num_micro_batch\n        else:\n            from ...plugins.trainer_plugins.batching import BatchingPlugin\n\n            batch_size = BatchingPlugin(self.batching_strategy).get_data_provider_batch_size(self._batch_info)\n\n        generator_seed = torch.Generator()\n        generator_seed.manual_seed(self.seed)\n\n        self._data_provider = StatefulDataLoader(\n            self.dataset,\n            batch_size=batch_size,\n            sampler=sampler,\n            num_workers=self.batching_workers,\n            collate_fn=self.renderer.process_samples,\n            pin_memory=self.pin_memory,","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/utils/batching.py#L214-L250","documentation":"_init_data_provider uses len(self.dataset) != -1 to detect map-style datasets; an iterable dataset reports length -1 in this convention, and the stateful distributed sampler + checkpoint-resume design requires indexable, length-known data. Iterable datasets are therefore not implemented yet and raise NotImplementedError.","triggerScenarios":"Passing a torch IterableDataset (or a dataset wrapper whose __len__ returns -1) into the v1 batching scheduler during trainer initialization.","commonSituations":"Streaming datasets (HF datasets load_dataset(streaming=True), webdataset, Kafka/tf.data-style sources); custom generator-based corpora.","solutions":["Materialize the dataset to a map-style dataset (e.g. list(streamed_items) or datasets.Dataset.from_generator) before training","Cache the streamed data to disk (arrow/parquet) and load it as a regular dataset","Use the v0 pipeline, which supports iterable datasets, until v1 adds support"],"exampleFix":"# before\n ds = load_dataset(\"big_corpus\", split=\"train\", streaming=True)  # iterable -> NotImplementedError\n\n# after\n ds = load_dataset(\"big_corpus\", split=\"train\").map(identity)  # map-style, len() known","handlingStrategy":"validation","validationCode":"def is_map_style(dataset) -> bool:\n    try:\n        return len(dataset) != -1 and len(dataset) >= 0\n    except TypeError:\n        return False","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Materialize streamed corpora to arrow/parquet before v1 training","Check hasattr(dataset, '__getitem__') and a real __len__ when writing custom datasets"],"tags":["dataset","iterable-dataset","streaming","not-implemented","v1"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}