unslothai/unsloth · error · ValueError

scan_dataset requires a materialized Dataset, not an Iterabl

Error message

scan_dataset requires a materialized Dataset, not an IterableDataset. Load without streaming=True, or materialize a slice first: Dataset.from_list(list(dataset.take(N))).

What it means

ValueError from scan_dataset's second input guard: the object is a streaming IterableDataset, which has no len() or column_names and cannot be randomly scanned. Rather than failing later with an opaque TypeError deep in the scan logic, the guard refuses up front and tells you how to materialize a slice (Dataset.from_list(list(dataset.take(N)))).

Source

Thrown at studio/backend/utils/datasets/dataset_none_detect.py:529

    except ImportError:
        pass
    try:
        from datasets import IterableDatasetDict as _IterableDatasetDict
        _dict_types.append(_IterableDatasetDict)
    except ImportError:
        pass
    if _dict_types and isinstance(dataset, tuple(_dict_types)):
        raise ValueError(
            "scan_dataset requires a single Dataset split, not a DatasetDict. "
            f"Available splits: {list(dataset.keys())}. "
            "Pass dataset[<split>] or use load_dataset(..., split='train')."
        )
    # Streaming IterableDataset has no len()/column_names; give a clear error
    # instead of a confusing downstream TypeError.
    try:
        from datasets import IterableDataset as _IterableDataset
        if isinstance(dataset, _IterableDataset):
            raise ValueError(
                "scan_dataset requires a materialized Dataset, not an IterableDataset. "
                "Load without streaming=True, or materialize a slice first: "
                "Dataset.from_list(list(dataset.take(N)))."
            )
    except ImportError:
        pass
    fmt = FORMAT_ALIASES.get(fmt, fmt)
    was_auto = fmt == "auto"
    # Zero-row dataset: return a trivially clean stats dict.
    if was_auto and len(dataset) == 0:
        return {
            "format": "unknown",
            "total_rows": 0,
            "findings": [],
            "bad_row_indices": [],
        }
    # Always probe so detection and column selection share one scan pass.
    conv_info = _probe_conversation(dataset)

View on GitHub (pinned to 203007d190)

Solutions

1. Materialize a sample for auditing: scan_dataset(Dataset.from_list(list(ds.take(10_000))))
2. Load non-streaming just for the scan: load_dataset('repo', split='train') without streaming=True
3. Keep the streaming dataset for training but maintain a separate materialized handle for audit tools that need len()/column_names

Example fix

# before
ds = load_dataset('repo', split='train', streaming=True)
stats = scan_dataset(ds)  # raises

# after
from datasets import Dataset
stats = scan_dataset(Dataset.from_list(list(ds.take(10_000))))

from datasets import Dataset, IterableDataset

def is_materialized(dataset) -> bool:
    return isinstance(dataset, Dataset) and not isinstance(dataset, IterableDataset)

def is_materialized(dataset) -> bool:
    """True only for a materialized Dataset with len()/column_names."""
    from datasets import Dataset as D, IterableDataset as I
    return isinstance(dataset, D) and not isinstance(dataset, I)

When it happens

Trigger: scan_dataset(load_dataset('repo', streaming=True)); also datasets converted to streaming for memory reasons then passed to the audit tool.

Common situations: Streaming used to avoid downloading huge datasets, then reusing that handle for the None-turn audit; memory-constrained pipelines that switched everything to IterableDataset.

Related errors

• dataset_streaming requires a plain split name in {field_name
• dataset_streaming streams from the Hub and cannot use the lo
• scan_dataset requires a single Dataset split, not a DatasetD
• This execution artifact is outside the Recipe Studio dataset
• Execution artifact path is not a dataset folder.

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/ceef195415add908. Report an issue: GitHub.