{"record":{"id":"ceef195415add908","repo":"unslothai/unsloth","slug":"scan-dataset-requires-a-materialized-dataset-not","errorCode":null,"errorMessage":"scan_dataset requires a materialized Dataset, not an IterableDataset. Load without streaming=True, or materialize a slice first: Dataset.from_list(list(dataset.take(N))).","messagePattern":"scan_dataset requires a materialized Dataset, not an IterableDataset\\. Load without streaming=True, or materialize a slice first: Dataset\\.from_list\\(list\\(dataset\\.take\\(N\\)\\)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"studio/backend/utils/datasets/dataset_none_detect.py","lineNumber":529,"sourceCode":"    except ImportError:\n        pass\n    try:\n        from datasets import IterableDatasetDict as _IterableDatasetDict\n        _dict_types.append(_IterableDatasetDict)\n    except ImportError:\n        pass\n    if _dict_types and isinstance(dataset, tuple(_dict_types)):\n        raise ValueError(\n            \"scan_dataset requires a single Dataset split, not a DatasetDict. \"\n            f\"Available splits: {list(dataset.keys())}. \"\n            \"Pass dataset[<split>] or use load_dataset(..., split='train').\"\n        )\n    # Streaming IterableDataset has no len()/column_names; give a clear error\n    # instead of a confusing downstream TypeError.\n    try:\n        from datasets import IterableDataset as _IterableDataset\n        if isinstance(dataset, _IterableDataset):\n            raise ValueError(\n                \"scan_dataset requires a materialized Dataset, not an IterableDataset. \"\n                \"Load without streaming=True, or materialize a slice first: \"\n                \"Dataset.from_list(list(dataset.take(N))).\"\n            )\n    except ImportError:\n        pass\n    fmt = FORMAT_ALIASES.get(fmt, fmt)\n    was_auto = fmt == \"auto\"\n    # Zero-row dataset: return a trivially clean stats dict.\n    if was_auto and len(dataset) == 0:\n        return {\n            \"format\": \"unknown\",\n            \"total_rows\": 0,\n            \"findings\": [],\n            \"bad_row_indices\": [],\n        }\n    # Always probe so detection and column selection share one scan pass.\n    conv_info = _probe_conversation(dataset)","sourceCodeStart":511,"sourceCodeEnd":547,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/utils/datasets/dataset_none_detect.py#L511-L547","documentation":"ValueError from scan_dataset's second input guard: the object is a streaming IterableDataset, which has no len() or column_names and cannot be randomly scanned. Rather than failing later with an opaque TypeError deep in the scan logic, the guard refuses up front and tells you how to materialize a slice (Dataset.from_list(list(dataset.take(N)))).","triggerScenarios":"scan_dataset(load_dataset('repo', streaming=True)); also datasets converted to streaming for memory reasons then passed to the audit tool.","commonSituations":"Streaming used to avoid downloading huge datasets, then reusing that handle for the None-turn audit; memory-constrained pipelines that switched everything to IterableDataset.","solutions":["Materialize a sample for auditing: scan_dataset(Dataset.from_list(list(ds.take(10_000))))","Load non-streaming just for the scan: load_dataset('repo', split='train') without streaming=True","Keep the streaming dataset for training but maintain a separate materialized handle for audit tools that need len()/column_names"],"exampleFix":"# before\nds = load_dataset('repo', split='train', streaming=True)\nstats = scan_dataset(ds)  # raises\n\n# after\nfrom datasets import Dataset\nstats = scan_dataset(Dataset.from_list(list(ds.take(10_000))))","handlingStrategy":"type-guard","validationCode":"from datasets import Dataset, IterableDataset\n\ndef is_materialized(dataset) -> bool:\n    return isinstance(dataset, Dataset) and not isinstance(dataset, IterableDataset)","typeGuard":"def is_materialized(dataset) -> bool:\n    \"\"\"True only for a materialized Dataset with len()/column_names.\"\"\"\n    from datasets import Dataset as D, IterableDataset as I\n    return isinstance(dataset, D) and not isinstance(dataset, I)","tryCatchPattern":null,"preventionTips":["Keep a materialized sample handle (Dataset.from_list(list(stream.take(N)))) for audits","Never pass streaming handles to tools that call len() or column_names","Gate all scan entry points with an is_materialized type guard for a clear early error"],"tags":["datasets","streaming","huggingface","validation"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}