huggingface/pytorch-image-models · error · ValueError

Dataset length is unknown, please pass `num_samples` explici

Error message

Dataset length is unknown, please pass `num_samples` explicitly. The number of steps needs to be known in advance for the learning rate scheduler.

What it means

ReaderHfids (HuggingFace image datasets reader) needs the dataset length to size the learning-rate schedule, and neither an explicit num_samples nor split_info.num_examples was available, so construction fails fast rather than producing an unknown-length iterator.

Source

Thrown at timm/data/readers/reader_hfids.py:82

        self.builder = datasets.load_dataset_builder(
            name,
            cache_dir=root,
            trust_remote_code=trust_remote_code,
        )
        if download:
            self.builder.download_and_prepare()

        split_info: Optional[SplitInfo] = None
        if self.builder.info.splits and split in self.builder.info.splits:
            if isinstance(self.builder.info.splits[split], SplitInfo):
                split_info: Optional[SplitInfo] = self.builder.info.splits[split]

        if num_samples:
            self.num_samples = num_samples
        elif split_info and split_info.num_examples:
            self.num_samples = split_info.num_examples
        else:
            raise ValueError(
                "Dataset length is unknown, please pass `num_samples` explicitly. "
                "The number of steps needs to be known in advance for the learning rate scheduler."
            )

        self.remap_class = False
        if class_map:
            self.class_to_idx = load_class_map(class_map)
            self.remap_class = True
        else:
            self.class_to_idx = {}

        # Distributed world state
        self.dist_rank = 0
        self.dist_num_replicas = 1
        if dist.is_available() and dist.is_initialized() and dist.get_world_size() > 1:
            self.dist_rank = dist.get_rank()
            self.dist_num_replicas = dist.get_world_size()

View on GitHub (pinned to 9a5261e31b)

Solutions

Pass num_samples explicitly to the reader / create_loader (e.g. num_samples=len(ds) or a known train-set size).
If streaming, compute the count once (iterate or use non-streaming mode) and pass it in.
Upgrade the datasets/huggingface_hub libraries so split metadata is populated.

Example fix

# before
reader = ReaderHfids(ds, split='train')

# after
reader = ReaderHfids(ds, split='train', num_samples=25_000)

Defensive patterns

Strategy: validation

Validate before calling

n = num_samples or (split_info.num_examples if split_info else None)
if not n:
    n = len(ds)  # non-streaming HF dataset
reader = ReaderHfids(ds, num_samples=n, ...)

Prevention

Always pass num_samples for streaming HF datasets.
Log dataset size before training starts.
Pin known train-set sizes in config for reproducibility.

When it happens

Trigger: Instantiating ReaderHfids with num_samples=None on a HF datasets split whose info does not expose num_examples (streaming datasets, some hub datasets with missing metadata), typically via create_loader/create_dataset with the hfids reader.

Common situations: Using dataset_name='hfds/hfids' with streaming=True; hub datasets whose split info lacks example counts; older huggingface_hub/datasets versions that don't populate split info.

Related errors

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/3b9b7e36b88aa5e5. Report an issue: GitHub.