huggingface/pytorch-image-models · error · RuntimeError

Invalid split definition, num_samples not specified in train

Error message

Invalid split definition, num_samples not specified in train mode.

What it means

In training mode the WebDataset reader must know the dataset length (for the LR scheduler and epoch semantics); neither num_samples nor the split's num_samples in info.json provided a positive value, so construction aborts.

Source

Thrown at timm/data/readers/reader_wds.py:308

        self.common_seed = seed  # a seed that's fixed across all worker / distributed instances
        self.shard_shuffle_size = 500
        self.sample_shuffle_size = sample_shuffle_size or SAMPLE_SHUFFLE_SIZE
        self.sample_initial_size = sample_initial_size or SAMPLE_INITIAL_SIZE

        self.input_key = input_key
        self.input_img_mode = input_img_mode
        self.target_key = target_key
        self.filename_key = filename_key
        self.key_ext = '.JPEG'  # extension to add to key for original filenames (DS specific, default ImageNet)

        self.info = _load_info(self.root)
        self.split_info = _parse_split_info(split, self.info)
        if num_samples is not None:
            self.num_samples = num_samples
        else:
            self.num_samples = self.split_info.num_samples
        if is_training and not self.num_samples:
            raise RuntimeError(f'Invalid split definition, num_samples not specified in train mode.')
        self.remap_class = False
        if class_map:
            self.class_to_idx = load_class_map(class_map)
            self.remap_class = True
        else:
            self.class_to_idx = {}

        # Distributed world state
        self.dist_rank = 0
        self.dist_num_replicas = 1
        if dist.is_available() and dist.is_initialized() and dist.get_world_size() > 1:
            self.dist_rank = dist.get_rank()
            self.dist_num_replicas = dist.get_world_size()

        # Attributes that are updated in _lazy_init
        self.worker_info = None
        self.worker_id = 0
        self.worker_seed = seed  # seed unique to each worker instance

View on GitHub (pinned to 9a5261e31b)

Solutions

1. Pass num_samples explicitly to the reader / train script.
2. Add "num_samples": <int> to the split entry in the dataset's info.json.
3. Regenerate info.json with the webdataset build tooling that records sample counts.

Example fix

# info.json (before): {"splits": {"train": {"filenames": [...]}}
# after:
{"splits": {"train": {"num_samples": 1281167, "filenames": [...]}}}
# or in code:
reader = ReaderWds(root, is_training=True, num_samples=1_281_167)

assert num_samples or (info.get('splits',{}).get(split,{}) or {}).get('num_samples'), 'pass num_samples'
reader = ReaderWds(root, split=split, is_training=True, num_samples=num_samples)

When it happens

Trigger: ReaderWds(..., is_training=True) where num_samples is None and info.json's split entry has num_samples missing, 0, or null.

Common situations: Hand-written or converted info.json that omits num_samples; using a validation-only shard layout for training; a split-name mismatch resolved to an empty split entry.

Related errors

• Dataset length is unknown, please pass `num_samples` explici
• split {split} not found in info ({info.get('splits', {}).key
• Please install webdataset 0.2.x package `pip install git+htt
• Input image must have positive dimensions, got H={height}, W
• Invalid class map file, expected a dict ({class_map_path}).

AI-assisted analysis of huggingface/pytorch-image-models@9a5261e31b (2026-08-27). Data as JSON: /api/errors/4b1bb47eee3b0341. Report an issue: GitHub.