{"record":{"id":"3b9b7e36b88aa5e5","repo":"huggingface/pytorch-image-models","slug":"dataset-length-is-unknown-please-pass-num-sample","errorCode":null,"errorMessage":"Dataset length is unknown, please pass `num_samples` explicitly. The number of steps needs to be known in advance for the learning rate scheduler.","messagePattern":"Dataset length is unknown, please pass `num_samples` explicitly\\. The number of steps needs to be known in advance for the learning rate scheduler\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"timm/data/readers/reader_hfids.py","lineNumber":82,"sourceCode":"        self.builder = datasets.load_dataset_builder(\n            name,\n            cache_dir=root,\n            trust_remote_code=trust_remote_code,\n        )\n        if download:\n            self.builder.download_and_prepare()\n\n        split_info: Optional[SplitInfo] = None\n        if self.builder.info.splits and split in self.builder.info.splits:\n            if isinstance(self.builder.info.splits[split], SplitInfo):\n                split_info: Optional[SplitInfo] = self.builder.info.splits[split]\n\n        if num_samples:\n            self.num_samples = num_samples\n        elif split_info and split_info.num_examples:\n            self.num_samples = split_info.num_examples\n        else:\n            raise ValueError(\n                \"Dataset length is unknown, please pass `num_samples` explicitly. \"\n                \"The number of steps needs to be known in advance for the learning rate scheduler.\"\n            )\n\n        self.remap_class = False\n        if class_map:\n            self.class_to_idx = load_class_map(class_map)\n            self.remap_class = True\n        else:\n            self.class_to_idx = {}\n\n        # Distributed world state\n        self.dist_rank = 0\n        self.dist_num_replicas = 1\n        if dist.is_available() and dist.is_initialized() and dist.get_world_size() > 1:\n            self.dist_rank = dist.get_rank()\n            self.dist_num_replicas = dist.get_world_size()\n","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/huggingface/pytorch-image-models/blob/9a5261e31b3b5128526eb2658333b4c0a54464ae/timm/data/readers/reader_hfids.py#L64-L100","documentation":"ReaderHfids (HuggingFace image datasets reader) needs the dataset length to size the learning-rate schedule, and neither an explicit num_samples nor split_info.num_examples was available, so construction fails fast rather than producing an unknown-length iterator.","triggerScenarios":"Instantiating ReaderHfids with num_samples=None on a HF datasets split whose info does not expose num_examples (streaming datasets, some hub datasets with missing metadata), typically via create_loader/create_dataset with the hfids reader.","commonSituations":"Using dataset_name='hfds/hfids' with streaming=True; hub datasets whose split info lacks example counts; older huggingface_hub/datasets versions that don't populate split info.","solutions":["Pass num_samples explicitly to the reader / create_loader (e.g. num_samples=len(ds) or a known train-set size).","If streaming, compute the count once (iterate or use non-streaming mode) and pass it in.","Upgrade the datasets/huggingface_hub libraries so split metadata is populated."],"exampleFix":"# before\nreader = ReaderHfids(ds, split='train')\n\n# after\nreader = ReaderHfids(ds, split='train', num_samples=25_000)","handlingStrategy":"validation","validationCode":"n = num_samples or (split_info.num_examples if split_info else None)\nif not n:\n    n = len(ds)  # non-streaming HF dataset\nreader = ReaderHfids(ds, num_samples=n, ...)","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always pass num_samples for streaming HF datasets.","Log dataset size before training starts.","Pin known train-set sizes in config for reproducibility."],"tags":["timm","huggingface","dataset","num-samples"],"backgroundTag":"unknown-dataset-length","analyzedSha":"9a5261e31b3b5128526eb2658333b4c0a54464ae","analyzedAt":"2026-08-27T02:34:25.417Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}